You won’t wanna keep running like that with a wobble. Eventually it’s gonna wear your motor shaft from rockin.
With all the things wrong with that pulley. Bore loose, keyway to big, only one setscrew. I would just make or buy a new one and stop pulling your hair out on this. That pulley should be easily sourced and really overall size can be changed for your speed request.
You know it has no sideways play on the shaft just rotational as the key way in the pulley is a tad big so when the set screw comes loose as it does every month I pull out my hair.
But you're probably right, I probably ought to change it.