Definitely no set screw, either knurl or shim stock to a tight fit. Not much turning force on the bushing so loctite may not be needed in either case.
or https://www.mcmaster.com/6391k673 turn down to fit.
or https://www.mcmaster.com/6391k673 turn down to fit.