As you mention, the 6mm encoder shaft is designed to work with a set screw, which works for that end of the connection. The other end of it is the pulley. I solved the problem by machining a 12mm shaft down to 10mm for the pulley and the threading a hole into the shaft end. Now I can use a M5 screw and washer to bear on one side of the pulley and the other side bears on the shaft step. The pulley is squeezed between the two, so it's held nicely in place.My encoder doesn't have a keyway, it has a flat on the 6 mm shaft. I bought an aluminum pulley, 40 tooth, 15 mm wide with 6 mm bore. I turned the hub off of it, then drilled and tapped through the pulley teeth for a set screw.
Once you make for spacer for the shaft and have it installed in the pulley you should be able to do the same.
Of course, as I note in the post above, it probably will end up being a pulley that doesn't work with the spindle "gear". So I may be restarting from scratch again.