There is a cape town guy who did very good plans for a rotary broach. He is on chaski. Its a simple design using two normal roller bearings. I added a thrust bearing to his design. Its more bulky than just using one single ball bearing. I just took a morse taper three and tig welded it to a flange . Then this bolts to the broach holder that holds the two roller bearings and thrust bearing but the base is machined offset at 1,5 degrees tocreate the wobble. It has two bolts on the outer diameter that are used too set the broach true to the hole you want to broach. You loosen the bol,ts ,set the broach in the hole and then nip up the bolts. Simple.
I will try find the drawing but its on HMEM.(home shop engine machinest )