The guide grooves on your drawing will go a long way toward keeping the bore horizontal. If you move the upright nearest the viewer further away from the bore (widen the base), you would have room for a jackscrew beside your clamping bolt, probably on the side closest to the point of the shaft.
The slots for mounting to the mill table should stay directly below the bore.