Doesnt seem right....lets think about that for a min....pilot hole is round....vrs the broach will be square....theres a correlation
The following process has always worked for me. Provided your required hole is a standard HSS tool bit size. Make a holder from a length of square HSS tool bit.
Put a short length of steel in the lathe two or three x dia. of the diagonal measurement of the square hole Drill a hole the dia of the diagonal size in the center, deep enough to hold the tool bit but allowing enough to stick out to punch your hole.
Drill a hole in the part where you want the square hole, I always drill on size, but you can go larger if you wish.
Grind both ends of the tool bit square, then hollow grind one end. Place the square end into the holder, set the job up on the bed of a hydraulic. press, with the hollow ground end over your pre drilled hole, and simply push the tool through, never had a failure. Then drill and tap for a grub screw to hold the tool bit.
I have used this method many times to make a square hole across the end of a boring bar to hold a HSS tool, I have even used it to make a diagonal hole about 45 deg. so it can bore to end of the hole, To do this you must drill a pilot hole to get the square tool bit started, same size as diagonal of the tool.
It is of course important that it is set up square.