I needed to cut a square hole some time ago. Made a tool identical to a shaper keyway cutter, and used my mill quill to gently broach tiny amounts of material out. Only a little at a time, and rotate the spindle for each side. This is a good way to get the bulk out, then filing for a close fit.
I wouldn't do it this way again, but it did work. If i were to do it again I would make a different shape broach and do it in the lathe using the carriage instead of the quill. But, I didn't have the lathe at the time when I needed that square hole.