I had to go back to this as two things stand out.
1: The guy you had over is smart enough to understand that houses "Float" on the surface of the earth.
2: Deflection. Your work holding (Collet vs 3 jaw chuck) and stickout will be the greatest factors you are likely to run into and not understand.
It takes a huge amount of force to make a cutter cut aluminum or steel. Just try to hold a tool bit in your hand and make a cut.
Second think, don't do that, there will be blood.
But you get the picture, it takes tons of force to make metal cut metal and every action has an equal and opposite reaction.
Wonky chuck jaws will induce taper in your part, this is why we have collets, for times when you need more accuracy.
And stickout is teh debbil.