gottcha...
BTW, if you run into that situation again, consider a step drill for expanding holes. Lots of oil. I find that they don't bite in the way a twist drill does.
I use them for plastic to prevent it from digging in, I have another set for metal. I only use a step drill for sheet metal.. And I agree your situation requires you to think about your wrists first. You can use you mill as a drill press, you don't need to mount it in the vise. Put a stud in your table to prevent the plate from spinning... open the jaws and drill over the open jaws...