I used to work on an assembly line as a driller, ( No Crap really I did ) I had 12 - 1/4" holes to drill on each door panel. These were roll-up Semi-trailer doors. So you can imagine. fast and furious drilling was required with an air drill.
The best perk of the job , we got to keep the burned Dormer drill bits because they cost too much time to resharpen them. haaa ! I must have 60 1/4" bits left.
At home I've had this happen so much, I have had clamped items tossed off the table or vise, been nipped by parts and here and there. after the last sliced finger I have tried to investigate this carefully.
This is what I have concluded.
What happens is as the bit breaks through the " chip " or " flashing " that is left just before the final dia. is cut acts like a screw thread on the flutes of the drill bit. coincidentally the larger the bit the more violently it can happen.
Clamping with sacrificial woods or a type of backer can help to prevent this.
As well as learning to listen to the bit & "feel the cut as it goes" .
I tend to "peck " with my drill press if I can. depending on the materials and the drill speeds.
I learned to listen to the cutting of the bit. I Know it sounds different as the bit gets closer the break through point.
just my point of view...
I forgot to mention I also use copious amounts of wd40 when drilling ally.
Another factor I haven't seen mentioned is the possibility of an off-center web/point. This will cause the bit to run out (wobble) and actually cut over its actual size. Usually this can be observed by unequal chips coming off the bit. Then, once the material gets too thin to support the actual point, or breaks through, the wobble more or less disappears and the drill cuts close to size, meaning it can be grabbed in the hole. If this is part of the problem, you should see a slight step where the hole size changes just above the exit. Also check the hole size on the entry and compare it to the exit.
I forgot to mention I also use copious amounts of wd40 when drilling ally.