Don't blame the drill just yet. An x-y drill vise is way different than an x-y table. Way too tall and floppy. Any flex and it will grab. You also need to use the right mill and depth of cut. What was your depth of cut? Were you climb milling or conventional? For something like that i would be using a roughing mill.
Depth of cut, not sure. It was very little each pass though. I'd say probably 30 passes to make each 1/4" groove, which works out to .0083".
Climb seemed to work best. It resulted in much less flopping around of the vise. That is, until I got out to the end of the work piece, and then then it wanted to take a 90 degree turn and follow the square on around. That's what sent the vise flying, and also knocked my drill press table sideways.
I'm not not sure but I don't think I was using a roughing mill. It was a 4 flute 1/2" end mill, looked like a regular end mill to me.