I actually think you might be better off with a milling vise. I bought a nice drill press on a friend's recommendation several years ago, along with the vise he recommended. It's nice, but sometimes a bit too cumbersome. There have been many instances in which I used it simply because it was there, but would have preferred something smaller - like a 4" milling vice.
Regards,
Terry