Hey Strantor
First, I don't blame you for trying. I did the same thing about 20 years ago. Same nasty drill press vice.
And worse, it was a tongue'd Morse #2 spindle! No drawbar! Bad idea!! I lost the chuck and bit once! But I didn't even have a lathe at that time to try to adapt.
Anyway- quick questions.
-You say you actually tossed the vice. I wonder if this was part of the problem? Was it bolted down properly?
-I wonder if your screws were adjusted properly for the backlash? We're the screws in the compound vice turned the right way to accommodate conventional milling (pulling in the screw)? That might explain why climb milling appeared smoother (pushing against the screw).
You would have to set the screw, and spin it -backward- until it stopped to keep it from feeding in (I think?)
-Also- for this setup, I don't see .009 as a really light cut with a 1/2 inch mill. Especially in what looks like steel T-nut. (Steel, right?). I would try .003 in aluminum first.
Also- I would ABsoLUTEly do SOMETHING to adapt your lathe in the meantime. Absolutely better set up for the different forces involved.
Bernie