The slowest speed may not be your friend!
You need to set your spindle rpm according to:
1) the tool material
2) the diameter of your tool
3) the material to be cut, and
4) the rigidity of the machine
For a HSS tool cutting aluminum, I'd aim around 110 sfpm.
For a HSS tool cutting mild steel, I'd aim around 80 sfpm.
There are various on line calculators, or see here:
http://www.hobby-machinist.com/threads/soft-jaws.55689/#post-460870
Also, if you can use the table locks for the direction you are not feeding then you can remove some motion of the table. For instance, if you are cutting with the X-feed, use the table locks for the y and z axis to help rigidity. If your mill does not have this feature you may be able to add it.
Please provide the tool material and diameter, maybe there will be more ideas.
-brino