Hi Tigtorch,
Should you decide to go with ER collets, for a few dollars an extra MT ER chuck for the tailstock of your lathe is one of the best investments you can make, I wouldn't hold a center drill or reamer any other way.
G'Day Earl,
I also have a mill without a spindle brake and use ER32 collets without any problems, when changing cutting tools I put a rag on the table in case the cutter drops, hold the chuck body with a 36mm open end wrench and use the "C" wrench to loosen the nut.
Regards,
Martin