My Tormach mill uses a 3/4" shank ER20 collet chuck which I mount in a 3/4" R8. Tormach also sells ER32 collet chucks. The advantage of the Tormach system is you have accurate and repeatable tool offset heights, providing you change out the chuck with the collet. I have about twenty different tool holders so I can have the whole set of pre-referenced milling tooling set up and switching out tooling during a job doesn't require re-referencing.
I modified my old mill to use the Tormach system as well. It simply involved removing some material from the face of a r8 R8 collet. My spindle face ran true so no modification was needed there. Tormach has a white paper on making this modification.
The tool holders will also work in a lathe but the disadvantage is the lack of a through hole. The solution would be to make or buy a spindle mounted ER chuck.
I opted for a 5C collet chuck for my lathe because I already had a set of 5c collets. Unfortunately, I had overlooked the limited grip range of the 5C collet and in order to have a complete set for work holding in the lathe, I would have to add, at least, the odd 1/32" collets and ideally, the odd 1/64" collets which would be 17 additional collets by 32nds or 51 additional by 64ths. Doing it over, I would have opted for an ER32 chuck or even an ER40 chuck.