I would like to have all the above also. IMHO YMMV, etc.
I have R8 collets on the mill. They work great. Since my mill (PM-25V) isn't very tall, operating the drawbar is not a big deal. Buy a set of R8 collets to use on your mill/drill. It will easily run at feed and speed with a 1/2 end mill. I run larger end mills and achieve great finish and respectable accuracy. With little effort, I can make chips faster that I wish to remove them. You have to learn your machines limitations. They all have them.
If your lathe is a new-ish asian lathe like mine, you might EVENTUALLY go with collets (ERxx), but until then, stick with the 3 and 4 jaw and learn from that process.
To get more productive on the mill, get a drill chuck on a shank so that you can easily change from drill bit to end mill of a certain size shank (say 1/2 inch), an edge finder of the same size and the tap center / assist. With that, one can easily locate, drill and tap holes in a lot of parts. In certain cases, having a 1/2 inch shank end mill holder to allow quick change to a smaller end mill might be a good thing.
While you could share collets between the lathe and the mill, you might always find (or not find) the collet you want because it is on the other machine.
For bar stock work on the lathe, I am looking seriously at ER32/40 to reduce runout. I find it unlikely that I would go with ER collets on the mill