Here is something else to consider. ER collets can be sprung when holding diameters that are far from their nominal size. Yes, ER collets can compress down about 0.040" and still hold but do this enough and their accuracy suffers eventually. On a lathe, in a hobby shop, we are often using stock that is off size or non-nominal so it is common to compress ER collets to hold them. Since this is the case, it is wise to use cheaper import collets on the lathe.
Now, on the mill, accuracy counts. ER collets are most accurate when holding tools at or very near to their stated sizes. Fortunately, most tools have shanks that fall on size and distorting a collet to hold an off size shank is not commonly encountered. Given that accuracy of the tool holding system has a major impact on tool life, accuracy and finishes it stands to reason that you should use high quality collets on the mill to hold your tools. Most of us who use ER chucks on the mill will own several sizes of chucks because no one chuck size fits all. Quite often, a smaller chuck is needed to gain access to a feature on a part that prevents the use of a larger chuck. I have an ER40, ER32 and ER20 because of this demand. This can get expensive because you have to buy separate sets of collets for each of these chucks. I happen to use Techniks collets for each ER chuck used on my mill and yes, that cost a lot but it cannot be helped.
My point is that for the lathe, an ER40 chuck with import collets makes sense. On the mill, I would go with an ER32 system (max shank capacity is 3/4") and add other chucks as needed. Buy high quality collets for use only on the mill and do not use them for work holding on the lathe.