Robin,
Just so that I'm sure you understand, you only use the vise off of the milling attachment. It will fit in place of the compound (as will the table off of the Universal Compound Vise).
If all of your cutters had 1/2" shanks, no, you wouldn't need any collets. However, that's unlikely. Atlas sold a set of four reducing bushings to go with the cutter holder, one each with an ID of 7/16", 3/8". 5/16" and 1/4". However, they tend to be what most would call "expensive". I bought one set new from Clausing that cost as I recall about $63. For about the same money, there's an outfit in Hong Kong called CTC Tools who sell an excellent quality set of 3MT (and 2MT) collets in 1/16" increments. The 3MT set covers 1/16" to 3/4" in sixteenth increments. It uses the same 3/8"-16 drawbar as the cutter holder does. If you find that you need smaller increments that 1/16" (most collets have very little "adjustment" range) you can use 3AT (what was originally sold by Atlas), 3C, 5C or that short European type whose name always escapes me. But you will have to acquire the appropriate draw tube for the first two or a front-mounted collet chuck for the latter two. All of which are more expensive to much more expensive than the 3MT or Atlas cutter holder and bushings. But any of the 4 are either available in 1/64" increments or have a have a larger operating range. And all can hold much longer work if it will pass through the spindle or somewhat longer work if it won't.
If all that you need collets for is to hold milling cutters, the cheapest solution is the 3MT set.
Robert D.