I don't care for end mill holders. They take too much room and it's not a big deal, but they do force the end mill slightly off center. The only time I've ever had an end mill slip in a collet was with an old Index mill. It had a wimpy draw bar that was hard to get tight.
1) A vise should be first on your list. I have a 5" Kurt Angle-Lock with a swivel base. They are very nice vises but quite expensive. I can't speak for the knock offs but you probably pretty much get what you pay for. Maybe a used one?
2) You'll need an assortment of parallels. The Chinese junk is fine for those since they take a beating anyway. I like the thin ones. They normally come in sets from 1/2" to 1 1/2" by 1/8" wide.
3) A good 1/2" Jacobs drill chuck with the appropriate taper for your mill.
4) An edge finder (Starrett)
5) A boring head
6) An angle plate
7) An assortment of hold down clamps, studs, hex nuts and tee nuts. Go crazy here, the stuff is cheap and you never know what you'll need.
8) A small sine bar is nice. I have a 2.5" one that is small enough to fit under the work in the vise. Of course that means you'll need a set of gauge blocks to set it (shop grade).
9) Adjustable parallels are handy for set-up and measurement on the mill.
10) Rotary table
11) Dividing head
12) A collet block is very handy if you have a set of 5C collets. I don't think I'd go out and buy a set of collets though just for the block.
13) Of course you'll need end mills. Buy the best you can afford and steer clear of the Chinese junk. They are cheap, but the geometry is poor and they don't last near as long as name brand American, European and Japanese cutters. The type you choose depends entirely on what you intend to use them on.
14) A good dial test indicator is a must. I like Inerapid, but Mitutoyo, Tesa and SPI are also good. Stick with Swiss or Japanese indicators and you won't be dissappointed.
That should be enough to get you started and definitely put a nice big dent in your bank account.
Tom