I use end mill holders whenever an end milling pulling down in the collet will wreck something. Every time you are roughing or milling to depth. I only use collets when end mill holder will not fit, or the cutting diameter is the primary important feature necessary.
I allways have my quill jammed in position when using end mill holder for bottom cuts with solid u-shaped block above the adjustable stop, the quill up against the stop, and the adjustable stop tight against the stop. Nothing can pull down to wreck your piece that way. Especially important when using indexable cutters or fly cutters.