If you are using collets rather than a chuck, you would probably want to have a pin permanently mounted in a end mill holder. When I need to change a collet, I just move the table so I am clear of the work. With a DRO, this is trivial. More of an issue if you are positioning by the dial.
The need to adjust the column height comes up when I have a great difference in tool lengths as in a spotting drill and a reamer and I don't have enough z axis travel to accommodate both. Needless to say, I try to set my head height so it will work with all the required tooling. It certainly got easier when I started using collets rather than a drill chuck.
I also started using my Tormach TTS tooling on the mill. It uses ER20 collets in a chuck that has a 3/4" shank that mounts in a 3/4" R8 collet. The shank length is only about 1`.4" of clearance for a swap. The other advantage of the TTS system is accurate z axis registration si that zsroing out the tool height with every tool change isn't required. (you do have to have multiple TTS tool holder to take advantage of this feature)