This is my current layout, in a bit of a disarray at the moment due to not having cleaned up in a while but should get the idea across.
For the lathe I have a small(and ugly) shelf for tailstock accessories and screwdrivers(for insert tooling).
This makes sure when I change between drill chuck and center I don't end up placing things on the table/ways.
Tool holders with my most used tools are on the wall/door next to lathe.
I've moved the lathe further away from this part so I'm probably going to move these holders to either above the lathe or to the right of it.
I try to avoid having things above the lathe as any falling items can cause a big accident and they're also subjected to dirt and grease from the lathe(ie coolant/chips/stuff thrown from chuck).
For my mill I have 6 ISO40 holders and various clamps on a shelf behind it. These slots are for the "currently used holders" and I empty it when I'm done with that current job.
Everything then goes back to my covered storage to keep things nice and clean.
There's a second shelf(also on linear bearings) underneath that holds chucks, vices etc for both mill and lathe.
Most of the time I get by with just what's already tooled up for both mill and lathe. If not I have this "spare" drawer.
Endmills, boring bars, inserts and whatever else is not a "daily use" item.
Below that is then the spare endmill/large morse taper drills drawer.
Stuff here rarely gets used but is at the same time invaluable to have.
I've considered having everything even closer to my mill/lathe but this has worked quite well for me and I feel no need to have more things cluttering up the work area.