Instead of a rotary table, I have found that collect blocks ( I use er40, but 5c are common) and are more useful to me. You still have a sizeable investment in collects, but they can also be shared with the lathe ( the collect blocks work in the lathe Chuck too).
I put a 6" Chuck in my rotary table, it's gets enough use that I leave it bolted to the table.
Get an awesome set of drill bits
Collects or endmills,( I prefer collets)
I use 1/8"parallels alot
End mills roughers, finishers (get good ones, they are a good investment)
Flycutter
Calipers, mics, tape, ruler
I went thru this some 4 years ago, little money and a crazy desire to own everything, I just could not afford it. Most the above items were what I had. But I made many clamps, hold downs, t nuts. Books, Harold hall has some good ones, he has many great ideas for making tooling.
Everyone's ideas here are pretty spot on.
I just thought of this, it pays to find fleemarkets and such, we have a couple decent ones around here that I visit every couple of weeks, I usually find a trinket or 2. One guy shows up every Sunday with machinist tooling. I usually give him 60-100 when I visit his booth
Hth