I've been in this hobby about 5 years. Was a woodworker for at least 40 years prior.
I've found that mass and rigidity is important. Can't have too much of either.
Same for horsepower. I have a 2hp rf45 clone mill. It's 'adequate' for my needs. Weighs about 700#. No where enough to take 'heavy' cuts without chatter.
It's a dovetail column. Much better than a round column where you lose your positioning when you raise/lower the head.
My lathe is a 2500# 14x40 Takisawa with a D1-6 camlock spindle. Very nice for my novice abilities. It's ridgid and has a 5 hp, 3 phase motor. Don't let 3 phase bother you. I have a 10hp rotary phase converter setup I built. If you are handy it's not a problem to build one. Mine has a 220v/50a input. Need to be comfortable doing electric work though.
Cam lock spindles are nice. D1-4 is your typical smaller one. I love my D1-6.
eBay is a good place to find used tooling. Stay away from D1-5 camlock spindles. There is a lot less on eBay for D1-5 stuff.
Aloris type wedge lock tool posts are typically the tool post of choice once you get out of the lantern tool post style typically on the 'smaller' machines.
Lastly, expect to spend at least as much money for tooling as you do for the basic machines. That could be a subject unto itself.
Just realize, you can do small stuff in a big machine. Typically can't do the reverse.