I'd probably aim for having the table top of the mill (not the bench) at about elbow height, I'm 6'2" so for me that would be around 42". Bench height would be what ever puts the table of the mill at that height.
Size of the mill plays a part as well, a larger mill I'd try to keep close to that height as a balance between lifting and mill head height concerns. On a mini-mill I might go a little higher since parts will be smaller and the mill isn't as tall. On a large bench mill, perhaps a bit lower.
On a mini-mill having it a little higher means less leaning over to get at the spindle and overall height isn't an issue. On a large bench mill you might go a little lower to ensure you can still get easy access to the top of the mill head at full extension.
That assumes standing at the mill when operating, not sitting.