Lots of people on here have their machines on casters. Do it if you need too. Heck, my 10F Atlas is on a cabinet on casters. Multiple more guys have thier lathes mounted to a butcher board on top of a roll around tool box.
It’s not the “ideal” for a lathe, but you gotta do what you gotta do.
You just have to accept that when you move it you loose the leveling and consequently, a loss of accuracy. Truthfully, its not so much about the lathe being level to the outside world as it is about having the ways parallel and level to each other and the headstock/tailstock aligned to the ways. Using a precision level makes the pieces level to the world, but if you make both ways level to the world, they are level with each other. It’s just an easier, convenient way to true up the bed ways to each other.
I’d recommend you find a spot where the lathe can live most of the time and level it there. Put screw down feet on you bench so you can raise it up off the casters so it doesn’t wobble or shift around while using it. Keep the casters and move it when you need to, but remember to re-level it when putting it back or if you keep it in a new location.
You need a machinist level to get it right. A carpenters level doesn’t have the same level of accuracy or sensitivity.
I’d be more concerned about the rigidity in your table/bench. Wood isn’t the best choice as it “breaths” with changing temperatures and humidity. Despite the fact the lathe is cast iron, it will bend, twist and sag if the base its bolted to isn’t rigid. Same as the work stands you used, kinda flimsy for the weight and forces a lathe feeds into it.
The whole thing just looks a little bit too flexi for me. I’d recommned at least grabbing a 3/8-1/2 piece of plate steel and mount the lathe to that, then bolt that to the table. The steel plate will make for a nice rigid base and then the table underneath can warp and twist as it will with minimal effect on the lathe itself.
good luck.