That doesn’t look bad, don’t let perfect be the enemy of good. Can always take off more but putting it back on is much harder.
You won’t be looking at this surface, and the spindle should be much harder than the bearing. I’d put it back together and see how it runs.
My old Seneca Falls Star had bearings that looked worse than that and it ran fine. Remember, it should really be running on a film of oil, not really metal on metal.
John