Do you have any provision for pre-loading the bearings Hawkeye? If not, that could be part of the issue, as those look like deep groove ball bearings. Without a preload they will precess within the outer ring.
If you already have the bearings preloaded, then another tip I picked up is to cut the final spindle taper with the spindle assembled. When I did it on the small grinding spindle I made a while back I mounted the spindle body in my steady rest, and grab the drive end of the spindle in my chuck. This is quite effective, but it relies on the bearings not moving, so if you ever disassemble, you will need to re-cut the taper again.