concentricity of the cutter body with the spindle center line shouldn't have too big effect unless it's way out and causes vibrations to show up in the work. The cutter body is hanging quite a ways out of the spindle and the cutter quite a ways out of the body, so there's a lot of tool hangout there. The body is also quite long compared to the diameter of the shank that goes in the collet, which will add to potential flex. Finally, if you look at the clamp and body, there's a lot more meat behind the cutter when the cutting face is on the clamp side, ie. rotating clockwise, which might help too.
Lots of other things like speeds, feeds and tool grind. Remember that a fly cutter does most of it's cutting on the outside edge, not the bottom (unless you're boring a hole), so make sure it has appropriate angles on that edge.