If you are getting those results from a short piece held just by the chuck then you may have some issues with the chuck or possibly from excessive cutting forces from your tool that results in deflection or your spindle is really worn. Try mounting a short piece, maybe 1-2" sticking out of the chuck and try again. If you still have that much error then you have some issues to work out. I suggest something soft, like aluminum or 12L14 for this test.
For use with a 3 jaw, lock the work into the chuck and bring your steady rest as close to the chuck as you can. Bring the two lower arms up so that they juuust, touch and lock them down. Then lower the top arm down so it just touches. Theoretically, the work is aligned fairly well by the chuck and you have fairly closely duplicated that alignment with the steady rest. Now unlock the steady rest and move it toward the end of the work piece and lock it down. You have now transferred the alignment of the work piece to what it has at the chuck end. Oil the work and turn the lathe on, then drill your center hole with a center drill. You can now remove the steady rest and engage a live center into the hole you just drilled and proceed to turn your work piece.
When using a 4 jaw independent chuck, you can get things much more precisely aligned. You dial in the work at the chuck and then dial in each arm of the steady rest that is positioned near the end of the work. There are other ways to do it. Joe Pi (previously linked video) has a neat way to do it; check his videos.
Each arm of the steady rest should be the same with regard to the contact points. They don't have to be precisely the same but at least close. The ball bearing thing is a good idea if you can incorporate it but not necessary; plain brass arms work well, too.
What lathe do you have? What kind of turning tools are you using? What is the material you're cutting? The more you give us, the more we can help.