Yes you should rough everything out. OD and ID. Then go back and bring the OD and ID features to final size. I believe the warpage and distortion will occur as the material is removed.
IE you don't have to wait days. Just rough everything and then finish.
This spring I made the adapter for collets to the spindle on my SB Heavy 10.
I used the compound to cut the tapers.
Here is the sequence of operations I used.
1. Install indicator on compound with tip and spindle center height Exactly.
Indicate along the spindle taper on the back side of taper (away from operator). Adjust compound until indicator has 0 motion.
2. Install 3 jaw chuck and Put material for adapter into chuck. I used 4140 Pre heat treated.
3 Turn OD to diameter of flange +0.020 inch. Turn down part that will become taper. Turn cylindrical to 0.020 over the large diameter of taper.
4. Drill ID and bore to diameter .020 smaller than the back of the collet.
5. Bore a relief in the inside of collet about .060 larger. start the relief about 1/2 inch in from end of part.
6. Rough the taper using the compound.
7. Finish turn the taper to size.
8. Go back and finish bore the end of the bore to the size for the collets. It needs to be a close sliding fit .0005-001 clearance.
9. Remove part, Install 4 jaw chuck and install a collet, indicate collet in to 0 rounout. Stick a dowel pin or other precise piece of stock in collet and close collet with a hose clamp.
10. Mount the indicator on center height and adjust the compound to cut the taper of the collet.
11. Remove chuck and put your part directly into the spindle. Tap it in with a soft mallet.
12 Taking small cuts, turn the OD of flange to size, Face flange to size.
13. Use compound to cut taper for collet.
14. Done!
This approach cuts the OD taper and the rear collet mounting feature in 1 setup. Therefore no runout.
The front tapered surface is cut with the OD taper mounted directly in the spindle. Again 0 runout.