Thank you guys,
Many points learnt by reading your replies. I list them below.
1. Determine the exact shaft diameter you want to get based on "light interference fit"
2. Take very small cuts
3. Measure after each cut
4. When it is close to desired diameter go more slow in taking cuts.
5. Measure but remember the temperature can cause issues. Wait for cool down if necessary.
6. Never try to insert the bearing by hand - it may not go straight and give you an erroneous feeling that there is more machining needed. In reality the shaft may be ready to take the bearing. Depend on the measurement than you insertion by hand.
7. When it read for final mounting freeze the shaft and heat the bearing. (I am not sure if opposite of this will work for inserting bearing into the bore. I remember reading somewhere that bore shrinks when heated. Would it still shrink if the bore is about 5.5"?)
Thanks
Prasad