Also, its not even really like sanding it down, its more like wiping it out, .0001 isnt much at all. In old machinist terms from the guys I used to work with, its a very light kind of hair as the amount that has to come out. Coming from me though when I have done fits on different things so many times over the years, I see how it can be frustrating especially when new to this. This is why machinists that do it for a living end up being very very cranky haha.
It is the fit of the taper not letting it come all the way to the spindle face, so it is the area that is blued that is being checked, that is correct.