I have mounted a lot of chucks in my approximately 58 years of machining, both as a apprentice, A journeyman, an employee, and a shop owner, and I have never had any problem with doing it well. I would use a metal to metal fit, not an interference fit.
The poster said that he uses .0005" clearance, personally I use zero clearance "metal to metal".
Metal to metal is an old way to say that there is neither clearance not interference; thus the recess and back plate would be the same dimension.
Wondering how that is possible without precision grounding …..
What would be the problem in machining the same diameter on two parts? It just takes a bit of care to sneak up on the dimension.
My experience is very limited but I have never been able to get a perfectly smooth surface by turning so some clearence is always needed to allow the parts to be assembled and disassembled without excessive force. I did have come across some commercial parts having zero clearence ( a boring head) but the mating surfaces are finely grounded. Again. This is just my impression and experience, can be wrong.