If you are using a hand-held drill, it isn't unusual to have drift when using jobber drills. If it was me and I had to make multiple pieces all fit, I would make a jig to fit a sharp prick punch so marking out is consistent. Then I would use a rotabroach to drill the holes. A rotabroach has a sharpened pin to orient the cutter and it will locate in the prick punch hole accurately. You can also visually see when a rotabroach is cutting evenly so you are less likely to cant the drill. A rotabroach also cuts burr-free holes so you have less clean up to do.