Best practice in aluminum for straight holes is:
1) drill to remove metal, (through holes are your friend)
2) bore to clean up drill marks to locate hole - leave at least .004, or more for finish, if holes persist in being oversize, increasing feed and/or decreasing hole size can help, as well,
either,
3a) finish bore to size (1,000,000,000 jig borers can't be wrong)
or
3b) ream to size using a floating helical reamer, using plenty of lubrication, but don't remove under power.
If you do this and still get oversiz holes, it is time to talk about:
Rigidity, worn ways, tool holding issues, adjustment of tool slides, or errors in indexing.....