I feel your pain. How much material were you taking with the reamer? You really don't want to take too much, but you also don't want to take too little. I'm sure there are plenty of formula and tables. My guess is that aluminum would be pretty tolerant of light cuts with a reamer. On a critical hole, I'll drill quite a bit under size, then bore it to near size, then finish with the reamer.
With this being your driving wheel, make the axle between centers, then just secure the shaft with Loctite. You can then skim the wheel OD and it will be very concentric (should run nice and smooth, and the chatter in the bore will not be an issue - makes a nice surface for the Loctite to grab).