If you go over it a few times you should get a good clean up. I am just fond of the stiffest set up I can get! You may have a bit of a smaller slot in the bottom from 'push off', but should be in your tolerance range no problem.
I would have drilled a series of 15/32 holes to near point depth to remove the bulk of the material from the slot. Then go in with your end mill again to near depth and clean out and straighten out the slot. Last go to depth and start working the sides .010" to the side, then .005", then .001-.002" at a time until final size is reached.
Couldn't agree more on this. There are specific end mills for aluminum, I usually use a 3 flute high rake/variable rake which does well. Chip evacuation is very important, so use air or vacuum to remove the chips. A little lubricant helps significantly, and in aluminum you should use climb cutting direction as opposed to regular feed. There will be a significant difference in the final cut finish and dimension. Try cutting the edge of some aluminum in one direction, and then the other and look at the finish, with climb cutting the chips are kicked away as opposed to getting recut again be the end mill.One word of caution, don't use 4-flute end mills on aluminum