That slot you see is a piece of oval shaped tube welded in place and by the time you mill it out 5mm there will not be a lot of wall left. What equipment do you have? The oval shape extrusion is a standard shape available in different sizes, have you checked what the next size up is and if it is close,(Metric or Imperial),to what you need? You may have to make a custom piece and I'd be sorting that out first. Alumiweld is a low-temp brazing rod that would excellent for this job.
Re-tramming the head is not a big deal and as you are going to have to do it sometime just accept it as something you have to do on some jobs.
If you have a drill press it would be easier to drill out and blend with a die grinder.
I strongly suggest you get an insert that fits your new axle before doing anything else though.