Here is a badly drawn picture. I used a square block for illustration. Rectangle would be better. Cut, file, mill one side to a tab that fits an AXA holder. Width some wheres a little more that 1/2". Thickness anywhere between 7/16-5/8" There is plenty adjustment in the screws and you can get an oversize 5/8" holder easily. The only criteria is that the screws dig in and hold securely. It does not matter within reason if it is a little crooked or whoppy-jawed as it will be corrected when you drill out the hole.
Mount the block in the tool holder and lock the height screw, bore out the hole. and all crookedness will be taken care of. Saw a slot in the side for clamping pressure.
Couple of points. If you ever plan on using this over, do the following. Orient the tool post to the chuck perfectly. use a 1-2-3 block up against the face. Mount the holder on the tool post all the way at the bottom. Do not use the height screws at all for adjustment. Hold the adapter up to the holder to get a general reference for locating the tab. The tab does not have to be in the center of the block. When you drill the hole using the chuck. the hole will be centered automatically for use. And straight. Give yourself about .002-3 clearence. You can now take this apart and store it. To use, clamp it in any holder bottomed out and orient the tool post to the angle you want to drill. Since the hole in the holder is perpendicular or parallel depending on your point of view to the tool holder, if you want a 30 degree angle, just use a 30 degree standard at the bottom of the tool post.
I have done this before in aluminum for quick holders. In the field I used a 2/4 in a vise to hold EMT. Just drill and clamp, no tab needed. No damage to the pipe either. I currently have collected about 20 lbs of cut up milk jugs. I plan to melt and make some square blocks and thick sheets for doing stuff like this. I'm cheap, why pay for 6061 when I can do plastic for free.
Just realized I oriented the block in the holder wrong in finished picture. That happens when you are get in a hurry.