OK Gerry, I've seen something similar before. it is possible that there is a super simple fix: Please forgive me, but many guys don't tighten their D1-4 camlock properly and then grind down the spindle, like you were tempted to do.
Please try this procedure, and recheck the shim spacing behind the chuck to see: 1) if the gap disappears, or 2) if the thickness of the shim is less and if it is even.
Remove the chuck and reclean all surfaces with a solvent and wipes.
Put the chuck on very lightly, so the camlocks 'just catch'.
Rotate the camlocks, one at a time to the same rotation, but not hard (yet). On mine, it 1/2 way to the line and things begin to feel 'just snug' equally.
-- this, believe me is a necessary sub step, forD1-4 and D1-6 camlocks: everything should be lined up at that point.
If you wonder why, it is because if the chuck goes on cocked, it will never seat all the way down, even on perfectly fitting tapers. that's because the taper is much larger than it is long, and if it is already on diagonally, it cant seat all the way down.
You've checked everything is still parallel and straight. then tighten each -not all the way, but to about 3/4 of full tightness.
THEN and only then, tighten all the way.
I do this every time I use either camlock - and the guy who sold me my D1-4 machine reported the exact problem (of course after I paid for it).
When you do the 3 rotations like this it becomes automatic and they align perfectly every time. I was taught this 3 turn procedure nearly 40 years ago on the D1-6 lathe I just bought. The tool and die maker I got it from always did this procedure, and the chucks registered perfectly each time and the spindles show zero wear. And his chucks are always marked so the cams always meet the same studs.
Get back to me if this hasn't improved. There are a few other things to check. An offshore chuck adapter On someone else's lathe from a big name company showed the exact opposite problem (taper too loose), but in that case it requred remachining the backplate to make it fit properly.
The chances your spindle is out are miniscule. the next check is for spindle roundness in half tenths. That's far more likely than the taper being the wrong angle or too large. If the spindle is out by more than a half or 3/4 tenth , it will lead to the chucks always being on crooked.
If, however you've used the lathe for 10 years by putting your chucks on crooked, you will have worn your spindle out and made it non-round. This is an advanced fix.