True the piece could come off and not affect operation of the chuck. But if it does come off, it would not fall off sitting still, it would be at speed and become a bullet.
Since you mentioned Brazing in the OP I assume you have that ability. If you are going to repair it, I would first completely disassemble, then stop drill it, then grind a V groove at least 3/4 of the way thru the material, overfill it with braze. then grind a groove on the back side till the braze from the front is showing and fill the backside with braze. This will completely remove the crack, and its stress risers, so that it can not try to propagate in the future. Machine it back to smooth and round to reduce vibration and make it pretty.
JB Weld is great stuff and I use a lot of it, In this case JB weld would only be a cosmetic fix, It is not near strong enough to join cast iron, and there is no way to get it into the actual crack, it would just be a cosmetic patch for the stop drill hole.