From what I have seen of rotary broaching, and I have never actually done it, the drill size is a smidge larger than the width across the flats of the hex. In this way the broach has less to clean up, the chips are individual ears, but the hex is imperfect with slight remnants of the original drill hole on the flats of the hex.
This is probably a process you need to sample with and figure out what process works well for you. I've also seen that if you grove the bottom of the hole with a very tiny grooving tool (probably a Micro100 solid carbide bar), then the chips can cut free and fall out.