For a hole saw as shown, that's probably okay. For any type of drill, though, the deflection caused by the lever at the chuck will cause the center of the cutting tool to climb up and to the rear, away from the work's center. I would allow axial twist in the tool and tailstock before trying to support the work as such.
Somebody is going to scream bloody sacrilege, but here's a trick. Go over to your carbide grinder and find some tungsten carbide grinding dust. Mix the dust with a few drops of oil or grease to make a paste of a similar consistency as anti-seize. Rub a *little* dab of that into your taper and install it- it will hold tight. Some of the tiny carbide dust will embed into the spindle/tailstock/tool shank without causing any type of runout, making a lasting hold long after the paste is wiped off. Some of you may be familiar with carbon fiber paste, this is the same concept except with hard steel. I am comfortable doing this with my own machines. The carbide paste is useful in so many applications, including gripping Philips head screws against a cross tip bit, even when the slots are stripped beyond recognition.