You can, and
most of the time, if you're disciplined enough to
only ever take very light passes with shallow DOC, you'll
probably be fine...
most of the time.
But the mixture of hardened chuck jaws and hardened endmill shank is a very risky recipe for tool grip.
It's not just rotational slip that could happen. The forces will tend to pull the endmill axially towards your workpiece and outwards from the chuck. That would be bad for your work piece and potentially even dangerous.
Lathe collet chucks generally aren't expensive. There's no need to be putting on chaps and spurs and yee-hawing one's way into sub-par practice in this case.