I've been hearing this argument for years, I respectfully disagree. At least in the sense that I doubt the bearings in a drill press spindle, are going to suffer some sort of damage if side load is applied. Especially not form normal milling operations, as I don't even think there is a lot of side load involved. The rotation of the tool is doing most of the cutting, not the advance. When I mill at a normal feed, I don't feel that much resistance.
Also drill presses usually have ball bearings that can easily take a reasonable amount of radial, side load.
I think the biggest issue with milling on a drill press is the usual lack of a drawbar and if the chuck is only held with a taper fit, that might make it prone to popping off, under side load.
Yes mills are designed to have much greater capacity for such forces but occasional milling on a drill press isn't going to kill it, IMHO...