The VFD is everything. 3 phase supply, contactor, control switch, direction, speed.... it has to have the final say in everything. These things work by taking line current, untwisting it back to DC, then chopping it back up and shoving it down the three wires. To do this, it must have a small amount of electron storage, a buffer, that it fills based on predicted need. A sudden open circuit from a contactor or switch will lead to a sudden oversupply, often followed by bad smells. And smoke.