I'm not a VFD expert by any means and I'm confident that one of our more experienced and knowledgeable sparkies will chime in soon.
However, I believe my comment is VFD 101 basic knowledge.
The difference is that an RPC doesn't care if it has a load or not. It will idle without a care. An RPC will accept multiple motors being randomly stopped and/or started (within the capacity of the RPC).
On the other hand, a VFD will not be happy if it is instructed to output 3 phase into an open circuit (no load) or if the load on the 3 phase output is increased or decreased (I don't know the tolerance). Unless there are new protections that I'm not aware of, disconnecting the motor from the operating VFD will release the magic smoke.
I don't see why you couldn't use the existing rotary switch to control the contactor you described and that can control the VFD ON/Off, but the VFD 3 Phase output must be wired directly one (and only one) load.