You have more of a balance issue with heavy pulleys with a lot of rotating mass and this will be exacerbated at speed. You should not need the heavy flywheel affect if you are running sensorless vector on a good VFD, you should be able to maintain around +/-1 RPM under varying load. On the lathe systems I have done, which has a lot more rotational mass and load with deeper cuts, on a 3 Hp inverter motor system there is essentially no RPM change with cutting load. If your seeing that, then you could have an issue sizing the motor or pulley ratios correctly. Compared to tradition V belts, the poly V and cogged belts run so much more smoothly with almost no belt vibration.
One other factor is that running your inverter motor to 120 Hz (and higher) will give you full Hp from 60-120 HZ, compared to a stock 60 Hz motor which needs to be pulleyed up to speed, you cut the Hp by 50% when you belt up to the same speed as opposed to running a motor at 120 Hz, the torque is the same when you factor the doubling of the pulley ratio needed for a 60Hz motor. I use a 2 Hp Baldor Inverter/Vector motor to 120 Hz on my lathe, it is rock solid to within +/- 1 RPM on the spindle speed to 1800 RPM. The motor is rated to 600o RPM for full Hp. The inverter motors with a sensorless vector VFD hold suprisingly tight RPM control, on one install we looked at using the attached motor encoder, but it would have not improved the speed control we where achieving. One other factor is that with Invertor motors you can push the overload point to around 180%, which they can usually sustain for ~1 minute, this can pump a lot of added motor control power for short periods.
I was looking at the use of cogged belts for various drive systems a couple of years ago, there was a ton of factory pulleys that could be bored or sheaved to the shaft size. The outfit carried all types of pulleys and drives, I need to dig through to find the link. There also may be some issues with the maximum speed ratings of standard V belts and pulleys, since the motor end will be running up to 4500 RPM. The maximum rim speed for iron pulleys is 6500 FPM, this is exceeded significantly if you are using a 6.75" pulley at this RPM, at least by my calculations.
Belt drive system on Grizzly VFD mill heads, not much rotational mass and small pulleys, maximum speed is 5000 RPM.