You have heard a lot of good/ valid reasons why smaller less ridgid/less powerful lathes will perform better with HSS cutting tools but as long as you watch your depth of cut and speed/feed rate,lock the axis not in use (in order to have more rigidity during any operation), you'll do very well with that tool set.
But I'm just a hobbyist, you don't have to take my word for it, just wait and read the responses of all the smaller mini lathe owners and find out for yourself what they normally use, you'll be surprised how popular these tool sets are.
Not many lathes can reach 5000 RPM and beyond., not everyone has those production type/ heavy machines but that never stops anyone from using carbide cutting tools.