Sorry if I ramble and get off on a tangent that you're not interested in. I'm not a machinist by trade and still learning. I have always wanted a mill and when I finally got a small benchtop model, I realized how little I knew about the machining world. My thoughts after playing with the mill for a few weeks was "why in the world would someone want to sit here turning these handles and making all these measurements and adjustments if you could just hook up some electronics and motors and let it do by itself"? Well obviously knowing how CNC works doesn't mean I knew all the tricks of milling and machining.
I knew something about CNC from other hobby machines I had built but I knew little about machining with a mill. You have a big advantage, you know what processes need to be done. Making the computer/motors end of the CNC do it for you is much easier than you think. A solid background and understanding of manual machining would help tremendously for anyone getting into CNC machining. Like everything else, in the hobby and retrofit CNC world you can spend as much as you want. I know a lot of folks buy a small mill (grizzly, HF, LMS etc) and either purchase or build/assemble the CNC components or kits with great results. If I could do it (and I did), anyone can and I sincerely mean that. One of the most daunting tasks initally if you're not already familiar with CAD and CAM software is learning the software. Someone can show/teach you in a few hours what would take weeks by reading and trying to "figure it out". Find someone in your town that's willing to spend a little time to help you in person.
I bought a High Torque mini mill from the little machine shop and converted it to CNC. Cost directed my decision against a kit of any kind. I'm very happy with the results and avoided spending at least twice as much a retrofit kit/parts would have. In some cases what I used was as good or better than available kits offered. If you don't want to use your current drill/mill get a machine big enough machine to suit your expected needs, and with your manual machine knowledge, you should have no trouble building the motor mounts, bearing mounts, machining lead screws ends (if you have a lathe) etc for a machine. A 4 axis gecko controller (like the G540) is a great controller and fairly inexpensive. On the lower cost end, ball screws can be found on ebay from China that are fairly accurate and work well for many folks. Then of course it goes up from there. As you're probably already aware there are a few folks who offer partial kits to complete retrofits for small/mid size mills. I guess it's all a matter of how much money you want to spend and how much work building parts you want to do. For me I enjoyed making all the parts myself, it was part of the fun (and remember I had no prior mill experience). I know you said you'd rather not retrofit the machine you have, but I think you'll find doing so would be the most cost effective and rewarding route to take. I can guarantee you that (even if you make the parts yourself) what seems extremely confusing or out of your abilities now regarding CNC will all become clear and look MUCH less complex in just a short time. Especially the machine and controller aspects (even the controller software like Mach). The CAD and CAM software takes some folks a little more time to become proficient, but that applies if you have a complete CNC mill ready to run or assemble it yourself, and troubleshooting accuracy or other issues (and they will arise without a doubt) will be much easier if you know how all the CNC components are assembled and what they do. Just Do it! The complete retrofit kits that include the controller, motors, screws and software etc., would be the 2nd best route in my opinion. It'll cost a lot more, but will simplify things somewhat, but not as much as you may think. Sorry so long winded and best of luck no matter which route you take. If you enjoy machining, you'll have a blast with CNC machining!