In answer to your original question, there are several more sets than what you are asking about. What you are calling two sets are actually three. There is the fractional 1/16 to 1/2 by 64ths. There is a number set from around 0.040(~1mm) to about 0.200. Then there is the letter set than goes from just above Nr 1 to a little over 0.400.(~7/16). Thats what you are asking after.
I spent 50 odd years building my sets. Anything that needs a good drill was bought individually. Some time back, when times were good, I purchased the full sets of questionable quality. If I use it once for steel and it dulls, it's thrown out and replaced with a known good drill. Full sets of good drills are horrendously expensive. In essence, you get what you pay for and you pay for what you get. There are many good name brands like Chicago Latrobe and Precision Twist Drill. (my personal preference) And there are many not so good and junkie sets. And many good drills of names of recent origon that I'm not familiar with. Here is as good a recommendation as any. Cheap sets still work for wood and plastic, and the boxes make good starter boxes. Huot is top line there.
Now, on to other sets, many of which are specific to what you work on. An extension to the numbered set runs fron Nr 61 to Nr 80. But in reality, how many(few) need to drill 0.013.(Nr 80) I build models in small scale, I do use them. As would someone doing carburator jets or gas furnace jets. Not the sort of thing found in the average tool box.
And to top off everything else, there are detailed metric sets up to 13mm. If you do a lot of metric work, get metric drills. If only occasionally, most metric drills have comparable imperial sizing. If you need closer than 0.001 for a hole, you'll be using a reamer anyway. I have some fairly complex metric sets because I had some spare cash at one time and needed somewhere to spend it. The US will go metric eventually, like it or not, but will be a slow conversion. I got a moderately good set so I'll have then when. . .
Then there are the job specific drills. Technically, a 1/4 capscrew rates a Nr F drill, a couple thou larger than 1/4. But, like most people, if i'm drilling for a 1/4 bolt, I use a 1/4 drill. I buy them in packs of 10. They are good for, though not properly sized for, Nr 14 machine screws and 6mm cap screws. And stuffing small wires through wood and the like. Or 1/8 rivets, where the proper drill is a Nr 30(0.128), not 1/8. And for 3/16 rivets, where a 3/16 drill is a fuzz too small and there's not another drill that is 0.003 oversized. But there is, I found some at an aircraft salvage house.
There are more drills than any normal person would ever need. My advice, such as it is, is buy a good set of 1/16 to 1/2 by 64ths. That will do 90% or more of your work. Then add here and there as you need something not in that case. I buy 1/16 drills in 10 packs at Harbor Freight for working with wood, ZAMAK, and plastic. A suitable size as a tap drill for 2-56 and 2mm. Not exact for either but usable for both.
I could lecture for hours when I get wound up. A copy of Machinery's Handbook, a Drill Doctor and a set of fractionals will get you by for a long time. Years. . . The rest you can get as you need.
.