The big problem with wood is not friction from the RPMs, but friction from the chips. Small bits load up quickly, and small bits at high spindle speeds load up very quickly. Backing out the bit won't necessarily clear the flutes, and this can be true with certain metals as well.
Pull your 1/8" bit into a piece of resinous tropical hardwood (eg. rosewood) at 16,000 RPM, and the flutes will be jammed instantly, and now you're burning your way through the hole instead of cutting. This can get out of hand and become a smoky mess quickly. I've done more than one project where I had to stop and rake the flutes clear multiple times to get through one hole in a piece of wood. The same is true using Forsner or sawtooth bits. I'm new to the lathe and mill, but I've used my drill press on all kinds of wood and metal for 20 years, and I usually leave the spindle running on the first or second slowest speed possible. I very, very rarely ever take it up above 800 RPM for anything ever.
You CAN go faster, but that doesn't mean you HAVE to go faster. This will depend on your production schedule, of course.
Routers spin at around 50,000 RPM, but router bits have a totally different geometry. They're much more efficient at any size for clearing waste than twist drills.