A couple other things that occurred to me over coffee:
* check out
riffler files and
finishing stones - these are just examples, cheaper ones can be had
* a
hand-powered shaper might interest you, though good luck finding one.
Goodell-Pratt also made a hand-powered drill press
If you plan to do work with hand tools, your most important purchases up-front are going to be a vise and a grinder. Build a bench or pedestal stand for the vise so it is at proper filing height (roughly, vise jaws at elbow level). If you have these already, you can start making your own hand tools. A cold chisel could be ground to a different shape, though you won't be able to get a cape chisel out of it without some forge work. A broaching-style tool could probably be made, roughly the size of the blind hole with one edge sharpened, and pounded into the hole with a hammer to form/smooth the sides of the hole. There's a lot to experiment with, just make sure you are grinding hardened tools and that they have the proper edge geometry for what you are trying to do.