Mechanically, you are absolutely correct. The 1/4-20 is one of a few taps that the closest drill size is undersized compared to the theoretical hole diameter requirements.I've always had more trouble with a 1/4-20 tap. Almost no material left after the flutes are cut.
Perhaps I am just very careful with a #6 tap, since it is so small. I always use a very small tap handle for #10 or smaller taps.