Voltage drop is an important consideration, but I am not sure I agree with your calculations. Assuming the OP has a standard 120/240 split phase service, then only the unbalanced current between the hot legs ends up travelling back on the neutral.
Using your example above, with 20A of 120V for receptacles and another 15A for lighting, assuming that the single pole breakers are connected to different poles in the panel (and any panel should have its 120V breakers balanced!) then you only have a 5A delta between the legs, and the neutral will end up carrying that 5A.
In other words, you would end up with 15A of 240V and only 5A of 120V in that example. Add in the 240V loads and you have a total of 45 amps of 220V and only 5A of 120V.
In most cases the feeds to outbuildings end up carrying very little current over the neutral. In big installs, it is allowed to undersize the neutral compared to ungrounded conductors for this reason.