ER32 seems to be the most common size so there is more variety in ER32 accessories and options. ER32 also seems to be a little cheaper than ER40.
I have a set of collet blocks, and a spindexer in ER32. I now see ER40 collet blocks as an option but I don't think they were until recently, and I've never seen an ER40 spindexer.
I have several sizes though with the ability to use ER32 on several machines, ER16 on my Taig and Sherline, and ER50 on my 11" Powermatic / Logan. I have several chucks for the different machines, but share the same collet sets across them.
I might be collet crazy, because I also have MT2, 8mm watchmaker collets, Taig collets and 4NS collets. Lately I've been considering 5C...
Anyway going all ER40 works, but you may find benefit to having both ER32 and ER40 available to you.