For a social species, we sure do like to exclude each other. Many people like to be comforted by being better than others, rather than being in a group. If we weren't so competitive, perhaps we could happily accept that we are fools, so long as we knew we weren't the only one.
But we are a competitive species, and do not like the idea that we are a fool, even if it were common to be one. Therefore, the bigger insult is saying that all men are fools.