Firstly, it is easily understandable that people rather would fall for the idea of striving towards pleasure than higher values: this is also where the problem occurs when reasoning it trough. There isn't really a reason to choose responsibility for suffering - but if responsibility is preferred in this matter, you're good.
Hedonism comes with a cost - guilt. To act upon reason is to act upon the categorical imperative of Kant - meaning that you then have the obligation to act as you wish others would act. Hedonism is pure egoism, you do not wish a society full of people that choose pleasure over maintaining order. If we were to sacrifice everything to make the most pleasurable life, we would end up losing everything and have no real core-value to our way of life (also a reason to why many are sceptical to modernisation.)