The scare for me is that the "right not to be offended" or "hate speech victimhood" can be extended to those who are not the target of siad speech or even part of the conversation. you see this with the definition of hatespeech being peddled all over the place on most news outlets and even on the UK West Yorkshire Police website:

"A Hate Incident is any non-crime incident which is perceived by the victim or any other person to be motivated by hostility or prejudice based on a persons disability, race, religion, sexual orientation or gender identity or perceived disability, race, religion, sexual orientation or gender identity."
- https://www.westyorkshire.police.uk/advice/abuse-anti-social-behaviour/hate-crime/hate-crime-hate-incidents/hate-crime-hate-incidents

Perceived hate... by anyone is a "Hate Incident". This is where it gets out of hand and is indefensible, how do you put up a defense against such an accusation? banter with a good friend could be perceived badly by someone else and you've then commited a "Hate Incident". If people have a "right" not to be offended its nt hard to see "hate incidents", which are also reportable anyway, become a "hate crime".