Great question, I think this is the most important topic of our day.

Aside: The mainstream media has been one of the most important sense-making institutions in our society, like the senses of a human body. As it has deviated from truth, as its analysis and emphasis on problems, causes, and solutions has become driven more by agenda than truth, it's like having senses that are hallucinating. Directly, not the end of the world, but if you can't make sense of the world accurately, it becomes easier for internal and external threats to critically damage you. I think the general sense of decline we feel in many ways in the west can be traced to problems with mainstream media, which is why I think this topic is so important. Elaborating on this is a whole other topic though.

Some people would disagree with the premise of this question, that anything bad is happening with the mainstream media. Many have documented this phenomenon very well, names like Batya Ungar-Sargon, Bari Weiss, Matt Taibbi come to mind immediately. I'll start from the premise that this is happening, and give two causal explanations.

Technological/Economic: Mainstream media doesn't just have to compete with other mainstream media, it has to compete with all media. There are only 24 hours in a day, and if people's eyeballs are on something else, that has an economic impact on mainstream media. So where are people's eyeballs? Largely, social media. Facebook came onto the scene, and could provide more instantaneous, more personalized, and more engaging and interactive content. It's more emotionally stimulating, and immediately so. If traditional mainstream media is like healthy vegetables, social media is like sugary colorful Jell-O. News could only survive if it became shareable, more prone to arouse strong emotions, more real-time (faster to "press", less time for "shoe-leather journalism").

Hundreds of years ago, the technology to cheaply mass-produce, distribute, and store fun-colored, easily slurpable, super sugary Jell-O simply didn't exist. But everything from plastics to refrigeration and more created the potential for this. And once it became possible, a child is naturally going to need Jell-O far more than they'll want broccoli, and parents are stuck fighting an uphill battle to keep their kids healthy, and avoid obesity. So, assuming technological advancements are going to happen, there's a bit of an inevitability/historicity to this situation.

I think the same is true of the technological advancements in information technology inevitably creating the economic forces driving mainstream media to need to be more arousing and enraging, and more narrative-driven than driven by facts, data, and research.

Cultural/Economic: While the liberal west has a notion of a free and independent press which can hold government accountable as essential to a good and just society, what arises in practice is influenced by culture (and economics) as well. Think of this notion as a guiding principle, which hopefully points journalism in the right direction, but the specifics of what a given journalistic operation does, which ones successfully become mainstream, etc. depend on lots of other factors too.

There was a time when journalism in America was dominated by cheap, highly circulated papers that the common man would read, and was written for the common man, often holding the wealthy to account. The "comfortable" (as in "afflict the comfortable, comfort the afflicted") were the wealthy folks at this time. The journalists at these papers were typically working class folks as well.

The New York Times was founded on a different business model, rather than revenue from a large number of people purchasing the paper, it would be based on ad revenue. The aim was to cultivate an elite, high-spending audience and thereby command a premium from advertisers who wanted access to this lucrative consumer segment.

To cater to this audience, the writing had to be more high-brow, and journalism became a profession for the highly educated rather than the working class. As academia increasingly became a political echo chamber, and a place detached from reality and everyday people, that eventually flowed into journalism as it became a job almost exclusively for the very highly educated.

I think this graph sums it up best:


(Source)

On the far left you'll see academia, newspapers and print media, online computer services (the tech industry), and the entertainment industry ("Hollywood"). While Hollywood is not full of highly educated people, the tech industry is. But more importantly, what all four of these have in common is the same sorts of divisive, aggressively ideological, and self-righteous attitudes deeply woven into their internal cultures which, in the case of journalism, manifests itself in what we're seeing today.

Today, ad revenue is no longer the dominant source of revenue, but the profession remains one (for now) for the highly educated elite. They're driven to produce content that lets their readers feel they're better than half of society, that their readers can share and virtue signal to their group, so papers still need to produce articles that look and sound smart, in addition to being enraging.

In summary:

IT advances -> fast/emotional/real-time social media -> news/mainstream media more about emotion/narrative/agenda and less based on facts and research

??? -> academia becomes a left-leaning echo chamber -> professions that demand the most highly-educated new college grads fill their ranks of content producers with people who don't value neutrality

ad revenue targeting wealthy consumers who want to consume high-brow content proves to be a lucrative business model for news -> journalism becomes a profession that demands highly-educated college graduates -> with the social media model, subscription revenue overtakes ad revenue but the ranks are already filled with agenda-driven writers, plus the articles that get shared and drive engagement still need to sound smart and sophisticated even if the facts and research behind them are thin.