Danger of a Break Down of Social Justice Rent is coercive for exactly the reasons you describe. You are not free to just exist somewhere except on someone else's terms, payment or otherwise, unless you have enough wealth to own a place that is yours. That means that people who don't own land of their own, as a class, have to do whatever those who own enough land to lend out, as a class, want, which is not freedom. Yeah, I'm not forced to pay my landlord in particular, unless I want to live on his land, but if I don't live on his land I will live somewhere else, and have to pay someone else to live on their land. Even if I want to buy land of my own, if I don't already have an equivalent amount of wealth to trade for it, I have to borrow from a bank, and pay them rent on that money -- interest -- if I want to continue living in "my" house, which makes it really their house. I cannot avoid somehow or another owing someone money just to be allowed to exist in some place, just to be left alone. Which leaves me severely disadvantaged when it comes to saving money with which to buy a place that's truly my own, because I have to pay so much to borrow someone else's place in the mean time.
Land is the primary example, because you cannot help but exist somewhere at all times and so always are in immediate need of some place in which you are allowed to exist. But all manner of rent and especially interest, which is just rent on money, enable a coerced transfer of wealth from those who have less than they need to those who have more than they need. That's because the lenders necessarily have more than they themselves think they need (if they have enough to be renting or lending out and so not using themselves), and the borrowers necessarily have less than they themselves think they need (if they're willing to be exploited like that to get it immediately instead of saving however long it would take to buy outright).
It's coerced and exploitative because it is not a straight-up equitable trade of one thing for another. At the start of the transaction, the lender has whatever he's lending out and the borrower has nothing. At the end of the transaction, when the lease is expired or the loan paid back, the lender has what he started with plus the rent or interest payments, and the borrower still has nothing, minus what he paid in rent or interest. But because the lender has enough to lend, more than he has to be using himself, and the borrower is in immediate need of it, the borrower has no choice but to accept those terms.
And that mechanism where the whole transaction moves wealth from those with less than they need to more than what they need is what breaks the naturally distributive nature one would naively expect form a free market. It's supposed to work so that the rich buy labor from the poor so that they don't have to work so much themselves, and the poor labor for the rich so that they can get richer, until the hardworking poor have wealth enough that they only need to labor to fund their ongoing consumption, and the idle rich lose wealth until they run out of excess to sell and have to start working to fund their own ongoing consumption too. If the rich want to stay rich, they'd need to work as hard as the hardworking poor do. That's how it's supposed to work in a truly free market, wealth goes to those who are doing the work, and if you slack off you lose it. But with rent and interest, so much of what the poor "buy" with the proceeds of their labor just gets returned to its rich owners, who can then use the proceeds from "selling" that "service" of lending it to pay for more of the labor of the poor, who then spend that back on rent and interest again, and so on, so the rich can sit idly forever making money off of the same wealth lent out over and over, while the poor keep working and working forever never making any headway.
If the gap between them is big enough, at least. If the difference in wealth is small enough then extra hard work or extra good luck or some combination thereof can surmount it still. But the point is that the existence of rent and interest systemically transfers wealth from the already-poor to the already-rich, in the process creating a pressure away from the middle class (defined here as those who have exactly as much as they need, and are neither lenders nor borrowers). That increases the gap between rich and poor more and more and makes it harder and harder for more and more people to cross it.
So long as that pressure away from center exists, some kind of counter-pressure toward the center is warranted. Ideally, there would be neither, but we're far from ideal right now.
Put another way: rent and interest enable what's basically a multi-nodal feudalism. Instead of working on one lord's land to generate profit (crops) only to then have to pay that same lord most of that profit in exchange for the right to have land to live and work on, we're "free" to work one "lord's" "land" (some employer) for our profit, only to have to pay most of that profit in exchange for the right to live on some other literal lord's land. We're still serfs, we've just got multiple lords now. We're not free until we own the things we need to work and live ourselves, and don't have to borrow them from others at interest.