"Maybe the answer is something really sort-of fuzzy. It seems to me that people tend to buy a house when they want to lay down "roots" in a place, but then that wouldn't explain why so many people just rent for a long time despite not needing the mobility."

Wouldn't this come down to the risk appetite of renters vs. homeowners? Perhaps long-term renters overestimate the risks/trouble of homeowning and overvalue their mobility. In a sense, renters may be willing to pay a perceived premium (i.e. "you're throwing your money away") because they feel that they are safer. This feeling would only be reinforced if they've had favourable historical renting experiences.

I can't help drawing an analogy to buying insurance; you hope nothing bad happens, but buy insurance anyway.