The good thing with Matrix is that it's room-based, and I haven't really seen rooms with over 10k users. Presumably most of these nodes would be offline most of the time.

I can easily see routing being delegated to more powerful homeservers. That said, you do not need perfect routing to be better than hub and spoke at using the network effectively, and it seems to be what most current mesh networks aim for.