> I wonder if there is ever going to be an encoding that replaces UTF-8?
Possibly if CJK keeps gaining importance there may be a new encoding which provides smaller encoding of these characters rather than the current primacy of western/european character sets.
Most commonly-used CJK characters are in the Basic Multilingual Plane, and take 2 bytes to represent in UTF-16 and 3 in UTF-8. So you're only really saving 33%, not 50%. Plus, if you're storing e.g. HTML, all the HTML markup characters go from one byte to two bytes, which is going to offset the CJK advantage to some degree (or maybe even outweigh it). I think most web documents are served compressed anyway, which makes the difference even smaller.
No, Unicode fares terribly for CJK languages, see Han Unification. I can't stress enough how it is a lie that Unicode can represent all languages in use, when it cannot even distinguish Japanese from Chinese.
It can't distinguish between English, French, German, Swedish, Norwegian, Spanish, etc. either. And there's no charset that does that. Do you find that a problem? If so, you're very much in the minority; if not, why distinguish between Japanese and Chinese but not English and French?
Swedish ä and German ä are the same character. 令 is a different character (of the same origin) in Japanese, Traditional Chinese and Simplied Chinese (and it depends on your browser setup which you get on screen, which is absolutely insane). It is like saying we should use English p as Cyrillic р, which only look similar, or as п, because they have the same origin or sound?
And no, CJK language users is not a minority, although many of us have become used to the font inconsistensies (because while wrong, they are legible to us).
The question comes down to if Chinese hanzi and Japanese kanji are the same alphabet or not. It's not clear what the answer should be, but for Unicode, it was absolutely necessary to treat them as the same alphabet, if they were not to break the 64K character limit (it's dubious Unicode would have seen its modern universality if it weren't a compact fixed 16-bit encoding, especially since the genius of UTF-8 encoding was a rather later addition).
Another point that is commonly missed is that Han unification was first proposed by East Asians (not ignorant Westerners), specifically the Chinese (which is why the Chinese don't really object to Unicode but the Japanese do).
A side note: it's interesting that you bring up ä as an example, because ä is actually a typographic unification of two very distinct letters. In languages like English and French, the diacritic is a diaeresis: a mark, derived from Ancient Greek indicating that the vowel is to be pronounced separately rather than as a diphthong. In languages like German, it's an umlaut, where it's a typographic reduction of a superscript e. They are in fact very different characters that merely look the same.
Whether the ideographs are the "same" or not is definitely a can of worm, both academically and politically speaking.
I am aware of the limits that the original Unicode specs were subjected to, which adds to my grief on how things could have been. Now that it has taken over the world we have to deal with wrong characters displayed for many applications. (Even modern day iOS has problems displaying mixed language content in system text fields.)
I don't know much about the history of western languages, so I cannot comment on the issue, just that it feels unjustified that during development CJK languages got crammed together and messed up (no matter who did it). More and more emojis and other symbols just rub salt into the wound.
I think it makes sense to "compress" the CJK characters to points where their appearance is the same, because there are on the order of one hundred thousand of them.
For western alphabets that have thirty or so symbols, the need is not nearly so acute.
And that is effectively what happened. Han unification has already done its damage and will continue to perform a lossy compression on all current and future CJK texts. Not that there is a way around...
Because the characters are different enough that it causes actual day-to-day problems (I see text all the time on printouts/signs where some incorrect font substitution has taken place and you get Chinese instead of Japanese characters). You can (probably correctly) argue that these are really just deficiencies in text editing/markup tools where you can't mark the language of text but the fact that that's required for correct presentation only in CJK languages seems to indicate that it's a problem with this unification in particular.
How would you like if there was Latin/Greek/Cyrillic unification? I mean they're all just alphabets that make the same sounds they're all basically the same right?
Possibly if CJK keeps gaining importance there may be a new encoding which provides smaller encoding of these characters rather than the current primacy of western/european character sets.