Version 18.0 of the Unicode Standard is now available. The Unicode Standard is the foundation for all digital communications, and this new version supports a wider range of text encoding needs. This major update includes new characters and code charts, updated data files, and updated specifications that define many fundamental aspects of text processing. This version adds 13,007 new characters, including nine new emoji characters as well as many other characters and symbols, bringing the total number of encoded characters to 172,808. Among the most anticipated new characters are three new currency symbols: * U+20C2 RUFIYAA SIGN * U+20C3 UAE DIRHAM SIGN * U+20C4 OMANI RIAL SIGN Each of these symbols was authorized for public use by the respective monetary authority over a year ago, but usage has been hindered by lack of a standardized encoding. With the release of Unicode 18.0, vendors are now able to implement support for these symbols. The largest set of new characters is for the historical Seal (or “Small Seal”) script. This set of 11,328 ideographic characters has important cultural significance in China, dating back to the Qin Dynasty (around 200 BCE). Another new cultural heritage script from China is Jurchen, used in northeastern China during the Jin Dynasty. See the delta code charts for details on all the new scripts and characters. For additional details regarding new emoji, see Emoji Recently Added, v18.0 No new algorithms have been introduced in this release, but new data files have been added for Seal and other East Asian scripts, along with a new Unicode Standard Annex documenting these new data files: UAX #60, Data for East Asian Scripts. Conformance language related to variation selectors has been updated, making clearer which uses of variation selectors are or are not conformant to the Unicode Standard. Recommendations for implementations were also added to make non-conformant uses of variation selectors visible in text. This is important as research has shown that sequences of invisible variation selector characters can be used to attack modern AI applications. For complete details on Unicode Version 18.0, see https://www.unicode.org/versions/Unicode18.0.0/.

For too many people, moving the digits around in some variation of Patriots69Lover is their idea of a strong password. So you might expect something complicated like” “ji32k7au4a83” would be a great password. But according to the data breach repository Have I Been Pwned (HIBP), it shows up more often than one might expect. This interesting bit of trivia comes from self-described hardware/software engineer Robert Ou, who recently asked his Twitter followers if they could explain why this seemingly random string of numbers has been seen by HIBP over a hundred times. > Fun thing I learned today regarding secure passwords: the password "ji32k7au4a83" looks like it'd be decently secure, right? But if you check e.g. HIBP, it's been seen over a hundred times. Challenge: explain why and how this happened and how this password might be guessed > > — R (@ArcaneNibble) March 1, 2019 Have I Been Pwned is an aggregator that was started by security expert Troy Hunt to help people find out if their email or personal data has shown up in any prominent data breaches. One service it offers is a password search that allows you to check if your password has shown up in any data breaches that are on the radar of the security community. In this case, “ji32k7au4a83″ has been seen by HIBP in 141 breaches. Several of Ou’s followers quickly figured out the solution to his riddle. The password is coming from the Zhuyin Fuhao system for transliterating Mandarin. The reason it’s showing up fairly often in a data breach repository is because “ji32k7au4a83″ translates to English as “my password.” I reached out to my friend Ben Macaulay to help us verify what’s going on here. Macaulay is a graduate student in linguistics and Taiwan enthusiast who is focusing on endangered language documentation. He also regularly uses a Zhuyin (aka Bopomofo) keyboard, which is a key to this riddle. Macaulay told us it’s the most commonly used system for typing in Taiwan. The phonetic system is recognized by Unicode, and Macaulay confirmed that this is the general, simplified version of how the translation breaks down when the characters interact: > ji3 -> 我 -> M > 2K7 -> 的 -> Y > au4 -> 密 -> PASS > a83 -> 碼 -> WORD (Here are two charts explaining how the Unicode works.) That’s the only translation most of us need, but Macaulay elaborated on how it works. I’m just going to quote it all for no other reason than it makes me appreciate the fact that I only ever have to type in one language. > There are four tones: 1st tone (hold Space), 2nd tone (6), 3rd tone (3), 4th tone (4), unstressed/lack of tone (7). > Then, consonants for the beginning of the syllable, arranged by place of articulation: b (1) p (q) m (a) f (z); d (2) t (w) n (s) l (x); g (e) k (d) h (c); j (r) q (f) x (v); zh (5) ch (t) sh (g) r (b); z (y ) c (h). > Then, the vowels/semivowels: i/yi/y (u) u/wu/w (j) ü/yo (m); a (8) o (i) e (sounds like ‘uh’; k) e (sounds like ‘eh’; _). > Then, some syllable-final consonants and vowel+vowel/vowel+consonant combinations: ai (9) ei (o) ao (l) ou (!); an (0) en/-n (p) ang (;) eng/-ng (?). > To type in zhuyin, you type one of each (in that order, except the tone is last). > I = 我 = wo3 = u (j) + o (i) + 3rd tone (3). > Then the possessive marker 的 (like English ‘s) = de (toneless) = d (2) + e (k) + toneless (7). > Password = 密碼 = 密 ‘secret’ + 碼 (the second half of 號碼 ‘number’) 密 = mi4 = m (a) + i (u) + 4th tone (4) 碼 = ma3 = m (a) + a (8) + 3rd tone (3). What’s the lesson here? Well, you might conclude that people in Taiwan appear to have some bad password habits, just like the rest of us—but who knows what’s really going on here. They may be in on the joke. Also, you should add a quick check on the HIBP database to your good password creation practices just to make sure that your seemingly random string of characters doesn’t actually have another meaning. And above all, it’s a big wide world out there just waiting to pwn us all. [Robert Ou, Have I Been Pwned]

The Physics Behind the Poo Emoji’s Shape: And how lugworms tweak the rules by pooping upside down

A spatial design tool where everything renders as Unicode box-drawing characters. Create wireframes, diagrams, and mockups. Share as text.

A deep dive into how Punycode works, and why I think it's so neat.

Rahul's Blog

A collection of bad practices in HTML, copied from real websites.

An extensive guide on how to style for RTL in CSS

cartoon with two panels. The first one shows a sheriff with a curly mustache, curly cowboy hat, curly eyebrows, and additional shadows that make it 'fancy'; it is titled Sheriff. The second one is the same sheriff, but the mustache is straight also the eyebrows and the cowboy hat; it is titled Sans-Sheriff.

In a recent educational trick, curl contributor James Fuller submitted a pull-request to the project in which he suggested a larger cleanup of a set of scripts. In a later presentation, he could show us how not a single human reviewer in the team nor any CI job had spotted or remarked on one of the changes he included: he replaced an ASCII letter with a Unicode alternative in a URL. This was an eye-opener to several of us and we decided we needed to up our game. We are the curl project. We can do better. ## GitHub The replacement symbol looked identical to the ASCII version so it was not possible to visually spot this, but the diff viewer knows there is a difference. In this GitHub website screenshot below I reproduced a similar case. The right-side version has the Latin letter ‘g’ replaced with the Armenian letter co. They appear to be the same. GitHub shows a diff. But what is actually the difference? The diff viewer says there is a difference but as a human it isn’t possible to detect what it is. Is it a flaw? Does it matter? If done “correctly”, it would be done together with a _real_ and expected fix. The impact of changing one or more letters in a URL can of course be devastating depending on conditions. When I flagged about this rather big omission to GitHub people, I got barely no responses at all and I get the feeling the impact of this flaw is not understood and acknowledged. Or perhaps they are all just too busy implementing the next AI feature we don’t want. ## Warnings When we discussed this problem on Mastodon earlier this week, Viktor Szakats provided me with an example screenshot of doing a similar stunt with Gitea which quite helpfully highlights that there is something special about the replacement: Gitea warns that the replacement is using “ambiguous Unicode characters” I have been told that some of the other source code hosting services also show similar warnings. As a user, I would actually like to know even more than this, but at least this warns about the proposed change clearly enough so that if this happens I would get the code manually and investigate before accepting such a change. ## Detect While we wait for GitHub to wake up and react (which I have no expectation will actually happen anytime soon), we have implemented checks to help us poor humans spot things like this. _To detect malicious Unicode._ We have added a CI job that scans all files and validates every UTF-8 sequence in the git repository. In the curl git repository most files and most content are plain old ASCII so we can “easily” whitelist a small set of UTF-8 sequences and some specific files, the rest of the files are simply not allowed to use UTF-8 at all as they will then fail the CI job and turn up red. In order to drive this change home, we went through all the test files in the curl repository and made sure that all the UTF-8 occurrences were instead replaced by other kind of escape sequences and similar. Some of them were also used more or less by mistake and could easily be replaced by their ASCII counterparts. The next time someone tries this stunt on us it could be someone with less good intentions, but now ideally our CI will tell us. ## Confusables There are plenty of tools to find similar-looking characters in different Unicode sets. One of them is provided by the Unicode consortium themselves: https://util.unicode.org/UnicodeJsps/confusables.jsp ## Reactive This was yet another security-related fix _reacting_ on a demonstrated problem. I am sure there are plenty more problems which we have not yet thought about nor been shown and therefore we do not have adequate means to detect and act on automatically. We want and strive to be proactive and tighten everything _before_ malicious people exploit some weakness somewhere but security remains this never-ending race where we can only do the best we can and while _the other side_ is working in silence and might at some future point attack us in new creative ways we had not anticipated. That future unknown attack is a tricky thing.