Sort and deduplicate

Chinese does not sort by code point

This is the thing most line-sorting tools get wrong, and it is invisible until you use one on Chinese.

A plain sort orders strings by their Unicode code points. For Chinese that produces an order no reader recognises — it is neither by stroke count, nor by pinyin, nor by radical, nor by anything a dictionary uses. It is simply the order the characters happened to be assigned numbers in.

So the collation here is an explicit choice, and it is named under the result:

• Stroke count — how a Traditional Chinese index is usually ordered, and how names are listed in Taiwan and Hong Kong.
• Pinyin — alphabetical by Mandarin reading, which is the mainland convention and how most Simplified dictionaries are arranged.
• Code point — kept, because it is what a bare sort does and seeing the difference is the point.

Two plausible orders that disagree is exactly the situation where a tool has to say which one it used.

Natural order for numbers

The default puts item2 before item10. A plain string sort does the opposite, because 1 sorts before 2 character by character, and that is almost never what anybody means by sorting a list of files or versions.

Deduplication keeps the first occurrence

Order is information. Removing later duplicates preserves whatever meaning the original order carried, where keeping the last would quietly reshuffle it.

Case-insensitive by default, because Apple and apple are usually the same entry in a list of names, tags or domains. Turn it on when case is significant — identifiers, passwords, base64.

Trimming is separate and on by default: apple and apple are the same item in every list anyone actually pastes, and a stray trailing space is the most common reason a duplicate survives.

Shuffle uses real randomness

Fisher–Yates, drawing from the platform’s cryptographic generator rather than Math.random. That is more than a list of names needs, but it costs nothing and means the result is genuinely unbiased — a naive shuffle using sort(() => Math.random() - 0.5) is not, and produces visibly skewed orders.

Nothing is uploaded

Sorted in this page. Lists pasted into a sorter are often email addresses, customer names or internal hostnames.