Visitar URL original
UCS2-based lowercasing breaks on certain scripts and combining sequences · Issue #2202 · DSharpPlus/DSharpPlus · GitHub
Skip to content

UCS2-based lowercasing breaks on certain scripts and combining sequences #2202

Open

Description

@akiraveliara

DSharpPlus.Commands currently lowercases user input if necessary based on System.Char, which is an UCS2 code point. (I am aware that UCS2 is not "up to date" terminology, the intent is to express that while the characters in the full string are represented as correct UTF16 characters, enumerating them yields them without accounting for combining characters or characters that require four bytes in UTF16 encoding.) This may produce incorrect results on any sufficiently complex script using diacritics in a way that is asymmetrically represented by Unicode, will produce incorrect results on any script where the relationship between uppercase and lowercase versions cannot be symmetrically represented in UCS2 code points or is asymmetric entirely and will produce incorrect results on any script that cannot be represented as merely UCS2 characters, with each char mapping to exactly one character (that is, alphabets using any characters greater than 0xFFFF). Affected scripts include, but are not limited to:

  • Osmanya, one of the scripts used for the Somali Language
  • Osage, used by the Osage language
  • Garay, occasionally used in the Wolof language
  • Khema, used by the Gurung language
  • Bamum, rarely used by the Bamum and Shümom languages
  • Mro, occastionally used by the Mru language
  • Tangsa, rarely used by the Tangsa language
  • Pahawh Hmong, used by the Hmong Daw and Hmong Njua languages
  • Kirat Rai, used by the Bantawa language
  • Nyiakeng Puachue Hmong, used by the White Hmong and Green Hmong languages
  • Toto, used by the Toto language
  • Wancho, rarely used by the Wancho language
  • Mundari Bani, rarely used by the Mundari language
  • Ol Onal, rarely used by the Bhumij language
  • Mende Kikakui, rarely used by the Mende language
  • Adlam, used by the Fula language

Furthermore, other scripts may need to rely on "extensions" in affected unicode planes, which will exhibit similar issues. Planes 1 (Supplementary Multilingual Plane), 2 (Supplementary Ideographic Plane) and 3 (Tertiary Ideographic Plane) are handled incorrectly today, and behaviour exhibited should be considered erroneous by default.

Since System.Char will slice UTF16 surrogate pairs into two System.Chars, which will then be treated as separate UCS2 characters, the value of the character during lowercasing may change to where it now represents two entirely distinct characters or it may render the string incomprehensible in its entirety: attempting to use Adlam characters, for example, will break the command in its entirety and it will not be uploaded to Discord at all, irrespective of its validity otherwise.

Activity

  1. added this to the Future milestone on Dec 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    commandsFor issues related to DSharpPlus.Commands

    Type

    Fields

    Effort

    None yet

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions