Add Unicode Property Parser in Text.Megaparsec.Char module #588

Activity

1. Pagwin2 commented on Dec 27, 2025

   

   Pagwin2

   on Dec 27, 2025

   Author

   More actions

   Unicode is somewhat monsterous so I understand if this is closed as Won't Implement
2. Pagwin2

   changed the title [-]Unicode Property Parser in `Text.Megaparsec.Char` module[/-] [+]Add Unicode Property Parser in `Text.Megaparsec.Char` module[/+] on Dec 27, 2025
3. mrkkrp commented on Dec 31, 2025

   

   mrkkrp

   on Dec 31, 2025

   Owner

   More actions

   I won't implement this myself, but it should not be too hard to do (I will gladly accept a PR). Take a look at how various combinators in Text.Megaparsec.Char are implemented—most of them are thin wrappers around satisfy. So, it seems like you just need a correct predicate function Char -> Bool. It does look like GHC.Unicode from base could be of help, it defines GeneralCategory, which seems relevant, although I am not sure whether it is the same concept as properties.
4. mrkkrp

   added

   feature-request

   on Dec 31, 2025
5. mrkkrp commented on Dec 31, 2025

   

   mrkkrp

   on Dec 31, 2025

   Last edited by mrkkrp

   Owner

   More actions

   I see now that GeneralCategory only classifies a Char into one of the categories, so it is more crude than properties. Possibly unicode-data or text-icu could allow for more precise analysis, but I haven't looked at them in detail. I have to warn you that I am reluctant to add more dependencies to Megaparsec at this point.

   I'm not sure what this parser's API should look like e.g. should it allow parsing for a codepoint dependant on it having multiple properties or one of a selection of properties or only allow validation based on one property

   I guess in its most general form it the combinator should accept a predicate that takes a set of properties and returns a Bool.
6. mrkkrp commented on Nov 14, 2026

   

   mrkkrp

   on Nov 14, 2026

   Owner

   More actions

   @Pagwin2 After thinking about it more, I don't think this belongs in Megaparsec. Keeping Unicode property data complete and up to date is a job of its own, and the unicode-data package already does it (it is currently at Unicode 18.0.0). Since character parsers in Megaparsec are thin wrappers around satisfy, you can combine the two directly. For the ECMAScript grammar you linked:

   import Unicode.Char.Identifiers (isIDContinue, isIDStart)
   
   identifierStartChar :: Parser Char
   identifierStartChar = satisfy (\c -> isIDStart c || c == '$' || c == '_')
   
   identifierPartChar :: Parser Char
   identifierPartChar =
     satisfy (\c -> isIDContinue c || c == '$' || c == '\x200C' || c == '\x200D')

   The same module also provides isXIDStart, isXIDContinue, isPatternSyntax, and isPatternWhitespace, and Unicode.Char.General provides isAlphabetic and isWhiteSpace, among others.

   I'm going to close this, but thanks for the suggestion.
7. mrkkrp

   closed this as not plannedon Nov 14, 2026
8. Pagwin2 commented on Nov 15, 2026

   

   Pagwin2

   on Nov 15, 2026

   Author

   More actions

   @Pagwin2 After thinking about it more, I don't think this belongs in Megaparsec. Keeping Unicode property data complete and up to date is a job of its own, and the unicode-data package already does it (it is currently at Unicode 18.0.0). Since character parsers in Megaparsec are thin wrappers around satisfy, you can combine the two directly. For the ECMAScript grammar you linked:

   import Unicode.Char.Identifiers (isIDContinue, isIDStart)

   identifierStartChar :: Parser Char
   identifierStartChar = satisfy (\c -> isIDStart c || c == '$' || c == '_')

   identifierPartChar :: Parser Char
   identifierPartChar =
   satisfy (\c -> isIDContinue c || c == '$' || c == '\x200C' || c == '\x200D')

   The same module also provides isXIDStart, isXIDContinue, isPatternSyntax, and isPatternWhitespace, and Unicode.Char.General provides isAlphabetic and isWhiteSpace, among others.

   I'm going to close this, but thanks for the suggestion.

   Understood, thank you for taking the time to reply and give an alternative.