Visitar URL original
Fix repo diff when diff=cpp attribute is used with utf-8 locales (on macOs) by herrerog · Pull Request #7399 · libgit2/libgit2 · GitHub
Skip to content

Fix repo diff when diff=cpp attribute is used with utf-8 locales (on macOs) - #7399

Open
herrerog wants to merge 2 commits into
libgit2:mainfrom
herrerog:fix/macos-diff-cpp
Open

herrerog wants to merge 2 commits into
libgit2:mainfrom
herrerog:fix/macos-diff-cpp

Conversation

@herrerog

@herrerog herrerog commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

I recently had to switch to macOS and while using our tools based on libgit2 (and pygit2), I faced below issue while trying to diff a c file from the linux kernel:

      for patch in repo.diff(commit.parents[0], commit):
                   ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  _pygit2.GitError: illegal byte sequence

Here is a simple reproducer:

import locale
from pathlib import Path
from tempfile import TemporaryDirectory
import pygit2

locale.setlocale(locale.LC_CTYPE, 'UTF-8')
with TemporaryDirectory() as directory:
    repo = pygit2.init_repository(directory)
    root = Path(directory)
    (root / '.gitattributes').write_text('*.c diff=cpp\n')
    source = root / 'test.c'
    source.write_text('int value = 1;\n')
    repo.index.add('test.c')
    repo.index.write()
    source.write_text('int value = 2;\n')
    print(repo.diff().patch)

When cpp driver is used, it seems that libgit2 compiles the regex pattern with this C string: "[\xc0-\xff][\x80-\xbf]+" (for example)
But it produces invalid utf-8 character that are rejected by regcomp_l on MacOS.
Given those patterns are not used, we can simply remove their compilation.

I've used AI for fixing the issue.

With *.c diff=cpp, generating an ordinary patch can fail on macOS with
"illegal byte sequence" under a UTF-8 locale. The built-in word pattern
contains this C string fragment:

    "[\xc0-\xff][\x80-\xbf]+"

The C compiler turns these escapes into raw bytes that are not valid UTF-8
in the pattern, so regcomp_l rejects it while loading the driver.

The word patterns were imported from Git alongside the function name
rules. The compiled word_pattern is stored and freed, but never used for
matching. Hunk headers use the separate fn_patterns list, so omitting
word-pattern compilation does not change hunk contents or headers.

Commit ab96ca5 ("Make sure we use the `C` locale for `regcomp` on
macOS.") addressed this error by switching to regcomp_l, but the failure
is still reproducible under a UTF-8 locale on macOS.

Stop compiling built-in word patterns when loading a driver and add a
regression test for diff=cpp under a UTF-8 locale.

Reproduce on macOS with pygit2 linked to libgit2 using regcomp_l:

python3 - <<'PYTHON'
import locale
from pathlib import Path
from tempfile import TemporaryDirectory
import pygit2

locale.setlocale(locale.LC_CTYPE, 'UTF-8')
with TemporaryDirectory() as directory:
    repo = pygit2.init_repository(directory)
    root = Path(directory)
    (root / '.gitattributes').write_text('*.c diff=cpp\n')
    source = root / 'test.c'
    source.write_text('int value = 1;\n')
    repo.index.add('test.c')
    repo.index.write()
    source.write_text('int value = 20;\n')
    print(repo.diff().patch)
PYTHON

Before the fix, generating the patch raises "illegal byte sequence".

Signed-off-by: Gregory Herrero <gregory.herrero@oracle.com>
Diff drivers compile diff.<driver>.wordregex and store the result in
word_pattern, but no matching code reads it. An invalid expression can
therefore prevent an ordinary line diff even though the pattern is unused.

Remove the compiled field and its compilation and disposal. Keep checking
whether wordregex is set, since a wordregex-only configuration selects a
custom driver. Dropping that lookup would allow fallback to a built-in
driver and change the function context in hunk headers.

Add tests to preserve driver selection with a valid wordregex and verify
that an invalid wordregex does not prevent patch generation.

Signed-off-by: Gregory Herrero <gregory.herrero@oracle.com>
@herrerog
herrerog force-pushed the fix/macos-diff-cpp branch from cbd0f38 to 837671f Compare October 4, 2026 20:06

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant