Visitar URL original
gh-96268: Fix loading invalid UTF-8 by mdboom · Pull Request #96270 · python/cpython · GitHub
Skip to content

gh-96268: Fix loading invalid UTF-8 #96270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gvanrossum merged 11 commits into from
Sep 7, 2022
Merged

gh-96268: Fix loading invalid UTF-8 #96270

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
e4aaa14
gh-96268: Fix loading invalid UTF-8
mdboom Aug 25, 2022
407eef7
Add blurb
mdboom Aug 25, 2022
e453819
Merge remote-tracking branch 'upstream/main' into fix-valid-utf8
mdboom Aug 25, 2022
3d60ff7
Fix blurb
mdboom Aug 25, 2022
5cc57ad
Use assertIn instead
mdboom Aug 25, 2022
ad4de7a
Fix reference to other decoding function
mdboom Aug 31, 2022
18927b1
Fix coding style
mdboom Aug 31, 2022
f741a9d
Add comments about handled code ranges in each branch
mdboom Aug 31, 2022
f8e9e6e
Fix line number in error message
mdboom Aug 31, 2022
ace4a8c
Remove obsolete comment
mdboom Sep 6, 2022
df074a8
PEP7
mdboom Sep 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
gh-96268: Fix loading invalid UTF-8 
This makes tokenizer.c:valid_utf8 match stringlib/codecs.h:decode_utf8.

This also fixes the related test so it will always detect the expected failure
and error message.
  • Loading branch information
@mdboom
mdboom committed Aug 25, 2022
commit e4aaa14a427bf4797259b68cac6f79fd8e934719
13 changes: 10 additions & 3 deletions Lib/test/test_source_encoding.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,16 +236,23 @@ def test_invalid_utf8(self):
# test it is to write actual files to disk.

# Each example is put inside a string at the top of the file so
# it's an otherwise valid Python source file.
template = b'"%s"\n'
# it's an otherwise valid Python source file. Put some newlines
# beforehand so we can assert that the error is reported on the
# correct line.
template = b'\n\n\n"%s"\n'

fn = TESTFN
self.addCleanup(unlink, fn)

def check(content):
with open(fn, 'wb') as fp:
fp.write(template % content)
script_helper.assert_python_failure(fn)
rc, stdout, stderr = script_helper.assert_python_failure(fn)
# We want to assert that the python subprocess failed gracefully,
# not via a signal.
self.assertGreaterEqual(rc, 1)
self.assertTrue(b"Non-UTF-8 code starting with" in stderr)
Comment thread
mdboom marked this conversation as resolved.
Outdated
self.assertTrue(b"on line 5" in stderr)
Comment thread
mdboom marked this conversation as resolved.
Outdated

# continuation bytes in a sequence of 2, 3, or 4 bytes
continuation_bytes = [bytes([x]) for x in range(0x80, 0xC0)]
Expand Down
30 changes: 21 additions & 9 deletions Parser/tokenizer.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -489,25 +489,37 @@ static void fp_ungetc(int c, struct tok_state *tok) {

/* Check whether the characters at s start a valid
UTF-8 sequence. Return the number of characters forming
the sequence if yes, 0 if not. */
the sequence if yes, 0 if not. The special cases match
those in stringlib/codecs.h:decode_utf8.
Comment thread
mdboom marked this conversation as resolved.
Outdated
*/
static int valid_utf8(const unsigned char* s)
Comment thread
mdboom marked this conversation as resolved.
Outdated
{
int expected = 0;
int length;
if (*s < 0x80)
if (*s < 0x80) {
/* single-byte code */
return 1;
if (*s < 0xc0)
/* following byte */
return 0;
if (*s < 0xE0)
} else if (*s < 0xE0) {
if (*s < 0xC2) {
Comment thread
gvanrossum marked this conversation as resolved.
return 0;
}
expected = 1;
else if (*s < 0xF0)
} else if (*s < 0xF0) {
if (*s == 0xE0 && *(s + 1) < 0xA0) {
return 0;
} else if (*s == 0xED && *(s + 1) >= 0xA0) {
return 0;
}
expected = 2;
else if (*s < 0xF8)
} else if (*s < 0xF5) {
if (*(s + 1) < 0x90 ? *s == 0xF0 : *s == 0xF4) {
return 0;
}
Comment thread
gvanrossum marked this conversation as resolved.
expected = 3;
else
} else {
/* invalid start byte */
return 0;
}
length = expected + 1;
for (; expected; expected--)
if (s[expected] < 0x80 || s[expected] >= 0xC0)
Expand Down