Sandbox::https://mail.python.org/archives/list/python-dev@python.org/2000/8/

[PEP 223] Change the Meaning of \x Escapes
by Tim Peters Feb. 15, 2001

Feb. 15, 2001

An HTML version of the attached can be viewed at http://python.sourceforge.net/peps/pep-0223.html This will be adopted for 2.0 unless there's an uproar. Note that it *does* have potential for breaking existing code -- although no real-life instance of incompatibility has yet been reported. This is explained in detail in the PEP; check your code now. although-if-i-were-you-i-wouldn't-bother<0.5-wink>-ly y'rs - tim PEP: 223 Title: Change the Meaning of \x Escapes Version: $Revision: 1.4 $ Author: tpeters(a)beopen.com (Tim Peters) Status: Active Type: Standards Track Python-Version: 2.0 Created: 20-Aug-2000 Post-History: 23-Aug-2000 Abstract Change \x escapes, in both 8-bit and Unicode strings, to consume exactly the two hex digits following. The proposal views this as correcting an original design flaw, leading to clearer expression in all flavors of string, a cleaner Unicode story, better compatibility with Perl regular expressions, and with minimal risk to existing code. Syntax The syntax of \x escapes, in all flavors of non-raw strings, becomes \xhh where h is a hex digit (0-9, a-f, A-F). The exact syntax in 1.5.2 is not clearly specified in the Reference Manual; it says \xhh... implying "two or more" hex digits, but one-digit forms are also accepted by the 1.5.2 compiler, and a plain \x is "expanded" to itself (i.e., a backslash followed by the letter x). It's unclear whether the Reference Manual intended either of the 1-digit or 0-digit behaviors. Semantics In an 8-bit non-raw string, \xij expands to the character chr(int(ij, 16)) Note that this is the same as in 1.6 and before. In a Unicode string, \xij acts the same as \u00ij i.e. it expands to the obvious Latin-1 character from the initial segment of the Unicode space. An \x not followed by at least two hex digits is a compile-time error, specifically ValueError in 8-bit strings, and UnicodeError (a subclass of ValueError) in Unicode strings. Note that if an \x is followed by more than two hex digits, only the first two are "consumed". In 1.6 and before all but the *last* two were silently ignored. Example In 1.5.2: >>> "\x123465" # same as "\x65" 'e' >>> "\x65" 'e' >>> "\x1" '\001' >>> "\x\x" '\\x\\x' >>> In 2.0: >>> "\x123465" # \x12 -> \022, "3456" left alone '\0223456' >>> "\x65" 'e' >>> "\x1" [ValueError is raised] >>> "\x\x" [ValueError is raised] >>> History and Rationale \x escapes were introduced in C as a way to specify variable-width character encodings. Exactly which encodings those were, and how many hex digits they required, was left up to each implementation. The language simply stated that \x "consumed" *all* hex digits following, and left the meaning up to each implementation. So, in effect, \x in C is a standard hook to supply platform-defined behavior. Because Python explicitly aims at platform independence, the \x escape in Python (up to and including 1.6) has been treated the same way across all platforms: all *except* the last two hex digits were silently ignored. So the only actual use for \x escapes in Python was to specify a single byte using hex notation. Larry Wall appears to have realized that this was the only real use for \x escapes in a platform-independent language, as the proposed rule for Python 2.0 is in fact what Perl has done from the start (although you need to run in Perl -w mode to get warned about \x escapes with fewer than 2 hex digits following -- it's clearly more Pythonic to insist on 2 all the time). When Unicode strings were introduced to Python, \x was generalized so as to ignore all but the last *four* hex digits in Unicode strings. This caused a technical difficulty for the new regular expression engine: SRE tries very hard to allow mixing 8-bit and Unicode patterns and strings in intuitive ways, and it no longer had any way to guess what, for example, r"\x123456" should mean as a pattern: is it asking to match the 8-bit character \x56 or the Unicode character \u3456? There are hacky ways to guess, but it doesn't end there. The ISO C99 standard also introduces 8-digit \U12345678 escapes to cover the entire ISO 10646 character space, and it's also desired that Python 2 support that from the start. But then what are \x escapes supposed to mean? Do they ignore all but the last *eight* hex digits then? And if less than 8 following in a Unicode string, all but the last 4? And if less than 4, all but the last 2? This was getting messier by the minute, and the proposal cuts the Gordian knot by making \x simpler instead of more complicated. Note that the 4-digit generalization to \xijkl in Unicode strings was also redundant, because it meant exactly the same thing as \uijkl in Unicode strings. It's more Pythonic to have just one obvious way to specify a Unicode character via hex notation. Development and Discussion The proposal was worked out among Guido van Rossum, Fredrik Lundh and Tim Peters in email. It was subsequently explained and disussed on Python-Dev under subject "Go \x yourself", starting 2000-08-03. Response was overwhelmingly positive; no objections were raised. Backward Compatibility Changing the meaning of \x escapes does carry risk of breaking existing code, although no instances of incompabitility have yet been discovered. The risk is believed to be minimal. Tim Peters verified that, except for pieces of the standard test suite deliberately provoking end cases, there are no instances of \xabcdef... with fewer or more than 2 hex digits following, in either the Python CVS development tree, or in assorted Python packages sitting on his machine. It's unlikely there are any with fewer than 2, because the Reference Manual implied they weren't legal (although this is debatable!). If there are any with more than 2, Guido is ready to argue they were buggy anyway <0.9 wink>. Guido reported that the O'Reilly Python books *already* document that Python works the proposed way, likely due to their Perl editing heritage (as above, Perl worked (very close to) the proposed way from its start). Finn Bock reported that what JPython does with \x escapes is unpredictable today. This proposal gives a clear meaning that can be consistently and easily implemented across all Python implementations. Effects on Other Tools Believed to be none. The candidates for breakage would mostly be parsing tools, but the author knows of none that worry about the internal structure of Python strings beyond the approximation "when there's a backslash, swallow the next character". Tim Peters checked python-mode.el, the std tokenize.py and pyclbr.py, and the IDLE syntax coloring subsystem, and believes there's no need to change any of them. Tools like tabnanny.py and checkappend.py inherit their immunity from tokenize.py. Reference Implementation The code changes are so simple that a separate patch will not be produced. Fredrik Lundh is writing the code, is an expert in the area, and will simply check the changes in before 2.0b1 is released. BDFL Pronouncements Yes, ValueError, not SyntaxError. "Problems with literal interpretations traditionally raise 'runtime' exceptions rather than syntax errors." Copyright This document has been placed in the public domain.

9 16

Application for developer status
by Eric S. Raymond Oct. 9, 2000

Oct. 9, 2000

May I have developer status on the SourceForge CVS, please? I maintain two standard-library modules (shlex and netrc) and have been involved with the development of several others (including Cmd, smtp, httplib, and multifile). My only immediate plan for what to do with developer access is to add the browser-launch capability previously discussed on this list. My general interest is in improving the standard class library, especially in the areas of Internet-protocol support (urllib, ftp, telnet, pop, imap, smtp, nntplib, etc.) and mini-language toolkits and frameworks (shlex. netrc, Cmd, ConfigParser). If the Internet-protocol support in the library were broken out as a development category, I would be willing to fill the patch-handler slot for it. -- <a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a> See, when the GOVERNMENT spends money, it creates jobs; whereas when the money is left in the hands of TAXPAYERS, God only knows what they do with it. Bake it into pies, probably. Anything to avoid creating jobs. -- Dave Barry

8 16

Re: [I18n-sig] Unicode experience
by Guido van Rossum Sept. 5, 2000

Sept. 5, 2000

> Toby Dickenson wrote: > > > > I'm just nearing the end of getting Zope to play well with unicode > > data. Most of the changes involved replacing a call to str, in > > situations where either a unicode or narrow string would be > > acceptable. > > > > My best alternative is: > > > > def convert_to_something_stringlike(x): > > if type(x)==type(u''): > > return x > > else: > > return str(x) > > > > This seems like a fundamental operation - would it be worth having > > something similar in the standard library? Marc-Andre Lemburg replied: > You mean: for Unicode return Unicode and for everything else > return strings ? > > It doesn't fit well with the builtins str() and unicode(). I'd > say, make this a userland helper. I think this would be helpful to have in the std library. Note that in JPython, you'd already use str() for this, and in Python 3000 this may also be the case. At some point in the design discussion for the current Unicode support we also thought that we wanted str() to do this (i.e. allow 8-bit and Unicode string returns), until we realized that there were too many places that would be very unhappy if str() returned a Unicode string! The problem is similar to a situation you have with numbers: sometimes you want a coercion that converts everything to float except it should leave complex numbers complex. In other words it coerces up to float but it never coerces down to float. Luckily you can write that as "x+0.0" while converts int and long to float with the same value while leaving complex alone. For strings there is no compact notation like "+0.0" if you want to convert to string or Unicode -- adding "" might work in Perl, but not in Python. I propose ustr(x) with the semantics given by Toby. Class support (an __ustr__ method, with fallbacks on __str__ and __unicode__) would also be handy. > BTW, could you elaborate a bit on your experience with adding > Unicode support to Zope ? Such a report would certainly make > a nice complement to the Unicode tutorial and help other > people adding Unicode support to their apps. Yes, that's what we need. Thanks to Toby for pioneering this! --Guido van Rossum (home page: http://dinsdale.python.org/~guido/)

11 17

Re: [Python-Dev] stack check on Unix: any suggestions?
by Skip Montanaro Sept. 3, 2000

Sept. 3, 2000

Charles> I get the exact same value. Of course the amount of other Charles> stuff running makes no differemce, you get the core dump Charles> because you've hit the RLIMIT for stack usage, not because Charles> you've exhausted memory. Amount of RAM in the machine, or swap Charles> space in use has nothing to do with it. Do "ulimit -s Charles> unlimited" and see what happens... Makes no difference: % ./python Python 2.0b1 (#81, Aug 31 2000, 15:53:42) [GCC 2.95.3 19991030 (prerelease)] on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam Copyright 1995-2000 Corporation for National Research Initiatives (CNRI) >>> % ulimit -a core file size (blocks) 0 data seg size (kbytes) unlimited file size (blocks) unlimited max locked memory (kbytes) unlimited max memory size (kbytes) unlimited open files 1024 pipe size (512 bytes) 8 stack size (kbytes) unlimited cpu time (seconds) unlimited max user processes 2048 virtual memory (kbytes) unlimited % ./python Misc/find_recursionlimit.py ... Limit of 2300 is fine recurse add repr init getattr getitem Limit of 2400 is fine recurse add repr Segmentation fault Skip

4 7

Namespace collision between lib/xml and site-packages/xml
by Mark Favas Sept. 3, 2000

Sept. 3, 2000

On July 26 I reported that the new xml package in the standard library collides with and overrides the xml package from the xml-sig that may be installed in site-packages. This is still the case. The new package does not have the same functionality as the one in site-packages, and hence my application (and others relying on similar functionality) gets an import error. I understood that it was planned that the new library xml package would check for the site-package version, and transparently hand over to it if it existed. It's not really an option to remove/rename the xml package in the std lib, or to break existing xml-based code... Of course, this might be fixed by 2.0b1, or is it a feature that will be frozen out <wry smile>? Fred's response was: " I expect we'll be making the package in site-packages an extension provider for the xml package in the standard library. I'm planning to discuss this issue at today's PythonLabs meeting." -- Mark

2 2

Pragma-style declaration syntax
by M.-A. Lemburg Sept. 2, 2000

Sept. 2, 2000

I've been tossing some ideas around w/r to adding pragma style declarations to Python and would like to hear what you think about these: 1. Embed pragma declarations in comments: #pragma: name = value Problem: comments are removed by the tokenizer, yet the compiler will have to make use of them, so some logic would be needed to carry them along. 2. Reusing a Python keyword to build a new form of statement: def name = value Problem: not sure whether the compiler and grammar could handle this. The nice thing about this kind of declaration is that it would generate a node which the compiler could actively use. Furthermore, scoping would come for free. This one is my favourite. 3. Add a new keyword: decl name = value Problem: possible code breakage. This is only a question regarding the syntax of these meta- information declarations. The semantics remain to be solved in a different discussion. Comments ? Thanks, -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

7 18

Re: [Python-Dev] test_gettext.py fails on 64-bit architectures
by bwarsaw＠beopen.com Sept. 2, 2000

Sept. 2, 2000

>>>>> "MF" == Mark Favas <m.favas(a)per.dem.csiro.au> writes: MF> Close, but no cigar - fixes the miscalculation of BE_MAGIC, MF> but "magic" is still read from the .mo file as MF> 0xffffffff950412de (the 64-bit rep of the 32-bit negative MF> integer 0x950412de) Thanks to a quick chat with Tim, who is always quick to grasp the meat of the issue, we realize we need to & 0xffffffff all the 32 bit unsigned ints we're reading out of the .mo files. I'll work out a patch, and check it in after a test on 32-bit Linux. Watch for it, and please try it out on your box. Thanks, -Barry

2 1

lookdict
by Vladimir.Marangozov＠inrialpes.fr Sept. 1, 2000

Sept. 1, 2000

I'd like to request some clarifications on the recently checked dict patch. How it is supposed to work and why is this solution okay? What's the exact purpose of the 2nd string specialization patch? Besides that, I must say that now the interpreter is noticeably slower and MAL and I were warning you kindly about this code, which was fine tuned over the years. It is very sensible and was optimized to death. The patch that did make it was labeled "not ready" and I would have appreciated another round of review. Not that I disagree, but now I feel obliged to submit another patch to make some obvious perf improvements (at least), which simply duplicates work... Fred would have done them very well, but I haven't had the time to say much about the implementation because the laconic discussion on the Patch Manager went about functionality. Now I'd like to bring this on python-dev and see what exactly happened to lookdict and what the BeOpen team agreed on regarding this function. -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

3 7

threadmodule.c comment error? (from comp.lang.python)
by Fredrik Lundh Sept. 1, 2000

Sept. 1, 2000

as noted by curtis jensen over at comp.lang.python: the parse tuple string doesn't quite match the error message given if the 2nd argument isn't a tuple. on the other hand, the args argument is initialized to NULL... thread_PyThread_start_new_thread(PyObject *self, PyObject *fargs) { PyObject *func, *args = NULL, *keyw = NULL; struct bootstate *boot; if (!PyArg_ParseTuple(fargs, "OO|O:start_new_thread", &func, &args, &keyw)) return NULL; if (!PyCallable_Check(func)) { PyErr_SetString(PyExc_TypeError, "first arg must be callable"); return NULL; } if (!PyTuple_Check(args)) { PyErr_SetString(PyExc_TypeError, "optional 2nd arg must be a tuple"); return NULL; } if (keyw != NULL && !PyDict_Check(keyw)) { PyErr_SetString(PyExc_TypeError, "optional 3rd arg must be a dictionary"); return NULL; } what's the right way to fix this? (change the error message and remove the initialization, or change the parsetuple string and the tuple check) </F>

2 1

Cookie.py security
by A.M. Kuchling Sept. 1, 2000

Sept. 1, 2000

[CC'ed to python-dev and Tim O'Malley] The Cookie module recently added to 2.0 provides 3 classes of Cookie: SimpleCookie, which treats cookie values as simple strings, SerialCookie, which treats cookie values as pickles and unpickles them, and SmartCookie which figures out if the value is a pickle or not. Unpickling untrusted data is unsafe. (Correct?) Therefore, SerialCookie and SmartCookie really shouldn't be used, and Moshe's docs for the module say so. Question: should SerialCookie and SmartCookie be removed? If they're not there, people won't accidentally use them because they didn't read the docs and missed the warning. Con: breaks backward compatibility with the existing cookie module and forks the code. (Are marshals safer than pickles? What if SerialCookie used marshal instead?) --amk

11 14

Python-Dev

August 2000