Visitar URL original
UnicodeDecodeError when reading Python 2 objects from memcache · Issue #70 · GoogleCloudPlatform/appengine-python-standard · GitHub
Skip to content

UnicodeDecodeError when reading Python 2 objects from memcache #70

Closed
Closed
UnicodeDecodeError when reading Python 2 objects from memcache#70
#91 (+2)
Assignees
estrellis

Description

@dave-2
dave-2
opened on Mar 13, 2023

Expected Behavior

memcache.get() returns an object from memcache, even if it was added to memcache using the python27 runtime.

Actual Behavior

Traceback (most recent call last):
  ... <truncated> ...
  File "/srv/services/secrets_svc.py", line 59, in GetSecrets
    secrets = memcache.get(GLOBAL_KEY)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/appengine/api/memcache/__init__.py", line 583, in get
    results = rpc.get_result()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/appengine/api/apiproxy_stub_map.py", line 648, in get_result
    return self.__get_result_hook(self)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/appengine/api/memcache/__init__.py", line 652, in __get_hook
    value = _decode_value(returned_item.value,
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/appengine/api/memcache/__init__.py", line 289, in _decode_value
    return do_unpickle(value)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/appengine/api/memcache/__init__.py", line 425, in _do_unpickle
    return unpickler.load()
UnicodeDecodeError: 'ascii' codec can't decode byte 0x93 in position 55: ordinal not in range(128)

Steps to Reproduce the Problem

  1. Call memcache.set() on an ndb.Model object in python27.
  2. Call memcache.get() to get the object in python39.

This reproduces even with MEMCACHE_USE_CROSS_COMPATIBLE_PROTOCOL set in app.yaml. This sets the pickling protocol to 2, which only affects pickling and not unpickling, since the protocol is autodetected upon unpickling.

env_variables:
  MEMCACHE_USE_CROSS_COMPATIBLE_PROTOCOL: "2"

Workaround

There is a workaround by setting the encoding to 'bytes' in the memcache unpickler. It must be 'bytes' instead of 'latin1' because the ndb.Model deserializer expects a bytes object.

import functools
import six

from google.appengine.api import memcache

unpickler = functools.partial(six.moves.cPickle.Unpickler, encoding='bytes')
memcache.setup_client(memcache.Client(unpickler=unpickler))

The Python docs and bug tracker indicate that the encoding argument of pickle.Unpickler() is used to help with the awkwardness in differences in str between Python 2 and 3.
https://docs.python.org/3/library/pickle.html#pickle.Unpickler
https://bugs.python.org/issue22005
https://stackoverflow.com/questions/28218466/unpickling-a-python-2-object-with-python-3/28218598#28218598

Specifications

  • Version: appengine-python-standard==0.3.1 and appengine-python-standard==1.0.0
  • Platform: python39

Activity

  1. pnico commented on Apr 27, 2023

    @pnico
    pnico
    on Apr 27, 2023

    does this happen with any ndb.Model? I thought this was only an issue with datetime/date objects?

  2. dave-2 commented on Apr 27, 2023

    @dave-2
    dave-2
    on Apr 27, 2023
    Author

    does this happen with any ndb.Model? I thought this was only an issue with datetime/date objects?

    I'm not sure if it's any ndb.Model. I encountered it with a Model containing 3 properties of type ndb.StringProperty(required=True) and nothing else.

  3. pnico commented on Apr 28, 2023

    @pnico
    pnico
    on Apr 28, 2023

    what?? oh no.. I thought ok maybe DateTimeProperty.. but yeah, no 😄
    I don't know how to "escalate" an issue here but that sounds like a big problem.

    it doesn't seem like encoding='bytes' is really acceptable either, or is it? won't that break other stuff that needs to be unicode? does this happen with the normal, automated memcache caching ndb does or only when you do it manually like this?

    Otherwise maybe Python 3 ndb just needs its own memcache namespace or something, it would double memcache use during the transition but it should work..?

    Edit: if the automated ndb caching works, and this is only when manually storing things in memcache, I think the best way might be to just use different memcache keys for your Python 2 and Python 3 deployments. But it does raise questions about other things, like deferred, pickling ndb.Key objects with string ids, etc

  4. pnico commented on Apr 28, 2023

    @pnico
    pnico
    on Apr 28, 2023

    Can you post a small example with code that shows this issue? Edit: actually I'm wondering about the string stored in the property - if it's something that can't be decoded to utf-8

  5. dave-2 commented on May 2, 2023

    @dave-2
    dave-2
    on May 2, 2023
    Author

    Sorry, I won't be able to at the moment, though I think the issue description contains enough information to create a small test case.

  6. ludoch
    assigned
    shreejad
    and
    estrellis
    and unassigned
    estrellis
    and
    shreejad
    on May 3, 2023
  7. ludoch
    closed this as completedin #88on May 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions