Repository navigation
Firestore listener stops receiving changes after a long time but without throwing an error #18
Description
Environment details
- OS type and version: debian-10-buster-v20191210 (GCP image)
- Python version: 3.8.1
- pip version: 19.2.3
google-cloud-firestoreversion: 1.6.1
Steps to reproduce
I have a python script that sets up a listener to new documents in a firestore collection using on_snapshot on a query. To keep the script alive I have an infinite while loop in the end. Everything works fine but after several hours it stops receiving new documents (or the callback is not executed at least) - yesterday it was after ~7 hours.
In an earlier version of google-cloud-firestore (1.5.0) there was a related issue where the listener got disconnected after 1 hour. I made my own ugly hack back then but it worked at least:
Class ...
.
.
.
def _connect(self):
now = None
query = self.query
if self.add_time_restriction:
now = datetime.datetime.utcnow()
query = query.where('updated', '>=', now)
logger.info('Subscribing to query{}'.format(" after {}".format(now) if self.add_time_restriction else ""))
query_watch = query.on_snapshot(callback=self.callback)
if self.query_watch is not None:
logger.info("Unsubscribe to former watch")
self.query_watch.unsubscribe()
self.query_watch = query_watch
threading.Timer(3000, self._connect).start()Code example (sample)
Since the 1 hour issue was fixed I now do
class Firestore:
def __init__(self):
self.client = Client()
self._unsubscribe_news = lambda: None
def watch_news(
self
):
def on_snapshot(col_snapshot, changes, read_time):
logger.debug(f"Parsing news changes at {read_time}")
self._unsubscribe_news()
now = datetime.datetime.utcnow()
query_watch = self.client.collection("news").where("updated", ">", now).on_snapshot(
on_snapshot
)
logger.info(f"News listener set up at {self.client.project} from {now}")
self._unsubscribe_news = query_watch.unsubscribeand then execute a script like
firestore = Firestore()
firestore.watch_news()
while True:
passStack trace
Unfortunately no stack trace since the listener just stops receiving messages.
I see this as the core feature of Firestore so hopefully it'll get the right attention right away (some of our core business is to trigger python applications on news from firestore so we are extremely dependent on this).
Activity
NixBiks commentedon Feb 6, 2020
Just noticed this which seems like the same issue: firebase/firebase-admin-python#294. It concerns me that is has been open since May last year - isn't this considered a core feature?
NixBiks commentedon Feb 6, 2020
Btw. I always start two processes running in the background. Both listen to the same news but handle it differently. In the end both processes stopped receiving news at the same time.
BenWhitehead commentedon Feb 14, 2020
@crwilcox or @tseaver Any ideas here about what could be going on as far as the python code is concerned? I've not yet heard about this happening in other languages.
37 remaining items
crwilcox commentedon Sep 21, 2020
May be a tall ask, but do you have logs to match the failure? That has been the sticking point for fixing this: we know something is causing failures for users, but aren't aware of what that failure is.
MaticConradi commentedon Sep 25, 2020
@crwilcox the general consensus I believe has been that there are no logs produced by this issue whatsoever. I have all function code enclosed inside try except blocks with Google Error reporting integrated, and there just isn't anything related to it coming up, not even network related errors, nothing.
MaticConradi commentedon Sep 25, 2020
It also looks like the issue has been coming up more frequently for me lately, but this could be because of higher number of changes listeners had to have been handling on my end.
crwilcox commentedon Sep 25, 2020
@tseaver is your repro WIP PR capturing the failure at this point?
crwilcox commentedon Sep 25, 2020
@MaticConradi if you have the ability could you try
2.0.0dev1? I am hoping to release it in the next ~30 days, but have it in dev as it also brings along some new async surface and wanted a chance to experiment first.The underlying protobuffer code is generated differently also so it is possible it has an effect.
MaticConradi commentedon Sep 25, 2020
@crwilcox sure, I can look into migrating to v2 and seeing if that fixes it. Is there a list of known issues (if any) with existing functionality that I should be on the lookout for?
crwilcox commentedon Sep 25, 2020
I am not aware of issues, but there are some changes that are breaking. Particularly the way params are taken to some methods to protect against future API changes causing usability issues/breaks.
We have some scripts you can run on a codebase to move to this as well: https://github.com/googleapis/python-firestore/tree/master/scripts
Depending on what paths you use this could vary in the amount of change.
MaticConradi commentedon Sep 28, 2020
Doesn't look like the issue is fixed in
2.0.0dev1, I can confirm the snapshot listeners break after a few hours with no error messagesNixBiks commentedon Oct 2, 2020
Btw. I imagine that this is related; googleapis/google-cloud-node#7490 (comment)
tseaver commentedon Oct 6, 2020
@crwilcox
Nope.
tseaver commentedon Oct 9, 2020
@BenWhitehead, @crwilcox I'm not sure what is to be done here, beyond constructing the scripts to (attempt) reproducing this issue in googleapis/nodejs-firestore#158.
crwilcox commentedon Nov 4, 2020
@tseaver I also don't know what can be done to get a more firm repro. Anyone who is experiencing this, if you could try installing the candidate builds for v2.0.0 it is possible the work done to overhaul things a bit may have squashed this bug.
Also, if anyone has suggestions on how to reproduce the failure we would be interested to try to track this down.
dmahugh commentedon Jan 21, 2021
Closing this issue because it looks like v2.0.0 (released Nov 10, 2020) may have fixed it. If anyone experiences this behavior with v2.0 or above going forward, please open a new issue. Thanks!