Visitar URL original
RST_STREAM error keeps showing up · Issue #7490 · googleapis/google-cloud-node · GitHub
Skip to content

RST_STREAM error keeps showing up #7490

Open

Description

@jakeleventhal

Environment details

  • OS: macOS Catalina 10.15.5 Beta (19F53f)
  • Node.js version: 13.7.0
  • npm version: 6.13.6
  • @google-cloud/firestore version: 3.7.4

Steps to reproduce

This error keeps appearing over and over in my logs (not regularly reproducible):

Error: 13 INTERNAL: Received RST_STREAM with code 2
at Object.callErrorFromStatus (/api/node_modules/@grpc/grpc-js/src/call.ts:81)
at Object.onReceiveStatus (/api/node_modules/@grpc/grpc-js/src/client.ts:324)
at Object.onReceiveStatus (/api/node_modules/@grpc/grpc-js/src/client-interceptors.ts:439)
at Object.onReceiveStatus (/api/node_modules/@grpc/grpc-js/src/client-interceptors.ts:402)
at Http2CallStream.outputStatus (/api/node_modules/@grpc/grpc-js/src/call-stream.ts:228)
at Http2CallStream.maybeOutputStatus (/api/node_modules/@grpc/grpc-js/src/call-stream.ts:278)
at Http2CallStream.endCall (/api/node_modules/@grpc/grpc-js/src/call-stream.ts:262)
at ClientHttp2Stream.<anonymous> (/api/node_modules/@grpc/grpc-js/src/call-stream.ts:532)
at ClientHttp2Stream.emit (events.js:315)
at ClientHttp2Stream.EventEmitter.emit (domain.js:485)
at emitErrorCloseNT (internal/streams/destroy.js:76)
at processTicksAndRejections (internal/process/task_queues.js:84)

Activity

  1. changed the title [-]Bandwidth exhausted consistently showing up[/-] [+]RST_STREAM error keeps showing up[/+] on Apr 16, 2020
  2. added
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on Apr 16, 2020
  3. schmidt-sebastian commented on Apr 17, 2020

    @schmidt-sebastian
    Contributor

    @jakeleventhal Thanks for filing this. Do you know at all what type of request/API call this is related to?

  4. jakeleventhal commented on Apr 19, 2020

    @jakeleventhal
    Author

    The only place i use grpc is via firestore so it must be something from there. I don't really have more info unfortunately

  5. added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    and removed
    🚨This issue needs some love.
    triage meI really want to be triaged.
    on May 8, 2020
  6. whyvrafvr commented on May 15, 2020

    @whyvrafvr

    I also got this one with firebase-admin 8.12.1 :

    {"error":"Error: 13 INTERNAL: Received RST_STREAM with code 2
        at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
       at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
        at Object.onReceiveStatus ...
  7. whyvrafvr commented on May 15, 2020

    @whyvrafvr

    This is a weird jeopardize issue, thrown during delete as create or whatever request type...

  8. schmidt-sebastian commented on May 18, 2020

    @schmidt-sebastian
    Contributor

    All document modifications (e.g. delete(), create()) use the same underlying request type. This request type is not safe for automatic retry, so unfortunately we cannot simply retry these errors.

  9. whyvrafvr commented on May 20, 2020

    @whyvrafvr

    Hi @schmidt-sebastian.
    I do confirm the request type is not safe.
    Retrying many times the same call (and so, the same request, I guess) has no significant effect.
    But there is no error when the nodejs instance is restarted : I guess a GRPC subroutine is hanging out.

  10. whyvrafvr commented on May 20, 2020

    @whyvrafvr

    After a while, errors, as mentioned by @jakeleventhal, are thrown and it’s not possible to persist data in firestore and an instance restart is required. That’s a real problem folks 😄
    FYI, I’ve no issue using firestore delete(), create() methods with Go...

  11. 160 remaining items

  12. Jhon-Idrovo commented on Jul 5, 2024

    @Jhon-Idrovo

    FIXED: While I was looking at the files I noticed the env variable NODE_ENV was unset. So I was trying to reach the emulator instead of the production DB. So if you're having this issue, check what you're connecting to

  13. leeyongeun9 commented on Nov 28, 2024

    @leeyongeun9

    Is there any update or progress regarding this issue? In my case, I encounter this problem frequently (~ 30 times per day) across various parts of my project where the reference.update() function is called. These errors pose a significant risk to my service, as they can lead to data inconsistencies.

    Below are logs from a single day, each occurring at different points in the code:

    Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)
        at callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
        at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client.js:193:76)
        at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
        at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
        at /workspace/node_modules/@grpc/grpc-js/build/src/resolving-call.js:129:78
        at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
    for call at
        at ServiceClientImpl.makeUnaryRequest (/workspace/node_modules/@grpc/grpc-js/build/src/client.js:161:32)
        at ServiceClientImpl.<anonymous> (/workspace/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
        at /workspace/node_modules/@google-cloud/firestore/build/src/v1/firestore_client.js:237:29
        at /workspace/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
        at repeat (/workspace/node_modules/google-gax/build/src/normalCalls/retries.js:82:25)
        at /workspace/node_modules/google-gax/build/src/normalCalls/retries.js:125:13
        at OngoingCallPromise.call (/workspace/node_modules/google-gax/build/src/call.js:67:27)
        at NormalApiCaller.call (/workspace/node_modules/google-gax/build/src/normalCalls/normalApiCaller.js:34:19)
        at /workspace/node_modules/google-gax/build/src/createApiCall.js:112:30
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    Caused by: Error
        at _firestore._traceUtil.startActiveSpan (/workspace/node_modules/@google-cloud/firestore/build/src/write-batch.js:438:27)
        at DisabledTraceUtil.startActiveSpan (/workspace/node_modules/@google-cloud/firestore/build/src/telemetry/disabled-trace-util.js:16:16)
        at WriteBatch.commit (/workspace/node_modules/@google-cloud/firestore/build/src/write-batch.js:436:43)
        at /workspace/node_modules/@google-cloud/firestore/build/src/reference/document-reference.js:396:18
        at DisabledTraceUtil.startActiveSpan (/workspace/node_modules/@google-cloud/firestore/build/src/telemetry/disabled-trace-util.js:16:16)
        at DocumentReference.update (/workspace/node_modules/@google-cloud/firestore/build/src/reference/document-reference.js:390:43)
        at addChatDetailToReservationIfNeeded (/workspace/index.js:1819:6)
        at /workspace/index.js:1470:9
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    
    Unhandled error Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)
        ... duplicate lines
        ...
    Caused by: Error
        ...
        ...
        at DocumentReference.update (/workspace/node_modules/@google-cloud/firestore/build/src/reference/document-reference.js:390:43)
        at /workspace/index.js:4901:10
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async /workspace/node_modules/firebase-functions/lib/common/providers/tasks.js:83:17 {
      code: 13,
      details: 'Received RST_STREAM with code 2 (Internal server error)',
      metadata: Metadata { internalRepr: Map(0) {}, options: {} },
      note: 'Exception occurred in retry method that was not classified as transient'
    }
    
    Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)
        ... duplicate lines
        ...
    Caused by: Error
        ...
        ...
        at DocumentReference.update (/workspace/node_modules/@google-cloud/firestore/build/src/reference/document-reference.js:390:43)
        at updateChatWithReservationUpdate (/workspace/index.js:1794:28)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    
    Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)
        ...
        ...
    Caused by: Error
        at _firestore._traceUtil.startActiveSpan (/workspace/node_modules/@google-cloud/firestore/build/src/write-batch.js:438:27)
        at DisabledTraceUtil.startActiveSpan (/workspace/node_modules/@google-cloud/firestore/build/src/telemetry/disabled-trace-util.js:16:16)
        at WriteBatch.commit (/workspace/node_modules/@google-cloud/firestore/build/src/write-batch.js:436:43)
        at updateChatlogData (/workspace/index.js:5597:25)
        at /workspace/index.js:3194:9
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    
  14. BrodaNoel commented on Nov 28, 2024

    @BrodaNoel

    I never got this issue again. Are you sure you are in the latest version of all the dependencies?

  15. leeyongeun9 commented on Nov 30, 2024

    @leeyongeun9

    I never got this issue again. Are you sure you are in the latest version of all the dependencies?

    Yes I updated all the dependencies and deplyed the functions after I wrote the reply but the errors still occured

    Environment Details

    • node : v20.9.0
    • npm : 10.1.0
    • firebase-admin : 13.0.1
    • firebase-functions : 6.1.1
    • @google-cloud/firestore : 7.10.0
  16. snallapa commented on Dec 19, 2024

    @snallapa

    running into this as well when doing a rather heavy migration

  17. AndersonFerreiraSantos commented on May 5, 2025

    @AndersonFerreiraSantos

    I have the same problem, apparently we don't have any solution at the moment, right? In my case, it seems that this problem occurs when my client has a very long duration.

  18. AndersonFerreiraSantos commented on May 8, 2025

    @AndersonFerreiraSantos

    Hi, I have a data import tool, the instances are used for long periods of time and I ended up getting this error.

    My solution was to add a time limit for the new Datastore, in this case I set it to 1 minute, but that's due to the large number of operations I perform, I tested with smaller modules and was successful with 30 minutes.

    Here's an example:

    ` const CLIENT_TTL = 1 * 60 * 1000;

    const datastoreClients = new TTLCache({ ttl: CLIENT_TTL });

    const getDatastoreClient = (_projectId = undefined) => {
    const projectId = getProjectId(_projectId);
    if (!datastoreClients.has(projectId)) {
    datastoreClients.set(projectId, new Datastore(Object.assign(Object.assign({}, datastoreConfig), { projectId: projectId })));
    }
    return datastoreClients.get(projectId);
    };`

  19. Akash16s commented on Jun 29, 2025

    @Akash16s

    I’ve run into this exact problem when hammering Firestore with very high write volumes (over 10 000 writes/sec per collection).

    To help, I built firestore-retry (https://www.npmjs.com/package/firestore-retry), which wraps your write calls in exponential backoff and retry logic.

    It won’t remove Firestore’s rate limits, but it will smooth out bursts and give you some breathing room while you scale. Let me know how it works for you!

  20. danbaechtold commented on Dec 16, 2025

    @danbaechtold

    Gemini 3 Pro seems to know where the bug comes from:

    The error @firebase/firestore: Firestore (12.6.0): GrpcConnection RPC 'Listen' stream ... error. Code: 13 Message: 13 INTERNAL: Received RST_STREAM with code 2 occurs because Google Cloud Firestore forcibly closes gRPC connections after 60 minutes.

    This is a known behavior of the Google Cloud infrastructure (specifically the Google Front End load balancers) which terminates long-lived connections after 1 hour. The Firebase SDK usually reconnects automatically, but it logs this error when the stream is reset.

    Why it happens:
    Google Cloud Load Balancers automatically terminate idle or long-lived gRPC connections after exactly 60 minutes (1 hour). Since the Firebase SDK for Node.js uses gRPC for real-time listeners, your connection hits this limit, and the server sends a RST_STREAM (reset stream) signal. The Firebase SDK logs this as an "Internal server error", although it typically reconnects automatically immediately after.

    It suggested to me to restart the listeners every 50min or so.. but is that really required and a smart thing to do?

  21. added
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    priority: p2Moderately-important priority. Fix may not be included in next release.
    🚨This issue needs some love.
    api: firestoreIssues related to the Firestore API.
    on Mar 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    🚨This issue needs some love.api: firestoreIssues related to the Firestore API.library: nodejs-firestoreIssues transferred from another repositorypriority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions