Visitar URL original
Ruby 3.3.5 - Consumer Pub Sub - Crashing with Cancelled error recurrently · Issue #27448 · googleapis/google-cloud-ruby · GitHub
Skip to content

Ruby 3.3.5 - Consumer Pub Sub - Crashing with Cancelled error recurrently #27448

Open
Open
Ruby 3.3.5 - Consumer Pub Sub - Crashing with Cancelled error recurrently#27448
Assignees
torreypayne
Labels
api: pubsubIssues related to the Pub/Sub API.priority: p3Desirable enhancement or fix. May not be included in next release.type: questionRequest for information or clarification. Not an issue.

Description

@rmzoni
rmzoni
opened on Oct 15, 2024

Environment details

OS: linux
Ruby version: 3.3.5
Gem name and version: google-cloud-pubsub v2.19.0

Steps to reproduce

  • just start the ConsumeEvents.new.consume method bellow using ruby 3.3.5
  • After a few minutes the error will happen and the process will stop
  • The error is: 1:CANCELLED. debug_error_string:{UNKNOWN:Error received from peer {grpc_message:"CANCELLED", grpc_status:1, created_time:"2024-10-15T13:48:37.628426185+00:00"}} (GRPC::Cancelled)

obs. This error occurs in a interval of 5 to 10 minutes and the POD in our K8s got restarted.

Code example

class ConsumeEvents
  def initialize(config, overrides = {})
    @config = config
    @subscriber_service = overrides.fetch(:subscriber_service) { ::Events::PubSub::Subscriber.new }
    @subscribers = {}
    @logger = overrides.fetch(:logger) { GRPC.logger }
  end

  def consume
    # Gracefully shut down the subscriber on program exit, blocking until
    # all received messages have been processed or n seconds have passed
    at_exit { stop_subscribers }

    loop do
      subscribe_to_configured_topics
      # Block, letting processing threads continue in the background
      sleep(15.seconds)
    end
  end

  def subscribe_to_configured_topics
    @config.each do |consumer_config|
      next if @subscribers.key?(consumer_config.subscription)

      subscriber = nil
      begin
        subscriber = @subscriber_service
          .with_topic_id(consumer_config.topic)
          .with_subscription_name(consumer_config.subscription) ## Error happen here
          ## This methos run this code herer
          # @pubsub = overrides.fetch(:pubsub) { Google::Cloud::Pubsub.new(project_id: ENV.fetch("PUBSUB_PROJECT_ID"), emulator_host: ENV.fetch("PUBSUB_EMULATOR_HOST", nil)) }
          # @pubsub.subscription(subscriber_name) || topic.subscribe(subscriber_name, **opts)
          .listen do |received_message|
            process_message(received_message, consumer_config)
          end
      rescue Events::PubSub::Subscriber::TopicNotFound => e
        set_error_on_span("Could not subscribe to topic: `#{consumer_config.key}`. Reason: #{e.message}")
        next
      rescue => e
        message = "General error #{e.class}: `#{consumer_config.key}` - `#{consumer_config.topic}` - `#{consumer_config.subscription}`. Reason: #{e.message}"
        set_error_on_span(message)
        @logger.error(message)

        raise StandardError.new(message)
      end

      subscriber.on_error { |exception| handle_error(exception, consumer_config) }

      subscriber.start

      @logger.info("Subscribed to #{consumer_config.topic}")
      @subscribers[consumer_config.subscription] = subscriber
    end
  end

  ...

end

Full backtrace

usr/local/bundle/gems/grpc-1.66.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:29:in check_status': 1:CANCELLED. debug_error_string:{UNKNOWN:Error received from peer {grpc_message:"CANCELLED", grpc_status:1, created_time:"2024-10-15T13:48:37.628426185+00:00"}} (GRPC::Cancelled) from /app/app/infra/events/pub_sub/subscription.rb:18:in subscribe_to'
│ from /app/app/infra/events/pub_sub/subscriber.rb:22:in with_subscription_name' │ from /app/app/infra/events/pub_sub/consumer/consume_events.rb:32:in block in subscribe_to_configured_topics'
│ from /app/app/infra/events/pub_sub/consumer/config.rb:53:in each' │ from /app/app/infra/events/pub_sub/consumer/config.rb:53:in each'
│ from /app/app/infra/events/pub_sub/consumer/consume_events.rb:25:in `subscribe_to_configured_topics'

Activity

  1. blunderbuss-gcf
    assigned
    dazuma
    on Oct 15, 2024
  2. AlanGasperini
    assigned
    aandreassa
    and unassigned
    dazuma
    on Nov 11, 2024

  3. aandreassa commented on Nov 12, 2024

    @aandreassa
    aandreassa
    on Nov 12, 2024
    Contributor

    @rmzoni or any others experiencing the problem, does the error still occur when downgrading below v2.19.0?

    Please try downgrading the underlying GAPIC as well (google-cloud-pubsub-v1) for testing the previous versions. We haven't been able to reproduce the problem yet, so this would be useful info.

  4. rmzoni commented on Nov 12, 2024

    @rmzoni
    rmzoni
    on Nov 12, 2024
    Author

    @rmzoni or any others experiencing the problem, does the error still occur when downgrading below v2.19.0?

    Please try downgrading the underlying GAPIC as well (google-cloud-pubsub-v1) for testing the previous versions. We haven't been able to reproduce the problem yet, so this would be useful info.

    Hi @aandreassa we´ve already made this test for lower versions. The problem still happen on version 2.15.1.

  5. davidgisbey
    added a commit that references this issue on Nov 21, 2024

    Bump Ruby to 3.2.6

    055f8e2

  6. rmzoni commented on Nov 28, 2024

    @rmzoni
    rmzoni
    on Nov 28, 2024
    Author

    @rmzoni or any others experiencing the problem, does the error still occur when downgrading below v2.19.0?
    Please try downgrading the underlying GAPIC as well (google-cloud-pubsub-v1) for testing the previous versions. We haven't been able to reproduce the problem yet, so this would be useful info.

    Hi @aandreassa we´ve already made this test for lower versions. The problem still happen on version 2.15.1.

    HI @aandreassa, any news about this issue? I saw that are other repos experience the same issue.

  7. kevindew commented on Jan 28, 2025

    @kevindew
    kevindew
    on Jan 28, 2025

    Has there been any progress on this? We had hit this issue before Christmas with Ruby 3.3.5 and have now tried with Ruby 3.4.1 and are experiencing the same.

    However we're not experiencing it in google-cloud-pubsub but the google-analytics-data-v1beta library (v0.15.0) with a stack trace of:

    Google::Cloud::CanceledError: 1:CANCELLED. debug_error_string:{UNKNOWN:Error received from peer  {grpc_message:"CANCELLED", grpc_status:1, created_time:"2025-01-28T16:20:18.190381328+00:00"}} (Google::Cloud::CanceledError)
/usr/local/bundle/ruby/3.4.0/gems/google-analytics-data-v1beta-0.15.0/lib/google/analytics/data/v1beta/analytics_data/client.rb:392:in 'Google::Analytics::Data::V1beta::AnalyticsData::Client#run_report'
/app/lib/analytics/ga4_import/data_fetcher.rb:43:in 'Analytics::Ga4Import::DataFetcher#get_data'
/app/lib/analytics/ga4_import/data_fetcher.rb:18:in 'block in Analytics::Ga4Import::DataFetcher#call'
/app/lib/analytics/ga4_import/data_fetcher.rb:17:in 'Analytics::Ga4Import::DataFetcher#call'
/app/lib/analytics/ga4_import/relevance_report_generator.rb:9:in 'Analytics::Ga4Import::RelevanceReportGenerator#call'
/app/lib/tasks/page_traffic.rake:10:in 'block (2 levels) in <top (required)>'
/usr/local/bundle/ruby/3.4.0/gems/rake-13.2.1/exe/rake:27:in '<top (required)>'
Caused by:
GRPC::Cancelled: 1:CANCELLED. debug_error_string:{UNKNOWN:Error received from peer  {grpc_message:"CANCELLED", grpc_status:1, created_time:"2025-01-28T16:20:18.190381328+00:00"}} (GRPC::Cancelled)
/usr/local/bundle/ruby/3.4.0/gems/grpc-1.69.0/src/ruby/lib/grpc/generic/active_call.rb:29:in 'Struct::BatchResult#check_status'
/usr/local/bundle/ruby/3.4.0/gems/grpc-1.69.0/src/ruby/lib/grpc/generic/active_call.rb:189:in 'GRPC::ActiveCall#attach_status_results_and_complete_call'
/usr/local/bundle/ruby/3.4.0/gems/grpc-1.69.0/src/ruby/lib/grpc/generic/active_call.rb:383:in 'GRPC::ActiveCall#request_response'
/usr/local/bundle/ruby/3.4.0/gems/grpc-1.69.0/src/ruby/lib/grpc/generic/client_stub.rb:174:in 'block (2 levels) in request_response'
/usr/local/bundle/ruby/3.4.0/gems/grpc-1.69.0/src/ruby/lib/grpc/generic/interceptors.rb:170:in 'GRPC::InterceptionContext#intercept!'
/usr/local/bundle/ruby/3.4.0/gems/grpc-1.69.0/src/ruby/lib/grpc/generic/client_stub.rb:173:in 'block in request_response'
/usr/local/bundle/ruby/3.4.0/gems/gapic-common-0.25.0/lib/gapic/grpc/service_stub/rpc_call.rb:130:in 'Gapic::ServiceStub::RpcCall#call'
/usr/local/bundle/ruby/3.4.0/gems/gapic-common-0.25.0/lib/gapic/grpc/service_stub.rb:217:in 'Gapic::ServiceStub#call_rpc'
/usr/local/bundle/ruby/3.4.0/gems/google-analytics-data-v1beta-0.15.0/lib/google/analytics/data/v1beta/analytics_data/client.rb:388:in 'Google::Analytics::Data::V1beta::AnalyticsData::Client#run_report'
/app/lib/analytics/ga4_import/data_fetcher.rb:43:in 'Analytics::Ga4Import::DataFetcher#get_data'
/app/lib/analytics/ga4_import/data_fetcher.rb:18:in 'block in Analytics::Ga4Import::DataFetcher#call'
/app/lib/analytics/ga4_import/data_fetcher.rb:17:in 'Analytics::Ga4Import::DataFetcher#call'
/app/lib/analytics/ga4_import/relevance_report_generator.rb:9:in 'Analytics::Ga4Import::RelevanceReportGenerator#call'
/app/lib/tasks/page_traffic.rake:10:in 'block (2 levels) in <top (required)>'
/usr/local/bundle/ruby/3.4.0/gems/rake-13.2.1/exe/rake:27:in '<top (required)>'
Tasks: TOP => page_traffic:load

  8. JosephLing commented on Jan 23, 2026

    @JosephLing
    JosephLing
    on Jan 23, 2026

    Likewise is there any progress on this? we are running into this issue with the latest version of the gem (3.1.1) and previously (2.6.1). We have only seen this on our projects within the US environment though over the last few weeks

    /bundle/ruby/3.3.0/gems/google-cloud-pubsub-v1-1.14.2/lib/google/cloud/pubsub/v1/subscription_admin/client.rb:1154:in `rescue in pull': 1:CANCELLED. debug_error_string:{UNKNOWN:Error received from peer  {created_time:\2026-01-23T11:59:02.950098342+00:00\, grpc_status:1, grpc_message:\CANCELLED\}} (Google::Cloud::CanceledError)
    from /bundle/ruby/3.3.0/gems/google-cloud-pubsub-v1-1.14.2/lib/google/cloud/pubsub/v1/subscription_admin/client.rb:1116:in `pull'
    from /bundle/ruby/3.3.0/gems/google-cloud-pubsub-3.1.1/lib/google/cloud/pubsub/service.rb:132:in `pull'
    from /bundle/ruby/3.3.0/gems/google-cloud-pubsub-3.1.1/lib/google/cloud/pubsub/subscriber.rb:220:in `pull'
    from /app/lib/pubsub/listener.rb:23:in `pull'
    from /app/bin/pubsub/listener.rb:9:in `block in <main>'
    from <internal:kernel>:187:in `loop'
    from /app/bin/pubsub/listener.rb:8:in `<main>'
    from /bundle/ruby/3.3.0/gems/railties-7.1.5.2/lib/rails/commands/runner/runner_command.rb:41:in `load'
    from /bundle/ruby/3.3.0/gems/railties-7.1.5.2/lib/rails/commands/runner/runner_command.rb:41:in `block in perform'
    from /bundle/ruby/3.3.0/gems/activesupport-7.1.5.2/lib/active_support/execution_wrapper.rb:92:in `wrap'
    from /bundle/ruby/3.3.0/gems/railties-7.1.5.2/lib/rails/commands/runner/runner_command.rb:40:in `perform'
    from /bundle/ruby/3.3.0/gems/thor-1.4.0/lib/thor/command.rb:28:in `run'
    from /bundle/ruby/3.3.0/gems/thor-1.4.0/lib/thor/invocation.rb:127:in `invoke_command'
    from /bundle/ruby/3.3.0/gems/railties-7.1.5.2/lib/rails/command/base.rb:178:in `invoke_command'
    from /bundle/ruby/3.3.0/gems/thor-1.4.0/lib/thor.rb:538:in `dispatch'
    from /bundle/ruby/3.3.0/gems/railties-7.1.5.2/lib/rails/command/base.rb:73:in `perform'
    from /bundle/ruby/3.3.0/gems/railties-7.1.5.2/lib/rails/command.rb:71:in `block in invoke'
    from /bundle/ruby/3.3.0/gems/railties-7.1.5.2/lib/rails/command.rb:149:in `with_argv'
    from /bundle/ruby/3.3.0/gems/railties-7.1.5.2/lib/rails/command.rb:69:in `invoke'
    from /bundle/ruby/3.3.0/gems/railties-7.1.5.2/lib/rails/commands.rb:18:in `<main>'
    from /bundle/ruby/3.3.0/gems/bootsnap-1.18.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
    from /bundle/ruby/3.3.0/gems/bootsnap-1.18.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
    from bin/rails:4:in `<main>'

  9. JosephLing commented on Jan 29, 2026

    @JosephLing
    JosephLing
    on Jan 29, 2026

    Has anyone had success in upgrading the grpc gem? as it looks like other people have run into the same issue here: grpc/grpc#39147

  10. amanda-tarafa
    assigned
    torreypayne
    and unassigned
    aandreassa
    on Oct 9, 2026
  11. amanda-tarafa
    added
    type: questionRequest for information or clarification. Not an issue.
    priority: p3Desirable enhancement or fix. May not be included in next release.
    on Oct 9, 2026

  12. amanda-tarafa commented on Oct 9, 2026

    @amanda-tarafa
    amanda-tarafa
    on Oct 9, 2026

    @torreypayne this seems to have been an issue with gRPC that was fixed in 1.74. Can you check that our gRPC dependencies are at that or later?

  13. amanda-tarafa
    added
    api: pubsubIssues related to the Pub/Sub API.
    on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: pubsubIssues related to the Pub/Sub API.priority: p3Desirable enhancement or fix. May not be included in next release.type: questionRequest for information or clarification. Not an issue.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions