Possible race condition in credential refresh

We are seeing a possible race condition around credential refresh. We use the following code in an integration test suite to fetch OAuth2 tokens:

    GoogleCredentials credentials = ...;
    try {
      if (forceRefresh) {
        credentials.refresh();
      } else {
        credentials.getRequestMetadata();
      }

      AccessToken token = credentials.getAccessToken();
      return token.getTokenValue();
    } catch (IOException e) {
      throw new RuntimeException(e);
    }

We expect the credentials to contain a valid access token at the end of the if-else block. However, our tests intermittently fail by throwing a NullPointerException at token.getTokenValue(). Any thoughts on what might be going on here? What should we do to ensure that getAccessToken() always returns a non-null value?

Note: In the failing tests, forceRefresh has been set to false. So it should execute the credentials.getRequestMetadata() path.

Seeing more strange behavior. Was trying to debug the above issue a bit further by running the following simple test case:

    GoogleCredentials cred = loadServiceAccount();
    
    cred.refresh();
    String t1 = cred.getAccessToken().getTokenValue();

    // Thread.sleep(1000);

    cred.refresh();
    String t2 = cred.getAccessToken().getTokenValue();
    assertNotEquals(t1, t2);

But this would fail most of the time:

java.lang.AssertionError: Values should be different. Actual: ya29.c.KqgBBwh5fCszBWUTEhptmtMA-1AAjGMcJK1XoNbOmH1rJDIOqYErmTCF1c3w1VIx1vc_1CiAB2AlBWuxkGHF75QSJ18mZkXX_54IcW9skiyA5Plmx26USngo4kiqsj1Wbk5IoT3A-...
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failEquals(Assert.java:187)
	at org.junit.Assert.assertNotEquals(Assert.java:163)
	at org.junit.Assert.assertNotEquals(Assert.java:177)
	at com.google.firebase.auth.FirebaseAuthIT.testMe(FirebaseAuthIT.java:106)

Introducing some delay before the second refresh operation (just uncomment the sleep), increases success rate greatly.

This happens randomly for us about once or twice per week. It occurs both locally on developer machines as well as in deployed environments such as staging and production.
We use the Cloud SQL postgres-socket-factory for IAM authentication between our applications and our Postgres databases. This library contains code to the effect of (see CloudSqlInstance.java):

GoogleCredentials credentials = ...;
credentials.refresh();
String token = credentials.getAccessToken().getTokenValue();

From this we see exceptions with these sample stack traces:

HikariPool-1 - Exception during pool initialization.
Error:	org.postgresql.util.PSQLException: Something unusual has occurred to cause the driver to fail. Please report this exception.
	at org.postgresql.Driver.connect(Driver.java:280)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
<snip>
Caused by: java.lang.NullPointerException: Cannot invoke "com.google.auth.oauth2.AccessToken.getTokenValue()" because the return value of "com.google.auth.oauth2.OAuth2Credentials.getAccessToken()" is null
	at com.google.cloud.sql.core.CloudSqlInstance.fetchEphemeralCertificate(CloudSqlInstance.java:524)
	at com.google.cloud.sql.core.CloudSqlInstance.lambda$performRefresh$0(CloudSqlInstance.java:330)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

The effect of this for us is that our applications randomly fail to start. Please fix this.

fix: Proposal on how to fix issue googleapis#692

Revert "fix: Proposal on how to fix issue googleapis#692"

fix: Resolve race condition reported in #692 (#1031)

chore(main): release 1.12.1 (#1059)

Navigation Menu

Repository navigation

Possible race condition in credential refresh #692

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Possible race condition in credential refresh #692

Description

Activity

hiranya911 commented on Jul 9, 2021

Ironlink commented on Jun 24, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions