mTLS host rewrite skips Agent Platform sandboxEnvironments calls ('sandbox' in url matches the path), causing 401 under Agent Identity; base_url workaround then doubles to mtls.mtls #3031

Environment

• google-genai 2.23.0 (reproduced unchanged on 2.26.0 and on main at 41b1ea6)
• google-cloud-aiplatform 2.2.0 (agentplatform package, which builds on google.genai._api_client.BaseApiClient)
• Python 3.13
• Runtime: Gemini Enterprise Agent Platform / Agent Runtime, BYOC container, identity_type=AGENT_IDENTITY, us-central1. The client cert config is present at /var/run/secrets/workload-spiffe-credentials/credentials.json, mtls.should_use_client_cert() and mtls.has_default_client_cert_source() both return true, and the client uses the mTLS AuthorizedSession path added in feat: Autoenable mTLS in environment with bound token (Agent Engine with AgentAuthority) through google-auth migration (except custom client args, custom client or custom ClientSession) #2172 (a95d08a).

Description

When mTLS is active, BaseApiClient rewrites the request host like this (google/genai/_api_client.py on main, lines ~1497, 1575, 1648, 2057 and 2431: sync, async, streaming, upload and download paths):

if self._authorized_session._is_mtls and 'googleapis.com' in url:
  if 'sandbox' in url:
    url = url.replace('sandbox.googleapis.com', 'mtls.sandbox.googleapis.com')
  else:
    url = url.replace('googleapis.com', 'mtls.googleapis.com')

The 'sandbox' in url branch looks meant for the *.sandbox.googleapis.com staging hosts (the tests use e.g. us-central1-autopush-aiplatform.sandbox.googleapis.com). But it tests the whole URL, path included. That causes two problems:

1. Agent Platform sandbox calls skip the mTLS host. Every sandbox API path has sandbox in it (.../reasoningEngines/<id>/sandboxEnvironments..., used by sandboxes.create/get/execute/delete). So these calls take the staging branch, the replace finds nothing to match, and the request goes to the plain {region}-aiplatform.googleapis.com host with a certificate-bound Agent Identity token. The server rejects it with:

   401 UNAUTHENTICATED: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential.
   

   This happens on every call for the life of the process. The same calls work with ordinary user ADC (no cert, no mTLS). Other calls from the same client (e.g. to the reasoning engine itself) are rewritten correctly and work.

2. The rewrite isn't idempotent, so base_url can't work around it. If the caller sets http_options.base_url to https://{region}-aiplatform.mtls.googleapis.com/, sandbox calls do go through. But any call whose path lacks sandbox, such as polling the .../operations/<id> returned by sandboxes.create, takes the else branch and becomes {region}-aiplatform.mtls.mtls.googleapis.com. That fails with SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] ... Hostname mismatch. No single base_url works for both kinds of call.

Reproduction (no credentials, no network)

The script below patches the two mtls checks to report a client cert. It then swaps in a fake mTLS session that records the final URL instead of sending the request.

from google.oauth2.credentials import Credentials
from google.auth.transport import mtls
from google.genai import _api_client, version

# Pretend we're on Agent Runtime with an Agent Identity client cert.
mtls.should_use_client_cert = lambda: True
mtls.has_default_client_cert_source = lambda: True


class Sent(Exception):
  pass


class FakeMtlsSession:
  """Stands in for google.auth AuthorizedSession after configure_mtls_channel()."""
  _is_mtls = True

  def request(self, method, url, **kwargs):
    raise Sent(url)


def final_url(path, base_url=None):
  client = _api_client.BaseApiClient(
      vertexai=True,
      project='my-project',
      location='us-central1',
      credentials=Credentials(token='placeholder'),  # never sent anywhere
      http_options={'base_url': base_url} if base_url else None,
  )
  client._authorized_session = FakeMtlsSession()
  try:
    client.request('get', path, {})
  except Sent as e:
    return e.args[0]


print('google-genai', version.__version__)
engine = 'reasoningEngines/123'
mtls_host = 'https://us-central1-aiplatform.mtls.googleapis.com/'
cases = [
    ('(a) sandbox call, default base_url', f'{engine}/sandboxEnvironments', None),
    ('    control: non-sandbox call, default base_url', f'{engine}', None),
    ('(b) operation poll, base_url = mTLS host', f'{engine}/operations/456', mtls_host),
]
for label, path, base in cases:
  print(f'{label}\n    -> {final_url(path, base)}')

Output (2.23.0 and 2.26.0 are identical):

google-genai 2.26.0
(a) sandbox call, default base_url
    -> https://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-project/locations/us-central1/reasoningEngines/123/sandboxEnvironments
    control: non-sandbox call, default base_url
    -> https://us-central1-aiplatform.mtls.googleapis.com/v1beta1/projects/my-project/locations/us-central1/reasoningEngines/123
(b) operation poll, base_url = mTLS host
    -> https://us-central1-aiplatform.mtls.mtls.googleapis.com/v1beta1/projects/my-project/locations/us-central1/reasoningEngines/123/operations/456

Expected

• (a) https://us-central1-aiplatform.mtls.googleapis.com/.../sandboxEnvironments
• (b) https://us-central1-aiplatform.mtls.googleapis.com/.../operations/456, i.e. a host that is already mTLS is left alone.

Actual

• (a) stays on the plain host, which returns a 401 for a bound token.
• (b) gets mtls. added a second time, which fails TLS hostname verification.

Suggested fix

Base the decision on the hostname, not the full URL string, and skip hosts that are already mTLS. One helper used by all five call sites:

def _to_mtls_url(url: str) -> str:
  parsed = urlparse(url)
  host = parsed.netloc
  if '.mtls.' in host:
    return url
  if 'sandbox.googleapis.com' in host:
    host = host.replace('sandbox.googleapis.com', 'mtls.sandbox.googleapis.com')
  else:
    host = host.replace('googleapis.com', 'mtls.googleapis.com')
  return urlunparse(parsed._replace(netloc=host))

The existing 'googleapis.com' in url check at each call site still gates the call. urlparse/urlunparse are already imported in _api_client.py.

Workaround

Use two clients: one whose base_url is the regional *.mtls.googleapis.com host for sandboxEnvironments calls, and a second with the default host for polling the long-running operation that sandboxes.create returns. After making this change, creates under Agent Identity were accepted and returned an operation. That confirms the host is the cause of the 401s, not IAM.

Related: #1956, #2172.