Intermittent ExAws OIDC credential refresh error on Fly.io

[tofran]

I’m running an Elixir app on Fly.io using ex_aws with OpenID Connect (AssumeRoleWithWebIdentity) to access S3. This generally works fine, but intermittently I see the following error in Sentry. I haven’t observed any actual S3 failures, but the error does crash the ExAws auth cache process.

RuntimeError: Could't get credentials from auth adapter after 6 retries, last error was
{:http_error, 400,
 %{code: "ValidationError",
   message: "Value at 'webIdentityToken' failed to satisfy constraint: Member must have length greater than or equal to 4",
   request_id: "c0e5da71-d37c-4c59-937b-4977e130a69d"}}

Full stack trace:

RuntimeError: Could't get credentials from auth adapter after 6 retries, last error was {:http_error, 400, %{code: "ValidationError", message: "1 validation error detected: Value at 'webIdentityToken' failed to satisfy constraint: Member must have length greater than or equal to 4", type: "Sender", detail: "", request_id: "c0e5da71-d37c-4c59-937b-4977e130a69d"}}
    lib/ex_aws/config/auth_cache.ex:87: Elixir.ExAws.Config.AuthCache.ExAws.Config.AuthCache.attempt_credentials_refresh/5
    lib/ex_aws/config/auth_cache.ex:73: Elixir.ExAws.Config.AuthCache.ExAws.Config.AuthCache.refresh_awscli_config/3
    lib/ex_aws/config/auth_cache.ex:60: Elixir.ExAws.Config.AuthCache.ExAws.Config.AuthCache.handle_info/2
    gen_server.erl:2345: gen_server.:gen_server.try_handle_info/3
    gen_server.erl:2433: gen_server.:gen_server.handle_msg/6
    proc_lib.erl:329: proc_lib.:proc_lib.init_p_do_apply/3

This suggests that the OIDC token being read is sometimes empty or invalid, possibly during token rotation, but the system appears to recover on its own.

Dependency versions

* ex_aws 2.5.10 (Hex package) (mix)
  locked at 2.5.10 (ex_aws) 88fcd9cc
  ok
* ex_aws_s3 2.5.8 (Hex package) (mix)
  locked at 2.5.8 (ex_aws_s3) 84e512ca
  ok
* ex_aws_sts 2.3.0 (Hex package) (mix)
  locked at 2.3.0 (ex_aws_sts) f14e4c7d
  ok

Is this a known issue with OIDC token refresh on Fly.io + ExAws?
Might it be a bug in the ex_ libraries? Haven’t found anything similar.

Thank you.

[tofran]

For reference: I’ve also posted thus here and here.