Visitar URL original
Configurable default log_level and log_format for injected Vault Agent containers · Issue #417 · hashicorp/vault-k8s · GitHub
Skip to content

Configurable default log_level and log_format for injected Vault Agent containers #417

Open
Open
Configurable default log_level and log_format for injected Vault Agent containers#417
Labels
enhancementNew feature or requestgood first issueGood for newcomers

Description

@itspngu
itspngu
opened on Jan 28, 2023

Is your feature request related to a problem? Please describe.
While it is possible to configure the log level & format for the injector deployment itself using the command line or environment variables, and configuring the log level & format for injected Vault Agent sidecars on a per-pod basis using annotations, there is no way to configure a default log level & format for injected Vault Agent sidecars globally (e.g. setting the log format of all injected Vault Agent containers to json instead of default unless explicitly specified otherwise in the per-pod annotation).

Describe the solution you'd like
Add a means to override the defaults for injected Vault Agent containers' log level and format instead of hardcoding them to info and default.

Describe alternatives you've considered
Using the per-pod annotations to configure these settings is, albeit a working solution, getting cumbersome once you have a lot of workloads relying on Vault Agent. Adding vault.hashicorp.com/log-format: "json" yields is supposed to yield the needed result of getting JSON log output from the Agent containers, but having to do so in dozens of places (regardless of whether it's manually or through automation) instead of being able to configure the injector accordingly feels like a missing feature.

Additional context
If I understand the code correctly, these 2 lines are the culprit: https://github.com/hashicorp/vault-k8s/blob/main/agent-inject/agent/agent.go#L378-L379

Edited: Only the vault.hashicorp.com/log-level annotation works, vault.hashicorp.com/log-format appears to have no effect, so this is actually a bug report on top of a feature request.

Activity

  1. itspngu
    added
    enhancementNew feature or request
    on Jan 28, 2023

  2. itspngu commented on Jan 28, 2023

    @itspngu
    itspngu
    on Jan 28, 2023
    Author

    Actually, the annotation isn't working either - when vault.hashicorp.com/log-format: "json" is added to a pod template, the injected Vault Agent container still logs in the default format.

  3. itspngu commented on Jan 28, 2023

    @itspngu
    itspngu
    on Jan 28, 2023
    Author

    /kind bug

  4. tvoran commented on Jan 31, 2023

    @tvoran
    tvoran
    on Jan 31, 2023
    Member

    Hi @itspngu, when I set vault.hashicorp.com/log-format: "json" I do see json logging come from agent, though there are some non-json log lines (especially at the debug level) which is maybe what you're seeing? They look like they're coming from consul-template, which agent uses under the hood.

    Regardless, you should open an issue on the vault repo for issues with agent logging format. There's been some discussion over there about it recently: hashicorp/vault#18081

  5. tvoran
    added
    good first issueGood for newcomers
    on Jan 31, 2023

  6. tvoran commented on Jan 31, 2023

    @tvoran
    tvoran
    on Jan 31, 2023
    Member

    Being able to set the default log level and format for injected agents would be good to have. I think we'd probably want to add a couple new flags and env variables, probably something like agent-log-level and agent-log-format here:

    vault-k8s/subcommand/injector/flags.go

    Line 136 in 5827448

    func (c *Command) init() {

    The names of the environment variables for the setting gets a little more complicated, since there are already AGENT_INJECT_LOG_LEVEL and AGENT_INJECT_LOG_FORMAT for the injector service itself.

  7. itspngu commented on Feb 2, 2023

    @itspngu
    itspngu
    on Feb 2, 2023
    Author

    Hi @itspngu, when I set vault.hashicorp.com/log-format: "json" I do see json logging come from agent, though there are some non-json log lines (especially at the debug level) which is maybe what you're seeing? They look like they're coming from consul-template, which agent uses under the hood.

    Regardless, you should open an issue on the vault repo for issues with agent logging format. There's been some discussion over there about it recently: hashicorp/vault#18081

    You're right, I didn't look at it for long enough to realize that it's only messages of DEBUG and TRACE severity levels which unconditionally log in text format, rather than json. That'd indeed be something to report in the main Vault repository, however I'd like to focus on the original intent of this issue for now.

    If we can find names for the proposed environment variables (or skip them and just opt for CLI flags), I'd happily implement this and put it up for review. If I were to be nitpicky, I'd say that the AGENT_INJECT_LOG_LEVEL and AGENT_INJECT_LOG_FORMAT variable names are misleading, and could be changed to AGENT_INJECTOR_LOG_LEVEL and AGENT_INJECTOR_LOG_FORMAT, freeing the other ones for configuring the agent itself, but that would break existing configurations.

  8. Kyslik commented on Jul 13, 2023

    @Kyslik
    Kyslik
    on Jul 13, 2023

    We experience this very issue and it is a bit annoying as it produces a lot of logs that have no value. For clarity, in the end we want to use log-level: error, below example is just to show that some components of agent don't really respect the log-format configuration.

    In our case I configured the agent log level and log format using annotations on the workload being injected:

    spec.template.metadata.annotations:
  vault.hashicorp.com/log-format: json
  vault.hashicorp.com/log-level: debug

    and this is the snipped output:

    {"@level":"info","@message":"creating file sink","@module":"sink.file","@timestamp":"2023-07-13T10:16:55.917053Z"}
{"@level":"info","@message":"file sink configured","@module":"sink.file","@timestamp":"2023-07-13T10:16:55.917178Z","mode":416,"path":"/home/vault/.vault-token"}
{"@level":"info","@message":"starting template server","@module":"template.server","@timestamp":"2023-07-13T10:16:55.917410Z"}
2023-07-13T10:16:55.917Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-07-13T10:16:55.917Z [DEBUG] (runner) final config: {SNIPPED JSON}
2023-07-13T10:16:55.918Z [INFO] (runner) creating watcher
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.11.2, built 2022-07-29T09:48:47Z
             Version Sha: 3a8aa12eba357ed2de3192b15c99c717afdeb2b5

{"@level":"info","@message":"starting auth handler","@module":"auth.handler","@timestamp":"2023-07-13T10:16:55.918919Z"}
// SNIP "json" logs
{"@level":"info","@message":"template server received new token","@module":"template.server","@timestamp":"2023-07-13T10:16:56.035314Z"}
2023-07-13T10:16:56.035Z [INFO] (runner) stopping
// SNIP "standard" logs
2023-07-13T10:16:56.143Z [DEBUG] (runner) all templates rendered

    I would expect everything to be in JSON, including the ==> Vault agent started! Log data will stream in below:.

  9. eg-chdev commented on Dec 2, 2024

    @eg-chdev
    eg-chdev
    on Dec 2, 2024

    Any updates on this issue or if it was consumed as a fix in another deployment or release?

  10. j4ng5y commented on Aug 28, 2025

    @j4ng5y
    j4ng5y
    on Aug 28, 2025

    bump, would love to know a status

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestgood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions