yq - The Swiss Army Knife for Command-Line YAML/JSON Processing
If you work with Kubernetes regularly, or maintain any project using YAML for configuration, you know the pain: YAML's indentation rules are maddening, manual editing is error-prone, and using a general-purpose programming language feels like overkill. jq sets the standard for JSON processing, but the YAML ecosystem has long needed an equally capable tool.
yq fills that gap. It combines jq's expressive power with native YAML support, handling not just YAML but also JSON, XML, TOML, CSV, and Properties formats. Crucially, it preserves comments and formatting when processing YAML—a must-have for maintaining configuration files.
Core Positioning: The jq for YAML
yq's design philosophy is straightforward: let anyone familiar with jq use YAML with zero learning curve. It uses similar expression syntax but optimizes for YAML's unique features—supporting multi-document files, preserving comments, handling anchors and aliases, and seamlessly converting between YAML and JSON.
Written in Go as a single binary, yq has no runtime dependencies, is lightweight (about 10MB), and runs on any platform.
Basic Usage: Read and Modify
Reading nested fields:
# Basic path access
yq '.a.b[0].c' file.yaml
# Read from pipe
cat file.yaml | yq '.a.b[0].c'In-place file updates:
# Modify a single field (-i means in-place)
yq -i '.a.b[0].c = "cool"' file.yaml
# Multiple updates connected with pipes
yq -i '
.a.b[0].c = "cool" |
.x.y.z = "foobar" |
.person.name = "Alice"
' file.yamlUsing environment variables:
NAME=mike yq -i '.a.b[0].c = strenv(NAME)' file.yamlstrenv() is a built-in function provided by yq for safely reading environment variable values.
Multi-Format Support: Beyond Just YAML
yq's real advantage lies in its cross-format processing capabilities. It can automatically detect formats by file extension, or you can explicitly specify them with -p (input) and -o (output) flags:
| Format | Input Flag | Output Flag | Notes |
|---|---|---|---|
| YAML | -p yaml / -y | -o yaml / -y | Default format, supports multi-doc |
| JSON | -p json / -j | -o json / -j | Standard JSON |
| XML | -p xml / -x | -o xml / -x | With +@ attribute prefix |
| TOML | -p toml / -t | -o toml / -t | Common for config files |
| CSV | -p csv / -c | -o csv / -c | Supports header parsing |
| TSV | -p tsv / -t | -o tsv / -t | Tab-separated values |
| Properties | -p props / -p | -o props / -p | Java properties files |
| HCL | -p hcl / -h | -o hcl / -h | Terraform configurations |
Format conversion examples:
# JSON to YAML (pretty print)
yq -P -oy sample.json
# YAML to JSON
yq -o json file.yaml
# XML to YAML
yq -o yaml file.xml
# CSV to JSON
yq -p csv -o json data.csvAdvanced Operations: Merge, Filter, and Transform
Merging multiple files:
# Merge two files (later overrides earlier conflicts)
yq -n 'load("file1.yaml") * load("file2.yaml")'
# Merge multiple files using glob (ea = evaluate all)
yq ea '. as $item ireduce ({}; . * $item)' path/to/*.ymlireduce is yq's reduction operator, similar to jq's reduce, used for accumulating multiple documents.
Array filtering and updates:
# Find item where name is "foo" and modify its address field
yq -i '(.[] | select(.name == "foo") | .address) = "12 cat st"' data.yamlHow this expression works:
.[]expands the arrayselect(.name == "foo")filters matching elements| .addresslocates the address field= "12 cat st"assigns the new value
Preserving YAML Features
Handling multi-document YAML:
# Read the 2nd document
yq 'select(di == 1)' multi-doc.yaml
# Add a tag to each document
yq '(. | select(di == 0)).env = "prod"' multi-doc.yamldi is short for document index, starting from 0.
Manipulating comments and styles:
# Add/modify comments
yq -i '.foo = "bar" | .foo line_comment = "This is a comment"' file.yaml
# Change string style (double quote, single quote, no quote, literal)
yq -i '.foo style = "double"' file.yaml
# Options: double | single | literal | folded | ""Handling anchors and aliases:
# Explode all aliases (dereference)
yq 'explode(.)' file.yaml
# Create a new anchor
yq '.foo anchor = "myAnchor"' file.yamlUseful Built-in Functions
| Function | Purpose | Example |
|---|---|---|
strenv(NAME) | Read environment variable | yq '.version = strenv(VER)' |
env(NAME) | Read environment variable (any type) | yq '.count = env(NUM)' |
load(file) | Load external file | yq 'load("other.yaml")' |
select(condition) | Conditional filtering | yq 'select(.active == true)' |
explode(.) | Explode anchor aliases | yq 'explode(.)' |
sort_keys(.) | Sort by keys | yq 'sort_keys(.)' |
di | Document index | yq 'select(di == 0)' |
Comparison with Similar Tools
| Feature | yq (Go) | yq (Python) | jq + yq | dasel |
|---|---|---|---|---|
| Language | Go (single binary) | Python | Two tools combined | Go |
| Native YAML support | Full | Full | Requires conversion | Full |
| jq syntax compatible | Similar but not identical | Not compatible | Native jq | Not compatible |
| Format support | 10+ formats | YAML/JSON/XML | Primarily JSON | Multiple |
| Comment preservation | Yes | Limited | No | Partial |
| Performance | High | Medium | Medium | High |
Note: There's another yq on PyPI (Python implementation) that wraps jq for YAML support. Both have similar functionality but incompatible CLI arguments. This article covers the Go implementation mikefarah/yq, which is more feature-complete and the default version installed by most package managers.
Real-World Use Cases
| Scenario | yq Solution |
|---|---|
| K8s config batch modification | yq -i '.spec.replicas = 3' deployment.yaml |
| CI/CD dynamic injection | yq -i '.version = strenv(VERSION)' package.yaml |
| Multi-environment config merging | yq eval-all 'select(fileIndex == 0) * select(fileIndex == 1)' base.yaml prod.yaml |
| Helm values processing | yq '.image.tag' values.yaml |
| API response transformation | `curl api |
| Docker Compose management | yq '.services[].image' docker-compose.yml |
Command-Line Options Quick Reference
| Option | Description |
|---|---|
-i, --inplace | Modify file in place |
-n, --null-input | Don't read from file, useful for generating new docs |
-p, --input-format | Specify input format (yaml/json/xml/csv...) |
-o, --output-format | Specify output format |
-P, --prettyPrint | Pretty print (reset styles) |
-r, --unwrapScalar | Unwrap scalar values (enabled by default) |
-e, --exit-status | Return non-zero exit code if no matches |
-I, --indent | Set indentation spaces (default 2) |
-C, --colors | Force colored output |
-M, --no-colors | Disable colored output |
Usage Tips
- Start with
-Pfor unknown YAML: When facing a messy file, runyq -Pfirst to see formatted output - Test complex modifications step by step: Don't use
-iimmediately; test without it first, then apply once confirmed - Wrap common operations in shell functions: For frequently used operations, create functions like
yqver() { yq '.version' "$1"; } - GitHub Actions integration:
yqprovides an official Action for safely modifying config files in CI pipelines
The official documentation at mikefarah.gitbook.io/yq has extensive examples and advanced usage, including date handling, encoding/decoding, and recursive traversal. For daily operations and configuration management, yq is already the de facto standard tool.
Source: