`jq`: a lightweight and flexible command-line JSON processor

jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.

jq is written in portable C, and it has zero runtime dependencies.

jq is built around the concept of filters that work over a stream of JSON. Each filter takes an input and emits JSON to standard out. As we're going to see, there are many predefined filters that we can use. We can effortlessly combine these filters using pipes to quickly construct and apply complex operations and transformations to our JSON data.

Usage

jq [<options>] <jq_filter> [<file...>]

Prettify JSON

Let's start by taking a look at the simplest filter of all, which incidentally is one of the most useful and frequently used features of jq:

$ echo '{"fruit":{"name":"apple","color":"green","price":1.20}}' | jq '.'

We echo a simple JSON string and pipe it into our jq command. Then we use the identity filter '.' that takes the input and produces it unchanged as output with the caveat that by default jq pretty-prints all output.

This gives us the following output:

{
  "fruit": {
    "name": "apple",
    "color": "green",
    "price": 1.20
  }
}

We can also apply this filter directly to a JSON file:

$ jq '.' fruit.json

It seems that a file must consist of either one JSON objecto or a space-separated string of JSON objects, like:

{
  "fruit": {
    "name": "apple",
    "color": "yellow",
    "price": 1.2
  }
}

{
  "fruit": {
    "name": "avocado",
    "color": "green",
    "price": 4.0
  }
}

...

Let's hit a simple API using curl to see this in practice:

$ curl http://api.open-notify.org/iss-now.json | jq '.'

This gives us a JSON response for the current position of the International Space Station.

Accessing Properties

We can access property values by using another simple filter: the .FIELD operator. To find a property value, we simply combine this filter followed by the property name.

We can also chain property values together, allowing us to access nested objects:

$ jq '.fruit.color' fruit.json

As expected, this simply returns the color of our fruit:

"yellow"
"green"

If we need to retrieve multiple keys, we can separate them using a comma:

$ jq '.fruit.color,.fruit.price' fruit.json

This results in an output containing both property values:

"yellow"
1.2
"green"
4.0

Note that if one of the properties has spaces or special characters, we need to wrap the property name in quotes when accessing it from the jq command:

$ echo '{ "with space": "hello" }' | jq '."with space"'

Yielding Results without Double Quotes

The -r flag is used for stripping out the double quotes.

Using Shell Variables through `--arg`

Compressing JSON Objects through `-c` or `--compact-output`

$ jq -c '.employees[]' sample.json
{"name":"Tony","age":"35","gender":"male"}
{"name":"Jane","age":"30","gender":"female"}

JSON Arrays

Let's now look at how we can work with JSON arrays.

We typically use arrays to represent a list of items. Remember that we use square brackets to denote the start and end of a JSON/JavaScript array.

Iteration

We'll start with a basic example to demonstrate how to iterate over an array:

$ echo '["x","y","z"]' | jq '.[]'

Here, we see the object value iterator operator .[] in use, which will print out each item in the array on a separate line:

"x"
"y"
"z"

Now let's imagine we want to represent a list of fruit in a JSON document, where each item in the array is an object that represents a fruit.

[
  {
    "name": "apple",
    "color": "green",
    "price": 1.2
  },
  {
    "name": "banana",
    "color": "yellow",
    "price": 0.5
  },
  {
    "name": "kiwi",
    "color": "green",
    "price": 1.25
  }
]

Let's see how to extract the name of each fruit from each object in the array:

$ jq '.[] | .name' fruits.json

First, we iterate over the array using .[]. Then we can pass each object in the array to the next filter in the command using a pipe |. The last step is to output the name field from each object using .name:

"apple"
"banana"
"kiwi"

We can also use a slightly more concise version and access the property directly on each object in the array:

$ jq '.[].name' fruits.json

Accessing by Index

Of course, as with all arrays, we can access one of the items in the array directly by passing the index:

$ jq '.[1].price' fruits.json

Slicing

Finally, jq also supports slicing of arrays, another powerful feature. This is particularly useful when we need to return a subarray of an array.

Again, let's see this using a simple array of numbers:

$ echo '[1,2,3,4,5,6,7,8,9,10]' | jq '.[6:9]'

The result will be a new array with a length of 3, containing the elements from index 6 (inclusive) to index 9 (exclusive):

[
  7,
  8,
  9
]

It's also possible to omit one of the indexes when using the slicing functionality:

$ echo '[1,2,3,4,5,6,7,8,9,10]' | jq '.[:6]' | jq '.[-2:]'

Since we specified only the second argument in .[:6], the slice will start from the beginning of the array and run up until index 6. It's the same as doing .[0:6].

The second slicing operation has a negative argument, which denotes in this case that it counts backward from the end of the array.

Note the subtle difference in the second slice — we pass the index as the first argument. This means we will start two indexes from the end (-2), and since the second argument is empty, it will run until the end of the array.

Using Functions

jq has many powerful built-in functions that we can use to perform a variety of useful operations. Let's take a look at some of them now.

Getting Keys

Sometimes, we may want to get the keys of an object as an array instead of the values.

We can do this using the keys function:

$ jq '.fruit | keys' fruit.json

This gives us the keys sorted alphabetically:

[
  "color",
  "name",
  "price"
]

Returning the Length

Another handy function for arrays and objects is the length function.

We can use this function to return the array's length or the number of properties on an object:

$ jq '.fruit | length' fruit.json

Here, we get “3” since the fruit object has three properties.

We can even use the length function on string values as well:

$ jq '.fruit.name | length' fruit.json

We would see “5” as the resulting output since the fruit name property has five characters: “apple”.

Mapping Values

The map function is a powerful function we can use to apply a filter or function to an array:

$ jq 'map(has("name"))' fruits.json

In this example, we're applying the has function to each item in the array and finding if there is a name property. In our simple fruits JSON, we get true in each result item.

We can also use the map function to apply operations to the elements in an array.

Let's imagine we want to increase the price of each fruit:

$ jq 'map(.price+2)' fruits.json

This gives us a new array with each price incremented:

[
  3.2,
  2.5,
  3.25
]

Min and Max

If we need to find the minimum or maximum element of an input array, we can utilize the min and max functions:

$ jq '[.[].price] | min' fruits.json

Likewise, we can also find the most expensive fruit in our JSON document:

$ jq '[.[].price] | max' fruits.json

Note that in these two examples, we've constructed a new array, using [] around the array iteration. This contains only the prices before we pass this new list to the min or max function.

Selecting Values

The select function is another impressive utility that we can use for querying JSON.

We can think of it as a bit like a simple version of XPath for JSON:

$ jq '.[] | select(.price > 0.5)' fruits.json

This selects all the fruit with a price greater than 0.5.

Likewise, we can also make selections based on the value of a property:

$ jq '.[] | select(.color=="yellow")' fruits.json

We can even combine conditions to build up complex selections:

$ jq '.[] | select(.color=="yellow" and .price >= 0.5)' fruits.json

This will give us all yellow fruit matching a given price condition:

{
  "name": "banana",
  "color": "yellow",
  "price": 0.5
}

Additionally, we can extract values of attributes based on keys containing a specific string:

$ jq '.[] | to_entries[] | select(.key | startswith("name")) | .value' fruits.json
"apple"
"banana"
"kiwi"

In this example, we select the values of attributes in the fruits.json file that contain keys starting with the string “name”.

Here, we use the to_entries[] function to convert each object in the array into an array of key-value pairs. Next, select(.key | startswith(“name”)) filters the key-value pairs, and selects only those where the key starts with the string “name”. Finally, .value extracts the value from the filtered key-value pairs.

Support for Regular Expressions

Next, we're going to look at the test function, which enables us to test if an input matches against a given regular expression:

$ jq '.[] | select(.name | test("^a.")) | .price' fruits.json

Simply put, we want to output the price of all the fruit whose name starts with the letter “a”.

Finding the Type with `type`

$ jq '.age | type' tony.json
"number"

...

Finding Unique Values with `unique`

One common use case is to be able to see unique occurrences of a particular value within an array or remove duplicates.

Let's see how many unique colors we have in our fruits JSON document:

$ jq 'map(.color) | unique' fruits.json

We use the map function to create a new array containing only colors. Then we pass each color in the new array to the unique function using a pipe |.

This gives us an array with two distinct fruit colors:

[
  "green",
  "yellow"
]

Deleting Keys From JSON

We'll also sometimes want to remove a key and corresponding value from JSON objects. For this, jq provides the del function:

$ jq 'del(.fruit.name)' fruit.json

This outputs the fruit object without the deleted key:

{
  "fruit": {
    "color": "green",
    "price": 1.2
  }
}

Creating and Editing JSON Objects

You can create and modify json data using {} brackets. Here's how to create json data from scratch:

$ jq -n --arg my_username "john" --arg my_id "25" '{
    "username": $my_username,
    "id": $my_id
}'

Here we use the -n flag to tell jq to start with a null json object. Then we used the content inside the {} as a template to build out a custom json object. We also passed in some variables to the template using the --arg flags:

{
  "username": "john",
  "id": "25"
}

Instead of using the --arg flag, you can use the content inside an existing json file to generate a new json file. Lets use some sample.json as our source data:

$ cat sample.json
[{"name":"Tony","age":"35","gender":"male"},{"name":"Jane","age":"30","gender":"female"}]

Now let's say we only want us Tony's data to create our new json object. So we need to run:

$ cat sample.json | jq '.[] | select(.name=="Tony")' | jq '{
    "username": .name,
    "current_age": .age
}'

Notice this time we call variables using . so instead of $name we used .name.

When we run the above, we get:

{
  "username": "Tony",
  "current_age": "35"
}

Rewriting JSON Objects into JSON Objects

jq '. | {word: .term, definition: .definition}' terms.json

JSON Array(s) from JSON Object(s)

jq '. | [.term,  .definition]' terms.json

CSV from JSON File with Objects through Function `@csv`

jq '. | [.term,  .definition] | @csv' terms.json

Transforming JSON

Frequently when working with data structures such as JSON, we might want to transform one data structure into another. This can be useful while working with large JSON structures when we are only interested in several properties or values.

In this example, we'll use some Wikipedia JSON that describes a list of page entries:

{
  "query": {
    "pages": [
      {
        "21721040": {
          "pageid": 21721040,
          "ns": 0,
          "title": "Stack Overflow",
          "extract": "Some interesting text about Stack Overflow"
        }
      },
      {
        "21721041": {
          "pageid": 21721041,
          "ns": 0,
          "title": "Baeldung",
          "extract": "A great place to learn about Java"
        }
      }
    ]
  }
}

We're only really interested in the title and extract of each page entry. So, let's see how we can transform this document:

$ jq '.query.pages | [.[] | map(.) | .[] | {page_title: .title, page_description: .extract}]' wikipedia.json

We'll take a look at the command in more detail to understand it properly:

First, we begin by accessing the pages array and passing that array into the next filter in the command via a pipe.
Then we iterate over this array and pass each object inside the pages array to the map function, where we simply create a new array with the contents of each object.
Next, we iterate over this array and for each item create an object containing the two keys: page_title and page_description.
The .title and .extract references are used to populate the two new keys.

This gives us a new, lean JSON structure:

[
  {
    "page_title": "Stack Overflow",
    "page_description": "Some interesting text about Stack Overflow"
  },
  {
    "page_title": "Baeldung",
    "page_description": "A great place to learn about Java"
  }
]

Appendix

Some Useful Switches

--null-input / -n:

Don´t read any input at all. Instead, the filter is run once using null as the input. This is useful when using jq as a simple calculator or to construct JSON data from scratch.

--slurp / -s:

Instead of running the filter for each JSON object in the input, read the entire input stream into a large array and run the filter just once.

--compact-output / -c:

By default, jq pretty-prints JSON output. Using this option will result in more compact output by instead putting each JSON object on a single line.

--raw-output / -r:

With this option, if the filter´s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.

--ascii-output / -a:

jq usually outputs non-ASCII Unicode codepoints as UTF-8, even if the input specified them as escape sequences (like \u03bc). Using this option, you can force jq to produce pure ASCII output with every non-ASCII character replaced with the equivalent escape sequence.

-f filename / --from-file filename:

Read filter from the file rather than from a command line, like awk´s -f option. You can also use ´#´ to make comments.

--arg name value:

This option passes a value to the jq program as a predefined variable. If you run jq with --arg foo bar, then $foo is available in the program and has the value "bar". Note that value will be treated as a string, so --arg foo 123 will bind $foo to "123".

Named arguments are also available to the jq program as $ARGS.named.

--argjson name JSON-text:

This option passes a JSON-encoded value to the jq program as a predefined variable. If you run jq with --argjson foo 123, then $foo is available in the program and has the (numerical) value 123.

--rawfile variable-name filename:

This option reads in the named file and binds its contents to the given global variable. If you run jq with --rawfile foo bar, then $foo is available in the program and has a string whose contents are the texts in the file named bar.

--args:

Remaining arguments are positional string arguments. These are available to the jq program as $ARGS.po‐ sitional[].

--jsonargs:

Remaining arguments are positional JSON text arguments. These are available to the jq program as $ARGS.positional[].

Filters*

Identity (.)

Object Identifier-Index (.some-key)

A filter of the form .foo.bar is equivalent to .foo | .bar.

Some examples:

jq ´.foo´
   {"foo": 42, "bar": "less interesting data"}
=> 42

jq ´.foo´
   {"notfoo": true, "alsonotfoo": false}
=> null

jq ´.["foo"]´
   {"foo": 42}
=> 42

Optional Object Identifier-Index (.some-key?)

Just like .some-key, but does not output an error when . is not an object.

Object Index (.[some-string])

You can also look up fields of an object using syntax like .["foo"] (.foo above is a shorthand version of this, but only for identifier-like strings).

Array Index (.[number])

When the index value is an integer, .[number] can index arrays. Arrays are zero-based, so .[2] returns the third element.

Negative indices are allowed, with -1 referring to the last element, -2 referring to the next to last element, and so on.

Array/String Slice (.[number:number])

The .[number:number] syntax can be used to return a subarray of an array or substring of a string. The array returned by .[10:15] will be of length 5, containing the elements from index 10 (inclusive) to index 15 (exclusive). Either index may be negative (in which case it counts backwards from the end of the array), or omitted (in which case it refers to the start or end of the array). Indices are zero-based.

jq ´.[2:4]´
              ["a","b","c","d","e"]
           => ["c", "d"]

           jq ´.[2:4]´
              "abcdefghi"
           => "cd"

           jq ´.[:3]´
              ["a","b","c","d","e"]
           => ["a", "b", "c"]

           jq ´.[-2:]´
              ["a","b","c","d","e"]
           => ["d", "e"]

Array/Object Value Iterator (.[])

If you use the .[index] syntax, but omit the index entirely, it will return all of the elements of an array. Running .[] with the input [1,2,3] will produce the numbers as three separate results, rather than as a single array. A filter of the form .foo[] is equivalent to .foo | .[].

You can also use this on an object, and it will return all the values of the object.

Note that the iterator operator is a generator of values.

jq ´.[]´
              [{"name":"JSON", "good":true}, {"name":"XML", "good":false}]
           => {"name":"JSON", "good":true}, {"name":"XML", "good":false}

           jq ´.[]´
              []
           =>

           jq ´.foo[]´
              {"foo":[1,2,3]}
           => 1, 2, 3

           jq ´.[]´
              {"a": 1, "b": 1}
           => 1, 1

(.[]?)

Like .[], but no errors will be output if . is not an array or object. A filter of the form .foo[]? is equivalent to .foo | .[]?.

Comma (,)

If two filters are separated by a comma, then the same input will be fed into both and the two filters´ output value streams will be concatenated in order: first, all of the outputs produced by the left expression, and then all of the outputs produced by the right. For instance, filter .foo, .bar, produces both the "foo" fields and "bar" fields as separate outputs.

The comma operator is one way to contruct generators.

jq ´.foo, .bar´
              {"foo": 42, "bar": "something else", "baz": true}
           => 42, "something else"

           jq ´.user, .projects[]´
              {"user":"stedolan", "projects": ["jq", "wikiflow"]}
           => "stedolan", "jq", "wikiflow"

           jq ´.[4,2]´
              ["a","b","c","d","e"]
           => "e", "c"

Pipe (|)

The | operator combines two filters by feeding the output(s) of the one on the left into the input of the one on the right. It´s similar to the Unix shell´s pipe, if you´re used to that.

If the one on the left produces multiple results, the one on the right will be run for each of those results. So, the expression .[] | .foo retrieves the "foo" field of each element of the input array. This is a cartesian product, which can be surprising.

Note that .a.b.c is the same as .a | .b | .c.

Note too that . is the input value at the particular stage in a "pipeline", specifically: where the . expression appears. Thus .a | . | .b is the same as .a.b, as the . in the middle refers to whatever value .a produced.

jq ´.[] | .name´
   [{"name":"JSON", "good":true}, {"name":"XML", "good":false}]
=> "JSON", "XML"

Parenthesis((...))

Parenthesis work as a grouping operator just as in any typical programming language.

Recursive Descent (`..`)

Recursively descends ., producing every value. This is the same as the zero-argument recurse builtin (see below). This is intended to resemble the XPath // operator. Note that ..a does not work; use .. | .a instead. In the example below we use .. | .a? to find all the values of object keys "a" in any object found "below" ..

This is particularly useful in conjunction with path(EXP) (also see below) and the ? operator.

jq ´.. | .a?´
   [[{"a":1}]]
=> 1

Builtin Operators and Functions*

Format strings and escaping

The @foo syntax is used to format and escape strings, which is useful for building URLs, documents in a language like HTML or XML, and so forth. @foo can be used as a filter on its own, the possible escapings are:

@text:: Calls tostring, see that function for details.
@json:: Serializes the input as JSON.
@html:: Applies HTML/XML escaping, by mapping the characters <>&'" to their entity equivalents <, >, &, and ', ".
@uri:: Applies percent-encoding, by mapping all reserved URI characters to a %XX sequence.
@csv:: The input must be an array, and it is rendered as CSV with double quotes for strings, and quotes escaped by repetition.
@tsv:: The input must be an array, and it is rendered as TSV (tab-separated values). Each input array will be printed as a single line. Fields are separated by a single tab (ascii 0x09). Input characters line-feed (ascii 0x0a), carriage-return (ascii 0x0d), tab (ascii 0x09) and backslash (ascii 0x5c) will be output as escape sequences \n, \r, \t, \\ respectively.
@sh:: The input is escaped suitable for use in a command-line for a POSIX shell. If the input is an array, the output will be a series of space-separated strings.
@base64:: The input is converted to base64 as specified by RFC 4648.
@base64d:: The inverse of @base64, input is decoded as specified by RFC 4648. Note\: If the decoded string is not UTF-8, the results are undefined.

This syntax can be combined with string interpolation in a useful way. You can follow a @foo token with a string literal. The contents of the string literal will not be escaped. However, all interpolations made in‐ side that string literal will be escaped. For instance,

@uri "https://www.google.com/search?q=\(.search)"

will produce the following output for the input {"search":"what is jq?"}:

"https://www.google.com/search?q=what%20is%20jq%3F"