gitlab

Learn JSON superpowers with jq - automate GitLab Personal Access Token management

Michael Friedrich

• 5 min read
Learn JSON superpowers with jq - automate GitLab Personal Access Token management

I've seen a question on the GitLab forum about managing Personal Access Tokens (PATs) with the API. While trying things, I had a peek into filtering the output and quickly getting access to values and IDs for automation.

This blog post covers the basics around listing PATs, working with jqfilters and lastly revoking PATs, practicing the filters even more :)

Requirements

  • Export your full admin personal access token into an environment variable called GITLAB_TOKEN. The examples below reference this with setting the header to -H "PRIVATE-TOKEN: $GITLAB_TOKEN".
  • Persist this in your user's environment, e.g. with ZSH and the .env plugin.

List all your personal access tokens

$ curl -H "PRIVATE-TOKEN: $GITLAB_TOKEN" https://gitlab.com/api/v4/personal_access_tokens

This returns raw JSON and is hard to read. jq allows you to format the JSON body in a readable way. Learn more about jq and its usage on the official website.

On macOS, you can install jq using Homebrew. Alternatively, use brew bundle. 

$ brew install jq

Pretty print JSON with jq

Let's try retrieving the PATs again, and pipe the JSON body into jq. In addition, we remove the curl stderr verbose output to /dev/null . Note: I have modified the user_id value for privacy reasons.

$ curl -H "PRIVATE-TOKEN: $GITLAB_TOKEN" https://gitlab.com/api/v4/personal_access_tokens 2&>1 >/dev/null | jq
[
  {
    "id": 2992610,
    "name": "ci-monitoring-webcast",
    "revoked": false,
    "created_at": "2020-08-24T09:30:21.088Z",
    "scopes": [
      "api",
      "read_user",
      "read_api",
      "read_repository",
      "read_registry"
    ],
    "user_id": 1234567,
    "active": true,
    "expires_at": null
  }
]

The JSON body contains all personal access tokens as an array with dictionary items. The more data is received the longer the list gets.

Filter JSON based on key/value

jq provides its own query language for parsing JSON into objects and inspect its attributes. Let's mimic the PAT example above into a simple data structure:

$ vim pat.json
[
  {
    "id": 1,
    "name": "pat1"
  },
  {
    "id": 2,
    "name": "pat2"
  }
]

Cat this file and pipe the content into jq. First, access all array elements:

$ cat pat.json | jq -c '.[]'
{"id":1,"name":"pat1"}
{"id":2,"name":"pat2"}

Add a filter with | select (...) and specify the comparison condition. The dot indexer . allows to access dictionary keys and their value. Try to filter .name by the string value pat2:

$ cat pat.json | jq -c '.[] | select (.name == "pat2")'
{"id":2,"name":"pat2"}

The syntax needs practice, repeat this with checking

  • equality on pat1 for .name
  • print only the .id with the value of 1

Filter with contains and regex

Sometimes you do not know the exact string. .name | contains(...) provides checking if the given string is inside .name.

$ cat pat.json | jq -c '.[] | select (.name | contains("2") )'
{"id":2,"name":"pat2"}

In order to use a regex, invoke the test() function. Note: Backslashes for matching a number \d+ need to be escaped with an additional backslash, becoming \\d+.

$ cat pat.json | jq -c '.[] | select (.name | test("^pat\\d+") )'
{"id":1,"name":"pat1"}
{"id":2,"name":"pat2"}

Real example with filtering GitLab PATs

Back to our example: Select the personal access token which name contains the string monitoring.

$ curl -H "PRIVATE-TOKEN: $GITLAB_TOKEN" https://gitlab.com/api/v4/personal_access_tokens 2&>1 >/dev/null | jq -c '.[] | select( .name | contains("monitoring"))'

{"id":9876543,"name":"ci-monitoring-webcast","revoked":false,"created_at":"2020-08-24T09:30:21.088Z","scopes":["api","read_user","read_api","read_repository","read_registry"],"user_id":1234567,"active":true,"expires_at":null}

This renders a raw JSON output again. Pretty print this with again piping to jq 💡

$ curl -H "PRIVATE-TOKEN: $GITLAB_TOKEN" https://gitlab.com/api/v4/personal_access_tokens 2&>1 >/dev/null | jq -c '.[] | select( .name | contains("monitoring"))' | jq

{
  "id": 9876543,
  "name": "ci-monitoring-webcast",
  "revoked": false,
  "created_at": "2020-08-24T09:30:21.088Z",
  "scopes": [
    "api",
    "read_user",
    "read_api",
    "read_repository",
    "read_registry"
  ],
  "user_id": 1234567,
  "active": true,
  "expires_at": null
}

The reason for going here was to finally select the value of the .id index. This is the token ID which we want to revoke.

$ curl -H "PRIVATE-TOKEN: $GITLAB_TOKEN" https://gitlab.com/api/v4/personal_access_tokens 2&>1 >/dev/null | jq -c '.[] | select( .name | contains("monitoring"))' | jq -c '.id'
9876543

For automation purposes, you can store the output in a shell variable to process in the next step, REVOKE_ID as a example.

$ REVOKE_ID=$(curl -H "PRIVATE-TOKEN: $GITLAB_TOKEN" https://gitlab.com/api/v4/personal_access_tokens 2&>1 >/dev/null | jq -c '.[] | select( .name | contains("monitoring"))' | jq -c '.id')

Revoke a GitLab Personal Access Token

Note that you cannot delete PATs, only revoke them. They will be hidden from the UI by default.

-X or --request defines DELETE as request method. If not provided, this uses GET by default. The URL path needs to add the ID as the last path string. 

$ curl -X DELETE -H "PRIVATE-TOKEN: $GITLAB_TOKEN" https://gitlab.com/api/v4/personal_access_tokens/$REVOKE_ID

You can check success with the HTTP response code 204 explained in the docs, using -w 204.

Practice: List only revoked PATs

Inspect the attributes in the GitLab API docs. revoked is of the boolean type for the jq filter.

$ curl -H "PRIVATE-TOKEN: $GITLAB_TOKEN" https://gitlab.com/api/v4/personal_access_tokens 2&>1 >/dev/null | jq -c '.[] | select( .revoked == true)' | jq
{
  "id": 9876543,
  "name": "ci-monitoring-webcast",
  "revoked": true,
  "created_at": "2020-08-24T09:30:21.088Z",
  "scopes": [
    "api",
    "read_user",
    "read_api",
    "read_repository",
    "read_registry"
  ],
  "user_id": 1234567,
  "active": false,
  "expires_at": null
}

This works the other way around with .revoked == false to only list active PATs.

Conclusion

While writing this blog post, I again learned great new things. Let me know on Twitter if the learning curve is the same for you, or what other cool things you learned :)

Tip for practicing: Download the JSON output from the API with -o pat.json and use the methods above to learn more about jq filters. This avoids pulling remote APIs too often. 

$ curl -H "PRIVATE-TOKEN: $GITLAB_TOKEN" https://gitlab.com/api/v4/personal_access_tokens 2&>1 >/dev/null -o pat.json

$ cat pat.json | jq