Parse Jamf Pro API JSON data in Shell Scripts

Shell scripts are probably not a first choice for API programming, but if it’s what you know, it’s what you know. You can do some pretty cool stuff in bash and save yourself a ton of work. Plus bash is available on every Mac so shell scripts are super portable.

Shell programming is good at working on the kinds of data people used back when it was created… like lines in a file or delimiter-separated fields. Modern APIs use more structured data formats like XML and JSON that didn’t exist when shell was invented.

It’s all good. We have tricks up our sleeves, at least for the simple stuff.

Choices

Kick it Old School

Shell tools like grep/awk/sed can be used to parse structured data if you really have to, but it’s not very reliable, and regex takes some getting used to. Your parse will break when the structure of the source data changes in a way you didn’t expect where real parsers will not.

For XML parsing in bash, you can use the “xpath -e” (or its “xmllint -xpath” wrapper) since those come built into macOS. That’s legit parsing.

jq / XMLStarlet

For JSON, you can install something like jq. It’s really easy to use so it’s great if you need it to run your own scripts on your own machine, but it doesn’t come pre-installed on macOS so you’d have to tell people to install it if they want to use your script. Same with XMLStarlet which does things with XML that the built-in xpath can’t touch.

Use a different language

Ruby/Perl/Python/golang/etc. are great at parsing JSON. Some macadmins mostly just know bash, though, so they stick with that and shell out to something else that has a capability they need once in a while. For example, you can use Python to do something quick in a shell script…

For example, if you have json {“token”:”349fhuqp3ieorf30f”} and you want to extract the “349fhuqp3ieorf30f” value…

api_token=$(echo "$authToken" | python -c 'import sys, json; print json.load(sys.stdin)["token"]')

Older versions of macOS came with some languages built-in, but they’re going away or gone. So, just as with binaries like jq, using them introduces dependancies and reduces the portability of your scripts.

plutil -extract

So, what if you’re going to ride or die with bash and don’t want to mess with installing other tools? The first advice is that you reconsider, but… moving on…

plutil stands for “Property List Utility” and it comes with macOS. Here’s a json parse using the plutil -extract command

#!/bin/bash

# Set these vars or read them from env vars, or from a password store file, etc. 
jssUrl='https://my.jamfcloud.com'   # omit trailing slash
apiUser='api_read_only_user'
apiPass='mycomplexpassword'


# #########################################################
curlCmd="/usr/bin/curl --silent --show-error"
url="${jssUrl}/api/v1/auth/token"
echo "Calling ${url}"
authTokenJson=$(${curlCmd} \
  --user "${userPass}" \
  --request "POST" \
  "${url}")
echo "authTokenJson: ${authTokenJson}"
# that sets "authTokenJson" to 
# {
#   "token" : "eyJhbGciOiJIUzI1NiJ9.verylongstring...",
#   "expires" : "2022-01-27T03:13:26.769Z"
# }

# Use plutil to extract the token value from the json.
#  ("-o - -" here means "output to stdout, accept input from the <<< $var")

authToken=$(/usr/bin/plutil -extract "token" raw -expect "string" -o - - <<< "${authTokenJson}")
echo "apiToken: ${apiToken}"
# output will be "eyJhbGciOiJIUzI1NiJ9.verylongstring..."

Check out that “authToken=$(/usr/bin/plutil -extract "token" raw -expect "string" -o - - <<< "${authTokenJson}")” line… nothing to install, no stacked invocation of binaries. Once you manage to decipher what all those dashes are doing, it’s pretty simple. (Thanks Armin!)

plutil doesn’t offer the complex attribute targeting of xpath or Python or jq, but for top-level attributes, it’s pretty simple. Using the “-extract keypathfmt [-expect expect_type]” option, you may be able to use plutil to pull out a child object, then send that back into plutil again to work your way down to what you actually want, but that gets kind of involved.

plutil -convert

If you need to do some more complex json parsing, you can use plutil to convert your json to a plist. I don’t get to use bash much anymore, but if I could, this is probably the method I’d use if I really needed the solution to be portable (i.e., no python/jq/etc.). It fits well with the things I knew how to do when I worked with the XML from Jamf’s Classic API.

plutil might not be great at extracting a value buried 10-levels deep in a complex json object, but it does have a conversion feature that can get us from json to plist. And once we have plist xml, plistbuddy offers a way to address nested data elements.

For example, if the plist version of a page of Jamf Pro computer records is in the $plist variable, we can get the MDM capable users for the second computer with this:

/usr/libexec/PlistBuddy -c "print :results:1:general:mdmCapable:capableUsers" /dev/stdin <<< "$plist"

PlistBuddy is the right tool for parsing plists. You’ll get your data a lot easier if you do it that way, so definitely try that first. But plistbuddy has its limits. It you hit them, there’s still another way. If your json is simple enough to wrap your head around, you can even use the xpath that comes with macOS, though it does feel like we’re getting in a time machine and traveling back 20 years…

#!/bin/bash

json='{
  "token" : "eyJhbGciOiJIUzI1NiJ9.verylongstring...",
  "expires" : "2022-01-27T03:13:26.769Z"
}'

xml=$(echo "$json" | /usr/bin/plutil -convert xml1 -o -  -- -)

# That sets $xml to...
# <plist version="1.0">
#   <dict>
#     <key>expires</key>
#     <string>2022-01-27T03:13:26.769Z</string>
#     <key>token</key>
#     <string>eyJhbGciOiJIUzI1NiJ9.verylongstring...</string>
#   </dict>
# </plist>

xpath="/plist/dict/string[preceding-sibling::key='token'][1]/text()"
token=$( echo $xml | /usr/bin/xpath -e  "${xpath}")
echo "token: \"$token\""
# That prints... token: "eyJhbGciOiJIUzI1NiJ9.verylongstring..."

xpath="/plist/dict/string[preceding-sibling::key='expires'][1]/text()"
expires=$( echo $xml | /usr/bin/xpath -e "${xpath}")
echo "expires: \"$expires\""
# That prints... expires: "2022-01-27T03:13:26.769Z"

The xpath is very flexible but there’s a technique in that xpath that some Jamf api scripters won’t have seen before. In Jamf API output, all attributes have a unique path. No computer ever has two names, for example. That being the case, you can always access values directly, e.g., “/computers/computer[1]/name“. But the plist flavor of xml does not work like that. You have to get used to working with dicts, arrays, and key-value pairs. If you look at the sample xml in the commented lines above, you’ll see that if you xpath to /plist/dict/string, you’re going to get two answers — both the expiration and token are have that path. It’s OK. XPath can deal with it.

/plist/dict/string[preceding-sibling::key='expires'][1]/text()

  • /plist/dict” is used to get as far into the hierarchy as we can until things get ambiguous
  • /string[preceding-sibling::key='token'][1]/text()” says “Give me the value (text()) of the first ([1]) string element (/string) that is preceded (preceding) by a “key” (::key=) element with the same parent (sibling) whose value is ‘expires’.”

This preceding element thing is called a predicate. They will drive you nuts for a while but that example has pretty much everything you’ll need to deal with plist data.


Extra for Experts… Shelling out to javascript

Scriptingosx recently pointed out that macOS does have JavaScript available and that you can call from a shell script. Using that doesn’t throw up a bunch of depreciation warnings like when you try to call an old version of Python, and JavaScript has modern parsing for everything.

(Credits… idea popularized by @pico on macadmins, see also: https://www.macblog.org/posts/how-to-parse-json-macos-command-line/ and https://paulgalow.com/how-to-work-with-json-api-data-in-macos-shell-scripts.)

Here is an example of using JavaScript to parse JSON API data…

The output will look like this…

You've got 118 computers

The script…

#!/bin/bash
# Jamf Pro's Classic API returns xml we can parse with xpath but the Jamf
# Pro API returns only json. How can we parse json without resorting to
# tools intended for unstructured data like sed/awk and without introducing
# dependencies that aren't installed on macOS systems by default, e.g., jq?
# (osascript javascript idea from @nico, @scriptingosx)


# ####################### AUTH #######################
# $jamf_url, $jamf_username, and $jamf_password are stored in environment vars
# $jamf_url is in the form "https://your.jamfcloud.com"
[[ -z $jamf_url || -z $jamf_username || -z $jamf_password ]] && { echo '[error] Missing environment vars'; exit; }


# ##################### FUNCTIONS #####################
do_curl() {
  curl_endpoint=$1
  [[ "$#" -ge 2 ]] && curl_method=$2 || curl_method='GET'
  [[ "$#" -ge 3 ]] && curl_token=$3 || curl_token=''
  headers='Accept: application/json '
  if [[ ! -z "$curl_token" ]]; then
    curl \
      --user "${jamf_username}:${jamf_password}" \
      --request "${curl_method}" \
      --header "Authorization: Bearer ${curl_token}" \
      --header "accept: application/json" \
      --silent --show-error \
      "${jamf_url}${curl_endpoint}"
  else
    curl \
      --user "${jamf_username}:${jamf_password}" \
      --request "${curl_method}" \
      --header "accept: application/json" \
      --silent --show-error \
      "${jamf_url}${curl_endpoint}"
  fi
}

getJsonValue() {
  # You can pass some json and a key into this function and get back the value
  # $1=json string, $2=key to return
  # e.g., json='{"key": "value"}'; val=getJsonValue "$json" "key"; echo $val => "value"
  JSON="$1" osascript -l 'JavaScript' \
    -e 'const env = $.NSProcessInfo.processInfo.environment.objectForKey("JSON").js' \
    -e "JSON.parse(env).$2"
EndOfScript
}


# ###################### SCRIPT ######################

debug=false  # true | false

auth_json=$( do_curl '/uapi/auth/tokens' 'POST' )
auth_token=$( getJsonValue "$auth_json" "token" )

computer_json_string=$( do_curl '/api/v1/computers-inventory?page-size=1&section=GENERAL' 'GET' "${auth_token}" )
[[ $debug == true || $computer_json_string == *'"errors" : [ {'* ]] && echo "[DEBUG] $computer_json_string"
computer_count=$( getJsonValue "${computer_json_string}" "totalCount" )
echo "You've got ${computer_count} computers"

That’s a bit much just to get one data item. But most of it is the overhead of the re-usable functions. You could also code almost all of the above logic right in JavaScript so all the shell script would need to do is make the call to OSAscript. That might actually be a totally reasonable approach since JavaScript is so capable, but very few Mac admins also happen to be JavaScript programmers.

Just to illustrate that JavaScript can do more than pluck out a single value, here’s an example where we pull a data series into a shell array we can use for looping:

The output will be something like…

You've got 118 computers

Fetching an array of all computer IDs
[DONE] Here's the array of computer IDs:
1, 2, 4, 5, 6, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 121, 122

The code… (most of this is just to deal with API result pagination… the javascript part is actually pretty simple.)

#!/bin/bash

# Get a list of computer IDs from Jamf Pro. 

# ####################### AUTH #######################
# Note: $jamf_url, $jamf_username, and $jamf_password are stored in environment vars
# $jamf_url is in the form "https://your.jamfcloud.com"
[[ -z $jamf_url || -z $jamf_username || -z $jamf_password ]] && { echo '[error] Missing environment vars'; exit; }

# ##################### FUNCTIONS #####################
do_curl() {
  curl_endpoint=$1
  [[ "$#" -ge 2 ]] && curl_method=$2 || curl_method='GET'
  [[ "$#" -ge 3 ]] && curl_token=$3 || curl_token=''
  headers='Accept: application/json '
  if [[ ! -z "$curl_token" ]]; then
    curl \
      --user "${jamf_username}:${jamf_password}" \
      --request "${curl_method}" \
      --header "Authorization: Bearer ${curl_token}" \
      --header "accept: application/json" \
      --silent --show-error \
      "${jamf_url}${curl_endpoint}"
  else
    curl \
      --user "${jamf_username}:${jamf_password}" \
      --request "${curl_method}" \
      --header "accept: application/json" \
      --silent --show-error \
      "${jamf_url}${curl_endpoint}"
  fi
}

getJsonValue() {
  # You can pass some json and a key into this function and get back the value
  # $1=json string, $2=key to return
  # e.g., json='{"key": "value"}'; val=getJsonValue "$json" "key"; echo $val => "value"
  JSON="$1" osascript -l 'JavaScript' \
    -e 'const env = $.NSProcessInfo.processInfo.environment.objectForKey("JSON").js' \
    -e "JSON.parse(env).$2"
    json.$my_key
EndOfScript
}

getComputerIDsFromPage () {
  # Returns ids from a block of computer.results
  local computer_json=$1
  /usr/bin/osascript -l JavaScript << EndOfScript 2>&1
    const json_obj = JSON.parse('${computer_json}');
    // Pull the results out (just for convenience)
    const computers = json_obj.results;
    // Create an empty array
    const ids = [];
    // Loop through the computers
    for (var i = 0; i < computers.length; i++){
      // Append this computer's ID to the ID array
      ids.push(computers[i].id);
    }
    // Return the array of IDs to the shell as a comma-separated string
    ids
EndOfScript
}

# ##################### SCRIPT #####################

debug=false  # true | false

auth_json=$( do_curl '/uapi/auth/tokens' 'POST' )
auth_token=$( getJsonValue "$auth_json" "token" )

computer_json_string=$( do_curl '/api/v1/computers-inventory?page-size=1&section=GENERAL' 'GET' "${auth_token}" )
[[ $debug == true || $computer_json_string == *'"errors" : [ {'* ]] && echo "[DEBUG] $computer_json_string"
computer_count=$( getJsonValue "${computer_json_string}" "totalCount" )
echo "You've got ${computer_count} computers"

# The above example is simple... just pulling a single data item out of the json
# But javascript has options for more complex json manipulation. Suppose we want to
# list the IDs for all computers so we can loop though them? If we have a lot of 
# computers we'll need to deal with the JP API being paginated. We can use the 
# computer_count from above to help with that. (This is obvs. way past the line 
# where you'd switch to a real programming language, but...)
echo; echo 'Fetching an array of all computer IDs...'
all_computer_ids_array=()
next_page=0
page_size=100
page_computers_count=9999
[[ $debug == true ]] && echo "[DEBUG] Creating an ID list from API computer pages..."
while [[ page_computers_count -ne 0 ]]; do
  [[ $debug == true ]] && echo "[DEBUG] Page [begin]"
  # Get a page of computers. 
  url="/api/v1/computers-inventory?page=${next_page}&page-size=${page_size}&section=GENERAL&sort=id%3Aasc"
  computer_json_string=$( do_curl "$url" 'GET' "${auth_token}" )

  # I saw some errors passing the text to the function. 
  # Probs some quoting needed but this fixes it.
  computer_json_string=$( echo "${computer_json_string}" | tr -d '\n' )

  # Use javascript to extract the computer IDs from the API json response
  page_computer_ids_csv=$( getComputerIDsFromPage "${computer_json_string}" )

  # Convert javascript's comma-separated list of computer IDs into a shell array
  IFS=', ' read -r -a page_computer_ids_array <<< "${page_computer_ids_csv}"
  page_computers_count=${#page_computer_ids_array[@]}

  # append this page's ids to the full id array
  all_computer_ids_array+=(${page_computer_ids_array[@]})

  # Prepare for the next loop iteration
  let next_page+=1  # iterate to the next page.
done

echo "[DONE] Here's the array of computer IDs:"
# These are some examples of bash array looping...
# for item in "${all_computer_ids_array[@]}"; do
#    echo " ${item}"
# done
# for ((i=0; i < ${#all_computer_ids_array[@]}; i++ )); do echo " Computer ID ${i}: ${all_computer_ids_array[$i]}"; done
printf -- "%s, " ${all_computer_ids_array[*]} | cut -d "," -f 1-${#all_computer_ids_array[@]}

Advertisement

One thought on “Parse Jamf Pro API JSON data in Shell Scripts

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: