Parse Jamf Pro API JSON data in Shell Scripts

Shell scripts are probably not a first choice for API programming, but if it’s what you know, it’s what you know. And it’s available on every Mac so shell scripts are super portable. Shell programming is good at working on the kinds of data people used back when it was created… like lines in a file or delimiter-separated fields. Modern APIs use more structured data formats like XML and JSON that didn’t exist when shell was invented.

Shell tools like grep/awk/sed can be used to parse structured data if you really have to, but it’s not very reliable, and regex takes some getting used to. For XML you can use the xpath or xmllint commands since those come built into macOS.

For JSON, you can install something like jq. It’s really easy to use so great if you need it to run your own scripts on your own machine, but it doesn’t come pre-installed on macOS so you’d have to tell people to install it if they want to use your script.

You can call Python from a bash script, but if you’re

Ruby, Perl, and Python are great at parsing JSON. Some macadmins mostly just know bash, though, so they’ll just shell out to one of those when they really have to. E.g., use python to do something quick in a shell script…

api_token=$(echo "$authToken" | python -c 'import sys, json; print json.load(sys.stdin)["token"]')

But Apple is depreciating the installs of those languages in macOS, and the python2 they still ship is past end-of-life. So you really are looking at installing a current version if you want to work with them so using them introduces dependancies and reduces the portability of your scripts.

So, what to do? Here’s a simple json parse using the plutil command

#!/bin/bash

# Set these vars or read them from env vars, or from a password store file, etc. 
jssUrl='https://my.jamfcloud.com'   # omit trailing slash
apiUser='api_read_only_user'
apiPass='mycomplexpassword'


# #########################################################
curlCmd="/usr/bin/curl --silent --show-error"
# iconv may help when using foreign character sets and special chars in passwords
userPass=$(printf "${apiUser}:${apiPass}" | /usr/bin/iconv -t ISO-8859-1)

# Use encoded username and password to request an API authentication token. 
url="${jssUrl}/api/v1/auth/token"
echo "Calling ${url}"
authTokenJson=$(${curlCmd} \
  --user "${userPass}" \
  --request "POST" \
  --connect-timeout "5" \
  "${url}")
echo "authTokenJson: ${authTokenJson}"
# Expect output in the form:
# authTokenJson: {
#   "token" : "eyJhbGciOiJIUzI1NiJ9.verylongstring...",
#   "expires" : "2022-01-27T03:13:26.769Z"
# }


# Extract the token from the output of /auth/token:
# Use plutil to extract the token value from the json.
# "-o - -" here means "output to stdout, accept input from the <<< $var"
authToken=$(/usr/bin/plutil -extract "token" raw -expect "string" -o - - <<< "${authTokenJson}")
echo "apiToken: ${apiToken}"
# output will be "eyJhbGciOiJIUzI1NiJ9.verylongstring..."

plutil doesn’t offer the complex attribute targeting of xpath or Python or jq, but for top-level attributes, it’s really simple. You may be able to use plutil to pull out an object, then send that back into plutil to work your way to what you want, but that gets kind of involved. It would be cool if something with a good json parser came built-into macOS.

Scriptingosx recently pointed out that macOS does have JavaScript available and that you can call from a shell script. Using that doesn’t throw up a bunch of depreciation warnings like if you call the antiquated Python Apple’s giving us, and JavaScript has modern parsing for everything.

(Credits… idea popularized by @pico on macadmins, see also: https://www.macblog.org/posts/how-to-parse-json-macos-command-line/ and https://paulgalow.com/how-to-work-with-json-api-data-in-macos-shell-scripts.)

Here is an example of using JavaScript to parse JSON API data…

The output will look like this…

You've got 118 computers

The script…

#!/bin/bash
# Jamf Pro's Classic API returns xml we can parse with xpath but the Jamf
# Pro API returns only json. How can we parse json without resorting to
# tools intended for unstructured data like sed/awk and without introducing
# dependencies that aren't installed on macOS systems by default, e.g., jq?
# (osascript javascript idea from @nico, @scriptingosx)


# ####################### AUTH #######################
# $jamf_url, $jamf_username, and $jamf_password are stored in environment vars
# $jamf_url is in the form "https://your.jamfcloud.com"
[[ -z $jamf_url || -z $jamf_username || -z $jamf_password ]] && { echo '[error] Missing environment vars'; exit; }


# ##################### FUNCTIONS #####################
do_curl() {
  curl_endpoint=$1
  [[ "$#" -ge 2 ]] && curl_method=$2 || curl_method='GET'
  [[ "$#" -ge 3 ]] && curl_token=$3 || curl_token=''
  headers='Accept: application/json '
  if [[ ! -z "$curl_token" ]]; then
    curl \
      --user "${jamf_username}:${jamf_password}" \
      --request "${curl_method}" \
      --header "Authorization: Bearer ${curl_token}" \
      --header "accept: application/json" \
      --silent --show-error \
      "${jamf_url}${curl_endpoint}"
  else
    curl \
      --user "${jamf_username}:${jamf_password}" \
      --request "${curl_method}" \
      --header "accept: application/json" \
      --silent --show-error \
      "${jamf_url}${curl_endpoint}"
  fi
}

getJsonValue() {
  # You can pass some json and a key into this function and get back the value
  # $1=json string, $2=key to return
  # e.g., json='{"key": "value"}'; val=getJsonValue "$json" "key"; echo $val => "value"
  local my_json=$1
  local my_key=$2
  /usr/bin/osascript -l JavaScript <<- EndOfScript
    const json=JSON.parse(\`${my_json}\`)
    json.$my_key
EndOfScript
}


getJsonValue_jsc() {
  # From: https://paulgalow.com/how-to-work-with-json-api-data-in-macos-shell-scripts
  # This one-liner uses macOS's JavaScriptCore. 
  # Would this be lower-overhead than invoking via OSAScript?

  # $1: JSON string to process, $2: Desired JSON key
  # Will return 'undefined' if key cannot be found
/System/Library/Frameworks/JavaScriptCore.framework/Versions/Current/Helpers/jsc \
  -e "print(JSON.parse(\`$1\`).$2);"
}



# ###################### SCRIPT ######################

debug=false  # true | false

auth_json=$( do_curl '/uapi/auth/tokens' 'POST' )
auth_token=$( getJsonValue "$auth_json" "token" )

computer_json_string=$( do_curl '/api/v1/computers-inventory?page-size=1&section=GENERAL' 'GET' "${auth_token}" )
[[ $debug == true || $computer_json_string == *'"errors" : [ {'* ]] && echo "[DEBUG] $computer_json_string"
computer_count=$( getJsonValue "${computer_json_string}" "totalCount" )
echo "You've got ${computer_count} computers"

That’s a bit much just to get one data item. But most of it is the overhead of the re-usable functions. You could also code almost all of the above logic right in JavaScript so all the shell script would need to do is make the call to OSAscript. That might actually be a totally reasonable approach since JavaScript is so capable, but very few Mac admins also happen to be JavaScript programmers.

Just to illustrate that JavaScript can do more than pluck out a single value, here’s an example where we pull a data series into a shell array we can use for looping:

The output will be something like…

You've got 118 computers

Fetching an array of all computer IDs
[DONE] Here's the array of computer IDs:
1, 2, 4, 5, 6, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 121, 122

The code… (most of this is just to deal with API result pagination… the javascript part is actually pretty simple.)

#!/bin/bash

# Get a list of computer IDs from Jamf Pro. 

# ####################### AUTH #######################
# Note: $jamf_url, $jamf_username, and $jamf_password are stored in environment vars
# $jamf_url is in the form "https://your.jamfcloud.com"
[[ -z $jamf_url || -z $jamf_username || -z $jamf_password ]] && { echo '[error] Missing environment vars'; exit; }

# ##################### FUNCTIONS #####################
do_curl() {
  curl_endpoint=$1
  [[ "$#" -ge 2 ]] && curl_method=$2 || curl_method='GET'
  [[ "$#" -ge 3 ]] && curl_token=$3 || curl_token=''
  headers='Accept: application/json '
  if [[ ! -z "$curl_token" ]]; then
    curl \
      --user "${jamf_username}:${jamf_password}" \
      --request "${curl_method}" \
      --header "Authorization: Bearer ${curl_token}" \
      --header "accept: application/json" \
      --silent --show-error \
      "${jamf_url}${curl_endpoint}"
  else
    curl \
      --user "${jamf_username}:${jamf_password}" \
      --request "${curl_method}" \
      --header "accept: application/json" \
      --silent --show-error \
      "${jamf_url}${curl_endpoint}"
  fi
}

getJsonValue() {
  # You can pass some json and a key into this function and get back the value
  # $1=json string, $2=key to return
  # e.g., json='{"key": "value"}'; val=getJsonValue "$json" "key"; echo $val => "value"
  local my_json=$1
  local my_key=$2
  /usr/bin/osascript -l JavaScript <<- EndOfScript
    const json=JSON.parse(\`$my_json\`)
    json.$my_key
EndOfScript
}

getComputerIDsFromPage () {
  # Returns ids from a block of computer.results
  local computer_json=$1
  /usr/bin/osascript -l JavaScript << EndOfScript 2>&1
    const json_obj = JSON.parse('${computer_json}');
    // Pull the results out (just for convenience)
    const computers = json_obj.results;
    // Create an empty array
    const ids = [];
    // Loop through the computers
    for (var i = 0; i < computers.length; i++){
      // Append this computer's ID to the ID array
      ids.push(computers[i].id);
    }
    // Return the array of IDs to the shell as a comma-separated string
    ids
EndOfScript
}

# ##################### SCRIPT #####################

debug=false  # true | false

auth_json=$( do_curl '/uapi/auth/tokens' 'POST' )
auth_token=$( getJsonValue "$auth_json" "token" )

computer_json_string=$( do_curl '/api/v1/computers-inventory?page-size=1&section=GENERAL' 'GET' "${auth_token}" )
[[ $debug == true || $computer_json_string == *'"errors" : [ {'* ]] && echo "[DEBUG] $computer_json_string"
computer_count=$( getJsonValue "${computer_json_string}" "totalCount" )
echo "You've got ${computer_count} computers"

# The above example is simple... just pulling a single data item out of the json
# But javascript has options for more complex json manipulation. Suppose we want to
# list the IDs for all computers so we can loop though them? If we have a lot of 
# computers we'll need to deal with the JP API being paginated. We can use the 
# computer_count from above to help with that. (This is obvs. way past the line 
# where you'd switch to a real programming language, but...)
echo; echo 'Fetching an array of all computer IDs...'
all_computer_ids_array=()
next_page=0
page_size=100
page_computers_count=9999
[[ $debug == true ]] && echo "[DEBUG] Creating an ID list from API computer pages..."
while [[ page_computers_count -ne 0 ]]; do
  [[ $debug == true ]] && echo "[DEBUG] Page [begin]"
  # Get a page of computers. 
  url="/api/v1/computers-inventory?page=${next_page}&page-size=${page_size}&section=GENERAL&sort=id%3Aasc"
  computer_json_string=$( do_curl "$url" 'GET' "${auth_token}" )

  # I saw some errors passing the text to the function. 
  # Probs some quoting needed but this fixes it.
  computer_json_string=$( echo "${computer_json_string}" | tr -d '\n' )

  # Use javascript to extract the computer IDs from the API json response
  page_computer_ids_csv=$( getComputerIDsFromPage "${computer_json_string}" )

  # Convert javascript's comma-separated list of computer IDs into a shell array
  IFS=', ' read -r -a page_computer_ids_array <<< "${page_computer_ids_csv}"
  page_computers_count=${#page_computer_ids_array[@]}

  # append this page's ids to the full id array
  all_computer_ids_array+=(${page_computer_ids_array[@]})

  # Prepare for the next loop iteration
  let next_page+=1  # iterate to the next page.
done

echo "[DONE] Here's the array of computer IDs:"
# These are some examples of bash array looping...
# for item in "${all_computer_ids_array[@]}"; do
#    echo " ${item}"
# done
# for ((i=0; i < ${#all_computer_ids_array[@]}; i++ )); do echo " Computer ID ${i}: ${all_computer_ids_array[$i]}"; done
printf -- "%s, " ${all_computer_ids_array[*]} | cut -d "," -f 1-${#all_computer_ids_array[@]}

One thought on “Parse Jamf Pro API JSON data in Shell Scripts

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: