Tip: Multiple REST API requests

Customizations
1

I had a request (no pun intended) around making multiple REST API requests more efficiently. Generally our REST API examples are written to be easy and instructive, without much error handling, XML parsing, or efficiency considerations.

Most high-level scripting languages provide some means for multiple REST API requests to be more efficient, by establishing and reusing a single HTTPS session for the multiple operations. This saves time by creating a single session, with a single authentication request, and then performing multiple operations over that one channel. In Python, the ‘requests.Session’ class implements this option.

I’ve posted an example at https://github.com/Jwalker107/BigFix/blob/master/API%20Samples/requests-session-example.py of using requests.Session to open a connection to the BigFix server, and perform multiple PUT operations. It loops through a directory containing one or more .bes files, where each file is named for a Task ID (123.bes, 456.bes, etc.) and each file is updated on the server’s ‘Temp’ Custom Site with a PUT to /api/task/custom/TEMP/{taskid}. As a bonus, the script also uses requests HTTPAdapter module to automatically retry operations when there is a retryable error like ‘Server Busy’ or ‘Server Unavailable’ (HTTP codes 500, 502, etc.)

I’m posting the script here as well for convenience & searchability, but be sure to check the GitHub link for the most recent version if I make any changes. I’d love to hear your feedback if you find this useful.

"""
Example using requests.Session to perform repeated operations over a single HTTPS session
This loops through each Task in a given directory and PUTS the task to the BigFix Server at the given URL
The 'localdir' should contain a number of Tasks in the format "123.bes", "456.bes", etc.
Each file should have the '.bes' extension and the filename component should be a numeric Task ID in the Site

As an added bonus, HTTPAdapter and Retry are used to automatically retry on retryable HTTP return codes such as 'server busy' or 'server unavailable'
"""
from requests import Session
from requests.adapters import HTTPAdapter
from urllib3 import Retry
import os


### Configuration settings ###
server="https://bes-root.local:52311"
username="mo"
password="Not Gonna Get Me Today!"

server_repo="/api/task/custom/Temp"
localdir="C:\\Temp\\apitest\\Temp\\task"


my_session=Session()
# Todo - add Certificate Verification
my_session.verify=False

# Provide the credentials that will be persisted in the Session object
my_session.auth = (username, password)

# Bonus - Add automatic retries & timeouts to the Session operations
adapter=HTTPAdapter(
        max_retries=Retry(total=4,
            backoff_factor=1,
            allowed_methods=None,
            status_forcelist=[429,500,502,503,504]
            )
        )

my_session.mount("http://", adapter)
my_session.mount("http2://", adapter)


# Test the session login
response=my_session.get(url=server + "/api/login")

if not response.ok:
    raise ValueError(f"Server login failed with {response.status_code} : {response.text}")

# Loop through each .bes file in local directory
for filename in os.listdir(localdir):
    if not filename.endswith('.bes'):
        print(f"Skipping {filename}, not a .bes file")
        continue
    # Split filename on the last period, expect the id should be an integer fixlet/task ID
    id=filename.rsplit(".",1)[0]
    if not id.isnumeric():
        print(f"Filename {filename} does not translate to a numeric ID, skipping")
        continue
    
    target_url=f"{server}{server_repo}/{id}"
    print(f"PUT {filename} to {target_url}")
    with open(os.path.join(localdir, filename), 'rb') as filecontent:
        response=my_session.put(url=target_url, data=filecontent)
        print(f"Result OK:{response.ok}, Code:{response.status_code}, Text:{response.text}")
3 Likes