Retrieving Paginated API Data With Axios

I recently encountered an API which would only permit a limited number of objects to be returned. The objects in this case were devices, with properties such as operating system, IP address, etc.

If there were more than 250 devices, the first 250 would be returned, along with a URL in a property called nextPageUrl for the next 250. If we were on the last ‘page’, or there were less than 250 devices in total, the nextPageUrl property would be null.

I created a recursive function using axios that would call itself repeatedly. We’re also using a lodash function to merge the objects along with a customizer function that will concatenate the arrays.

function customizer(objValue, srcValue) {
    if (_.isArray(objValue)) {
        return objValue.concat(srcValue);
    }
}

async function ExecuteRequest(url, data) {

    // As this is a recursive function, we need to be able to pass it the prevous data. Here we either used the passed in data, or we create a new objet to hold our data.
    data = data || {};

    await axios.get(url).then(response => {

        // We merge the returned data with the existing data
        _.mergeWith(data, response.data, customizer);

        // We check if there is more paginated data to be obtained
        if (response.data.pageDetails.nextPageUrl != null) {

            // If nextPageUrl is not null, we have more data to grab
            return ExecuteRequest(response.data.pageDetails.nextPageUrl, data);
        }
    });

    return data;
}

We can then make our API request and have it return all the data for further processing.

ExecuteRequest("https://api.company.com.au/v2/account/devices").then(data => {
    console.dir(data);
});

It should be noted that this is a very cut down version of the actual code, it is missing all the authentication and bearer token checking.

In the production code we called an another function (instead of calling ExecuteRequest directly) which would deal with the authentication and would then call ExecuteRequest, passing it the tokens to be used in the request.