Paging API Data with Flow

This draft page will be completed shortly. Come back soon for complete instructions!

Most APIs will return a truncated or limited data set leaving you with a token or URL for the next page or set of data. For example, let’s say you wanted to request the metrics for all of your YouTube or Facebook videos. YouTube will return only 50 results in the first page; Facebook will return up to 100 results in the first page. We need a Flow that iterates over each page and appends the data you’re looking for to a string or an array.

As an example we’ll create a Flow that paginates over multiple pages of YouTube analytics metrics. The method is similar for Facebook with a few modifications.

Start by creating a new Flow. It can be manually triggered at this point, or you can set it up on a recurring schedule. Insert the Initialize Variable action, and make it a string variable with the name Next. You can leave the initial value blank, we’ll set the variable in a later step. Add the HTTP step with GET as the method. For this example we’ll use a YouTube API request that gets a list of videos for a specified channel:[APIKEY]&channelId=[CHANNEL_ID]&part=id&order=date&maxResults=50

Replace [API_KEY] and [CHANNEL_ID] with your API key and the channel ID you want to collect information from.

Add the Parse JSON action to your Flow and select Body from the HTTP request as your content. Then generate the data schema by copying your request into a browser and pasting the results into Flow’s built-in schema generator.

If you scroll down in your JSON schema you’ll notice that there is a nextPageToken item. As you follow this token in the following requests a new one will be generated until there are no more pages of data to view. Add a Set Variable step, select our only variable, Next, then set the value to nextPageToken from the previous Parse JSON action.

Next, add a Condition control step and set the “Choose a value” field to our “Next” variable. The logical test should be set to “is equal to” and the next field should be left blank. You’ll understand why we leave the second “choose a value” field blank shortly. At this point your Flow should look like this:

In the “If No” side of the condition action add the “Do Until” action. Do the same thing here as above where our Next variable is equal to (blank).

Add a HTTP GET action in our Do Until action and in this case we’ll paste the YouTube API request that returns the next page of data:[nextPageToken]&key=[API_KEY]&channelId=[CHANNEL_ID]&part=id&order=date&maxResults=50

Replace [nextPageToken] with your Next variable, and [API_KEY] AND [CHANNEL_ID] with your applicable data. Insert the Parse JSON step, set the content to the HTTP request above it, then paste the schema from your very first Parse JSON action – they will be the same.

While still in your Do Until action add the Set Variable action, select our Next variable, then set the value to nextPageToken from the previous Parse JSON action. What happens is the HTTP request request the next page of data with each new page token until the nextPageToken field is empty. If the dataset is small enough to fit in the first request, Flow won’t even try to send a request for the next page since the Next variable will be empty.

Your finished Flow should look like this:

At this point you can customize the Flow to meet your needs. If you are saving paginated data make sure to do that within the Do Until loop, before the Flow moves on to the next page of data.