Quantcast
Channel: Active questions tagged rest - Stack Overflow
Viewing all articles
Browse latest Browse all 3667

Offset in API calls in Azure Data Factory

$
0
0

I am using ADF to retrieve data from a REST API and loading into an SQL DB. The problem I have is that the endpoints in the API return a max of 50 records per page.

I'm trying to use the built-in pagination in ADF by setting Pagination rules with QueryParameters, {offset} using range and the max value.

Some endpoints in the API does return total rows in the call. So I could possibly use this somehow to make the Copy Data activity dynamic.

The relative URL i use is:

@concat(pipeline().parameters.relative_url,'?offset={offset}')

And the {offset} in Pagination Rules is set up as:

@concat('RANGE:0:',activity('LookUp_Total').output.firstRow.Total,':50')

The problem I encounter is with endpoints where the total is larger than 50 I get duplicates. So if the total is for example 76 I still get 100 records.


Viewing all articles
Browse latest Browse all 3667

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>