I am using ADF to retrieve data from a REST API and loading into an SQL DB. The problem I have is that the endpoints in the API return a max of 50 records per page.
I'm trying to use the built-in pagination in ADF by setting Pagination rules with QueryParameters, {offset} using range and the max value.
Some endpoints in the API does return total rows in the call. So I could possibly use this somehow to make the Copy Data activity dynamic.
The relative URL i use is:
@concat(pipeline().parameters.relative_url,'?offset={offset}')
And the {offset} in Pagination Rules is set up as:
@concat('RANGE:0:',activity('LookUp_Total').output.firstRow.Total,':50')
The problem I encounter is with endpoints where the total is larger than 50 I get duplicates. So if the total is for example 76 I still get 100 records.