I need to implement scroll API on Azure Data Factory Copy Activity from OpenSearch via rest API (https://opensearch.org/docs/2.0/opensearch/search/paginate/).

The Scroll API can be used to iterate over a large amount of OpenSearch documents matching a query, or even all the matching documents.

Here you are an example (curl):

First request

GET products/_search?scroll=1m{"size": 10 }

Next request

GET _search/scroll{"scroll_id":"FGluY2x1ZGVfY29udGV4dF91dWlkDXF1ZXJ5QW5kRmV0Y2gBFlViblo1YXZTU2pxYkl1eE9zV3JLaEEAAAAAAAAAChZLem1HOElfcVRkT2JoZWljcnlWaFN3" }

I had troubles with ADF Copy Activity pagination rules. Please help.

I tried by first catching the scroll_id as output from the first call, storing it to a variable. But I couldn't figure out how to use that variable in Copy Activity's pagination rule (next request) in order to copy the OpenSearch Index json response.

Thanks