Quantcast
Channel: Active questions tagged rest - Stack Overflow
Viewing all articles
Browse latest Browse all 3637

Getting 403 error in non-public API but not blocked from domain

$
0
0

I hope everything is good with you!

I created a tool in C# that would run every day and was able to fetch me the prices of all the products in certain categories of my choice from the Cex website (uk.webuy.com). I used this tool to keep a price history for the products but one day, as the tool was executing, it started getting a 403 error. I haven't been able to diagnose why this is happening since other end-points from this domain work completely fine.

Here is the code I have been using for the last 6 months for the request:

public static List<Box> GetCategoryProducts(List<int> largeCategoriesList)        {            List<Box> boxes = new List<Box>();            foreach (int categoryId in largeCategoriesList)            {                int currentCategoryProducts = 0;                int firstRecord = 1;                int count = 50;                int totalRecords = 0;                int minDelay = 3;                int maxDelay = 6;                int retryDelay = 10; // Retry after 10 seconds if response isn't successful                bool hadError = false;                int retryCount = 0;                do                {                    try                    {                        string weBuyAPILink = $"https://wss2.cex.uk.webuy.io/v3/boxes?firstRecord={firstRecord}&count=50&sortBy=relevance&sortOrder=desc&categoryIds=[{categoryId}]";                        var client = new RestClient(weBuyAPILink);                        client.Timeout = -1;                        var request = new RestRequest(Method.GET);                        List<KeyValuePair<string, string>> headerList = new List<KeyValuePair<string, string>>()                        {                            new KeyValuePair<string, string>("Accept", "application/json, text/plain, */*"),                            new KeyValuePair<string, string>("Refer", "https://uk.webuy.com"),                            new KeyValuePair<string, string>("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"),                            new KeyValuePair<string, string>("Origin", "https://uk.webuy.com"),                            new KeyValuePair<string, string>("Accept-Encoding","gzip, deflate, br")                        };                        request.AddHeaders(headerList);                        IRestResponse response = client.Execute(request);                        if (!response.IsSuccessful)                        {                            Logger.Error($"Request response encountered an error! Error details: {response.StatusCode}");                            hadError = true;                            Thread.Sleep(retryDelay * 1000); // Retry after 10 seconds                            retryCount++;                        }                        else                        {                            //PROCESS RESPONSE                            //.......                            //.......                            hadError = false;                            var random = new Random();                            var delay = random.Next(minDelay * 1000, maxDelay * 1000);                            Logger.Info($"Waiting {delay / 1000} seconds before next request...");                            Thread.Sleep(delay);                        }                    }                    catch (Exception ex)                    {                        Logger.Fatal(ex, "Fatal error processing category list!");                        // Handle fatal error as needed                        hadError = true;                    }                } while ((currentCategoryProducts < totalRecords || totalRecords == 0) && !hadError && retryCount < 3); // Retry only 3 times                if (hadError)                {                    Console.WriteLine("Unable to get price info");                }            }            return boxes;        }

Trying to access the URL on a browser will give a Cloudflare page saying that I have been blocked and that I can no longer access webuy.io, but, for example, accessing the product detailendpoint that gives me only the details from a specific product, works fine.

Can someone point me in the right direction? I've been looking at Cloudflare bypasses but I haven't invested a lot of time in them because I think that won't solve my issue.

I looked into this GitHub repo when building my tool as it had some insightful information about the API.

I tried changing some request headers, tried using a VPN, and also tried using ScraperAPI and ScrapeOps to fetch me the information without getting the 403 error but to no avail.


Viewing all articles
Browse latest Browse all 3637

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>