I hope everything is good with you!
I created a tool in C# that would run every day and was able to fetch me the prices of all the products in certain categories of my choice from the Cex website (uk.webuy.com). I used this tool to keep a price history for the products but one day, as the tool was executing, it started getting a 403 error. I haven't been able to diagnose why this is happening since other end-points from this domain work completely fine.
Here is the code I have been using for the last 6 months for the request:
public static List<Box> GetCategoryProducts(List<int> largeCategoriesList) { List<Box> boxes = new List<Box>(); foreach (int categoryId in largeCategoriesList) { int currentCategoryProducts = 0; int firstRecord = 1; int count = 50; int totalRecords = 0; int minDelay = 3; int maxDelay = 6; int retryDelay = 10; // Retry after 10 seconds if response isn't successful bool hadError = false; int retryCount = 0; do { try { string weBuyAPILink = $"https://wss2.cex.uk.webuy.io/v3/boxes?firstRecord={firstRecord}&count=50&sortBy=relevance&sortOrder=desc&categoryIds=[{categoryId}]"; var client = new RestClient(weBuyAPILink); client.Timeout = -1; var request = new RestRequest(Method.GET); List<KeyValuePair<string, string>> headerList = new List<KeyValuePair<string, string>>() { new KeyValuePair<string, string>("Accept", "application/json, text/plain, */*"), new KeyValuePair<string, string>("Refer", "https://uk.webuy.com"), new KeyValuePair<string, string>("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"), new KeyValuePair<string, string>("Origin", "https://uk.webuy.com"), new KeyValuePair<string, string>("Accept-Encoding","gzip, deflate, br") }; request.AddHeaders(headerList); IRestResponse response = client.Execute(request); if (!response.IsSuccessful) { Logger.Error($"Request response encountered an error! Error details: {response.StatusCode}"); hadError = true; Thread.Sleep(retryDelay * 1000); // Retry after 10 seconds retryCount++; } else { //PROCESS RESPONSE //....... //....... hadError = false; var random = new Random(); var delay = random.Next(minDelay * 1000, maxDelay * 1000); Logger.Info($"Waiting {delay / 1000} seconds before next request..."); Thread.Sleep(delay); } } catch (Exception ex) { Logger.Fatal(ex, "Fatal error processing category list!"); // Handle fatal error as needed hadError = true; } } while ((currentCategoryProducts < totalRecords || totalRecords == 0) && !hadError && retryCount < 3); // Retry only 3 times if (hadError) { Console.WriteLine("Unable to get price info"); } } return boxes; }
Trying to access the URL on a browser will give a Cloudflare page saying that I have been blocked and that I can no longer access webuy.io, but, for example, accessing the product detailendpoint that gives me only the details from a specific product, works fine.
Can someone point me in the right direction? I've been looking at Cloudflare bypasses but I haven't invested a lot of time in them because I think that won't solve my issue.
I looked into this GitHub repo when building my tool as it had some insightful information about the API.
I tried changing some request headers, tried using a VPN, and also tried using ScraperAPI and ScrapeOps to fetch me the information without getting the 403 error but to no avail.