Every list on this site is checked before it goes up, and every list is half dead a day later. That is the nature of public proxies, and it sets a hard limit on what you can scrape with them. This page puts numbers on that limit so you can tell in advance whether a free list will carry your job or whether you are about to spend a weekend restarting it.
What a fresh list is worth
Take a batch of 100 proxies from the homepage. Within 24 hours about half stop answering. Of the rest, a third are transparent, which means the target sees your real address in the headers, and a third respond in three seconds or slower. You are left with 15 to 20 addresses that behave, and a good share of those are already rate-limited on Amazon, Google or any large site, because a thousand other people pulled the same list this morning.
Twenty usable addresses at one request every five seconds each give you roughly 14,000 requests in a day, if none of them dies mid-run. In practice they do, so plan on 5,000 to 8,000 pages from one list before you need the next one.
Jobs a free list handles
A one-off pull of a few thousand pages. Checking how a page renders from another country. Testing a new scraper before you point it at anything expensive. Learning how rotation, retries and timeouts behave. For all of that, grab the latest list, run it through the checker first, and go.
Jobs it does not
Anything that has to finish every day at the same size. A catalog of 200,000 pages, a daily crawl of a job board, a competitor's whole product range. Large sites block an address somewhere between 300 and 1,000 requests a day, so a 200,000-page crawl needs 200 to 700 addresses that all survive the day, and a free list will not give you 20 that do. The crawl stalls at page 8,000, you feed it a new list, it stalls again at 15,000, and by evening the data is from three different hours and two different lists.
At that size a pool is the cheaper option once you count the hours you stop spending on list maintenance. RemProxy has a page for this job with a thread calculator and a Python sample that rotates through the address list and retries on failure: a pool built for scraping. Thousands of addresses, each one touching the target a few dozen times a day, which stays under the limits that burn a shared IP in an hour.
How free proxies fail, so you can catch it
Dead mid-run. Connection refused or a timeout after the first hundred pages. Set a timeout of 10 seconds and drop the address after three failures in a row.
Transparent. The proxy forwards your real IP in X-Forwarded-For. Check a fresh
list against a page that echoes headers, and keep only the addresses that hide you. The anonymity
column on every list here is that check, done once at posting time.
Already banned. The address works, the target answers, and the answer is a captcha or an empty page. Detect it by looking for the field you came for. A product page with no price is a block, and the HTTP status will say 200 anyway.
Wrong country. Geo-restricted content and localized prices depend on the address. Filter the list by country before the run, and do not trust the label without checking: some addresses move.
Before you point a scraper at anything
- Read the target's robots.txt and respect the crawl delay if it names one.
- Cache what you fetched. Re-scraping a page you already have wastes the address budget twice.
- One request per address every five seconds or more, retries through a different address.
- Verify the list through the checker right before the run, not the night before.
- Count the pages first. Under 5,000 a free list works. Over 50,000 it does not, and the number in between depends on how hostile the target is.