Web scraping is automated collection of page content at a scale and speed that a person browsing the site would not reach.
Some collection is permitted research, and some copies a catalog so fast that it harms the site or breaks its terms.
How it works
A program requests pages and extracts content repeatedly.
It may hide behind a home style address or a client that looks ordinary, so volume and behavior matter more than a single address.
Rate limits, accounts for valuable data, and a computational cost per request make bulk collection more expensive.
Measure performance separately so scrapers do not distort what human visitors experience.
Concrete examples
- A script copies an entire product catalog overnight without a human reader.
- A competitor pulls prices every few minutes through automated requests.
- A researcher downloads public pages slowly with permission and a clear identity.
Protective measures
- State which collection is allowed and which volume is not.
- Rate limit valuable data and require an account where that fits the product.
- Detect sudden walks of an entire catalog rather than normal browsing.
- Add a computational cost so bulk collection is more expensive to automate.
How CDPCaptcha helps
Proof of work raises the cost of automated page collection because each fetch must spend computing time before the site returns the content.
CDPCaptcha is a self-hosted proof of work check in the browser, with no cookies, no tracking, and no picture puzzles.
It is made and hosted in the EU, in Germany, and the service is described at https://cdpcaptcha.com/.
Related articles
What is spam? describes bulk automation of messages, a different channel from automated page collection.
What is a static residential proxy? explains the stable home address that collectors use to resemble a household.
What is time to first byte? shows the performance metric that heavy automated collection can distort.