Web scraping for business: what it is and when it is allowed
Web scraping means collecting information from websites automatically instead of copying it by hand. Companies use it to track prices, monitor announcements, gather public listings and keep their own data up to date. It is useful, but it comes with rules. This guide explains the basics. It is general information, not legal advice.
Common business uses
- Price and stock monitoring across competitors or suppliers
- Collecting public announcements, tenders and calls, such as scholarship calls or utility notices
- Keeping a catalogue or directory in sync with a source you are allowed to use
- Turning documents on a website into structured data for analysis
Questions to ask before you start
- Is there an official API or data export? If yes, use it instead of scraping
- What do the site’s terms of use and robots.txt say about automated access?
- Is the data public, and does it contain personal data? Personal data brings GDPR duties
- Is the content protected by copyright or database rights, and how will you reuse it?
- Will your collection load the site gently, with limits on how often you request pages?
What good scraping looks like
A reliable scraper checks the source on a schedule, handles changes to the page layout, cleans messy text, stores the result in a database and alerts you when something breaks. The hard part is rarely the first download. It is keeping the data correct month after month.
Our approach
We build and run scrapers for our own products, including the public announcement feeds behind nestalo.ba and Scatch, and for clients. We start by checking whether an official source exists, then agree what may be collected and how it will be used. If a project does not look appropriate, we say so.