Sure. Let's call an example customer Dave.
Good question. Most of our scrapers are written using golang, which is our preferred language for web scraping. It's a compiled language which provides huge performance benefits over traditional web scraping tools (such as python's BeautifulSoup). We utilize existing packages as well as our own to create an efficient scraper, which fits your requirements.
Once we've unit tested the golang code, we deploy cloud server instances to run the scraping work. The amount of server instances we deploy depends on the amount of work required (quantity of data to be collected, complexity of websites to be crawled, etc). Often we rotate through proxies, and use threading (go routines) to control the rate at which we process data.
This data is then stored in a database, usually postgres or mysql.
Once complete, depending on the requirements - the database is analyzed and processed into the format requirements of the customer. Often this will be a simple CSV file, which may be shared via FTP, AWS s3, Google cloud, or directly uploaded to a clients server via rsync, scp or similar.