Simply hearing the word data collection often leads people to think of shady businessmen, evil meglomaniacs or data thieving pirates. They often associate data collection, or website scraping with privacy invading conglomerate sized companies such as Facebook, or situations such as the Cambridge Analytica data scandal
Others think we are hackers, stealing credit card information, breaking into banks and destabilising social structures.
Whilst these explanations make great fictional stories, in fact this service is run by your average nerd. I've got a life, run a couple of businesses, and this is one of them. Data collection can be perfectly ethical and legal if done correctly, in fact many of the worlds largest companies (search engines) business models rely on data collectors. For your average small business, scraping a small amount of data from a competitors website (for example) is perfectly ethical and can generate business intelligence unavailable thorough other means.
In reality, I'm just a small business owner who finds great satisfaction in writing a spider and watching it harvest data on a group of servers. It's almost a breathtaking thrill watching the http requests fly back and forth on different threads whilst mining data quietly and efficiently.
Written by Ollie Cox