← Projects

Client Prospector

LaunchedDeprecated

A Python and Tornado pipeline that turned raw WHOIS exports into a qualified call list — thousands of concurrent checks for which small businesses had a website that needed help.

pythontornadoasyncdata pipelinebashconsulting

Consulting work for a software company. The problem they brought me is the problem that every small services business has, which is finding the customers.

The insight was that the answer was already public. Every registered domain has a WHOIS record on it with a registrant, and an address, and dates. Every website either does or does not run WordPress, and it either does or does not work on a phone. If you cross those facts against each other then you have a list of local businesses with a site that needs work, and you have a named human to talk to about it.

WHOIS exportsnoisy proxyScrub and mergerecursive, one datasetFilter by zipa territory, not the worldEnrichthousands in flightProspect listgrouped by registrar, with a name to call
Public facts in, a call list out.

It started as Bash, because the first job was cleaning. A recursive scrubber walked a directory tree of WHOIS exports, and stripped the malformed trailing characters off them, and dropped the columns that carried nothing useful, and concatenated all of it into one dataset. That is the kind of job sed is good at, and it stays good at it right up until the point where it isn't.

The real tool was the Python rewrite, a command-line program with a handful of verbs:

  • Count the records in a dataset, because knowing the size of the thing you are working on is the first step.
  • Filter by zip code range, which is how a national dataset becomes a territory you could drive to.
  • Check for WordPress, which is the enrichment that made the whole thing worth building. It requests /wp-admin on every domain and the status is read back, across tens of thousands of hosts, using Tornado's async HTTP client over a curl backend. Done serially at 1 second per host, a large dataset is a week of waiting, and done with thousands of requests in flight it is a coffee break instead. This is where I learned concurrency as a practical discipline rather than as an exam question.
  • Check mobile readiness against Google's PageSpeed Insights API, so that a site could be qualified as needing work before anyone picked up a phone.
  • Group by registrar or administrative organization, and pull every record for one of them, because discovering that a single web shop administers 300 local domains changes who it is that you should be calling.

A header was put on every script that read "Created by Aaron Trank, for use by Jonesen LLC only," and that sums the arrangement up. I wrote the tooling and they ran the business on it.