Analyzing Who's Hiring Data on HackerNews with Interactive Python

- 3 mins

HackerNews has a monthly thread where employers can post job listings. Interactive Mode in Visual Studio is something I recently discovered that puts a notebook inside Visual Studio. Here, I’m going to put those two things together to use Interactive Mode to lightly analyze job postings on HN.

The Python Notebook in git

Some questions we will be able to answer with this analysis…

Interactive Mode in Visual Studio
Interactive Mode in Visual Studio is a notebook built into your IDE. A line with
# %%
creates a new cell for operations. That is it. Each # %% is a cell that can be run individually or put into debug mode.

Rough requirements for this analysis…


General process for this notebook

  1. Collect every HN Who’s Hiring posts. I’m using Google to query for what we hope is the correct HN link, take the first result & verify if it’s true. Selenium is used for the scraping.
  2. For every post on HN, get every item in that post. This is a time consuming GET request for every item, for those items not already loaded.
  3. Parse collected data into a dict.
  4. Save dict to a file on disk.
  5. Demormalize data into a pandas dataframe.
  6. Analyze data.

Notes for this notebook



rss facebook twitter github gitlab youtube mail spotify lastfm instagram linkedin google google-plus pinterest medium vimeo stackoverflow reddit quora quora