For my senior capstone class, I was a part of a 4-person team which decided to make a product that attempted to improve the job searching process. The "Resume Matcher" would take a resume, scan it, and compare the details to jobs on platforms like Linkedin or Indeed and find the best matching jobs for that resume. There aren't any tools on the market that provide this sort of automation, so this was an interesting product to explore.
There were multiple pieces that had to come together to make the matcher, and I was responsible for the part that collected the job information from the internet. Web scraping is not generalizable across different webpages, so I only ended up having time to develop the scraper specific to the Linkedin jobs page. At a high level, it was designed to take any sort of parameters in regards to "job title" and "location" and from the resulting page go through each card and collect the data from it.
The speed of the scraper itself is an interesting problem on its own. Thousands of jobs are uploaded to Linkedin every hour, so ideally we would want the scraper to keep up with this number or surpass it. Unfortunately due to various constraints, some outlined in the report extract below, this speed is not feasible from one single scraper instance. This is because one single scraper instance performs at best around 1000 jobs per hour (i7-12700k cpu). Again, there are multiple reasons for this, but the main bottleneck is on the Linkedin side, where data requests quickly become throttled. One idea to get around this that I had would be to utilize the built-in search parameters that Linkedin offers to run multiple scrapers at the same time for different locations or different job catagories.
That idea was never implemented though and this project was discontinued after the semester ended. If it was available it would be accessible at http://resume-matcher-mvp.herokuapp.com/. Despite this, the web scraper itself can be ran as a stand-alone program and its source code is provided below.
Product Demo
Linkedin Web Scraper Full Implementation Details - below are implementation details taken as an extract from the report for this project and the source code for the web scraper.