Parallel board crawlers
Naukri and LinkedIn workers run side by side toward one fresh-job target — selected keywords, AI top-up, or full auto mode.
Automation · 2026
Job Crawler
Parallel Naukri + LinkedIn crawler with live browser view, freshness filters, and keyword-driven discovery
PlaywrightReactMongoDBDockerSSENaukriLinkedIn
Overview
I built Job Crawler because manually refreshing Naukri and LinkedIn burns hours and still leaves you staring at stale listings. The job I wanted done: find fresh, relevant openings fast — then decide what to apply to without drowning in noise.
The stack is a Docker monorepo: a React dashboard, a REST API with SSE crawl logs, Playwright workers for Naukri and LinkedIn, and MongoDB for storage. Sessions persist after browser login so crawls can run with authenticated browsing — including a VNC-style view of the browser navigating job pages in real time.
The pipeline is deliberate: discover by keyword (selected, AI top-up, or full auto), filter freshness, validate open listings, score and tier matches against a resume profile, then export to Excel. Naukri and LinkedIn run in parallel toward a shared fresh-job target so progress is visible run by run.
This project lives on GitHub — clone the repo and run it locally with Docker. There is no public hosted demo.
Highlights
- Keyword targets per board with manual add and AI-assisted top-up
- Live crawl progress toward a fresh-job run target (e.g. 8 / 20)
- Browser login session persistence for Naukri and LinkedIn
- Job table with search, session filters, score, tier, and status
- Live crawl logs with timestamps and per-source tags
- Excel export for filtered or full crawl results
Technical depth
What I engineered
Systems and patterns behind Job Crawler— the parts that don't show in a screenshot.
Live browser + SSE logs
Watch Playwright navigate listings in real time while the API streams crawl events over SSE into a terminal-style log view.
Freshness → score → export
Skip aged postings, validate open listings, tier matches against a resume profile, then export filtered results to Excel.
Docker monorepo
Web, API, worker, and MongoDB compose up together — session login flows for both boards, cron-ready worker, and a single dashboard.
Walkthrough
Inside the build
Key screens from Job Crawler — what I built and why it mattered.
01
Dashboard crawler
Search settings, keyword panels for Naukri and LinkedIn, crawl mode (selected + AI top-up), and live counters — fresh jobs toward target, fetched totals, and recent crawl runs.
02
Live crawler
Session indicators, cancel controls, and a live browser pane so I can watch the worker navigate search results while metrics update in parallel.
03
Searched jobs by keyword
After a crawl, every listing lands in a filterable table — title, company, posted age, score, tier, status, and a direct open link — so I can triage fresh matches by keyword and session.
04
Live logs
SSE-backed log stream with timestamps and source tags — search URLs, listing counts, and crawl steps as they happen, without refreshing the page.