Atrium Job Scraper

A modular, polite, production-oriented scraper that collects publicly available internship/job listings for Atrium's student community.

⚠️ Read this first: a note on live-site verification

This project was built in a sandboxed environment with no network access to linkedin.com, indeed.com, naukri.com, or internshala.com — only package registries (PyPI/npm/GitHub) were reachable. That means:

• The architecture, pipeline, tests, error handling, and CLI are fully built and verified end-to-end using saved HTML fixtures (tests/fixtures/*.html) that mirror each site's real DOM structure.
• Internshala is the fully working reference implementation — its selectors are based on Internshala's known server-rendered markup and pass parsing tests against a realistic fixture.
• Indeed, Naukri, and LinkedIn are implemented with the same architecture and best-effort selectors, but you must run them against the live sites yourself (from a machine with normal internet access) and adjust selectors in scraper/sources/<name>.py if a site has changed its markup since this was written. Each source file isolates its selectors in one or two small methods specifically so this kind of update is quick and low-risk.
• Run python main.py --source internshala first, then check output/*.json and logs/scraper.log. If Internshala's HTML has drifted, the log will show 0 jobs found on page 1 — that's your signal to inspect the live page and update the selectors in scraper/sources/internshala.py.

This is standard practice for any scraper anyway: sites change their markup periodically, so selector maintenance is an ongoing task, not a one-time build step, regardless of who wrote the original code.

Architecture

scraper/
  sources/          # one file per site, isolated selectors
    base.py          # BaseSource ABC — pagination loop, error isolation
    internshala.py    # requests + BeautifulSoup (fully working reference)
    naukri.py          # requests, JSON-payload parsing + DOM fallback
    indeed.py            # Selenium (site requires JS / blocks plain requests)
    linkedin.py            # requests, guest endpoint only (see limitations)
  core/
    scraper.py       # orchestrator: runs sources, isolates failures
    normalizer.py     # shared text/date/skill cleanup helpers
    filters.py         # KEYWORDS / LOCATION / REMOTE_ONLY / etc.
    deduplicator.py     # stable UID-based dedup
    output.py            # JSON + CSV writers, Postgres stub
  models/
    job.py           # the Job dataclass every source normalizes into
  utils/
    config.py        # .env -> typed Config object
    logger.py          # console + rotating file logging
    http.py             # retries, backoff, rate limiting, robots.txt
    browser.py           # reusable headless Selenium manager
main.py              # CLI entrypoint
tests/               # pytest suite + HTML fixtures (no live network needed)

Data flow: main.py → core.scraper.run_sources() → for each source: BaseSource.scrape() paginates, calls the source's parse_listing_page() + normalize() → all Job objects are merged → Deduplicator removes duplicates → filters.apply_filters() applies your .env filters → core.output writes JSON + CSV.

requests vs. Selenium — which sources use which, and why

Setup & Installation

# 1. Install via pip
pip install atrium-job-scraper

# 2. Or, if installing from source:
git clone https://github.com/devkunal2812/Atrium-Scraper.git
cd Atrium-Scraper
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# 3. Configuration
cp .env.example .env
# Edit .env to set your KEYWORDS, LOCATION, MAX_PAGES, etc.

Selenium 4.6+ uses Selenium Manager, which auto-downloads the correct ChromeDriver the first time it runs — no manual driver setup needed. You do need Google Chrome (or Chromium) installed on the machine running the Indeed source.

Configuration (.env)

KEYWORDS=react,node.js,python,java   # matched against title/description/skills
SKILLS=                              # optional, narrower than KEYWORDS
LOCATION=india                       # substring match
REMOTE_ONLY=false
INTERNSHIP_ONLY=false
MAX_PAGES=5

(All config settings can be overridden per-run via CLI flags).

Usage Guide

The package provides multiple ways to run the scraper depending on your workflow:

1. Interactive Terminal (TUI) + Web UI 🌟

Run the following command to start both the modern Interactive Terminal Interface and the Web UI in the background:

atrium-scraper

• Interactive Terminal: An opencode-style interface will appear in your terminal. You can run commands like /scrape naukri, /scrape all, or even standard shell commands (e.g., !ls).
• Web UI: Automatically launches at http://localhost:8000. You can monitor scrape progress and results via the web dashboard simultaneously!

2. Standalone Web UI

If you only want to start the Web UI (and access the API/dashboard without the interactive terminal):

uvicorn app:app --host 0.0.0.0 --port 8000

3. Traditional CLI

If you prefer standard CLI arguments (great for cron jobs or CI/CD pipelines), use atrium-cli:

# Single source
atrium-cli --source internshala
atrium-cli --source linkedin

# All sources at once
atrium-cli --source all

# Override filters for one run
atrium-cli --source all --keyword "frontend developer" --location "india"
atrium-cli --source internshala --max-pages 5 --remote-only

Output goes to output/jobs_<timestamp>.json and .csv. Logs go to logs/scraper.log and the console.

Running the tests

python -m pytest -q

44 tests cover: normalization helpers (dates, remote-status detection, skill parsing), the Job UID/dedup logic, filter combinations, and HTML parsing for Internshala and Naukri against saved fixtures — so the suite runs fully offline and doesn't depend on live sites being up or unchanged.

Deduplication

Each Job computes a stable UID via Job.compute_uid():

1. source + source_job_id if the site exposes a real job ID (best).
2. source + application_url if no ID but a canonical URL exists.
3. sha256(source + title + company + location) as a last resort.

The Deduplicator runs across the combined results of all sources in one invocation, so --source all won't double-count a job that happens to be posted identically on two sites under different IDs (it will still count it once per source if the title/company/location differ even slightly — true cross-source identity matching would need fuzzy text matching, which isn't implemented).

Error handling & rate limiting

• Every page fetch and every per-job normalization is wrapped so one bad URL or malformed card never kills the whole run — it's logged and skipped.
• HttpClient retries transient failures (timeouts, 5xx, 429) with exponential backoff, respects robots.txt by default, and treats anything that looks like a CAPTCHA/bot-block page as a failure to skip, never something to "solve."
• A configurable delay is enforced between every request (REQUEST_DELAY_SECONDS).

Connecting to Postgres later

scraper/core/output.py has a save_to_postgres() stub with the intended implementation shape documented in its docstring. Because every source already normalizes into the same Job dataclass and job_uid is a natural primary key, wiring up a real DB (SQLAlchemy/psycopg2, an INSERT ... ON CONFLICT (job_uid) DO NOTHING) is a small, isolated change — no scraping code needs to change.

Known limitations (be upfront about these with stakeholders)

• Selector drift: sites redesign their frontends periodically. Expect to revisit selectors every so often, especially for Naukri/Indeed/LinkedIn. This is normal maintenance for any scraper.
• LinkedIn descriptions are unavailable without logging in, which this scraper deliberately does not do.
• Indeed depends on a headless browser succeeding past its bot-check; on a machine/IP with a poor reputation, more pages may come back blocked. This is expected and by design (we don't try to force through).
• Location filtering is a plain substring match, not geocoding — see the comment in scraper/core/filters.py.
• Job descriptions/skills from Internshala's listing page are limited to what the card shows; getting the full description would require an additional fetch per job to its detail page (not done by default to keep request volume low — can be added as an optional --deep flag if Atrium wants it).