Full-Stack · Agentic AI · LangGraph

Building Job Agent — an AI-powered job application system with human-in-the-loop control

How I built a full-stack AI system that automates job searching, scoring, resume generation, and submission — while keeping a human in control of every critical decision.

Trilochan Sharma

2024

~18 min read

First-year CS @ KU

LangGraphFastAPI Next.js 14PostgreSQL pgvectorRedis CeleryDPO/TRL PlaywrightWebSocket

The Problem
Two-Gate Pipeline
Why Not a Script?
Why LangGraph
Database Design
LLM Layer
Scraping Layer
Frontend
Security
RL / DPO Layer
Task Queue
MCP Server
Retrospect
Numbers
What's Next

01 — Origin

The Problem That Started It All

Job searching is one of the most time-consuming, demoralizing, and repetitive tasks a developer faces. You open LinkedIn. You scroll through 40 listings. 30 of them are irrelevant — wrong stack, wrong level, wrong location. You open the 10 relevant ones. You copy-paste your resume. You tweak the summary. You submit. You do this again tomorrow.

I was doing this for weeks. The process wasn't hard — it was just exhausting. Every application looked the same but required slightly different tailoring. Every job board had a different interface. Every ATS had a different form.

And the worst part: the boring parts were taking all the time. The interesting part — actually deciding whether a job was worth applying to — was maybe 5% of the effort. The other 95% was mechanical repetition.

What if an AI could handle the 95%, and I only dealt with the 5%? Not fully automated — but assisted. The AI does the legwork. I make the decisions. The AI does the paperwork. I review and confirm.

That became the core philosophy of Job Agent: amplify human judgment, don't replace it.

02 — Design

The Core Idea: A Pipeline with Two Human Gates

Before writing a single line of code, I sketched out what the system actually needed to do:

Scrape

50 jobs / day

→

Score 0–100

rubric + profile

⏸

Gate 1

You Review

approve / reject

→

Write CV

tailored per role

⏸

Gate 2

You Approve

review + edit

→

Submit

Playwright ATS

Two gates. Two places where nothing moves forward without my explicit approval. This was non-negotiable from the start.

Why two gates and not one?

"Is this job worth applying to?" — The AI gives a score and reasoning, but I know things it doesn't: company culture, whether the role is a step back, whether I've heard bad things about them.

"Is this resume good enough to send?" — An AI-generated resume might over-emphasise the wrong things. I should always have a chance to fix that before it goes out.

Gate 1 prevents wasted effort on bad matches. Gate 2 prevents embarrassing or inaccurate applications. Together they meant I could trust the system.

03 — Architecture

Choosing the Architecture: Why Not Just a Script?

The first temptation was to write a Python script. scrape_jobs() → score_jobs() → generate_resumes() → submit_applications(). Simple, linear, done.

But scripts don't handle the human gates well. A script that pauses and waits for user input is either blocking (bad for a server) or requires polling state from a file/database (fragile). I needed something that could:

Run asynchronously — scraping 50 jobs takes minutes. I shouldn't block a web server waiting for that.

Pause at defined points — wait for human approval without losing state.

Resume after interruption — if the server restarts mid-run, the agent shouldn't lose its progress.

Be inspectable — I should be able to see what step the agent is on at any time.

This pointed clearly to a stateful multi-agent framework with checkpointing.

04 — Framework

Why LangGraph?

I evaluated three options:

Option A

Raw Celery + Redis

I'd be reimplementing a state machine from scratch — checkpoint serialization, interrupt handling, graph traversal. Lots of boilerplate for something with well-tested libraries.

Option B

LangChain Agents

Great for tool-calling loops, but not designed for long-running pipelines with explicit human interrupt points. Agents assume they drive the flow — I wanted an explicit, defined graph.

✓ Winner

LangGraph

Built for stateful multi-step workflows. TypedDict state, interrupt_before nodes, MemorySaver checkpointing. Flow is a readable graph — not hidden in agent reasoning.

The interrupt_before parameter is where the magic happens — LangGraph automatically pauses before those nodes and waits for the graph to be resumed via an API call. State is fully persisted. You could stop the server, go on holiday, come back a week later, and the agent would resume exactly where it left off.

graph.add_node("scrape_jobs", scraper_node)
graph.add_node("score_jobs", scorer_node)
graph.add_node("human_job_review", _human_job_review_node)
graph.add_node("generate_resumes", resume_node)
graph.add_node("human_resume_review", _human_resume_review_node)
graph.add_node("submit_applications", application_node)

graph.set_entry_point("scrape_jobs")

# interrupt_before pauses here — resumes via API call
return graph.compile(
    checkpointer=MemorySaver(),
    interrupt_before=["human_job_review", "human_resume_review"],
)

05 — Data Layer

The Database Design: PostgreSQL + pgvector

Separating raw jobs from scored jobs

One key architectural decision: store raw jobs and scored jobs in separate tables. Raw jobs are immutable — they represent what was actually posted. Scored jobs are derived — they represent the system's interpretation per user.

-- raw_jobs: scraped data, immutable, deduplicated by SHA-256 fingerprint
raw_jobs    → (title, company, description, url, fingerprint, created_at)

-- scored_jobs: AI evaluation per user — can be recomputed if model improves
scored_jobs → (score, reasoning, matched_skills, missing_skills, status, user_id)

This separation means if I improve the scoring model later, I can re-score existing raw jobs without losing any original data. It also supports multi-user scoring from the same raw listings.

Why pgvector?

Semantic deduplication: Two job listings might have identical content but different URLs (different ATS mirrors). SHA-256 fingerprinting misses these. Embedding descriptions and comparing cosine similarity catches semantic duplicates.

Future semantic search: Instead of keyword matching ("Python developer"), you could search by embedding ("I want something like my current role but more leadership-focused") and retrieve semantically similar jobs.

The 11-table schema

users

Authentication

user_profiles

Job preferences, skills, resume, search defaults

qa_bank

Answers to common application questions (salary, visa, etc.)

raw_jobs

Scraped listings — immutable, deduplicated by fingerprint

scored_jobs

AI assessments scoped per user

job_embeddings

pgvector embeddings for semantic search/dedup

applications

Pipeline: draft → submitted → outcome

application_outcomes

Interview / rejection / offer tracking

resume_versions

Tailored resumes generated per job

resume_preferences

Chosen/rejected pairs — DPO training data

agent_runs

Pipeline execution history and current status

The resume_preferences table is the training data collection mechanism for the RL layer. Every time I edit a resume or rate one, a row gets inserted. These rows become the DPO fine-tuning dataset.

06 — AI Layer

The LLM Layer: Claude and Gemini

The abstraction

I built a thin LLM abstraction so every part of the system makes a single function call and remains provider-agnostic. The provider selection happens at the user level — stored in user_profiles.preferred_llm_provider.

async def get_llm_response(
    prompt: str,
    provider: Optional[str] = None,  # "claude" | "gemini"
    model: Optional[str] = None,
    max_tokens: int = 1024,
    json_mode: bool = False,
) -> str:
    provider = provider or settings.DEFAULT_LLM_PROVIDER
    if provider == "claude":
        return await _call_claude(prompt, model, max_tokens)
    elif provider == "gemini":
        return await _call_gemini(prompt, model, max_tokens)

Claude vs. Gemini — what I found

Tested both Claude Sonnet 4.6 and Gemini 2.0 Flash for job scoring. Both produced good scores, but Claude's reasoning was consistently more nuanced. When a job was 72/100, Claude would explain why — "Strong Python match, but missing the MLOps experience they emphasise in the second half." Gemini would score similarly but with less specific reasoning.

For resume generation, both were excellent. Gemini is noticeably faster and cheaper — matters when generating 10 resumes in a batch. The default is Claude for quality-critical tasks; Gemini is the right choice at higher volume.

Prompt engineering lessons

Show the exact JSON schema — not "return JSON with fields X, Y, Z." Showing exact key names and types reduced parsing failures significantly.

A scoring rubric anchors the model — "90–100 = perfect match, 75–89 = strong…" reduced score variance between runs dramatically.

Explicit negatives matter. "Do NOT invent experiences or skills" is in the resume prompt in bold. Without it, Claude occasionally extrapolates a skill "in passing" that wasn't in the original resume.

07 — Scraping

The Scraping Layer: JobSpy + Playwright

JobSpy for the heavy lifting

JobSpy is an open-source library providing a unified interface to LinkedIn, Indeed, Glassdoor, ZipRecruiter, and Google Jobs. It handles auth flows, pagination, rate limiting, and field normalisation. Since JobSpy is synchronous, I wrap it in a thread pool executor:

async def scrape_jobs(search_query, location, sites, results_wanted):
    loop = asyncio.get_event_loop()
    jobs = await loop.run_in_executor(None, _scrape_sync, search_query, location, sites)
    return jobs

Playwright for dynamic sites

Some company career pages aren't covered by JobSpy. For those, I use Playwright with playwright-stealth — patches browser automation fingerprints that bot-detection systems look for: randomising navigator.webdriver, spoofing navigator.plugins, and setting realistic user agent strings.

Two-layer deduplication

Layer 1 — URL fingerprinting: SHA-256 hash of the job URL stored as raw_jobs.fingerprint. O(1) lookup. Catches the vast majority of duplicates on every scrape cycle.

Layer 2 — Semantic similarity: For jobs with different URLs but identical content (different ATS mirrors), we compare pgvector description embeddings. cosine similarity > 0.95 → treated as duplicate.

08 — Frontend

The Frontend: Next.js 14 App Router

State management strategy

React Query (TanStack Query) for server state — API calls, caching, background refetching. Auto-caches results, deduplicates in-flight requests, and invalidates on mutation.

Zustand for global client state — auth token, selected job ID, agent WebSocket status. Minimal API with no reducers or actions needed.

Real-time agent status via WebSocket

Initial implementation polled every 5 seconds. Final implementation uses WebSocket connected to Redis pub/sub — Celery worker publishes to agent:events:{'{user_id}'}, FastAPI WebSocket forwards to the browser, React renders live status:

// Try WebSocket first
ws.onerror = () => setWsConnected(false);

// React Query only polls when WebSocket is NOT connected
refetchInterval: wsConnected ? false : 5000,

Falls back gracefully to 5-second polling if the WebSocket drops. A small indicator in the UI shows which mode is currently active.

09 — Security

Security: What Matters and Why

bcrypt password hashing — the only acceptable choice. bcrypt's cost factor makes brute-force attacks prohibitively slow.

JWT access + refresh tokens — access tokens expire in 60 minutes, refresh tokens in 7 days. Limits the damage window if an access token is stolen from a log.

Security headers middleware — X-Content-Type-Options: nosniff, X-Frame-Options: DENY, X-XSS-Protection, Strict-Transport-Security.

Rate limiting on login — 10 requests/minute per IP via slowapi. Prevents brute-force password attacks at the edge.

API docs disabled in production — Swagger and ReDoc expose your full API surface. One ENVIRONMENT=production env var disables them entirely.

CORS restricted — only the configured frontend origin can make cross-origin requests. Prevents other websites from making authenticated API calls using a user's cookies.

Things I deliberately left simple for the MVP: no email verification, no 2FA, no token revocation. Noted as future work, not oversights.

10 — Learning Loop

The RL/DPO Layer: Teaching the AI Your Preferences

This was the most intellectually interesting part of the project.

Claude generates good resumes. But "good" is subjective. The first few generated resumes might be 80% right. After I edit them, the AI has learned something about my preferences. Over time, generated resumes should need less editing.

How DPO works conceptually

Instead of training a separate reward model (full RLHF pipeline), DPO trains the language model directly on preference pairs. The training objective maximises the probability of generating the chosen response relative to the rejected one. Over enough pairs, the model learns to generate text more like what you prefer.

Automatic preference collection

Edit signal — When you save a modified resume, the system automatically records (edited_version, original_version) as a preference pair with signal_type="edit". No extra click.

Outcome signal — When an application gets an interview, the resume used gets marked as preferred. An interview implies the resume was effective.

Explicit rating — The UI has a thumbs up/down for deliberate feedback when you want to be more intentional.

The training pipeline

training_args = DPOConfig(
    beta=0.1,        # KL penalty — controls drift from base model
    max_length=1024,
    num_train_epochs=3,
)
trainer = DPOTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,   # [(prompt, chosen, rejected), ...]
    tokenizer=tokenizer,
)
trainer.train()

The beta parameter controls how aggressively the model adapts vs. staying close to base behaviour. Too high → overfit on 50 examples. Too low → training has no effect. Base model: google/flan-t5-base (~250MB), runs on CPU. Training requires 50+ pairs for statistical stability.

11 — Infrastructure

The Task Queue: Celery + Redis

The agent pipeline takes 5–30 minutes to run. Running this inside a FastAPI request handler would hold the HTTP connection open for that entire time. Celery solves this: the API endpoint enqueues a task and immediately returns a task_id, the worker runs the pipeline in the background, and the frontend streams updates via WebSocket.

@celery_app.task(bind=True)
def run_full_agent(self, user_id: str, scrape_config: dict):
    return _run_async(_run_full_agent_async(user_id, self.request.id, scrape_config))

def _run_async(coro):
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    try:
        return loop.run_until_complete(coro)
    finally:
        loop.close()

On Windows, --pool=solo is required because Windows doesn't support the fork process model. On Linux/macOS, --pool=prefork -c 4 gives parallel task execution.

12 — Composability

The MCP Server

The MCP (Model Context Protocol) server exposes the agent's tools in a standard format that any MCP-compatible AI client can use. You could ask Claude Desktop "what jobs did you find for me today?" and it would call the MCP tool, or build a VS Code extension that surfaces relevant jobs while you're coding.

@mcp.tool()
async def scrape_jobs(search_query: str, location: str, sites: list[str]) -> dict:
    """Scrape job listings from multiple job boards."""
    ...

The MCP server is optional — the main system works fine without it. But it makes the tool composable with the broader AI ecosystem, which matters for an open-source project.

13 — Reflection

What I'd Do Differently

Proud of

The two-gate architecture — a graph interrupt, not a flag. Structurally impossible to bypass.

LLM abstraction — switching Claude → Gemini is one line in .env.

Real-time pipeline: Celery → Redis → WebSocket with graceful polling fallback.

Unified setup — one .env.example, one requirements.txt, zero subdirectory installs.

Would change

AsyncPostgresSaver instead of MemorySaver — agent state should survive server restarts.

Integration tests from day one — pytest-asyncio against a test DB. Refactoring without tests is scary.

Better TypeScript types — replaced any casts with interfaces generated from Pydantic schemas.

Feature flag for RL — ENABLE_RL_TRAINING=false env var to cleanly disable DPO.

14 — Scope

The Numbers

100

source files

~50

Python files

~20

TSX files

DB tables

25+

API endpoints

9.3k

lines of code

build days

15 — Roadmap

What's Next

Workday / Greenhouse / Lever ATS support — Most companies use proprietary ATS with complex multi-page forms. The most impactful and hardest next feature.

Outcome-driven scoring — Record which jobs led to interviews and retrain the scorer to weight the factors that actually predict success.

Browser extension — "Save this job" button that adds any listing to the queue with one click, bypassing the scraping problem entirely.

Email notifications — Instant alert when a high-scoring (90+) job appears, rather than waiting to check the dashboard.

Conclusion

Job Agent started as a solution to a personal frustration and became a full-stack AI system I'm genuinely proud of. The architecture reflects a real design philosophy — not "use AI to do everything" but "use AI to do the mechanical work while keeping humans in control of the judgment calls."

What surprised me most wasn't the technical complexity — it was how much the constraints shaped the architecture. "Nothing submits without approval" forced the two-gate design → forced LangGraph checkpointing → enabled real-time WebSocket status → made the dashboard actually useful. Good constraints produce good architecture.

If you're building something similar: design your human gates first. Every architectural decision downstream flows from "where does the human decide, and where does the machine act?" Get that right, and the rest follows.

JOB AGENT