How to build a native SEO audit agent with browser usage and Cloud API

Every digital marketing agency has someone whose job involves opening a spreadsheet, looking at each client’s URL, checking the title tag, meta description, and H1, noting broken links, and pasting everything into a report. Then do it again next week.

That work is decisive. An agent can do this.

In this tutorial, you’ll build a native SEO audit agent from scratch using Python, Browser Usage, and the Claude API. The agent visits real pages in a visible browser window, extracts SEO signals using Claude, periodically checks for broken links, handles edge cases with human-in-the-loop pauses, and writes a structured report—which can be restarted if interrupted.

By the end, you’ll have a working agent that you can run against any list of URLs. It costs less than $0.01 per URL to run.

What will you make?

A seven-module Python agent that:

• Reads a list of URLs from a CSV file.

• Visits every URL in a real Chromium browser (not a headless scraper)

• Claude extracts title, meta description, H1s, and canonical tags via API.

• Checks for broken links asynchronously using httpx.

• Detects edge cases (404s, login walls, redirects) and pauses for human input

• Writes results to report.json Gradual – It is safe to stop and restart.

• Produces a plain English. report-summary.txt Upon completion

The full code is on GitHub. dannwaneri/seo-agent.

Conditions

Table of Contents

1. Why use a browser instead of a scraper?

2. Project structure

3. Setup

4. Module 1: State Management

5. Module 2: Browser Integration

6. Module 3: Cloud Extraction Layer

7. Module 4: Broken Link Checker

8. Module 5: Human in the Loop

9. Module 6: Report Writer

10. Module 7: Main Loop

11. Run the agent

12. Scheduling for agency use

13. What the results look like.

Why use a browser instead of scraping?

The standard approach to SEO auditing is to bring the page HTML with it. requests And pars it with beautiful soup. It works on static pages. It breaks on content served by JavaScript, misses dynamically injected meta tags, and fails completely on authenticated pages.

Using a browser (84,000+ GitHub stars, MIT license) takes a different approach. It controls a real Chromium browser, reads the DOM after the JavaScript has completed, and exposes the page through the PlayWrite access tree. The agent sees what a human would see.

Practical difference: A request-based scraper may miss the meta description that is inserted by the React component. The browser will not use

Another difference worth mentioning: Using a browser reads pages verbatim. A PlayWrite script breaks when the button’s CSS class changes. btn-primary To button-main. Browser usage recognizes that this is still a “Submit” button and acts accordingly. The extraction logic resides in the cloud prompt, not in the breakable CSS selectors.

Project structure

seo-agent/
├── index.py          # Main audit loop
├── browser.py        # Browser Use / Playwright page driver
├── extractor.py      # Claude API extraction layer
├── linkchecker.py    # Async broken link checker
├── hitl.py           # Human-in-the-loop pause logic
├── reporter.py       # Report writer
├── state.py          # State persistence (resume on interrupt)
├── input.csv         # Your URL list
├── requirements.txt
├── .env.example
└── .gitignore

Setup

Create a project folder and install the dependencies:

mkdir seo-agent && cd seo-agent
pip install browser-use anthropic playwright httpx
playwright install chromium

make input.csv With your URLs:

url

/about
/contact

make .env.example:

ANTHROPIC_API_KEY=your-key-here

Set your API key as an environment variable before running:

# macOS/Linux
export ANTHROPIC_API_KEY="sk-ant-..."

# Windows PowerShell
$env:ANTHROPIC_API_KEY = "sk-ant-..."

make .gitignore:

state.json
report.json
report-summary.txt
.env
__pycache__/
*.pyc

Module 1: State Management

The agent needs to track which URLs it has already audited. If the run is interrupted — a power cut, a keyboard interrupt, a network error — it must resume where it stopped, not restart.

state.py Handles this with a flat JSON file:

import json
import os

STATE_FILE = os.path.join(os.path.dirname(__file__), "state.json")

_DEFAULT_STATE = {"audited": (), "pending": (), "needs_human": ()}


def load_state() -> dict:
    if not os.path.exists(STATE_FILE):
        save_state(_DEFAULT_STATE.copy())
    with open(STATE_FILE, encoding="utf-8") as f:
        return json.load(f)


def save_state(state: dict) -> None:
    with open(STATE_FILE, "w", encoding="utf-8") as f:
        json.dump(state, f, indent=2)


def is_audited(url: str) -> bool:
    return url in load_state()("audited")


def mark_audited(url: str) -> None:
    state = load_state()
    if url not in state("audited"):
        state("audited").append(url)
    save_state(state)


def add_to_needs_human(url: str) -> None:
    state = load_state()
    if url not in state("needs_human"):
        state("needs_human").append(url)
    save_state(state)

Design is intentional: mark_audited() Called immediately after the URL is processed and written to the report. If the agent crashes mid-run, it loses at most one URL function.

Module 2: Browser Integration

browser.py The main page does the navigation. It directly uses PlayWrite (which the browser installs as a dependency) to open a visible Chromium window, navigate to URLs, capture HTTP status and redirect information, and extract raw SEO signals from the DOM.

Key Design Decisions:

A visible browser, not a headless one. Set headless=False So you can see the agent’s work. This is important for demos and debugging.

Status capture via response listener. The playwright took exception to the 4xx/5xx responses, but on("response", ...) The handler fires before the exception. We occupy the position there.

2 second delay between trips. Prevents rate limiting or bot detection on agency client sites.

Here is the basic navigation function:

import asyncio
import sys
import time
from playwright.sync_api import sync_playwright, TimeoutError as PlaywrightTimeout

TIMEOUT = 20_000  # 20 seconds


def fetch_page(url: str) -> dict:
    result = {
        "final_url": url,
        "status_code": None,
        "title": None,
        "meta_description": None,
        "h1s": (),
        "canonical": None,
        "raw_links": (),
    }

    first_status = {"code": None}

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        page = browser.new_page()

        def on_response(response):
            if first_status("code") is None:
                first_status("code") = response.status

        page.on("response", on_response)

        try:
            page.goto(url, wait_until="domcontentloaded", timeout=TIMEOUT)
            result("status_code") = first_status("code") or 200
            result("final_url") = page.url

            # Extract SEO signals from DOM
            result("title") = page.title() or None
            result("meta_description") = page.evaluate(
                "() => { const m = document.querySelector('meta(name=\"description\")'); "
                "return m ? m.getAttribute('content') : null; }"
            )
            result("h1s") = page.evaluate(
                "() => Array.from(document.querySelectorAll('h1')).map(h => h.innerText.trim())"
            )
            result("canonical") = page.evaluate(
                "() => { const c = document.querySelector('link(rel=\"canonical\")'); "
                "return c ? c.getAttribute('href') : null; }"
            )
            result("raw_links") = page.evaluate(
                "() => Array.from(document.querySelectorAll('a(href)'))"
                ".map(a => a.href).filter(Boolean).slice(0, 100)"
            )

        except PlaywrightTimeout:
            result("status_code") = first_status("code") or 408
        except Exception as exc:
            print(f"(browser) Error: {exc}", file=sys.stderr)
            result("status_code") = first_status("code")
        finally:
            browser.close()

    time.sleep(2)
    return result

A few things are worth noting:

gave raw_links The cap at 100 is deliberate. DEV.to profile pages contain hundreds of links — you don’t need all of them to spot a broken link.

gave wait_until="domcontentloaded" The setting is faster networkidle And enough to extract the meta tag. Content rendered by JavaScript needs the DOM to be ready, not to complete all network requests.

extractor.py Takes a snapshot of the raw page from browser.py and calls Claude to deliver a systematic SEO audit result.

This is where most tutorials go wrong. They either write complex parsing logic in Python (delicate) or ask Claude for a free-form response and try to parse the prose (unreliable). The correct approach: give Claude a strict JSON schema and tell it to return nothing else.

Quick engineering that makes it reliable:

import json
import os
import sys
from datetime import datetime, timezone
import anthropic

MODEL = "claude-sonnet-4-20250514"
client = anthropic.Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))


def _strip_fences(text: str) -> str:
    """Remove accidental markdown code fences from Claude's response."""
    text = text.strip()
    if text.startswith("```"):
        lines = text.splitlines()
        # Drop opening fence
        lines = lines(1:) if lines(0).startswith("```") else lines
        # Drop closing fence
        if lines and lines(-1).strip() == "```":
            lines = lines(:-1)
        text = "\n".join(lines).strip()
    return text


def extract(snapshot: dict) -> dict:
    if not os.environ.get("ANTHROPIC_API_KEY"):
        raise OSError("ANTHROPIC_API_KEY is not set.")

    prompt = f"""You are an SEO auditor. Analyze this page snapshot and return ONLY a JSON object.
No prose. No explanation. No markdown fences. Raw JSON only.

Page data:
- URL: {snapshot.get('final_url')}
- Status code: {snapshot.get('status_code')}
- Title: {snapshot.get('title')}
- Meta description: {snapshot.get('meta_description')}
- H1 tags: {snapshot.get('h1s')}
- Canonical: {snapshot.get('canonical')}

Return this exact schema:
{{
  "url": "string",
  "final_url": "string",
  "status_code": number,
  "title": {{"value": "string or null", "length": number, "status": "PASS or FAIL"}},
  "description": {{"value": "string or null", "length": number, "status": "PASS or FAIL"}},
  "h1": {{"count": number, "value": "string or null", "status": "PASS or FAIL"}},
  "canonical": {{"value": "string or null", "status": "PASS or FAIL"}},
  "flags": ("array of strings describing specific issues"),
  "human_review": false,
  "audited_at": "ISO timestamp"
}}

PASS/FAIL rules:
- title: FAIL if null or length > 60 characters
- description: FAIL if null or length > 160 characters  
- h1: FAIL if count is 0 (missing) or count > 1 (multiple)
- canonical: FAIL if null
- flags: list every failing field with a clear description
- audited_at: use current UTC time in ISO 8601 format"""

    response = client.messages.create(
        model=MODEL,
        max_tokens=1000,
        messages=({"role": "user", "content": prompt}),
    )

    raw = response.content(0).text
    clean = _strip_fences(raw)

    try:
        return json.loads(clean)
    except json.JSONDecodeError as exc:
        print(f"(extractor) JSON parse error: {exc}", file=sys.stderr)
        return _error_result(snapshot, str(exc))


def _error_result(snapshot: dict, reason: str) -> dict:
    return {
        "url": snapshot.get("final_url", ""),
        "final_url": snapshot.get("final_url", ""),
        "status_code": snapshot.get("status_code"),
        "title": {"value": None, "length": 0, "status": "ERROR"},
        "description": {"value": None, "length": 0, "status": "ERROR"},
        "h1": {"count": 0, "value": None, "status": "ERROR"},
        "canonical": {"value": None, "status": "ERROR"},
        "flags": (f"Extraction error: {reason}"),
        "human_review": True,
        "audited_at": datetime.now(timezone.utc).isoformat(),
    }

Two things make it reliable in production:

first, _strip_fences() Handles the case where Claude wraps his response. ```json Despite saying no, the fence is infrequent with the sonnet and is constantly broken. json.loads() If you don’t handle it.

Second, _error_result() Fallback means the agent never crashes on a bad cloud response—it logs the error and marks the URL for human review, then continues to the next URL.

Cost: Claude Sonnet 4 costs \(3 per million input tokens and \)15 per million output tokens. A typical page snapshot is around 500 input tokens. The generated JSON response is about 300 output tokens. This works out to about \(0.006 per URL — about \)0.12 for a 20-URL audit.

Module 4: Broken Link Checker

linkchecker.py takes raw_links Lists from a browser snapshot and checks single domain links for broken status using async HEAD requests.

Design Choices:

• Only one domain. Checking every external link on a page will take minutes and agency clients don’t need it. Filter links on the same domain as the page being audited.

• Head requests, not GETs. Fast, low bandwidth, sufficient for status code detection.

• Cap at 50 links. Pages like the DEV.to article list have hundreds of internal links. Checking all of these will dominate the runtime.

• Synchronize requests via asyncio. All links are checked in parallel, not sequentially.

import asyncio
import logging
from urllib.parse import urlparse
import httpx

CAP = 50
TIMEOUT = 5.0
logger = logging.getLogger(__name__)


def _same_domain(link: str, final_url: str) -> bool:
    if not link:
        return False
    lower = link.strip().lower()
    if lower.startswith(("#", "mailto:", "javascript:", "tel:", "data:")):
        return False
    try:
        page_host = urlparse(final_url).netloc.lower()
        parsed = urlparse(link)
        return parsed.scheme in ("http", "https") and parsed.netloc.lower() == page_host
    except Exception:
        return False


async def _check_link(client: httpx.AsyncClient, url: str) -> tuple(str, bool):
    try:
        resp = await client.head(url, follow_redirects=True, timeout=TIMEOUT)
        return url, resp.status_code != 200
    except Exception:
        return url, True  # Timeout or connection error = broken


async def _run_checks(links: list(str)) -> list(str):
    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(*(_check_link(client, url) for url in links))
    return (url for url, broken in results if broken)


def check_links(raw_links: list(str), final_url: str) -> dict:
    same_domain = (l for l in raw_links if _same_domain(l, final_url))

    capped = len(same_domain) > CAP
    if capped:
        logger.warning("Page has %d same-domain links — capping at %d.", len(same_domain), CAP)
        same_domain = same_domain(:CAP)

    broken = asyncio.run(_run_checks(same_domain))

    return {
        "broken": broken,
        "count": len(broken),
        "status": "FAIL" if broken else "PASS",
        "capped": capped,
    }

Module 5: Human in the Loop

This is the part that most automation tutorials skip. What happens when an agent hits a login wall? A page that returns a 403? A URL that redirects to a “Subscribe to Continue Reading” page?

Most scripts either crash or quit silently. Neither is acceptable in the context of the agency.

hitl.py handles this with two functions: one that detects whether a pause is needed, and another that handles the pause itself.

from state import add_to_needs_human

LOGIN_KEYWORDS = {"login", "sign in", "sign-in", "access denied", "log in", "unauthorized"}
REDIRECT_CODES = {301, 302, 307, 308}


def should_pause(snapshot: dict) -> bool:
    code = snapshot.get("status_code")

    # Navigation failed entirely
    if code is None:
        return True

    # Non-200, non-redirect
    if code != 200 and code not in REDIRECT_CODES:
        return True

    # Login wall detection
    title = (snapshot.get("title") or "").lower()
    h1s = (h.lower() for h in (snapshot.get("h1s") or ()))

    if any(kw in title for kw in LOGIN_KEYWORDS):
        return True
    if any(kw in h1 for kw in LOGIN_KEYWORDS for h1 in h1s):
        return True

    return False


def pause_reason(snapshot: dict) -> str:
    code = snapshot.get("status_code")
    if code is None:
        return "Navigation failed (None status)"
    if code != 200 and code not in REDIRECT_CODES:
        return f"Unexpected status code: {code}"
    return "Possible login wall detected"


def pause_and_prompt(url: str, reason: str) -> str:
    print(f"\n⚠️  HUMAN REVIEW NEEDED")
    print(f"   URL:    {url}")
    print(f"   Reason: {reason}")
    print(f"   Options: (s) skip  (r) retry  (q) quit\n")

    while True:
        choice = input("Your choice: ").strip().lower()
        if choice in ("s", "r", "q"):
            return {"s": "skip", "r": "retry", "q": "quit"}(choice)
        print("   Enter s, r, or q.")

gave should_pause() The function catches four cases: navigation failure, unexpected HTTP status, login keywords in the title, and login keywords in H1 tags. The login keyword check is what catches “Please sign in to continue” pages that return a 200 but are effectively inaccessible.

i --auto mode (for scheduled runs), skips the main loop. pause_and_prompt() Call the URL by logging in and handle these cases automatically. needs_human() In condition and ongoing.

Module 6: Report Writer

reporter.py Writes results gradually. This is important: the results are written after each URL is audited, not batched at the end. If the run is interrupted, you don’t lose completed work.

import json
import os
from datetime import datetime, timezone

REPORT_JSON = os.path.join(os.path.dirname(__file__), "report.json")
REPORT_TXT = os.path.join(os.path.dirname(__file__), "report-summary.txt")


def _load_report() -> list:
    if not os.path.exists(REPORT_JSON):
        return ()
    with open(REPORT_JSON, encoding="utf-8") as f:
        return json.load(f)


def write_result(result: dict) -> None:
    """Append or update a result in report.json."""
    entries = _load_report()
    url = result.get("url", "")

    # Update existing entry if URL already present (handles retries)
    for i, entry in enumerate(entries):
        if entry.get("url") == url:
            entries(i) = result
            break
    else:
        entries.append(result)

    with open(REPORT_JSON, "w", encoding="utf-8") as f:
        json.dump(entries, f, indent=2, ensure_ascii=False)


def _is_overall_pass(result: dict) -> bool:
    fields = ("title", "description", "h1", "canonical")
    for field in fields:
        if result.get(field, {}).get("status") not in ("PASS",):
            return False
    if result.get("broken_links", {}).get("status") == "FAIL":
        return False
    return True


def write_summary() -> None:
    entries = _load_report()
    passed = sum(1 for e in entries if _is_overall_pass(e))

    lines = ()
    for entry in entries:
        overall = "PASS" if _is_overall_pass(entry) else "FAIL"
        failed_fields = (
            f for f in ("title", "description", "h1", "canonical", "broken_links")
            if entry.get(f, {}).get("status") == "FAIL"
        )
        suffix = f" ({', '.join(failed_fields)})" if failed_fields else ""
        lines.append(f"{entry.get('url', 'unknown'):<60} | {overall}{suffix}")

    lines.append("")
    lines.append(f"{passed}/{len(entries)} URLs passed")

    with open(REPORT_TXT, "w", encoding="utf-8") as f:
        f.write("\n".join(lines))

Copy in write_result() Handles retries cleanly. If the URL is retried after a human has reviewed and verified the login wall, the new result replaces the old one instead of creating a duplicate entry.

Module 7: Main Loop

index.py Wires everything together. It reads the list of URLs, loads the state, skips the previously audited URLs, and runs the audit loop.

import csv
import os
import sys
import time
import argparse

from state import load_state, is_audited, mark_audited, add_to_needs_human
from browser import fetch_page
from extractor import extract
from linkchecker import check_links
from hitl import should_pause, pause_reason, pause_and_prompt
from reporter import write_result, write_summary

INPUT_CSV = os.path.join(os.path.dirname(__file__), "input.csv")


def read_urls(path: str) -> list(str):
    with open(path, newline="", encoding="utf-8") as f:
        return (row("url").strip() for row in csv.DictReader(f) if row.get("url", "").strip())


def run(auto: bool = False):
    if not os.environ.get("ANTHROPIC_API_KEY"):
        print("Error: ANTHROPIC_API_KEY environment variable is not set.")
        sys.exit(1)

    urls = read_urls(INPUT_CSV)
    pending = (u for u in urls if not is_audited(u))

    print(f"Starting audit: {len(pending)} pending, {len(urls) - len(pending)} already done.\n")

    total = len(urls)

    try:
        for i, url in enumerate(pending, start=1):
            position = urls.index(url) + 1
            print(f"({position}/{total}) {url}", end=" -> ", flush=True)

            # Browser navigation
            snapshot = fetch_page(url)

            # Human-in-the-loop check
            if should_pause(snapshot):
                reason = pause_reason(snapshot)

                if auto:
                    print(f"AUTO-SKIPPED ({reason})")
                    add_to_needs_human(url)
                    mark_audited(url)
                    continue

                action = pause_and_prompt(url, reason)
                if action == "quit":
                    print("Exiting.")
                    break
                elif action == "skip":
                    add_to_needs_human(url)
                    mark_audited(url)
                    continue
                # "retry" falls through to re-fetch below
                snapshot = fetch_page(url)

            # Claude extraction
            result = extract(snapshot)

            # Broken link check
            links = check_links(snapshot.get("raw_links", ()), snapshot.get("final_url", url))
            result("broken_links") = links

            # Write result immediately
            write_result(result)
            mark_audited(url)

            overall = "PASS" if all(
                result.get(f, {}).get("status") == "PASS"
                for f in ("title", "description", "h1", "canonical")
            ) and links("status") == "PASS" else "FAIL"

            print(overall)

    except KeyboardInterrupt:
        print("\n\nInterrupted. Progress saved. Re-run to continue.")
        return

    write_summary()
    passed = sum(
        1 for e in (r for r in ())
        if all(e.get(f, {}).get("status") == "PASS" for f in ("title", "description", "h1", "canonical"))
    )
    print(f"\nAudit complete. Report saved to report.json and report-summary.txt")


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--auto", action="store_true", help="Auto-skip URLs requiring human review")
    args = parser.parse_args()
    run(auto=args.auto)

gave KeyboardInterrupt The handler is the restart mechanism. When you press Ctrl+C the handler prints a message and exits gracefully. Because mark_audited() It is called after write_result() For each URL, the next run skips everything already processed.

Run the agent

Interactive mode (pause on edge cases):

python index.py

Auto mode (skips edge cases, adds needs_human()):

python index.py --auto

When it runs, you’ll see a browser window open for each URL and the terminal print progress:

Starting audit: 7 pending, 0 already done.

(1/7)  -> PASS
(2/7) /about -> FAIL
(3/7) /contact -> AUTO-SKIPPED (Unexpected status code: 404)
...
Audit complete. Report saved to report.json and report-summary.txt

To resume after an interruption:

python index.py --auto
# Starting audit: 4 pending, 3 already done.

Scheduling for agency use

For recurring weekly audits, create a batch file and schedule it with Windows Task Scheduler.

make run-audit.bat:

@echo off
set ANTHROPIC_API_KEY=your-key-here
cd /d C:\Users\yourname\Desktop\seo-agent
python index.py --auto

In Windows Task Scheduler:

1. Create a new base task

2. Set the trigger to Weekly, Monday at 7:00 AM

3. Set the action to “Start Program”.

4. Browse to your run-audit.bat File

Check. report-summary.txt Monday morning. URLs in needs_human() i state.json Manual review required — login walls, paywalls, or pages that return unexpected status codes.

For macOS/Linux, use cron:

# Run every Monday at 7am
0 7 * * 1 cd /path/to/seo-agent && ANTHROPIC_API_KEY=your-key python index.py --auto

What the results look like.

I ran this agent against seven of my published pages in Hashnode, freeCodeCamp, and DEV.to. Each failed.

                    | FAIL (h1)
     | FAIL (description)
 | FAIL (description)
   | FAIL (title, description)
     | FAIL (description)
         | FAIL (title)
     | FAIL (title)

0/7 URLs passed

FreeCodeCamp’s description issues are partly platform-level — freeCodeCamp’s template sometimes truncates or omits meta descriptions for article listing pages. The DEV.to title issues are mine. Article titles that serve as headlines are often longer than 60 characters.