Open SEO Crawler

Open SEO Crawler is a free, open-source SEO crawler you run locally: a self-hosted website SEO crawler and site audit tool that crawls any website for technical SEO issues. Fast, concurrent, and CMS-aware, it’s a drop-in alternative to Screaming Frog, Sitebulb, and Ahrefs Site Audit, with no accounts, no API keys, no cloud and no per-URL limits.

References

• GitHub

---

Make directory

mkdir -p {{DOCKER_PATH_VAR}}/open-seo-crawler && cd {{DOCKER_PATH_VAR}}/open-seo-crawler

Dockerfile

nano Dockerfile

Dockerfile

FROM python:3.11-slim AS builder

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    git \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

RUN git clone https://github.com/puneetindersingh/open-seo-crawler.git .

RUN python -m venv /opt/venv \
    && /opt/venv/bin/pip install --no-cache-dir -r requirements.txt

FROM python:3.11-slim

WORKDIR /app

COPY --from=builder /opt/venv /opt/venv
COPY --from=builder /app /app

ENV PATH="/opt/venv/bin:$PATH" \
    HOST=0.0.0.0 \
    PORT=5002 \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

EXPOSE 5002

CMD ["python", "app.py"]

compose.yml

nano compose.yml

compose.yml

services:
  open-seo-crawler:
    build: .
    container_name: open-seo-crawler
    ports:
      - "5002:5002"
    volumes:
      - ./crawls:/root/.site-crawler-crawls
    restart: unless-stopped

Start container

docker compose up --build -d

Open web ui

http://localhost:5002 or http://{{SERVER_IP_VAR}}:5002