SLM Web Scraper
Extract structured JSON directly from dirty, obfuscated HTML layouts offline using BeautifulSoup and local neural reasoning.
01 / Overview & Capabilities
The SLM Web Scraper parses raw HTML strings, stripping scripts, styles, duplicate nav tags, and ads. The sanitized flat content is compiled locally into structured formats using target JSON schemas.
02 / Standalone OS Harness Installers, CLI & Python SDK
In SLMAgents 2.0, SLM Web Scraper executes inside the 10-Plane Pure-CPU Agent Harness (slm_harness). Download a self-extracting Individual Agent + Harness OS Installer for macOS, Windows, or Linux (bundling slm_web_scraper + slm_harness + the ./slm CLI), install the full 26-agent desktop suite, or install via Python pip:
1. Individual SLM Web Scraper + 10-Plane Harness Installers (All OSes)
slm_web_scraper + slm_harness + ./slm CLIslm_web_scraper + slm_harnessslm_web_scraper + slm_harness + ./slm CLI2. Full 26-Agent Studio & Desktop OS Installers
3. 10-Plane Harness CLI (./slm) & Python pip Installation
# Option A: Install via PyPI or editable source with the 10-Plane Harness $ pip install slm-web-scraper $ pip install -e ./slm_harness -e ./slm_web_scraper # Option B: Run directly through the Unified 10-Plane Pure-CPU Harness CLI (./slm) $ ./slm ask "Scrape and sanitize https://www.slmagents.ai/vision_parser.html into structured Markdown" $ ./slm chat # Interactive Rich TUI with 18 slash commands (/trace, /models, /arena, /undo) $ ./slm serve # 100% Offline Web AI Studio + OpenAI /v1 API at http://127.0.0.1:7860
03 / 10-Plane Pure-CPU Harness Integration (slm_harness)
Rather than executing as an isolated prompt wrapper, SLM Web Scraper is natively orchestrated by the SLMAgents 2.0 10-Plane Pure-CPU Harness (via slm_harness.runtime.legacy_bridge and CascadeRouter). Every call benefits from hybrid llama.cpp GGUF (Q4_K_M) + Quantized INT4/INT8 CPU execution, watermarked LRU memory governance, grammar-constrained decoding, and deterministic verification at $0 API cost:
| Harness Plane | Subsystem | How It Powers SLM Web Scraper on Pure CPU |
|---|---|---|
| Plane 6 | CascadeRouter (<35ms) |
Stage 1 (HTML Payload / URL Signature) & Stage 2 (Scraping & Sanitization Intent) |
| Plane 2 & 5 | MemoryArena & Hybrid CPU Runtime |
Specialist Role: primary (Qwen3-4B GGUF Q4_K_M) + vision (Florence-2-base for inline . Managed inside Watermark LRU MemoryArena (lite 2.6 GB / balanced 4.7 GB / power 8.2 GB) with zero-copy session sharing and automatic OOM tier step-down (8B -> 4B -> 3B -> 0.8B). |
| Plane 7 & 8 | Deterministic Verifier & Governance | Plane 2 GBNF JSON Schema Compiler + Plane 3 HTML DOM Pruning & Artifact Scratchpad (artifact:// |
| Plane 1, 3 & 4 | Kernel, Artifact Scratchpad & 4-Tier Memory | SQLite WAL CheckpointStore (/fork, /rewind, /undo), content-addressed artifact://<sha12> handles to prevent prompt bloat, and hybrid SQLite FTS5 BM25 + HNSW vector memory. |
04 / Source Checkout (Git Sparse-Checkout)
Clone this agent alongside the 10-Plane Pure-CPU Harness (slm_harness) from the monorepo using Git sparse-checkout — no need to download all 26 agents:
Option 1 — Sparse Checkout with 10-Plane Harness (Recommended)
# 1. Create and enter a new directory $ mkdir slm_web_scraper_workspace && cd slm_web_scraper_workspace # 2. Initialise empty git repo and add remote $ git init $ git remote add origin https://github.com/t00114218-stack/SLMAgents.git # 3. Enable sparse-checkout for both slm_harness and slm_web_scraper $ git sparse-checkout init --cone $ git sparse-checkout set slm_harness slm_web_scraper # 4. Pull the 10-Plane Harness and agent source $ git pull origin main
Option 2 — Full Repository Clone
$ git clone https://github.com/t00114218-stack/SLMAgents.git $ cd SLMAgents $ ./slm doctor
Note: After checkout, install the harness and agent in editable mode with pip install -e ./slm_harness -e ./slm_web_scraper or download the pre-packaged OS Harness Installer above.
🤖 Truly Agentic Workflow
The Web Scraper filters raw DOM elements to produce clean extractions:
- DOM Pruning: Automatically strips scripts, styles, metadata tags, and ad headers using BeautifulSoup.
- Visual Image OCR: Identifies image tags (
<img>) on the page, downloads them, and runs them through the local SLMVisionParser to generate natural language text descriptions, replacing the tags with the descriptions. - Whitespaces Compacting: Collapses contiguous whitespace blocks into single space delimiters.
- Schema Alignment: Instructs the local neural model to match and structure the sanitized text into clean JSON.
CPU Performance Tuning Guidelines
Follow these configuration rules to maximize scraping speed on standard hardware:
- Sanitize First: Always clean webpage source code before parsing. Running raw HTML through the model increases token count and latency.
- Memory Management: Clear page memory scopes after heavy parsing cycles to keep footprint under 1.5 GB RAM.
Accuracy Improvement Tips
"Item 1 | Price 1") preserves formatting structure for the model, improving data extraction accuracy.
05 / Configuration & API Reference
SLMWebScraper Initialization
from slm_web_scraper.web_scraper import SLMWebScraper
scraper = SLMWebScraper()
| Parameter | Type | Description |
|---|---|---|
| model_path | str | Local path to model weights. |
| system_prompt | str | None | Optional custom system prompt instructions overriding the default template. |
| user_input | str | None | Optional additional user-supplied target parameters or variables. |
scrape_url Method
Fetches a URL, filters out header/footer menus, dropdown lists, sidebars, and ads, and automatically parses tables and image content:
Hybrid Visual Scraping (Tables & Images)
This example scrapes the vision parser documentation page https://www.slmagents.ai/vision_parser.html, describing its parameters table and flowchart image automatically:
from slm_web_scraper.web_scraper import SLMWebScraper
scraper = SLMWebScraper()
# Scrapes page and converts tables/images to descriptions automatically
clean_text = scraper.scrape_url("https://www.slmagents.ai/vision_parser.html")
print(clean_text)
SLM Vision Parser | Documentation
Overview
The SLM Vision Parser uses local multimodal vision models to extract image structures directly.
API Reference
SLMVisionParser Initialization
from slm_vision_parser.vision_parser import SLMVisionParser
parser = SLMVisionParser()
[Table Description: The HTML table represents the constructor parameter configuration for SLMVisionParser, detailing the "model_path" parameter of type string (str), which references the local directory containing pre-downloaded checkpoints and defaults to local model weights.]
parse_image Method
Run coordinate mappings or caption generation by supplying task tags. Below is the input flowchart diagram processed by the vision model:
[Image Description: A flowchart showing a start step ('Start Process') and a next step ('Next Step Link') connected with two arrows from the start step to the next step.]