slm_web_scraper · Web & Search · 10-Plane Pure-CPU Agent

SLM Web Scraper

Extract structured JSON directly from dirty, obfuscated HTML layouts offline using BeautifulSoup and local neural reasoning.

01 / Overview & Capabilities

The SLM Web Scraper parses raw HTML strings, stripping scripts, styles, duplicate nav tags, and ads. The sanitized flat content is compiled locally into structured formats using target JSON schemas.

02 / Standalone OS Harness Installers, CLI & Python SDK

In SLMAgents 2.0, SLM Web Scraper executes inside the 10-Plane Pure-CPU Agent Harness (slm_harness). Download a self-extracting Individual Agent + Harness OS Installer for macOS, Windows, or Linux (bundling slm_web_scraper + slm_harness + the ./slm CLI), install the full 26-agent desktop suite, or install via Python pip:

1. Individual SLM Web Scraper + 10-Plane Harness Installers (All OSes)

2. Full 26-Agent Studio & Desktop OS Installers

3. 10-Plane Harness CLI (./slm) & Python pip Installation

Terminal — 10-Plane Harness CLI & Pip
# Option A: Install via PyPI or editable source with the 10-Plane Harness
$ pip install slm-web-scraper
$ pip install -e ./slm_harness -e ./slm_web_scraper

# Option B: Run directly through the Unified 10-Plane Pure-CPU Harness CLI (./slm)
$ ./slm ask "Scrape and sanitize https://www.slmagents.ai/vision_parser.html into structured Markdown"
$ ./slm chat   # Interactive Rich TUI with 18 slash commands (/trace, /models, /arena, /undo)
$ ./slm serve  # 100% Offline Web AI Studio + OpenAI /v1 API at http://127.0.0.1:7860

03 / 10-Plane Pure-CPU Harness Integration (slm_harness)

Rather than executing as an isolated prompt wrapper, SLM Web Scraper is natively orchestrated by the SLMAgents 2.0 10-Plane Pure-CPU Harness (via slm_harness.runtime.legacy_bridge and CascadeRouter). Every call benefits from hybrid llama.cpp GGUF (Q4_K_M) + Quantized INT4/INT8 CPU execution, watermarked LRU memory governance, grammar-constrained decoding, and deterministic verification at $0 API cost:

Harness PlaneSubsystemHow It Powers SLM Web Scraper on Pure CPU
Plane 6 CascadeRouter (<35ms) Stage 1 (HTML Payload / URL Signature) & Stage 2 (Scraping & Sanitization Intent)
Plane 2 & 5 MemoryArena & Hybrid CPU Runtime Specialist Role: primary (Qwen3-4B GGUF Q4_K_M) + vision (Florence-2-base for inline tags). Managed inside Watermark LRU MemoryArena (lite 2.6 GB / balanced 4.7 GB / power 8.2 GB) with zero-copy session sharing and automatic OOM tier step-down (8B -> 4B -> 3B -> 0.8B).
Plane 7 & 8 Deterministic Verifier & Governance Plane 2 GBNF JSON Schema Compiler + Plane 3 HTML DOM Pruning & Artifact Scratchpad (artifact://)
Plane 1, 3 & 4 Kernel, Artifact Scratchpad & 4-Tier Memory SQLite WAL CheckpointStore (/fork, /rewind, /undo), content-addressed artifact://<sha12> handles to prevent prompt bloat, and hybrid SQLite FTS5 BM25 + HNSW vector memory.

04 / Source Checkout (Git Sparse-Checkout)

Clone this agent alongside the 10-Plane Pure-CPU Harness (slm_harness) from the monorepo using Git sparse-checkout — no need to download all 26 agents:

Option 1 — Sparse Checkout with 10-Plane Harness (Recommended)

Terminal — Git Sparse Checkout
# 1. Create and enter a new directory
$ mkdir slm_web_scraper_workspace && cd slm_web_scraper_workspace

# 2. Initialise empty git repo and add remote
$ git init
$ git remote add origin https://github.com/t00114218-stack/SLMAgents.git

# 3. Enable sparse-checkout for both slm_harness and slm_web_scraper
$ git sparse-checkout init --cone
$ git sparse-checkout set slm_harness slm_web_scraper

# 4. Pull the 10-Plane Harness and agent source
$ git pull origin main

Option 2 — Full Repository Clone

Terminal — Full Clone
$ git clone https://github.com/t00114218-stack/SLMAgents.git
$ cd SLMAgents
$ ./slm doctor

Note: After checkout, install the harness and agent in editable mode with pip install -e ./slm_harness -e ./slm_web_scraper or download the pre-packaged OS Harness Installer above.

🤖 Truly Agentic Workflow

The Web Scraper filters raw DOM elements to produce clean extractions:

  • DOM Pruning: Automatically strips scripts, styles, metadata tags, and ad headers using BeautifulSoup.
  • Visual Image OCR: Identifies image tags (<img>) on the page, downloads them, and runs them through the local SLMVisionParser to generate natural language text descriptions, replacing the tags with the descriptions.
  • Whitespaces Compacting: Collapses contiguous whitespace blocks into single space delimiters.
  • Schema Alignment: Instructs the local neural model to match and structure the sanitized text into clean JSON.

CPU Performance Tuning Guidelines

Follow these configuration rules to maximize scraping speed on standard hardware:

  • Sanitize First: Always clean webpage source code before parsing. Running raw HTML through the model increases token count and latency.
  • Memory Management: Clear page memory scopes after heavy parsing cycles to keep footprint under 1.5 GB RAM.

Accuracy Improvement Tips

Tip for Layout Cleaning: Condensing tables or pricing matrices into columnar string lists (e.g. "Item 1 | Price 1") preserves formatting structure for the model, improving data extraction accuracy.
Keep Schema Flat: Keep the target schema flat. Deep nesting can confuse smaller models, causing schema validation errors.

05 / Configuration & API Reference

SLMWebScraper Initialization

from slm_web_scraper.web_scraper import SLMWebScraper

scraper = SLMWebScraper()
ParameterTypeDescription
model_pathstrLocal path to model weights.
system_promptstr | NoneOptional custom system prompt instructions overriding the default template.
user_inputstr | NoneOptional additional user-supplied target parameters or variables.

scrape_url Method

Fetches a URL, filters out header/footer menus, dropdown lists, sidebars, and ads, and automatically parses tables and image content:

Hybrid Visual Scraping (Tables & Images)

This example scrapes the vision parser documentation page https://www.slmagents.ai/vision_parser.html, describing its parameters table and flowchart image automatically:

from slm_web_scraper.web_scraper import SLMWebScraper

scraper = SLMWebScraper()

# Scrapes page and converts tables/images to descriptions automatically
clean_text = scraper.scrape_url("https://www.slmagents.ai/vision_parser.html")
print(clean_text)
Cleaned Webpage Text Output with Visual Descriptions:
SLM Vision Parser | Documentation
Overview
The SLM Vision Parser uses local multimodal vision models to extract image structures directly.

API Reference
SLMVisionParser Initialization
from slm_vision_parser.vision_parser import SLMVisionParser
parser = SLMVisionParser()

[Table Description: The HTML table represents the constructor parameter configuration for SLMVisionParser, detailing the "model_path" parameter of type string (str), which references the local directory containing pre-downloaded checkpoints and defaults to local model weights.]

parse_image Method
Run coordinate mappings or caption generation by supplying task tags. Below is the input flowchart diagram processed by the vision model:
[Image Description: A flowchart showing a start step ('Start Process') and a next step ('Next Step Link') connected with two arrows from the start step to the next step.]