slm_vision_parser · Data & Vision · 10-Plane Pure-CPU Agent

SLM Vision Parser

Extract text nodes, bounding boxes, chart labels, and table grids locally on CPU using local CPU-optimized visual neural models.

01 / Overview & Capabilities

The SLM Vision Parser uses local multimodal neural models to translate image structures directly. It is designed to run offline on CPU under 1.5 GB memory footprint, making it ideal for visual PDF OCR, diagram relationship mappings, and whiteboard extractions.

02 / Standalone OS Harness Installers, CLI & Python SDK

In SLMAgents 2.0, SLM Vision Parser executes inside the 10-Plane Pure-CPU Agent Harness (slm_harness). Download a self-extracting Individual Agent + Harness OS Installer for macOS, Windows, or Linux (bundling slm_vision_parser + slm_harness + the ./slm CLI), install the full 26-agent desktop suite, or install via Python pip:

1. Individual SLM Vision Parser + 10-Plane Harness Installers (All OSes)

2. Full 26-Agent Studio & Desktop OS Installers

3. 10-Plane Harness CLI (./slm) & Python pip Installation

Terminal — 10-Plane Harness CLI & Pip
# Option A: Install via PyPI or editable source with the 10-Plane Harness
$ pip install slm-vision-parser
$ pip install -e ./slm_harness -e ./slm_vision_parser

# Option B: Run directly through the Unified 10-Plane Pure-CPU Harness CLI (./slm)
$ ./slm ask "Extract all nodes and arrow connections from @flowchart.png"
$ ./slm chat   # Interactive Rich TUI with 18 slash commands (/trace, /models, /arena, /undo)
$ ./slm serve  # 100% Offline Web AI Studio + OpenAI /v1 API at http://127.0.0.1:7860

03 / 10-Plane Pure-CPU Harness Integration (slm_harness)

Rather than executing as an isolated prompt wrapper, SLM Vision Parser is natively orchestrated by the SLMAgents 2.0 10-Plane Pure-CPU Harness (via slm_harness.runtime.legacy_bridge and CascadeRouter). Every call benefits from hybrid llama.cpp GGUF (Q4_K_M) + Quantized INT4/INT8 CPU execution, watermarked LRU memory governance, grammar-constrained decoding, and deterministic verification at $0 API cost:

Harness PlaneSubsystemHow It Powers SLM Vision Parser on Pure CPU
Plane 6 CascadeRouter (<35ms) Stage 1 (Image MIME / Extension Inspection: .png, .jpg, .webp, , )
Plane 2 & 5 MemoryArena & Hybrid CPU Runtime Specialist Role: vision (Florence-2-base Vision Encoder + Decoder INT4/FP16 CPU). Managed inside Watermark LRU MemoryArena (lite 2.6 GB / balanced 4.7 GB / power 8.2 GB) with zero-copy session sharing and automatic OOM tier step-down (8B -> 4B -> 3B -> 0.8B).
Plane 7 & 8 Deterministic Verifier & Governance Plane 5 MemoryArena Sequential Vision Pinning + Plane 7 Bounding-Box & Task-Tag (, , ) Verifier
Plane 1, 3 & 4 Kernel, Artifact Scratchpad & 4-Tier Memory SQLite WAL CheckpointStore (/fork, /rewind, /undo), content-addressed artifact://<sha12> handles to prevent prompt bloat, and hybrid SQLite FTS5 BM25 + HNSW vector memory.

04 / Source Checkout (Git Sparse-Checkout)

Clone this agent alongside the 10-Plane Pure-CPU Harness (slm_harness) from the monorepo using Git sparse-checkout — no need to download all 26 agents:

Option 1 — Sparse Checkout with 10-Plane Harness (Recommended)

Terminal — Git Sparse Checkout
# 1. Create and enter a new directory
$ mkdir slm_vision_parser_workspace && cd slm_vision_parser_workspace

# 2. Initialise empty git repo and add remote
$ git init
$ git remote add origin https://github.com/t00114218-stack/SLMAgents.git

# 3. Enable sparse-checkout for both slm_harness and slm_vision_parser
$ git sparse-checkout init --cone
$ git sparse-checkout set slm_harness slm_vision_parser

# 4. Pull the 10-Plane Harness and agent source
$ git pull origin main

Option 2 — Full Repository Clone

Terminal — Full Clone
$ git clone https://github.com/t00114218-stack/SLMAgents.git
$ cd SLMAgents
$ ./slm doctor

Note: After checkout, install the harness and agent in editable mode with pip install -e ./slm_harness -e ./slm_vision_parser or download the pre-packaged OS Harness Installer above.

🤖 Truly Agentic Workflow

The Vision Parser maps visual coordinate dimensions into clean layout logs:

  • OCR Node Extraction: Translates raster text strings from diagrams.
  • Region Mapping: Generates bounding box parameters mapping visual coordinates for web rendering.
  • Semantic Image Captioning: Extracts visual flow structures (e.g., arrows, shapes, labels) into a raw description.
  • Agentic LLM Refinement: Pipe raw visual description nodes directly into the local neural reasoning model. By guiding the text model with a template prompt, you can synthesize exact custom descriptions (e.g., describing step hierarchies and the exact count of connecting arrows).

CPU Performance Tuning Guidelines

Local vision models utilize visual transformers. To run model prediction tasks efficiently on CPU under 2.0 GB memory:

  • Sequential execution: Process page scans sequentially rather than in batches to avoid CPU memory thrashing.
  • Garbage Collection: Clear cache blocks after processing high-resolution images:
    import gc
    gc.collect()

Accuracy Improvement Tips

Tip for Scanned Diagrams: Contrast is key for visual OCR models. Preprocess low-contrast diagram images by resizing them to a standard width of 1024px and scaling contrast values using Pillow before parsing.
Task Tag Wrapper: Always encapsulate vision model instructions in explicit brackets (e.g., "<OCR>" or "<DETAILED_CAPTION>") to trigger correct task headers.

05 / Configuration & API Reference

SLMVisionParser Initialization

from slm_vision_parser.vision_parser import SLMVisionParser

parser = SLMVisionParser()
ParameterTypeDescription
model_pathstrLocal directory containing vision model checkpoints.
system_promptstr | NoneOptional custom system prompt instructions overriding the default template.
user_inputstr | NoneOptional additional user-supplied target parameters or variables.

parse_image Method

Run coordinate mappings or caption generation by supplying task tags. Below is the input flowchart diagram processed by the vision model:

Input Flowchart Diagram
from slm_vision_parser.vision_parser import SLMVisionParser

parser = SLMVisionParser()

# Describe the flowchart diagram in text using the local vision LLM
result = parser.parse_image("flowchart.png", task="<DETAILED_CAPTION>")
print(result)
Image Description Response Output:
"A flowchart showing a start step ('Start Process') and a next step ('Next Step Link') connected with two arrows from the start step to the next step."