slm_document_parser · Data & Utilities · 10-Plane Pure-CPU Agent

SLM Document Parser

Extract tabular metrics, metadata structures, and key paragraphs from PDFs, Word docs, and plain text files locally under strict MIT-licensing parameters.

 

01 / Overview & Capabilities

The SLM Document Parser leverages high-performance local neural reasoning and visual document parsing engines orchestrated by the SLMAgents 2.0 10-Plane Pure-CPU Harness (hybrid llama.cpp GGUF Q4_K_M + quantized INT4/INT8 ONNX Runtime CPU) to parse multi-formatted documents offline. It supports high-fidelity layout preservation for scanned PDFs and legacied formats (.pdf, .docx, .doc, .pptx, .ppt, .txt) to output structured text and semantic RAG database chunks.

02 / Standalone OS Harness Installers, CLI & Python SDK

In SLMAgents 2.0, SLM Document Parser executes inside the 10-Plane Pure-CPU Agent Harness (slm_harness). Download a self-extracting Individual Agent + Harness OS Installer for macOS, Windows, or Linux (bundling slm_document_parser + slm_harness + the ./slm CLI), install the full 26-agent desktop suite, or install via Python pip:

1. Individual SLM Document Parser + 10-Plane Harness Installers (All OSes)

2. Full 26-Agent Studio & Desktop OS Installers

3. 10-Plane Harness CLI (./slm) & Python pip Installation

Terminal — 10-Plane Harness CLI & Pip
# Option A: Install via PyPI or editable source with the 10-Plane Harness
$ pip install slm-document-parser
$ pip install -e ./slm_harness -e ./slm_document_parser

# Option B: Run directly through the Unified 10-Plane Pure-CPU Harness CLI (./slm)
$ ./slm ask "Parse @financial_report.pdf including tables and export semantic chunks"
$ ./slm chat   # Interactive Rich TUI with 18 slash commands (/trace, /models, /arena, /undo)
$ ./slm serve  # 100% Offline Web AI Studio + OpenAI /v1 API at http://127.0.0.1:7860

03 / 10-Plane Pure-CPU Harness Integration (slm_harness)

Rather than executing as an isolated prompt wrapper, SLM Document Parser is natively orchestrated by the SLMAgents 2.0 10-Plane Pure-CPU Harness (via slm_harness.runtime.legacy_bridge and CascadeRouter). Every call benefits from hybrid llama.cpp GGUF (Q4_K_M) + Quantized INT4/INT8 CPU execution, watermarked LRU memory governance, grammar-constrained decoding, and deterministic verification at $0 API cost:

Harness PlaneSubsystemHow It Powers SLM Document Parser on Pure CPU
Plane 6 CascadeRouter (<35ms) Stage 1 (File Extension / MIME Inspection: .pdf, .docx, .pptx)
Plane 2 & 5 MemoryArena & Hybrid CPU Runtime Specialist Role: vision (Florence-2-base INT4) + primary (Qwen3-4B GGUF Q4_K_M). Managed inside Watermark LRU MemoryArena (lite 2.6 GB / balanced 4.7 GB / power 8.2 GB) with zero-copy session sharing and automatic OOM tier step-down (8B -> 4B -> 3B -> 0.8B).
Plane 7 & 8 Deterministic Verifier & Governance Plane 3 Content-Addressed Artifact Scratchpad (artifact://) + Plane 7 Structured Markdown & Heading Hierarchy Verifier
Plane 1, 3 & 4 Kernel, Artifact Scratchpad & 4-Tier Memory SQLite WAL CheckpointStore (/fork, /rewind, /undo), content-addressed artifact://<sha12> handles to prevent prompt bloat, and hybrid SQLite FTS5 BM25 + HNSW vector memory.

04 / Source Checkout (Git Sparse-Checkout)

Clone this agent alongside the 10-Plane Pure-CPU Harness (slm_harness) from the monorepo using Git sparse-checkout — no need to download all 26 agents:

Option 1 — Sparse Checkout with 10-Plane Harness (Recommended)

Terminal — Git Sparse Checkout
# 1. Create and enter a new directory
$ mkdir slm_document_parser_workspace && cd slm_document_parser_workspace

# 2. Initialise empty git repo and add remote
$ git init
$ git remote add origin https://github.com/t00114218-stack/SLMAgents.git

# 3. Enable sparse-checkout for both slm_harness and slm_document_parser
$ git sparse-checkout init --cone
$ git sparse-checkout set slm_harness slm_document_parser

# 4. Pull the 10-Plane Harness and agent source
$ git pull origin main

Option 2 — Full Repository Clone

Terminal — Full Clone
$ git clone https://github.com/t00114218-stack/SLMAgents.git
$ cd SLMAgents
$ ./slm doctor

Note: After checkout, install the harness and agent in editable mode with pip install -e ./slm_harness -e ./slm_document_parser or download the pre-packaged OS Harness Installer above.

🤖 Truly Agentic Hybrid Visual Pipeline

The Document Parser handles layout mapping by combining visual coordinate classifiers with standard extraction loops:

  • LibreOffice Conversion Layer: Converts complex formats (DOC/PPT/PPTX) into intermediate PDFs in the background using headless soffice conversions.
  • Page Image Rendering: Renders PDF page indices to images sequentially using pypdfium2.
  • Visual OCR & Object Detection: Scans rendered page layouts, running Object Detection (<OD>) to locate table and figure boxes. Crops table boxes to OCR them individually, and runs detailed caption generation (<DETAILED_CAPTION>) for diagrams.
  • Layout Reconstruction: Passes text, parsed tables, and diagram captions to the local neural engine to synthesize a unified, formatted Markdown page representation.
  • Semantic Chunker & Linker: Segments markdown into structured paragraph chunks, extracts heading hierarchies, resolves product entities, and links related sections together.

Complex Table Visual OCR Extraction

When the visual parser encounters complex tables inside a document scan, it crops the table coordinates, routes it to the vision parser, and synthesizes a natural language text description. Below is an example of an input expense table and its corresponding parsed description:

Input Corporate Expense Table
Extracted Natural Language Table Description Output:
### Q2 FY2024 Company Expense Summary
The table outlines corporate expenditures across multiple categories:
- **Infrastructure**: Spent $1,195,450 against a $1,250,000 budget (a decrease of 4.36%).
- **AI Compute Nodes**: Spent $3,842,910 against a $3,500,000 budget (an increase of 9.80%).
- **Model Conversion**: Spent $412,300 against a $450,000 budget (a decrease of 8.38%).
- **Data Storage**: Spent $845,600 against a $800,000 budget (an increase of 5.70%).
- **GPU Clusters**: Spent $2,215,800 against a $2,100,000 budget (an increase of 5.51%).

CPU Performance Tuning Guidelines

Follow these configuration rules to optimize latency on standard CPUs:

  • Core Thread Cap: Set n_threads strictly to the number of physical cores to avoid context switching thread collisions.
  • Sequential Processing: Process multi-page images sequentially instead of in parallel batches to keep the memory footprint under 2.0 GB.
  • OpenMP Settings: Keep thread counts aligned in your shell environment:
    export OMP_NUM_THREADS=4
    export MKL_NUM_THREADS=4

05 / Configuration & API Reference

SLMDocumentParser Initialization

from slm_document_parser.document_parser import SLMDocumentParser

parser = SLMDocumentParser(n_ctx=4096, n_threads=4)
ParameterType / DefaultDescription
model_pathstr | NoneLocal path to model weight checkpoints.
n_ctxint | 4096Inference token window size. Supports deep multi-page context windows.
n_threadsint | 4Allocated physical CPU P-cores for 10-Plane Harness execution.
system_promptstr | NoneOptional custom system prompt instructions overriding the default template.
user_inputstr | NoneOptional additional user-supplied target parameters or variables.

extract_text Method

Converts document layouts and images into a single reconstructed Markdown text string:

markdown_content = parser.extract_text("financial_report.pdf")

chunk_document Method

Extracts markdown text and slices it into linked semantic chunks with metadata properties:

chunks = parser.chunk_document("financial_report.pdf")
print(chunks[0])
Extracted Semantic Chunk Output (JSON):
{
  "text": "SpaceX successfully launched the Falcon 9 rocket from Cape Canaveral Space Force Station, landing the booster return flight for the 15th time. The mission delivered communication payloads into low Earth orbit.",
  "metadata": {
    "source": "financial_report.pdf",
    "heading": "1. Launch Milestones",
    "subheading": "Falcon 9 Performance",
    "product": "SpaceX",
    "key_terms": ["Falcon 9", "Cape Canaveral", "booster"],
    "format": "pdf",
    "chunk_index": 0,
    "related_chunks": [1, 2]
  }
}

parse_and_chunk_stream Method

Streaming generator yielding semantic chunks page-by-page as they are processed to reduce first-chunk response times (TTFT):

for chunk in parser.parse_and_chunk_stream("multi_page_manual.pdf"):
    print(f"Processed Chunk: {chunk['text'][:100]}...")

export_chunks_to_excel Method

Saves the parsed chunks to an Excel spreadsheet:

# Exports to workbook. Set append=True to write to an existing sheet
parser.export_chunks_to_excel(chunks, "rag_dataset.xlsx", append=False)
Output Excel Table Layout:
Chunk IndexSource FileHeadingSubheadingProductRelated ChunksText
0financial_report.pdf1. Launch MilestonesFalcon 9 PerformanceSpaceX1,2SpaceX successfully launched the Falcon 9...