🧹 Data Sanitization Agent

SLM JSON Cleaner

Lightweight local JSON text sanitizer and structural repair tool that formats malformed, truncated, or unquoted LLM outputs offline on standard CPU.

💻 Installation

Terminal

              # Install in editable mode locally
pip install -e ./slm_json_cleaner

# Set performance threads
export SLM_JSON_CLEANER_N_THREADS=4
            

🐙 Checkout from GitHub

Clone only this agent's folder from the monorepo using Git sparse-checkout — no need to download the full repository:

Option 1 — Sparse Checkout (Recommended)

Terminal — Git Sparse Checkout

              # 1. Create and enter a new directory
$ mkdir slm_json_cleaner && cd slm_json_cleaner

# 2. Initialise empty git repo and add remote
$ git init
$ git remote add origin https://github.com/t00114218-stack/SLMAgents.git

# 3. Enable sparse-checkout and set target folder
$ git sparse-checkout init --cone
$ git sparse-checkout set slm_json_cleaner

# 4. Pull only that agent's source
$ git pull origin main
            

Option 2 — Full Repository Clone

Terminal — Full Clone

              $ git clone https://github.com/t00114218-stack/SLMAgents.git
$ cd SLMAgents/slm_json_cleaner
            

💡 Tip: After checkout, install the package locally with pip install -e ./slm_json_cleaner to run in editable mode without publishing to PyPI.

⚙️ Configuration API

Constructor Parameters

Parameter	Type / Default	Description
model_path	str \| None	Explicit path to the ONNX model directory. Defaults to caching/sharing the main model in the monorepo.
cache_dir	str \| None	HF model directory path. Also settable via SLM_JSON_CLEANER_CACHE_DIR.
n_ctx	int \| 2048	Context window size in tokens. Also settable via SLM_JSON_CLEANER_N_CTX.
n_threads	int \| 4	CPU threads for ONNX Runtime. Also settable via SLM_JSON_CLEANER_N_THREADS.
system_prompt	str \| None	Optional custom system prompt instructions overriding the default template.
user_input	str \| None	Optional additional user-supplied target parameters or variables.

clean_json() Parameters

Parameter	Type / Default	Description
malformed_text	str	Required. Unstructured or broken JSON input string to repair.
schema_dict	dict	Required. A dictionary mapping keys and types required in the final output (e.g. {"name": "string", "age": "number"}).
system_prompt	str \| None	Optional custom system prompt instructions overriding the default template.
user_input	str \| None	Optional additional user-supplied target parameters or variables.

Python Quick Start

              from slm_json_cleaner.json_cleaner import SLMJSONCleaner

cleaner = SLMJSONCleaner()

# Truncated, nested, and conversational malformed input log:
broken_input = """
Raw logs:
{
  "project_name": "Antigravity Pipeline",
  "build_status": "success",
  "metrics": {
     "duration_seconds": 124.5,
     "test_count": 48,
     "failed_tests": 0,
     "coverage": "98.5%
  },
  "contributors": [
     {"name": "Alice", "role": "lead"},
     {"name": "Bob", "role": "reviewer"
  ],
  "releases": [
     "v1.0", "v1.1", 
  
[LOG EXPIRED MID-TOKEN]
"""

schema = {
    "project_name": "string",
    "build_status": "string",
    "metrics": {
        "duration_seconds": "number",
        "test_count": "number",
        "failed_tests": "number",
        "coverage": "string"
    },
    "contributors": [
        {"name": "string", "role": "string"}
    ],
    "releases": ["string"]
}

parsed, success = cleaner.clean_json(broken_input, schema)
print(f"Success: {success}")
print(parsed)
            

JSON Recovery & Repair Capabilities

The sanitization model operates as a high-speed pattern matching engine. It cleans common structural errors caused by early tokens cutoff or memory limits:

Auto-closure: Repairs unclosed brackets } or list braces ].
Quote repair: Fixes missing double-quotes on JSON keys and string values.
Regex brace fallback: Includes a backup regex matcher to extract matching curly braces if standard JSON parsing triggers an exception.

Target Schema Enforcement

By supplying a target schema dictionary, you guarantee that the returned attributes match the expected data types. The repaired response returns exactly structured JSON data:

{
  "project_name": "Antigravity Pipeline",
  "build_status": "success",
  "metrics": {
    "duration_seconds": 124.5,
    "test_count": 48,
    "failed_tests": 0,
    "coverage": "98.5%"
  },
  "contributors": [
    { "name": "Alice", "role": "lead" },
    { "name": "Bob", "role": "reviewer" }
  ],
  "releases": [ "v1.0", "v1.1" ]
}