AutoRAG Agent

Now your agent can find anything in your computer.

= 24" /> = 1.2" />

[!IMPORTANT] Looking for the original AutoRAG (RAG AutoML / pipeline optimization tool)? This repository now hosts AutoRAG 2.0, a complete reimagining of AutoRAG as a self-evolving librarian agent. The original Python-based AutoRAG — the RAG AutoML tool for automatically finding an optimal RAG pipeline for your data — now lives in the legacy/ directory of this repository.

The legacy AutoRAG is NOT abandoned. It continues to be maintained (bug fixes, dependency updates, and PyPI releases via pip install AutoRAG) in maintenance mode. Existing users can keep using it exactly as before — see the legacy README for its documentation, and file issues in this repository as usual. New feature development is focused on AutoRAG Agent (2.0).

What is AutoRAG Agent?

Search tools dump file paths and matching lines — forcing you to open files, read context, and synthesize answers yourself.

AutoRAG Agent is a self-evolving librarian agent. Built on the Pi agent framework, a single configured model searches across multiple retrieval methods, opens source documents directly via bash to verify ground truth, and curates answers into clean, numbered knowledge units:

You ask:  "What changes were made to the cloud infrastructure contract in Q3?"

AutoRAG Agent:
[1] Enterprise Compute Discount — AWS annual discount tier increased to 28% based on commitment volume. (contracts/cloud-2026.pdf:p.4)
[2] Regional Data Residency — Explicit compliance clause pinning customer data storage to the Seoul AWS region. (/slack/legal-ops/chunks/318)
[3] SLA Guarantee — Guaranteed uptime threshold adjusted to 99.95% with penalty credits starting at 15 minutes downtime. (contracts/sla-addendum.md:lines 45-62)

Core Values

Three principles drive every design decision in AutoRAG Agent:

1. Never migrate your data to search it. Traditional RAG systems force you to upload, ETL, and duplicate your files into a centralized vector database. AutoRAG Agent federates your data in place, querying CLI-native stores (lazykatok, discrawl, slacrawl, mailcrawl, rclone, qmd) where your data already lives. Results retain opaque, source-native identities (/kakao/..., /slack/...) that preserve local access control and privacy. (See our Competitive Landscape Study on why in-place federation is the durable differentiator).

2. Just works — no RAG degree required. No pipeline tuning, no vector-DB maintenance, and no remote embedding keys required. AutoRAG Agent automatically manages MinSync for incremental Change Data Capture (CDC) chunking and provides a local embedding gateway out of the box with zero external telemetry. A remote rerank model (OpenRouter, voyageai/rerank-3-lite by default) is an opt-in extra.

3. Fast by design. Rather than coordinating slow multi-agent hierarchies, a single configured model owns the entire retrieval, direct-read, and curation loop. Local CDC chunks (BM25, vector, and hybrid modes) deliver rapid, low-latency turnaround across multi-turn research queries.

Architecture & How It Works

AutoRAG Agent orchestrates five integrated subsystems:

1. Retrieval Memory (check_memory): After each answer, cited evidence Jev judges to genuinely support it is stored in ~/.autorag/memory.json. Later searches receive this conversation's evidence, similar past questions, and long-term insights as advisory reference — it never reorders results or overrides current evidence. See docs/retrieval-memory.md.
2. Pluggable Multi-Method Retrieval:
   • BM25 Lexical Search: Fast keyword ranking via MinSync over parsed markdown mirrors.
   • Semantic Vector Search: Dense vector retrieval over CDC chunks via the built-in embedding gateway.
   • Hybrid Search: Combines BM25 and vector ranking via Reciprocal Rank Fusion (RRF).
   • Jikji Find-First Discovery: Local CLI-backed fast discovery answer packs.
   • Everything File-Name Search (Windows): The bundled voidtools Everything indexes file and folder names under your search roots so the agent finds files by name, extension, path, size, or date instantly (everything_search).
   • FSearch File-Name Search (macOS/Linux): fsearch-cli (FSearch) keeps a live, per-workspace name index of your search roots so the agent finds files by name, extension, path, size, or date instantly (fsearch_search); without fsearch-cli installed it degrades to a slow filesystem walk.
   • Datasource Skills: Federated retrieval across every configured external datasource.
3. Result Merger & Scope Narrowing: Cross-method deduplication, score normalization, and ordinary query-scope narrowing.
4. Direct Evidence Reading (bash): The agent directly opens and inspects promising files with cat, grep, or find to verify facts against ground truth.
5. Curation: The final answer is the model's plain reply; the harness resolves its inline [eN] citations into numbered, source-backed results; cited evidence that Jev judges to genuinely support the answer is remembered as reference context for later searches (docs/retrieval-memory.md).

Pi host boundary

AutoRAG uses @earendil-works/pi-coding-agent as the runtime host for model sessions. Pi owns provider credentials and OAuth storage, model-runtime dispatch, session JSONL persistence/resume, extension loading, lifecycle events, and the built-in read/bash/edit/write/grep/find/ls tools. AutoRAG registers only its domain tools and keeps orchestration outside the host: datasource orchestration, MinSync/Jikji preparation, memory hints, fast-to-verification two-phase search, structured result emission, and remote-session filtering.

The CLI TUI (autorag tui) is Pi's interactive mode hosted on the AutoRAG librarian: there is no separate AutoRAG renderer. Pi owns the terminal UI and its native in-session commands — /login//logout, /model, /resume, /new, /tree, /compact, and /settings — so provider sign-in and model selection work on first launch even when no AutoRAG model is configured. AutoRAG registers its retrieval tools and runs the two-phase search in the same Pi session, where the model's progress notes, first answer, and final answer are ordinary assistant messages, so resume and two-phase search keep working together.

Update notices are AutoRAG's, not Pi's: autorag tui suppresses Pi's "run pi update" banner (Pi is a bundled host, not a separate install) and instead checks the published @autorag/librarian release, showing a one-line notice in the session when a newer version exists. Set AUTORAG_NO_UPDATE_CHECK=1 to skip the lookup; AUTORAG_UPDATE_CHECK_URL overrides the registry endpoint.

🤖 For AI Agents & LLMs: Setup & Skills

If you are an AI coding agent or LLM (Claude Code, Cursor, Windsurf, Codex, Senpi, Pi, etc.) tasked with installing, configuring, or querying AutoRAG Agent, use our built-in skills located in the skills/ directory:

Install the skills into your coding agent

Skills are not auto-discovered — copy only the setup skill when an agent needs to bootstrap AutoRAG. Routine AutoRAG Lite retrieval is provided by MCP tools, not by a autorag-lite-search skill or shell command.

# From a clone of this repository
mkdir -p .claude/skills
cp -R skills/autorag-lite-setup .claude/skills/

# From a global install — the npm package ships the same skills/ folder
AUTORAG_SKILLS="$(npm root -g)/@autorag/librarian/skills"
# Bun global installs live at ~/.bun/install/global/node_modules/@autorag/librarian/skills
mkdir -p .claude/skills
cp -R "$AUTORAG_SKILLS/autorag-lite-setup" .claude/skills/

Run the setup skill once to register the stdio server with the host. Reload the agent, then discover the actual tool names and schemas with MCP tools/list. The normal Lite path is autorag.status → autorag.refresh when needed → autorag.search; use autorag.report and autorag.evidence for the optional curation lifecycle. Copy skills/autorag and skills/autorag-setup only when the agent should also drive the model-backed librarian, and skills/autorag-doctor for diagnostics. Other agents read their own skill directories; copy the same setup folder there and reload the agent session so it picks up the MCP registration instructions. Do not copy or retain the removed autorag-lite-search skill.

Quick Agent Workflow

1. Install CLI:

   command -v autorag >/dev/null || bun install -g @autorag/librarian
   

2. Inspect & Preflight:

   autorag status --json        # check corpus freshness and index readiness
   autorag duplicates --json    # scan for exact or near-duplicate document families
   

3. Perform Curated Search:

   autorag search "your question" --json
   

4. Safety Guidelines for Agents:
   • Never delete, move, or modify original user documents.
   • AutoRAG writes index artifacts strictly into /.autorag/ and .jikji/ caches.
   • Never print or leak API keys, tokens, or credential values into stdout or logs.

AutoRAG Lite MCP Server

Register the package's stdio server with the host; the host owns the process lifecycle and sends MCP tool calls:

AUTORAG_CONFIG=/absolute/path/to/.autorag/config.json autorag-mcp

For Claude Code and Codex registration commands, use skills/autorag-lite-setup/SKILL.md. The MCP contract source of truth is src/mcp/server.ts (tool registration, schemas, handlers) plus src/mcp/index.ts (stdio entrypoint). Discover tools and schemas with MCP tools/list; do not hard-code a tool count.

Core Lite tools include autorag.status, autorag.search, autorag.search.files, autorag.datasources.list, autorag.datasources.get, autorag.refresh, autorag.report, and autorag.evidence. Configured integrated datasources expose additional scoped search tools. Read-only MCP mode omits mutating tools such as refresh and report.

⚡ AutoRAG Lite: Model-Free Retrieval Engine

Need blazing fast local search without configuring an LLM or paying for API tokens? Use AutoRAG Lite.

AutoRAG Lite provides the exact same high-performance indexing, BM25 ranking, and local MinSync vector/hybrid retrieval engine as the full librarian, but runs 100% model-free:

• Zero LLM Token Usage: Run purely local BM25 and vector search offline.
• Agent Integration Ready: Use the AutoRAG Lite MCP server to supply raw context chunks to an external model; the CLI remains a bootstrap and maintenance interface.
• Fast Local CLI: The CLI remains available for terminal-only indexing and repair.

# Initialize a model-free workspace
autorag lite init --search-paths ./documents

# Index local files and datasources
autorag lite refresh

Normal Lite retrieval runs through the MCP tools (autorag.status, autorag.search); the CLI remains the bootstrap and maintenance interface for indexing and repair.

🔌 Supported Datasources

AutoRAG Agent connects to external tools and communication platforms using dedicated datasource skills. Data remains in each tool's native store — AutoRAG does not copy or ingest foreign databases into a centralized store:

For configuration syntax and connector details, see docs/datasource-skills.md. Non-interactive sync/index steps get a 30-minute per-connector budget (connector.indexTimeoutMs) because first-run imports routinely take minutes; interactive search keeps its 60-second default.

Installation & Setup

AutoRAG Agent is published as @autorag/librarian (requires Node.js ≥ 24 or Bun):

# Install CLI globally
bun install -g @autorag/librarian
# or with npm:
npm install -g @autorag/librarian

# Add as a TypeScript/JavaScript library
bun add @autorag/librarian

System Prerequisites

• No Java required: Document parsing (HWP/HWPX/HWPML, PDF, DOCX, XLSX/XLS) runs in-process through kordoc.
• Rust Toolchain (Optional): Automatically compiles Jikji (jikji-cli) if installed.
• MinSync: Automatically downloaded and installed into /.autorag/bin on first run.
• Everything (Windows only, bundled): The package ships the portable Everything 1.4.1.1032 and its ES 1.1.0.38 CLI (x64 and ARM64, SHA-256 pinned). On Windows, AutoRAG extracts them into /.autorag/everything/ and runs a private, user-level instance that indexes only your configured search roots: no administrator rights, no Everything service, no whole-drive scan, no HTTP/ETP server, and no change to any Everything you already run. Set "everything": false in the config to turn it off. macOS and Linux are not affected.
• FSearch (macOS/Linux only, separate install): Install fsearch-cli yourself (e.g. brew install NomaDamas/fsearch-mac/fsearch-mac (installs the fsearch-cli binary); GPL-2.0, spawned as a separate process, never bundled or linked). On macOS and Linux, AutoRAG builds a per-workspace database at /.autorag/fsearch/ indexing only your configured search roots, and keeps a fsearch-cli watch daemon live (FSEvents/inotify) for sub-second name search. It never touches the FSearch app's own database. Without fsearch-cli, name search falls back to a slow bounded filesystem walk. Set "fsearch": false in the config to turn it off. Windows is not affected (Everything covers it).

Quick Start

1. Initialize and Index

# Initialize configuration for your documents folder
autorag init --search-paths ~/Documents/research

# Korean + English is the default; set it explicitly for other corpora
autorag init --search-paths ~/Documents/research --languages ja,en

# Index documents (parses PDFs/Markdown, builds BM25 and MinSync vectors)
autorag refresh

# Check indexing health
autorag status

Document languages and parsers

One global languages setting describes the corpus. It is resolved from --languages, then AUTORAG_LANGUAGES, then languages in the config file, falling back to ["ko", "en"]. Supported tags: ko, en, ja, zh-hans, zh-hant, fr, de, es, ru, it, pt, vi, th, ar, hi.

.bmp and .tiff are not parsed: kordoc rejects both as unsupported.

Making photos and scans searchable

Image files and scanned PDF pages become searchable through one switch, parserOptions.ocr.enabled (default false). It is off by default so indexing downloads no recognition model and never runs OCR over a photo library. With it off, image files are not parsed at all and scanned pages stay unrecognized.

{ "languages": ["ko", "en"], "parserOptions": { "ocr": { "enabled": true } } }

kordoc owns both inputs. The configured languages pick the engine:

The built-in model has no kana, han, Cyrillic, Thai, Arabic or Devanagari characters and drops diacritics in accented Latin text, so those languages need Tesseract. Set parserOptions.ocr.cachePath to store the Tesseract models elsewhere, or KORDOC_OFFLINE=1 to block the built-in model download. An explicit parserOptions.kordoc.ocr overrides the language-based choice.

Photos and videos are also reachable through two other paths that are not part of this switch: the clawgallery datasource (captions and visual search over its own folders) and Jikji's media index (jikji.enableMediaIndex, file facts such as pixel size and format only, no content). Video content is not searchable yet.

2. Search from CLI

autorag search needs one chat model. autorag init does not pick one, so on first run either sign in and choose a model inside the TUI:

autorag tui      # then /login (OAuth or API key) and /model

or add the model entry to your existing AutoRAG config, preserving its other settings. Find provider/model ids with autorag models list --available:

"model": { "provider": "", "id": "" }

Then run autorag health. Do not re-run autorag init --force just to select a model: it replaces the existing config, including datasource and parser settings. Configure the provider's credentials through /login or its supported environment variables.

# Perform a curated search (uses your configured reasoning model)
autorag search "What are our primary Q3 deliverables?"

# Launch the interactive Terminal UI (Pi host: /login, /model, /resume, …)
autorag tui

3. Programmatic Usage (TypeScript API)

import { AutoRAGAgent } from "@autorag/librarian";

// Initialize librarian agent
const agent = new AutoRAGAgent({
  searchPaths: ["/path/to/documents"],
});

// Run curated search loop
const response = await agent.searchDocuments("Summarize recent compliance updates");

console.log("Answer:", response.answer);
for (const result of response.results) {
  console.log(`[${result.number}] ${result.title} (${result.source})`);
  console.log(`    ${result.summary}`);
}

Hosted AutoRAG Plan (Dazzi)

AutoRAG can run on the hosted AutoRAG plan as a first-class model provider named autorag. The provider speaks the server's OpenAI Responses API surface (/v1), so streaming, tool calls, and usage are handled by Pi's built-in openai-responses implementation, and the model catalog is discovered live from the server — no model id is baked into AutoRAG.

Sign in from the interactive TUI:

autorag tui
/login          # choose "AutoRAG"; the browser completes a PKCE login

Headless (no browser) with a long-lived API key:

export AUTORAG_API_KEY=dz_...

On a fresh agent home, pin the model with an explicit endpoint so it resolves offline without a catalog snapshot — this is the headless configuration:

{
  "model": {
    "provider": "autorag",
    "id": "anthropic/claude-haiku-5.5",
    "baseUrl": "https://api.dazziapp.com/v1",
    "api": "openai-responses",
    "apiKeyEnv": "AUTORAG_API_KEY"
  }
}

The provider/id-only shorthand ({ "provider": "autorag", "id": "anthropic/claude-haiku-5.5" }) only resolves once a TUI /login or a model refresh has persisted the catalog snapshot; with no snapshot there is nothing to restore offline, and resolution fails with Unknown configured model: autorag/.

The catalog is persisted after a /login (or any model refresh), so the provider/id shorthand model: { provider: "autorag", id } keeps resolving offline. When you are not signed in the provider still registers — it appears under /login, contributes no models, and makes no network call. See Hosted AutoRAG provider for the full flow.

The same plan can also serve the Jev judgment model and rerank stage, so the whole librarian loop runs on the hosted account:

{
  "jev": { "backend": "autorag", "model": "jev-1.13" },
  "rerank": { "provider": "autorag", "model": "qwen/qwen3-reranker-8b" }
}

jev.backend: "autorag" posts Jev's native dialect to /v1/jev (default model jev-1.13); rerank.provider: "autorag" reranks against /v1/rerank (default model qwen/qwen3-reranker-8b). Both read the account key from AUTORAG_API_KEY, and an explicit rerank.baseUrl, rerank.apiKeyEnv, or rerank.model still overrides the defaults.

CLI Command Reference

Documentation Links

Deep dive into AutoRAG Agent's architecture, security, and integration guides:

• MinSync Setup & Embedding QA: Automatic binary installation, CDC chunking, and local embedding verification.
• Local Embedding Runtime & Gateway: AutoRAG-owned local gateway, model prefetching, and zero-egress semantic search. Model card: qwen3-embedding-0.6b.
• Hosted AutoRAG Provider: The autorag (Dazzi) model provider — PKCE /login, AUTORAG_API_KEY, AUTORAG_BASE_URL, live model discovery, and offline catalog snapshots.
• Datasource Skills Reference: Full configuration contracts, connection aliases, and connector options.
• Jev Decisions: Jev query pipeline, on by default via OpenRouter: it routes each question to a direct answer (intrinsic knowledge) or local search (everything needing more information; web search is left to the agent after the fast answer), splits multi-part questions into up to five parallel searches (openrouter/qwen/qwen3.7-flash), picks which registered datasources to search before the fast answer, and ends the run after the fast answer when that answer is complete. Also covers the jev judgment tool, backends (OpenRouter, TypeSafe, Vercel AI Gateway), and how to opt out.
• Manual QA & Datasource Test Harnesses: Real-world testing guides for Discord, KakaoTalk, Slack, Notion, and email.
• P2P SimpleX Sharing & Path Standard: Decentralized peer query sharing with SimpleX, PII redaction, and approval queues.
• Supply Chain Security & License Audits: Software bill of materials (SBOM) and dependency gate policies.
• Competitive Landscape Study: In-depth analysis of why in-place federation outperforms centralized RAG.
• Legacy AutoRAG (AutoML): Documentation for the legacy Python AutoML pipeline optimizer (pip install AutoRAG).

Contributors & Community

AutoRAG Agent is an open-source project built by the community. We welcome contributions, bug reports, datasource connectors, and ideas!

• Contributing: Feel free to open an issue or pull request. Start with CONTRIBUTING.md — development setup, the make ci check we ask for, the Signed-off-by trailer every commit needs, and when to run the live end-to-end lanes. MAINTAINERS lists who reviews which area. AI tools are allowed; unverified dumps are not — see AI_POLICY.md.
• Security: Private vulnerability reports and the response SLA live in SECURITY.md.
• GitHub Contributors: View all contributors on GitHub.

Acknowledgements

AutoRAG Agent stands on the shoulders of fantastic open-source projects:

• Pi Framework by @earendil-works — The foundational agent framework powering AutoRAG's reasoning loop.
• MinSync — Ultra-fast incremental Change Data Capture (CDC) chunking and local BM25/vector indexing.
• Jikji by NomaDamas — High-performance find-first local document discovery.
• Everything and ES by voidtools (David Carpenter) — Instant Windows file-name indexing and its command-line interface, bundled under the MIT License.
• FSearch and fsearch-cli — Everything-style file-name search for macOS/Linux (GPL-2.0-or-later). Never bundled or linked: users install fsearch-cli separately and AutoRAG spawns it as a separate process (mere aggregation per the FSF GPL FAQ).
• dupey by NomaDamas — Fast duplicate and near-duplicate document family detection.
• Federated CLI Authors: External datasource tools lazykatok, discrawl, mailcrawl, wacrawl, telecrawl, slacrawl, notcrawl, qmd, and rclone.
• kordoc by chrisryugj — HWP/HWPX/HWPML, PDF, DOCX and XLSX parsing with nested-table fidelity, used as AutoRAG's default document parser.

Troubleshooting

When search returns nothing, a datasource disappears from results, refresh hangs, or the gateway will not start, run the autorag-doctor agent skill (skills/autorag-doctor/SKILL.md). Point your coding agent at it and say "check AutoRAG" — it walks the full diagnose-and-repair procedure and ends with a per-datasource status table showing what is indexed and what actually returns hits.

The first three commands cover most of it:

autorag status --json          # freshness, per-component state, diagnostics
autorag health --json          # model resolution + one live completion probe
autorag gateway status --format json   # on-demand embedding runtime

Indexing is not the same thing as searchability, so always confirm retrieval itself through the connected Lite MCP server:

autorag.status {}
autorag.refresh {}
autorag.search {"query":"a word that certainly appears","topK":3}
autorag.search {"query":"recent topic","scope":"/discord/local/**","topK":3}

The CLI equivalents remain available for terminal repair, but do not use them as the agent's normal Lite search path.

Common failures and their fix:

Native datasource stores stay owned by their CLIs — AutoRAG never rebuilds them. Fix a broken archive with lazykatok doctor, discrawl --json metadata, slacrawl --json doctor, wacrawl --json doctor, telecrawl --json doctor, notcrawl doctor, qmd status, or mailcrawl doctor, then re-run autorag refresh --method datasources --json.

License

• AutoRAG 2.0 (AutoRAG Agent): Released under the MIT License.
• Legacy Python AutoRAG (legacy/): Released under the Apache License 2.0.
• Production third-party notices and licenses are documented in NOTICE.
• The bundled Windows binaries keep their own licenses: Everything (MIT, plus PCRE BSD-3-Clause) in licenses/everything-MIT-and-PCRE-BSD.txt and ES (MIT) in licenses/es-MIT.txt. Everything's source code is not public; AutoRAG redistributes the unmodified voidtools portable ZIPs.
• FSearch/fsearch-cli (GPL-2.0-or-later) are not distributed with AutoRAG: no code is copied, linked, or bundled. Users install fsearch-cli themselves and AutoRAG communicates with it only through its command-line interface and daemon socket — separate programs, per the FSF mere-aggregation FAQ. Source: https://github.com/NomaDamas/fsearch-mac.