Hypergrep

Abstract

AI coding agents -- Claude Code, Cursor, Copilot, aider -- depend on text search as their primary codebase navigation tool. This creates a compounding failure: grep returns raw lines, the agent reads files to understand context, repeats 50+ times per session. Measured waste ratios reach 60-80% of all tokens consumed (Nesler, 2026).

Hypergrep is a codebase intelligence engine that unifies indexed text search, a live code graph, semantic result compression, and predictive query prefetch into a single daemon. It answers the questions agents actually ask -- "who calls this function?", "what breaks if I change this?", "does this project use Redis?" -- in microseconds to milliseconds, returning structured results that fit within token budgets.

On ripgrep's own source code (208 files, 52K lines), Hypergrep achieves 4.4ms median warm search latency (7x faster than ripgrep for repeated queries), 87% token reduction in a realistic agent investigation task (20,580 tokens reduced to 2,814), and enables query types -- call graph traversal in 2.5 microseconds, bloom filter existence checks in 291 nanoseconds -- that no text search tool can answer at any speed.

This is not a faster grep. It is a different tool for a different interaction model.

The Problem

AI agents waste most of their tokens on navigation, not on solving the actual problem.

The search-read-search loop: an agent greps for a pattern, gets 15 matching lines across 8 files (~500 tokens), reads each file for context (~8,000 tokens), reasons about relevance (~2,000 tokens), then acts on ~800 useful tokens. Total consumed: ~11,300 tokens. Useful: ~800. Waste ratio: 93%.

Nesler (2026) measured that 60-80% of tokens consumed by AI coding agents go toward figuring out where things are, not answering the actual question. A single question consumed ~12,000 tokens when the answer required ~800. The agent read 25 files to locate 3 functions.

The fundamental mismatch: agents think in tasks ("fix the auth bug") but grep answers "what lines contain this string?" Text search is a bad proxy for codebase understanding. Making grep faster does not fix this. A different tool is needed.

Novel Contributions

Six capabilities that do not exist in any other shipping tool.

Token Savings: Three Layers

Progressive disclosure -- agents start at L0 or L1 and drill down only as needed.

Example: hypergrep --layer 1 --budget 600 --json "fn search"

[
  {
    "file": "crates/searcher/src/searcher/mod.rs",
    "name": "Searcher",
    "kind": "impl",
    "line_range": [627, 828],
    "signature": "impl Searcher {",
    "tokens": 32
  },
  {
    "file": "crates/core/main.rs",
    "name": "search",
    "kind": "function",
    "line_range": [107, 151],
    "signature": "fn search(args: &HiArgs, mode: SearchMode) -> anyhow::Result<bool>",
    "calls": ["search_path", "searcher", "printer", "walk_builder", "matcher"],
    "called_by": ["search_parallel", "run", "try_main"],
    "tokens": 189
  }
]

~350 tokens. Without Hypergrep, getting this understanding requires reading main.rs (~2,000 tokens) and search.rs (~3,000 tokens). Budget fitting selects the top-ranked results that fit within the token limit using greedy selection.

Impact Analysis

BFS upstream through the call graph with severity classification.

$ hypergrep --impact "hash_password" src/

Impact analysis for 'hash_password' (depth 3):

  [depth 1] WILL BREAK   src/auth.rs:authenticate
  [depth 2] MAY BREAK    src/api.rs:login_handler
  [depth 3] REVIEW        src/main.rs:setup_routes

flowchart LR
    HP["hash_password"] --> A["authenticate"]
    A --> LH["login_handler"]
    LH --> R["router"]
    style HP fill:#2d2a24,color:#f5f0eb
    style A fill:#b91c1c,color:#fff
    style LH fill:#e8772e,color:#fff
    style R fill:#a09a90,color:#fff
      

16 Languages

Structural search + call graph for programming languages. Tree-sitter parsing for config/markup.

Structural + call graph

Parsed (text search + indexing)

Unsupported languages fall back to line-level text search (same as ripgrep).

Daemon Mode

For agent sessions with 50+ queries. Keeps the index in memory for sub-millisecond searches.

# Start in background (auto-stops after 30 min idle)
hypergrep-daemon --background /path/to/project

# Check status
hypergrep-daemon --status /path/to/project
# Running
#   PID:    18067
#   Socket: /tmp/hypergrep-f983e88f.sock
#   Memory: 8.5 MB

# Stop manually
hypergrep-daemon --stop /path/to/project

Installation

curl -sSfL https://github.com/marjoballabani/hypergrep/releases/latest/download/hypergrep-installer.sh | shCopy

git clone https://github.com/marjoballabani/hypergrep.git
cd hypergrep && ./install.shCopy

cargo build --release
cp target/release/hypergrep ~/.cargo/bin/Copy

CLI Reference

Limitations

Research and References

Full theoretical foundations, prior art analysis, and quantitative projections: RESEARCH.md