Is Grep All You Need? How Agent Harnesses Reshape Agentic Search

Notes

Sen, S., Kasturi, A., Lumer, E., Gulati, A. and Subbiah, V.K. (2026) 'Is Grep All You Need? How Agent Harnesses Reshape Agentic Search', *arXiv preprint* arXiv:2605.15184 [cs.CL]. Available at: https://arxiv.org/abs/2605.15184 (Accessed: 9 June 2026). Empirical study (PricewaterhouseCoopers, U.S.) of how retrieval strategy (lexical grep vs. semantic vector), agent harness (custom Chronos vs. provider-native CLI agents Claude Code, Codex, Gemini CLI), and tool-calling delivery (inline vs. file-based/programmatic) jointly determine end-to-end accuracy on a 116-question LongMemEval-S subset, graded by a GPT-4o auxiliary grader. Experiment 1 (factorial, full haystack): inline grep beats inline vector for every harness–model pair, but programmatic file-based delivery reorders the grep–vector comparison and swapping the harness moves accuracy as much as swapping the retriever (e.g. Claude Opus 4.6 reaches 93.1% on Chronos vs. 76.7% on Claude Code). Experiment 2 (noise scaling, session limits s5–full): both retrievers degrade only mildly as distractor sessions accumulate, grep leads on average, and the grep–vector crossover depends on harness and backbone rather than corpus size alone. Five inference models: Claude Opus 4.6, Claude Haiku 4.5, GPT-5.4, Gemini 3.1 Pro, Gemini 3.1 Flash-Lite. arXiv PDF page = printed page (no offset). In scope for AI KB: agentic/autonomous LLM systems, machine-learning retrieval, evaluation; relevant to remote-operations and human-centered-design KBs (linked, not duplicated).

```bibtex
@article{sen2026grep,
  author  = {Sen, Sahil and Kasturi, Akhil and Lumer, Elias and Gulati, Anmol and Subbiah, Vamse Kumar},
  title   = {Is Grep All You Need? How Agent Harnesses Reshape Agentic Search},
  journal = {arXiv preprint arXiv:2605.15184},
  year    = {2026},
  eprint  = {2605.15184},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url     = {https://arxiv.org/abs/2605.15184}
}
```

Related Articles

Notes