ORBIT — Operational Intelligence Benchmarking 

ORBIT is a benchmark developed by our co-founder during a research internship with IBM Research. It asks a simple question with hard answers: can an AI agent reason through a real operational incident, step by step?

IT operations problems unfold over many steps — reading telemetry, forming hypotheses, choosing actions. Single-turn benchmarks don't capture that, so it is hard to know whether an agent is actually ready for operational work.

1. 01Define multistep operational reasoning tasks grounded in realistic ITOps scenarios.
2. 02Build retrieval-grounded pipelines to generate task data tied to source material.
3. 03Design annotation workflows that keep evaluation consistent and auditable.

1. Layer 1

   Ground

   Retrieval over operational knowledge to anchor generated tasks.

2. Layer 2

   Generate

   LLM-assisted task and trajectory construction.

3. Layer 3

   Annotate

   Structured human review for correctness.

4. Layer 4

   Evaluate

   Step-level scoring of agent reasoning.

• A structured way to evaluate agents on multistep operational reasoning.
• Reusable retrieval-grounded data generation and annotation pipelines.
• Research in progress; details shared as publication allows.