DocOps

Autonomous LLM agents are moving from passive chat interfaces toward active participants in digital workspaces. Yet current document evaluations remain constrained by two dominant paradigms: (1) static document understanding, which treats documents as read-only repositories for extraction or question answering, and (2) workflow-oriented software evaluation, which treats the document as a passive payload moved between applications.

Consequently, it remains an open research question whether current agents can reliably execute end-to-end document-centric tasks while maintaining global document state consistency, and achieving user objectives without introducing destructive modifications.

To this end, we introduce DocOps, a rigorously verifiable evaluation framework designed to systematically assess LLM agents on complex end-to-end document manipulation. A primary contribution of this work is the formulation of a principled taxonomy that deconstructs the operational black box of document manipulation.

As illustrated in the overview figure, our taxonomy maps the operational space across two orthogonal axes: atomic capabilities and workflow depth. First, we isolate document interaction into three interdependent dimensions: content, format, and structure. Second, we establish a four-tiered complexity gradient: L1/L2 isolate single- and multi-dimensional atomic edits, while L3/L4 escalate to long-horizon, cross-document workflows inspired by real-world interaction scenarios.

This hierarchical taxonomy serves not merely as a categorization scheme, but as a fine-grained agentic diagnostic framework, enabling researchers to localize whether failures stem from localized structural disruption or a collapse in long-term state tracking. Each task is paired with a deterministic native-document verifier that checks target postconditions, native validity, and preservation constraints on the final artifact.