How reliable are autonomous agents across complete workflows?

Interactive benchmarks, repeated success, verification and the limits of transfer to marketing

This retrospective review considers source versions available by 24 July 2026. The cutoff defines the evidence included; preparation and source checking took place on 21 October 2026.

Keywords: agents; workflow reliability; benchmarks; tool use; verification; marketing

1. Autonomy makes reliability a system question

A language model that proposes an action and a system that carries it out have different failure surfaces. The latter must interpret the goal, inspect the environment, select a tool, provide valid arguments and determine whether the action succeeded. Its effect persists outside the generated text. A fluent account of completion can therefore diverge from the actual state of the task.

This review asks what agent research establishes about the reliability of multi-step workflows and what follows for marketing applications. It focuses on task completion, repeated success, verification and action boundaries. It does not rank current agent products or claim that historical benchmark scores measure the capabilities available in October 2026. The enduring question is how an evaluation connects a model's local decisions to the outcome of a complete process.

A campaign workflow can include research, drafting, asset adaptation and updates to external systems. These steps differ in observability and consequence. An internal summary can be reviewed before use; an external change may affect customers or expenditure immediately. Treating all steps as a single level of autonomy conceals that variation. An assessment needs to specify which states can change, which actions are reversible and which outcomes count as successful.

The distinction also affects the intervention being studied. A model with a calculator, a constrained database API or unrestricted computer access is not the same system. Tool descriptions, permissions and validation code can alter performance even when the underlying model is unchanged. Claims about an agent therefore need to identify the surrounding configuration that turns generated decisions into actions.

2. Review method and evidence boundaries

Targeted searches on 21 October 2026 located 20 primary research and institutional sources on reasoning-and-action architectures, tool use, interactive benchmarks, security and risk management. The τ-bench manuscript was inspected for environment construction, reward definition, repeated-trial metrics and reported results. Other sources were consulted through author abstracts, official benchmark descriptions and institutional guidance. This is a purposive narrative review without independent screening, replication or statistical pooling.

The evidence spans different tasks and system generations. Code repair, web navigation, desktop operation and simulated customer service measure different behaviours. Their success rates cannot be combined into an estimate of general agent reliability. The review retains the task and evaluation unit behind each claim. A result from a 2024 benchmark is a historical observation under that configuration, not a present-day ceiling or guarantee.

Engineering guidance is distinguished from controlled research. A developer's architectural recommendations can describe useful design choices without estimating their average effect across deployments. Similarly, security benchmarks expose classes of failure under test conditions; they do not directly estimate incident rates in a particular organisation. Transfer to marketing requires evaluating the actual tools, data, permissions and outcome criteria of the intended workflow.

3. Reasoning, tools and control over the next step

ReAct interleaves model-generated reasoning with task-specific actions, allowing the system to obtain information from an environment and revise its course [1]. Toolformer studies a model trained to decide when and how to use external APIs [2]. Both demonstrate ways to connect language processing with operations beyond text generation. They address mechanisms of tool use, rather than proving that an arbitrary sequence of actions is dependable.

Anthropic's engineering guidance distinguishes predefined workflows from agents that dynamically direct their own process and tool use [11]. The distinction concerns control over the path. A fixed workflow can contain model calls, and an autonomous agent can operate inside tightly constrained tools. Neither label alone describes the range of possible side effects or the standard of evidence used to accept completion.

Control can be distributed across components. A model may choose a tool, while application code checks the arguments and a human approves the resulting action. An evaluation that reports only the model name leaves these contributions implicit. The observed result belongs to the combination. Changing the approval arrangement or validation layer changes the system whose reliability is being measured.

Dynamic planning has a different resource profile from a fixed sequence. It can adapt to information that was not known in advance, but it also creates additional opportunities to choose an unnecessary or incorrect step. A simpler system may be adequate when the task path is known. A more flexible system may be necessary when the environment changes. The appropriate comparison concerns accepted outcomes and total resource use under the task's actual variation.

4. Interactive benchmarks change what is tested

AgentBench evaluates reasoning and decision-making across eight interactive environments [3]. WebArena supplies realistic, reproducible websites and tasks evaluated for functional completion [4]. These designs extend assessment beyond producing a plausible next answer. They require a system to navigate a stateful environment where earlier actions affect later possibilities.

OSWorld uses real computer environments, with initial-state configurations and execution-based evaluation scripts across desktop and web applications [5]. SWE-bench asks systems to edit codebases to resolve real GitHub issues [6]. Their contribution to this review is the emphasis on outcomes external to the model's explanation. A claimed repair can be evaluated against executable tests, and a claimed computer action can be evaluated against resulting state.

Benchmark realism has dimensions. A task can use an actual application while simplifying the surrounding organisation. It can contain realistic data while replacing a human user with a simulator. It can reproduce the interface while fixing the initial state so every trial starts consistently. These choices make comparison possible, but each changes the boundary of what the result represents.

The benchmark's evaluator is part of the measurement system. A narrow state check may miss an unintended change elsewhere; a broad outcome check may reject a legitimate alternative route. The target should reflect the task's valid outcomes, and the evaluator's coverage should be understood before a percentage is described as reliability. A reproducible number can still measure an incomplete success criterion.

5. What the τ-bench design adds

τ-bench combines simulated users, domain policies, API tools and databases in retail and airline settings. Its reward checks the final database state and required information in the response. The manuscript acknowledges that passing this check can still miss a policy violation, such as acting without required confirmation. Trials use a limit of 30 actions, with at least three runs per task for the main results [7].

The study distinguishes pass@k, where at least one of k trials succeeds, from pass^k, where all k succeed. These quantify solution discovery and consistency respectively. Its reported GPT-4o function-calling results are 61.2% for retail and 35.2% for airline, with a domain-weighted average of 48.2% [7]. The comparison demonstrates why the task domain and repeated-trial definition belong beside a success figure.

A simulated user is an experimental component. It creates controlled variation in dialogue, while also bringing assumptions about how a person states goals, supplies information and responds to the agent. Results from that interaction should retain the simulator and task design in their interpretation. Real users can have different knowledge, ambiguity or incentives from those encoded in a benchmark.

The policy requirement also exposes a general distinction between an acceptable end state and an acceptable process. A system can reach the requested final state through a route that exceeds its authority. Where process constraints matter, evaluating only the end state leaves an important part of the task unobserved. The success definition needs to reflect both the outcome and the constraints that govern how it may be produced.

6. Average success, repeated success and composition

Reliability has more than one denominator. A percentage can describe successful tasks, successful attempts, successful tool calls or users whose entire request was completed. These units produce different quantities. A system may make many correct tool calls while failing the final task, or complete most tasks while repeatedly failing a narrow class of requests. The intended decision determines which quantity is relevant.

Repeated attempts answer a different question from repeated dependable service. If an application can generate several candidates and cheaply verify them, the chance of finding at least one valid result may be useful. If it must handle every customer interaction correctly, failures across repeated uses become central. Reporting only the best run selects a different outcome from reporting typical or repeated performance.

For a sequence of required events, the probability that all succeed follows the ordinary probability chain rule. It depends on the probability of each event conditional on the preceding events. Multiplying unconditional step-success rates assumes independence that a real workflow may not have. A wrong early interpretation can affect several later steps, while a shared tool outage can make multiple operations fail together.

This distinction prevents a common numerical shortcut. A local test of one operation does not supply the joint success probability of an entire campaign process. The analyst needs either direct end-to-end observations or a justified model of dependence. A conceptual multiplication of convenient percentages can give a precise-looking answer without measuring the actual system.

Recovery changes the sequence as well. A failed operation followed by a successful repair can produce an acceptable final outcome while consuming additional time and tools. Counting only the final result hides recovery cost; counting every initial failure as a final failure understates the value of repair. Both the outcome and the path can be relevant, provided their definitions remain separate.

7. Verification and the difference between messages and state

An agent's final message is an observation of what it says, not direct evidence of what an external system contains. A tool can return an error, accept a request asynchronously or update only part of the intended record. Verification should therefore use evidence appropriate to the action. For a state-changing task, this often means inspecting the resulting state or an authoritative completion signal.

The choice of verifier depends on the outcome. Code can sometimes be checked with executable tests; a database update can be compared with an intended record; a visual asset can be inspected against a design specification. Strategic suitability and audience response require other forms of evidence. Passing an executable format check cannot establish the quality of a marketing idea, just as a favourable preference rating cannot establish that a database update occurred.

Verification also requires independence from the error being checked. Asking the same reasoning process to restate its confidence may reproduce the original mistake. A second model can provide a different assessment but may share training patterns or source omissions. An external specification, deterministic constraint or independent measurement can supply information absent from the original generation. The strongest arrangement depends on what can be observed reliably.

Evidence should distinguish initiation from completion. A request accepted by a service may still fail during processing. A workflow that needs the completed result must observe the later state before reporting success. Where the final state cannot be checked, the report should describe the action that is known to have occurred. This is a measurement boundary, not a stylistic preference for cautious language.

The cost of verification belongs inside the workflow. Some tasks are easy to generate and expensive to check. Others have clear machine-readable acceptance tests. This asymmetry affects the value of autonomous operation. A benchmark whose verifier is cheap and exact can support conclusions that transfer only partly to work where acceptance depends on substantial human judgement.

8. Untrusted content creates an additional failure surface

AgentDojo evaluates agents executing tools over untrusted data and includes adaptive attacks and defences. The author manuscript describes both ordinary task failures and security-property failures under prompt injection [8]. InjecAgent similarly tests malicious instructions embedded in external content, including attempts at harmful actions and data exfiltration [9]. These are primary demonstrations of a risk introduced when external material can influence tool-using behaviour.

The relevant distinction is between information needed for the task and authority to change the task. A customer record, web page or retrieved document can contain text resembling an instruction. Its presence in the evidence stream does not make it an authorised request. An agent must use the material as data while preserving the authority structure of the workflow.

For marketing, external content may include reviews, competitor pages, supplier documents and messages. Their relevance can be high even when their trustworthiness is limited. Restricting a workflow to trusted documents can reduce one exposure while also reducing its information coverage. Evaluating that trade-off requires both task-utility and adversarial tests; a secure but unusable system and a useful but easily redirected system fail different requirements.

Security results are conditional on the attack model and allowed actions. A successful attack against one configuration does not establish that every configuration fails in the same way. Conversely, the absence of a successful attack in a finite test set does not establish immunity. Reports should identify the attacker's access, the targeted property and the actions available to the agent.

9. Permissions determine the consequence of a mistake

A model can produce an incorrect recommendation even when it has no permission to act. Granting write or execution access changes what that error can cause. The permission boundary is therefore part of the system's risk profile, separate from its answer accuracy. A system with imperfect recommendations can remain useful in a workflow where consequential actions require effective review.

The principle is contextual rather than absolute. Requiring a person to approve every harmless action can make a process unusable, while allowing consequential changes without a suitable check can expose the organisation to avoidable loss. The appropriate action boundary depends on reversibility, magnitude, observability and the quality of the available verification. The reviewed literature supplies evaluation concerns, not a universal approval schedule.

Permissions also affect test interpretation. An agent evaluated with read-only access has not demonstrated reliability in making external changes. A system evaluated in a simulation has not encountered every failure of a live service. Expanding its authority should therefore be treated as a new configuration requiring evidence appropriate to the expanded effects.

NIST's AI Risk Management Framework situates such questions within intended use and lifecycle risk management [12]. It is voluntary guidance rather than a certificate of agent safety. Its relevance here is the connection between capability testing and the conditions under which capability is deployed. The same measured error can have different significance when the permitted action and consequence change.

10. Recovery, reversibility and incomplete tasks

A multi-step task can fail after making useful partial progress. It can also leave a state that makes a naive restart harmful, such as duplicating a submission. Reliable continuation therefore requires knowing what has already happened. The system needs observations of completed actions, pending actions and rejected actions, rather than relying only on an intended plan.

Reversibility changes the options after failure. A draft can often be discarded; an external communication may already have been received; a budget change may have begun affecting delivery. A workflow's recovery method should be evaluated against the actual side effects. The ability to undo one record does not establish that all consequences have been undone.

Idempotent operations, where repeating the same request does not create an additional effect, can simplify recovery when the service supports them. Where they are unavailable, the system needs a way to inspect whether the first request succeeded before retrying. This is ordinary software-reliability reasoning applied to a model-directed process. It does not depend on assuming that the model can infer the state from an ambiguous response.

Escalation is another possible successful outcome. If the task specification permits a system to stop and ask for information when the evidence is insufficient, doing so can be correct. A benchmark that counts every escalation as failure measures a different objective from a service designed to preserve accuracy by escalating difficult cases. The evaluation should state whether success means autonomous completion, appropriate resolution or an acceptable mix of automation and human handling.

The reporting burden should match that definition. A completion rate that excludes escalated tasks can overstate coverage, while an accuracy rate among completed tasks can remain high through selective abstention. Both may be informative if the denominator is clear. Coverage, correctness and escalation workload should remain visible when evaluating the complete service.

11. Time horizons, resources and the meaning of long tasks

Kwa and colleagues propose the 50%-task-completion time horizon: the human time associated with tasks that an AI system completes with a specified success probability. Their research concerns software tasks and introduces a way to relate benchmark capability to task length [10]. The metric does not measure how long an agent can run continuously without error or how long a marketing campaign can operate unattended.

Human duration and agent duration are different variables. A task that takes an expert a long time may be easy for a system with a particular tool; a short human task may depend on contextual judgement that is difficult for the model. A time-horizon measure describes the task distribution and reference humans used to construct it. Generalisation to another profession needs evidence about that profession's tasks.

Resource limits affect observed success. More attempts, a larger context, additional tools or more expensive reasoning can improve the chance of completion while changing cost and latency. A reliability comparison that omits those resources may reward a configuration unavailable under real operating constraints. The relevant system description includes its budget, action limits and stopping rule.

Long-running workflows also encounter changing state. A source document can be updated, a service can become unavailable or another person can modify the same record. A plan based on an earlier observation may then be stale. Evaluation should investigate whether the system rechecks the state at consequential points, rather than assuming that accurate initial information remains accurate throughout the task.

12. Transfer to marketing workflows

Agent benchmarks establish that tool-using systems can be evaluated against outcomes beyond their own explanations. Applying the approach to marketing requires a task specification grounded in the work itself. A workflow for preparing a research summary has different acceptance criteria from one updating a live campaign. The relevant source material, allowed actions and completion evidence should follow from that distinction.

A campaign also spans multiple kinds of outcome. Research can be factually supported, assets can meet their technical specification and media settings can match an authorised plan, while the campaign still produces little incremental response. Technical workflow reliability and marketing effectiveness remain separate empirical questions. Reliable execution of a weak intervention does not establish that the intervention is useful.

Human review can alter both reliability and resource use. Its contribution should be observed rather than assumed. A review stage may catch substantive failures, merely approve fluent material or become overloaded as output volume grows. A comparison of full workflows can measure accepted results and total effort under each arrangement. Evaluating only the agent's isolated output leaves that interaction unresolved.

The evidence needed for rollout should reflect ordinary work as well as stress cases. Routine cases reveal operational performance under the sampled workload. Difficult cases and adversarial inputs investigate failure modes. Combining them in a single undifferentiated success rate hides which distribution the figure represents. A clear account preserves both the workload estimate and the diagnostic tests.

There is no requirement that every task be autonomous for the system to have value. A reliable hybrid process can combine automated retrieval, constrained drafting, deterministic checks and human selection. Its value is an empirical question about the resulting work and resources. Describing a process as agentic adds no evidence about that value without the outcomes that define it.

12.1 Failure accounting and the validity of the test

A useful failure account identifies the stage at which an intended task ceased to meet its requirements. An incorrect goal interpretation, a wrong tool argument, an unavailable service and a misleading final report can all produce an unsuccessful task, but they imply different remedies. Logging only the final score discards this information. Recording the observed transition and the acceptance rule allows the analyst to separate a model decision from a tool or environment failure without assuming that either category is harmless.

The initial state is equally important. A task can become easier if a previous trial leaves a partially completed record, a cached answer or a prepared file. Repeated runs should restore the state required by the comparison or explicitly treat continuation as part of the intervention. Otherwise apparent improvement can reflect a changed problem. Interactive benchmarks use controlled environments partly to make these state differences observable [4], [5]. A live workflow requires an equivalent account of what was present before the agent acted.

Test exposure is another boundary. If a system or its builders repeatedly optimise against the same tasks, performance can improve on that set while providing less information about unseen work. A held-out set can investigate generalisation, but its relevance still depends on the workload it represents. Stable performance on familiar tasks and reliable adaptation to new tasks are distinct properties. Both can matter operationally, and the evaluation should identify which one a result measures.

Finally, the treatment of ambiguous requests should be specified. Some tasks have several acceptable outcomes; others require clarification before an authorised outcome exists. Forcing an agent to choose one answer can penalise appropriate uncertainty or reward unsupported assumptions. The task definition should make room for clarification where clarification is part of correct service. This preserves the difference between failing to complete an answer and correctly declining to invent missing requirements.

CheckList illustrates behavioural tests that isolate failures hidden by an overall score. [13] Model cards document the conditions and uses attached to a capability claim. [14] FActScore evaluates factual precision at the level of atomic claims, a different endpoint from completing an external action. [15]

Citation-generation research separates source support from fluent answer production. [16] Technical-debt research places models within wider dependency systems rather than treating a model call as the complete application. [17] The ML Test Score organises production tests and monitoring, complementing benchmark success with operational evidence. [18]

Underspecification research shows why similar held-out performance need not imply similar behaviour after deployment. [19] The automation taxonomy separates acquisition, analysis, decision and implementation. An agent can be reliable at one of these functions while failing at another. [20]

13. Conclusions and limitations

The reviewed literature treats agents as interactive systems whose reliability depends on tools, environment, evaluation and action boundaries. Historical benchmarks show the importance of end states, repeated trials and domain policies. Security studies add the possibility that untrusted material redirects tool use. These findings justify evaluating complete configured workflows rather than inferring reliability from fluent reasoning or isolated model scores.

The review is selective and does not reproduce benchmark results or audit deployment code. Most source access was limited to abstracts and official descriptions; the τ-bench manuscript supplied additional method detail. No general current success rate is estimated, and no benchmark is treated as a direct test of an end-to-end marketing campaign.

A defensible reliability claim states the task distribution, configuration, resource limits and completion rule. It also identifies how errors are detected, what recovery costs and whether constraints hold throughout the path. Those observations make it possible to distinguish an agent that sometimes finds a successful route from a workflow that repeatedly delivers acceptable work under its intended operating conditions.

Source access and appraisal register

The additional comparative sources in references 13–20 are used for the specific distinctions cited in the discussion. Access is limited to previously inspected primary-source abstracts, publication records or manuscripts. Their inclusion does not imply a fresh full-text risk-of-bias appraisal or direct validation in the application discussed here.

The 12 references below were located through targeted title and topic searches on 21 October 2026. Author or publisher abstracts, official research summaries and publication records were consulted. The discussion uses the claims available in those materials; full manuscripts, supplementary analyses and underlying data were not independently appraised. Institutional guidance is treated as guidance, rather than a controlled estimate of effectiveness. Where a working paper is cited, its provisional status remains part of the interpretation. The evidence map is selective and does not estimate the prevalence of results in the complete literature.

The τ-bench full manuscript was inspected for its simulated-user environment, state/output reward, confirmation-policy limitation, action cap, repeated-trial definitions and Table 2. Other sources remain at author-abstract, official-description or guidance level. Benchmark results are historical and configuration-specific.

References