Agent Reach: capability should survive provider failure

Short answer

Agent Reach is a repository that separates WHAT an agent needs to do from WHO does it: a capability is declared once (for example, send-email), and one or more providers implement it, with a preferred provider, optional fallback providers, and health status for each.

The pattern attacks a specific production failure: when a provider breaks - expired credentials, a changed API, a regional outage - the naive system's worker reports the task impossible, and the run dies. With capability-provider separation, the runtime demotes the unhealthy provider and invokes the fallback, and the run continues.

The central lesson is quotable: 'the worker cannot use X' does not prove 'X does not exist.' A worker's inability is an observation about one provider configuration, not a fact about the capability. Systems that conflate the two turn every provider hiccup into a failed task.

The failure mode: a provider outage becomes a dead task

Here is a scene that plays out in production every week. An agent is mid-task, three steps from done, and it needs to send an email. The email provider's API key expired at midnight. The agent calls the tool, gets a 401, retries twice, gets the same 401, and then writes a perfectly reasonable-sounding final message: 'I could not complete the task because email sending is unavailable.' The run ends. The task is marked failed. Somewhere in the logs, a human reads the summary and believes it.

The message is honest and wrong at the same time. Email sending was not unavailable. One provider's credentials were. The company has a second email provider configured, working credentials, and a documented fallback path. The agent could not see any of that, because the tool surface it was given was shaped like 'send email through Provider A' rather than 'send email.'

This is the failure mode the capability-provider pattern exists to prevent: the system's ability to do something is encoded as a single concrete integration, so every failure of that integration - outage, expiry, rate limit, breaking API change - is indistinguishable from the capability ceasing to exist. The agent is not lying when it reports the task impossible. The architecture lied to it first, by giving it a provider where it needed a capability.

What the repository actually implements

PRIMARY SOURCE RESULT: the Agent Reach repository (Panniantong, GitHub) implements a capability-provider abstraction for agent tooling. The core move is a registry: capabilities are declared as stable, named interfaces, and providers register as implementations of those capabilities. Each capability has a preferred provider and may have one or more fallback providers. The registry tracks provider health, so an unhealthy provider can be demoted and a fallback invoked in its place without the calling code - or the agent - learning any provider-specific details.

PRIMARY SOURCE RESULT: the repository's stated scope is the abstraction layer itself: capability naming, provider registration, preference ordering, fallback, and health. It is a small, focused codebase rather than a framework that wants to own your runtime, which matters for adoption - the pattern can be lifted into an existing system without a rewrite. We treat the repository as the primary source for the pattern, and note plainly that it is a community repository, not a peer-reviewed study with measured results; there is no benchmark in the repo demonstrating failover rates, and we did not find published evaluation numbers for it.

To be precise about what health means here, because the word is doing real work: health is an observable status per provider - reachable or not, authenticated or not, within rate limits or not - updated from invocation results and explicit checks. Health is not a judgment about the provider's output quality. A provider can be perfectly healthy and still produce bad results; that problem belongs to a different layer, and conflating the two is one of the mistakes we warn about below.

The failure taxonomy: configuration -> credentials -> adapter -> registry -> exposure -> invocation

Ernesta Labs found the most useful thing hiding in this pattern is not the failover mechanics but the discipline it forces on failure diagnosis. When a tool call fails, the question 'can the agent do X?' has at least six distinct answers, and each has a different owner and a different fix. We use this taxonomy daily, ordered from the environment inward.

Configuration failure: the environment is wrong - the tool is not installed, the wrapper is not on the path, the environment variable naming changed, the process has no network route. The capability was never reachable this run. Credentials failure: the environment is right, but authentication is broken - expired token, rotated key, revoked OAuth grant, missing scope. The provider exists and the agent's identity is the problem. Adapter failure: credentials are fine, but the translation layer broke - the provider changed its API shape, a schema drifted, a serialization bug. The provider is reachable and the glue is wrong.

Registry failure: adapter and credentials are fine, but the routing layer is wrong - the capability is not registered, the provider is not attached to it, the fallback list is empty, the health check mislabeled a good provider as bad. The system has the ability and cannot find it. Exposure failure: everything internal is fine, but the agent's tool surface never offered the capability - the tool was filtered out, denied by policy, or never wired into the run. The system can do X; the agent is not allowed to ask. Invocation failure: the agent had the tool and used it wrong - malformed parameters, wrong capability for the job, called it in a state where the operation is invalid. Everything works and the call still fails.

The point of the taxonomy is that 'the worker cannot use X' is ambiguous across all six. A 401 is a credentials failure dressed as a capability failure. A silent tool omission is an exposure failure the agent will misreport as impossibility. When your logs conflate these, your incident post-mortems conflate them too, and you fix the wrong layer.

Limitations: what this pattern does not solve

The pattern has real limits and we want them stated rather than discovered later. First, fallback only works for capabilities with more than one viable provider. Plenty of capabilities are effectively single-provider in practice - a specific SaaS with no equivalent, a piece of licensed infrastructure. The registry pattern still helps there (it makes the single point of failure explicit), but it cannot fail over, and pretending otherwise produces fallback configurations that have never run and will not work when tried.

Second, providers are not interchangeable at the semantics level. Two email providers differ in deliverability, attachment handling, template systems, and bounce semantics. A fallback that fires and loses messages-in-flight at the first provider creates a new failure mode: duplicate sends, or worse, a send recorded at provider A and a receipt generated at provider B. Fallback needs idempotency and per-provider state handling, which the abstraction encourages but cannot enforce.

Third, the repository itself is a small community project, not a study. We have no measured evidence that the pattern improves completion rates in long-horizon agents - that is an inference from the structure of the failure mode, not a result. Fourth, health checks add a moving part that can itself fail: a health-check bug that marks a healthy provider dead will route traffic to the fallback for no reason, and a health check that only tests reachability will happily route to a provider whose credentials broke two minutes ago. Health is approximate, and the design has to assume it.

Why builders care

Because provider failure is not an edge case in long-horizon work - it is a scheduled event. Runs that last hours or days will, with near-certainty, cross at least one provider hiccup: a token rotation, a rate window, a deploy-induced API change. Agent-degradation research in this same series (How Fast Do Agents Rot?) shows reliability compounding downward across dependent steps; a run that dies at step eleven because the email provider rotated a key wastes the ten steps before it. Fallback is not a luxury feature for long-horizon agents; it is what makes multi-step reliability multiplicative instead of fragile.

It also changes what your agent's failure reports mean. An agent with provider-shaped tools reports 'I cannot do X' when it means 'Provider A rejected me.' An agent with capability-shaped tools, backed by a registry, can report something far more useful: 'Capability email.send has no healthy provider; provider one failed credentials, fallback failed configuration.' That report routes to a human who can fix the actual layer. The taxonomy turns agent error messages from noise into telemetry.

And it composes with the attribution problem: when a run fails, you want to know whether your planner, your executor, or your provider was at fault. A registry that records which provider executed each capability call gives you that split for free. Credit-assignment work like CHIME depends on exactly this kind of provider-level labeling to avoid blaming the model for a provider's outage.

LABS INTERPRETATION: capability is stable; implementation is replaceable

LABS INTERPRETATION: the deep lesson of Agent Reach is an identity separation: a capability (what the system can do, named stably) and a provider (who does it, replaceable at runtime) are different objects with different lifecycles. Capabilities should outlive providers. Most agent architectures invert this - they harden the provider integration and let the capability be an emergent property of it - so the system's self-description, its failure reports, and its roadmap all inherit the provider's fragility. The registry pattern is the minimal fix: one indirection, a health signal, and an ordered fallback list. It is a small amount of machinery for a structural change in what your system can survive.

LABS RECOMMENDATION: what we would implement - and what we would not

LABS RECOMMENDATION: implement the pattern, in this order. One - inventory your tool surface as capabilities, not integrations: send-email, fetch-page, persist-record, publish-post. Name them for what they do; never name them after vendors. Two - attach at least one provider per capability, and where a second provider genuinely exists, register it as fallback with its own credentials and adapter. Three - add per-provider health derived from real invocation results (last success, last failure, failure class), not just pings. Four - on invocation failure, classify it through the taxonomy above before retrying: credentials failures should trigger fallback immediately, configuration failures should halt with a precise report, invocation failures should be surfaced to the agent as its own error to fix. Five - log which provider executed each call, so attribution between agent and provider is always answerable. Six - run the fallback path in a scheduled drill; an untested fallback is a hope, not a mechanism.

What we would not implement: automatic retry of a credentials failure against the same provider (it will fail identically, and burns the run's time budget); health checks that gate every invocation (stale health is worse than no health - cache it, use it as a hint, and let the invocation result be the final word); quality-aware routing that picks providers by output scores without a human-reviewable rubric (provider output quality is a learning problem, not a routing input); and a fallback for a capability that has no second provider - instead, mark those capabilities explicitly as single-provider in the registry so their outages are correctly reported as capability outages, which is the honest version of the same information.

The failover drill: break a provider on purpose and audit what your agent says

You can run this today against your own system in about an hour. You need one effectful capability with a real provider (email, a CRM write, a database insert) and the ability to break its credentials safely - rotate the key to an invalid value, or point it at a sandbox endpoint that returns 401.

Step one - baseline: give the agent a task that requires the capability and let it run; confirm it succeeds and note the wording of its tool calls. Step two - break the credentials and rerun the same task. Capture exactly what the agent reports: does it say 'email is unavailable' (a capability claim it cannot support) or 'provider authentication failed' (a taxonomy-labeled truth)? Step three - classify the failure yourself through the six layers: configuration, credentials, adapter, registry, exposure, invocation. If your logs cannot answer which layer failed, that is your first finding. Step four - if you have any fallback mechanism, verify whether it actually fired, and check for the two secondary failure modes: duplicate execution (both providers acted) and orphaned state (the record at provider A has no counterpart at provider B). Step five - restore the credentials and confirm recovery is detected: does the health signal flip back without a restart?

The audit question at the end is the whole point: find the sentence in your agent's failure report that claims a capability does not exist. Every such sentence, when the truth is a credentials failure, is the exact conflation this pattern removes. Count how many of your run-failure summaries from the last month contained a capability-impossibility claim, pull the logs, and check how many were actually provider-layer failures wearing a capability costume.

What remains unknown

• No published benchmark measures how much capability-provider failover improves long-horizon completion rates; the benefit is a structural inference from the failure mode, not a measured result.
• The health-check layer's failure modes are under-characterized: how often stale or shallow health signals cause unnecessary failover or mask real outages in production systems is not documented anywhere we found.
• Cross-provider semantic drift - how much a fallback provider's differing behavior (deliverability, idempotency, at-least-once semantics) silently corrupts higher-level guarantees - has no general treatment; each capability has to be audited individually.
• The repository is a small community project; how the abstraction performs at high invocation volume, with many capabilities and providers, is not evidenced in the repo or in any study we located.

Primary sources

• Agent Reach - capability-provider abstraction with health and fallback (Panniantong, GitHub repository)
• CHIME: Credit-Aware Hierarchical Memory Evolution (for provider-attribution context)