Contact Us 1-800-596-4880
  1. Homepage
  2. Agent Fabric
  3. Agent Fabric Use Cases

  4. Controlling LLM Costs in Agent Fabric

Controlling LLM Costs in Agent Fabric

Agentic systems make large language model (LLM) calls constantly, and every call costs tokens. As you add agents, teams, and providers, token spend grows faster than you can track, and a single agent can run up a large bill before you notice. To manage that cost, you need to see what’s driving it, cut the spend you don’t need, and cap the spend you can’t afford.

Why LLM Costs Grow Without Agent Fabric Controls

Without visibility and enforcement, LLM costs grow as an agentic system scales. Agent Fabric cost management addresses these cost factors:

  • Unpredictable token use: Token usage varies widely based on agent behavior and queries.

  • No caller visibility: Costs can’t be attributed to the providers, models, and callers that generated them.

  • Expensive models on simple queries: Teams use expensive models for simple queries when a cheaper model can do the work.

  • Budgets that don’t enforce: Without enforcement of a budget, a number in the plan isn’t a real limit.

  • Runaway agents: A compromised or malfunctioning agent can drive unexpected cost.

  • Optimization without data: Without data on usage patterns, optimization is guesswork.

You need visibility into what drives cost, and you need controls that reduce spend and cap it.

See, Reduce, Cap, and Contain Spend

Every governed LLM call in Agent Fabric flows through Model Proxy. Omni Gateway is the runtime enforcement point for Agent Fabric, and Model Proxy runs on Omni Gateway. Model Proxy is a single access point in front of your LLM providers. You use Model Proxy to see spend, reduce spend, cap spend, and contain an anomalous agent. Usage that never reaches Model Proxy isn’t included in these controls.

Model Proxy cost management gives you the visibility and controls you need to manage LLM costs by:

  • Helping make decisions based on spend data: See token consumption and cost by provider, model, and wallet, so you act on what drives cost.

  • Knowing the owner for each overrun: Trace tokens and dollars to the wallet that drove them.

  • Lowering cost without removing capability: Route routine queries to a cheaper model, and reuse responses to similar requests.

  • Having a cap that blocks requests: A Model Wallet blocks requests when a budget is reached, so spend stops at the cap.

  • Having a fast stop for a runaway agent: When an agent behaves anomalously, block its model access. Reverse the block if the flag was a false positive.

  • Having one control plane for every provider: Apply the same controls across every provider. Adding a model or a team doesn’t mean rebuilding those controls.

How Cost Management Works in Agent Fabric

Omni Gateway is the runtime enforcement point for Agent Fabric. Model Proxy runs on Omni Gateway and provides one governed endpoint for many LLM providers. Because all model traffic that you route through the proxy passes through this endpoint, cost management works in stages: see spend, reduce it, cap it, and contain anomalies. Usage that never reaches the proxy isn’t included.

Review and Attribute Spend

First, set a cost for each model so token usage translates into dollars. The Cost Management dashboard shows token consumption and spend across governed model calls, broken down by provider, model, and wallet. You can view usage as tokens or as dollars.

Token counts come from model traffic, but dollar figures depend on the per-model costs you set. Since cost tracking starts from the time you set costs, rather than retroactively, set model costs early so spend data reflects your full usage. A model with no configured cost shows zero spend, but Model Proxy still counts its token usage.

Reduce Spend with Routing and Caching

Not every request needs your most powerful, most expensive model. Two optimizations lower the cost curve without capping usage:

  • Semantic routing compares each request to prompt topics you define and routes it to the best-matching model path. Route routine intents, such as order status or account lookups, to a fast, cost-effective model, and reserve advanced models for complex work. You can also deny-list topics to block requests you don’t want served at all.

  • Semantic caching stores model responses and reuses one when a new request is semantically similar to an earlier one, returning the cached response instead of calling the provider again. That reduces latency and cost. On chat-style traffic, semantic caching typically cuts token usage substantially.

Semantic routing chooses a model path from the request, as in this example:

Simple query: "What's the status of order 12345?"
→ Route to: fast, cost-effective model

Complex query: "Analyze this quarter's sales trends, identify anomalies, and recommend strategic adjustments"
→ Route to: powerful, higher-cost model

Cap Spend with Model Wallets

A Model Wallet identifies a caller of Model Proxy and caps that caller’s token or dollar spend against a provider over a period you set. You can apply these cost control options to a Model Wallet:

  • Enforced limits: When a wallet reaches its budget, Model Proxy blocks further requests to that provider’s models.

  • Periodic caps: Set a daily, weekly, or monthly limit that resets on a schedule, so spend stays bounded each period.

  • Flexible metrics: Set limits in dollars (USD) or tokens, whichever your team tracks.

  • Identity-scoped controls: A wallet ties usage to a caller’s identity, so each application or agent can have its own budget. Wallets rely on an identity provider that issues signed tokens to identify callers. Enforcement is approximate rather than a real-time hard cutoff. Usage can briefly exceed a limit before Model Proxy blocks further requests.

The Cost Management dashboard shows usage against each cap. Budget rows show when usage is approaching the limit and when the wallet has reached it.

Contain Runaway Agents

Budgets stop normal usage at a planned cap. A compromised or malfunctioning agent is a different problem: it can drive unexpected cost through anomalous behavior that no single budget anticipates. To address such concerns, apply these policies:

  • Rogue Agent Detection policy: Monitor agent traffic and flag risky behavior, such as prompt injection or attempts to escalate privileges.

  • Agent Kill Switch policy: Block a flagged agent’s model access when you decide to contain it.

Detection never acts on its own. You review flagged activity before taking action, and containment is fully reversible, so you can restore an agent’s access if a flag turns out to be a false positive.

Apply both policies as described in Detect and Contain Rogue Agents.

Who This Is For

Agent Fabric cost management capabilities are ideal for:

  • FinOps teams managing cloud and AI spending

  • Engineering leaders optimizing infrastructure costs

  • Product teams building cost-effective agentic features

  • Enterprises scaling agent deployments beyond proof of concept

  • Any organization where LLM costs are a significant budget item

Example Cost Management Scenarios in Agent Fabric

A large organization runs agents in several business units and can’t tell which callers and providers drive LLM cost.

Scenario 1: Cost Visibility Across Providers and Callers

Challenge: A large organization has agents deployed across multiple business units but no visibility into which callers and providers are driving LLM costs.

With Agent Fabric you get cost visibility for traffic that goes through Model Proxy.

Here’s how it works:

  1. Route LLM traffic through Model Proxy.

  2. Set a cost for each model so token usage translates into spend.

  3. Use Cost Management to view token consumption and spend by provider, model, and wallet.

Result: Token usage and spend map to the providers, models, and wallets that generated them. Leaders can see which deployments cost the most and target optimization where it pays off.

For more information, see Setting Model Costs for Spend Tracking.

Scenario 2: Semantic Routing and Caching for Customer Support

Challenge: A customer support agent system uses expensive models for all queries, including simple lookups, and repeats similar calls for common questions.

With semantic routing and caching in Agent Fabric you cut cost on routine and repeated work.

Here’s how it works:

  1. Configure semantic routing rules:

    • Simple queries (order status, account lookups) route to a cost-effective model.

    • Complex queries (troubleshooting, technical issues) route to an advanced model.

  2. Enable semantic caching so repeated, similar questions return a stored response instead of a new model call.

Result: Routine lookups move to a cheaper model path, and repeated questions are served from cache without a provider call. Average token use drops on common support traffic.

For more information about semantic routing, see Configuring Semantic Caching Services.

Scenario 3: Budget Controls for Development Teams

Challenge: Development teams experimenting with agents have no spending limits, leading to budget surprises.

With the Model Wallets feature in Agent Fabric you get budgets agents can’t quietly overrun.

Here’s how it works:

  1. Create a wallet for each team, scoped to that team’s callers.

  2. Add a budget per provider, choosing a daily, weekly, or monthly period.

  3. Set the limit in dollars (USD) or tokens, so requests are blocked when the limit is reached.

  4. Watch budget consumption in the Cost Management dashboard.

Result: A runaway experiment stops near its cap instead of surfacing as a month-end invoice. Finance gets predictable per-team spend, and teams keep the autonomy to experiment within a limit they can see.

For more information about Model Wallets, see Capping Spend for Callers of Model Proxies.

Scenario 4: Multi-Model Optimization

Challenge: An organization uses multiple LLM providers but doesn’t know which models are most cost-effective for its use cases.

With Agent Fabric you get evidence-based model selection.

Here’s how it works:

  1. Deploy agents with flexible model configuration.

  2. Use Cost Management to compare spend by provider, model, and wallet.

  3. Adjust routing so each class of request uses the model that fits its cost and complexity.

Result: Model selection becomes a decision based on observed spend, not a default. Routing can change as providers change pricing or release new models.

For more information, see Managing Costs and Token Usage.

Scenario 5: Containing a Runaway Agent

Challenge: An agent is compromised or misconfigured and begins making unusual, high-volume model calls that drive unexpected spend.

With runaway-agent containment policies in Agent Fabric you stop the runaway agent without taking down the whole system.

Here’s how it works:

  1. Apply the Rogue Agent Detection and Agent Kill Switch policies to Model Proxy.

  2. Rogue Agent Detection flags the agent’s anomalous behavior for review.

  3. Review the flagged activity, then quarantine the agent. The Agent Kill Switch policy blocks its model access.

  4. Restore the agent’s access once you’ve corrected the underlying issue.

Result: A single misbehaving agent is contained instead of running up cost until someone notices the invoice. Other agents keep working, and the block is fully reversible if the flag was a false positive.

For more information about rogue agent detection, see Detect and Contain Rogue Agents.

Implementation Steps

Follow these steps to set up cost management in Agent Fabric.

Step 1: Deploy Model Proxy

  1. Set up Model Proxy as part of your Omni Gateway deployment.

  2. Configure integration with your LLM providers.

  3. Route agent traffic through Model Proxy.

Step 2: Set Model Costs

Set the input and output token cost for each model your proxies use so token usage translates into spend. Dollar figures accrue only from the time you set costs, not retroactively, so set them early. A model with no configured cost shows zero spend, though its token usage is still counted.

For the steps to set model costs, see Setting Model Costs for Spend Tracking.

Step 3: Turn On Cost Visibility

  1. Identify each caller at the proxy to allocate traffic costs to the right wallet.

  2. Use Cost Management to view token consumption and spend by provider, model, and wallet.

Step 4: Establish Baseline Costs

  1. Monitor usage patterns for one to two weeks without optimization.

  2. Identify current cost drivers.

  3. Categorize query types by complexity and frequency.

  4. Calculate baseline cost metrics.

Step 5: Optimize with Semantic Routing and Caching

  1. Define query complexity categories as prompt topics, and assign an appropriate model to each.

  2. Create routing rules so routine requests reach a cost-effective model and complex ones reach an advanced model.

  3. Enable semantic caching so similar requests reuse a stored response instead of calling the provider again.

  4. Test routing and caching with sample queries.

Step 6: Set Budget Controls with Model Wallets

  1. Create a wallet for each caller (team, application, or agent) you want to cap. Wallets require an identity provider that issues signed tokens to identify callers.

  2. Add a budget for each provider the caller routes to, choosing a daily, weekly, or monthly period.

  3. Set the limit in dollars (USD) or tokens, so requests are blocked when the limit is reached.

  4. Watch budget consumption in the Cost Management dashboard.

Step 7: Add Runaway-Agent Containment

  1. Apply the Rogue Agent Detection policy to the proxies you want to protect, and define the anomalies to flag.

  2. Register each agent with a unique identity so flags and actions attribute correctly.

  3. Apply the Agent Kill Switch policy so that when you quarantine a flagged agent, the proxy blocks its model access.

  4. Review flagged activity before you quarantine, and restore access if a flag is a false positive.

Step 8: Monitor and Optimize

  1. Track cost trends after optimization.

  2. Refine routing rules and budgets based on results.

  3. Share cost savings and optimization wins.