Skip to main content
Observation is the default. Passing apply=True opts a wrapped client in to Visceral’s cache-layout optimizations — request rewrites the backend has already proven safe and profitable for your traffic:
Providers with apply support today: Anthropic and OpenAI. On other providers the flag is ignored.

​
What gets applied

Visceral analyzes your traffic server-side and mints rules only where the same prompt structure recurs enough to prove the rewrite pays for itself. The SDK fetches your workspace’s active rules and applies the ones that match each outbound request:

Anthropic — cache breakpoints

Inserts cache_control markers at proven-stable prefix boundaries so Anthropic’s prompt cache actually hits. Markers are billing metadata — the tokens the model sees are identical.

OpenAI — cache routing

Sets prompt_cache_key on requests whose prefix matches a proven-stable bucket, so repeated prefixes land on the same cache shard instead of missing.
Both rewrites are output-neutral by construction: they change how the provider caches and bills the request, never the tokens the model reads or the response your agent receives.

​
How rules reach the SDK

  • On wrap(..., apply=True) the SDK fetches your workspace’s rules once, then refreshes them in a background thread every 60 seconds (VISCERAL_APPLY_RULES_TTL_SECONDS).
  • The hot path reads an in-memory snapshot — applying rules adds no network call and no lock to your LLM requests.
  • Rules carry a schema version; an SDK never applies a rule version it doesn’t understand.
  • Every applied rewrite is annotated on the trace (visceral.apply.decision_id), so you can see exactly which optimization touched which call.

​
Guardrails

The rewriter is deliberately conservative:
  • Your request objects are never mutated — rewrites happen on a copy, and only the provider sees them.
  • A caller-supplied prompt_cache_key or existing cache_control markers are respected, never overridden. Anthropic’s four-breakpoint budget is honored, counting any markers you already set.
  • prompt_cache_key is only sent to api.openai.com itself — OpenAI-compatible providers that would reject the parameter are left alone. If OpenAI ever rejects it anyway, the SDK retries the call without it.
  • Rules can be withdrawn centrally at any time; a fetch failure keeps the last known-good snapshot and never touches your call.
  • Any error while rewriting relays the original request unchanged. Apply is fail-open end to end.

​
Measuring the win

Realized savings show up in the dashboard and in GET /v1/workspaces/{id}/stats/summary as estimated_cache_savings_usd, alongside cached-token counts per model. Where Visceral finds waste it cannot prove safe to auto-fix — a prefix that almost caches but is broken by a volatile block, say — it files a finding instead of touching your traffic.