Notice: The Weights & Biases domain will change on September 30. This includes all websites and services that use a W&B domain. No action is needed today.
Reduce latency for repeated prompts with prefix caching, and isolate cache reuse with cache_salt when needed.
W&B Inference uses prefix caching on supported hosted models to speed up repeated requests with identical prompt prefixes.When a request shares the same prompt prefix as an earlier request on the same backend, the model reuses the previously computed key-value (KV) cache instead of recomputing the entire prefix. This reduces latency for repeated prompts, long system prompts, and workloads with a stable shared prefix.Prefix caching is automatic on supported models, so you don’t need to enable it in your request. This page describes when prefix caching is most effective and how to control cache isolation with cache_salt.
When prefix caching helps
Prefix caching is most useful when you repeatedly send requests that share a long common prefix, such as:
A large system prompt reused across many requests.
A long shared document followed by different user questions.
Repeated evaluation prompts with only small per-request changes.
Multi-turn workloads where much of the conversation history stays the same.
Cache isolation
In some environments, you might need to prevent cache reuse across different users or applications. The cache_salt parameter provides this control.By default, requests with identical prompt prefixes may reuse cache on shared infrastructure when the backend allows it.To isolate cache reuse to a specific trust boundary, set the cache_salt request parameter. Requests only reuse prefix cache when both the prompt prefix and the cache_salt match.Use cache_salt when you want cache reuse within a single user, tenant, session, or application boundary, but don’t want reuse across other callers.
How it works
The presence and value of cache_salt affect cache reuse as follows:
Same prompt prefix, no cache_salt: cache may be reused across matching requests.
Same prompt prefix, same cache_salt: cache can be reused.
Same prompt prefix, different cache_salt: cache is isolated and not reused across salts.
cache_salt must be a non-empty string when provided.
Examples
The following examples show how to send a chat completion request with cache_salt to isolate cache reuse.
Python
Bash
import openaiclient = openai.OpenAI( base_url="https://api.inference.wandb.ai/v1", api_key="<your-api-key>",)response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=[ { "role": "system", "content": "You are a careful assistant that answers concisely." }, { "role": "user", "content": "Summarize this document in one sentence: <long shared prefix here>" }, ], extra_body={ "cache_salt": "tenant-a-user-123-secret", },)print(response.choices[0].message.content)
curl https://api.inference.wandb.ai/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer <your-api-key>" \ -d '{ "model": "moonshotai/Kimi-K2.5", "messages": [ { "role": "system", "content": "You are a careful assistant that answers concisely." }, { "role": "user", "content": "Summarize this document in one sentence: <long shared prefix here>" } ], "cache_salt": "tenant-a-user-123-secret" }'
Response behavior
To confirm that prefix caching is active for a request, check the response usage details. On some models, usage details may include cached token counts in usage.prompt_tokens_details.cached_tokens when prefix cache is reused. The availability of that field varies by model and backend.
{"docsConfig":{"theme":"aspen","$schema":"https://mintlify.com/docs.json","name":"Weights \u0026 Biases Documentation","colors":{"primary":"#FCBC32","light":"#FCBC32","dark":"#FCBC32"},"logo":{"light":"https://mintcdn.com/wb-21fd5541/C7PMWo9Jl9GZ2Zki/icons/Endorsed_primary_blackwhite.svg?fit=max\u0026auto=format\u0026n=C7PMWo9Jl9GZ2Zki\u0026q=85\u0026s=a9d999235518399090dd8ddd0a9e1e4a","dark":"https://mintcdn.com/wb-21fd5541/C7PMWo9Jl9GZ2Zki/icons/Endorsed_primary_goldwhite.svg?fit=max\u0026auto=format\u0026n=C7PMWo9Jl9GZ2Zki\u0026q=85\u0026s=f4eaf37e934c12e5fd3e806fe2984c2e","href":"https://docs.wandb.ai/"},"favicon":"https://mintcdn.com/wb-21fd5541/HesivM0BUKMK8VeH/favicon.png?fit=max\u0026auto=format\u0026n=HesivM0BUKMK8VeH\u0026q=85\u0026s=0c96f60ed4edd2ece5e942ecde3c11fa","background":{"decoration":"gradient"},"navbar":{"links":[{"href":"https://app.wandb.ai/login?_gl=1*8ninq0*_ga*MTE3ODEwNDkyLjE3MzQwMjk0NjA.*_ga_JH1SJHJQXJ*MTczNDA2Mzc4OS4yLjEuMTczNDA2MzkxMC42MC4wLjA.*_ga_GMYDGNGKDT*MTczNDA2Mzc4OS4zLjEuMTczNDA2Mzg2Ny4wLjAuMA..*_gcl_au*MTc0Mjk3ODgzMi4xNzM0MDI5NDYw","label":"Log in"}],"primary":{"type":"button","label":"Sign Up","href":"https://app.wandb.ai/login?signup=true\u0026_gl=1*y15ckv*_ga*MTE3ODEwNDkyLjE3MzQwMjk0NjA.*_ga_JH1SJHJQXJ*MTczNDA2Mzc4OS4yLjEuMTczNDA2Mzk3MC42MC4wLjA.*_ga_GMYDGNGKDT*MTczNDA2Mzc4OS4zLjEuMTczNDA2Mzg2Ny4wLjAuMA..*_gcl_au*MTc0Mjk3ODgzMi4xNzM0MDI5NDYw"}},"footer":{"socials":{"discord":"https://discord.com/invite/RgB8CPk2ce","github":"https://github.com/wandb/docs","linkedin":"https://www.linkedin.com/company/wandb","x":"https://x.com/weights_biases?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor","youtube":"https://www.youtube.com/c/WeightsBiases"}},"seo":{"metatags":{"canonical":"https://docs.wandb.ai"},"indexing":"all"},"fonts":{"heading":{"family":"Source Serif 4","weight":400},"body":{"family":"Source Sans 3","weight":400}},"styling":{"eyebrows":"breadcrumbs","latex":true},"integrations":{"ga4":{"measurementId":"G-5JYCHZZP7K"},"gtm":{"tagId":"GTM-5BL5RTH"}},"banner":{"content":"**Notice:** The Weights \u0026 Biases domain [will change on September 30](https://wandb.ai/site/domain-update-faqs/). This includes all websites and services that use a W\u0026B domain. No action is needed today.","dismissible":true,"type":"warning"},"contextual":{"options":[{"title":"Install W\u0026B MCP","description":"Set up the W\u0026B MCP server in Claude Code, Codex, VS Code and or other LLM agents.","icon":"plug","href":"/platform/mcp-server"},{"title":"Install W\u0026B MCP in Cursor","description":"Automatically install W\u0026B MCP server to Cursor. Requires a W\u0026B API key.","icon":"plug","href":"cursor://anysphere.cursor-deeplink/mcp/install?name=wandb\u0026config=eyJ0cmFuc3BvcnQiOiJodHRwIiwidXJsIjoiaHR0cHM6Ly9tY3Aud2l0aHdhbmRiLmNvbS9tY3AiLCJoZWFkZXJzIjp7IkF1dGhvcml6YXRpb24iOiJCZWFyZXIgWU9VUl9XQU5EQl9BUElfS0VZIiwiQWNjZXB0IjoiYXBwbGljYXRpb24vanNvbiwgdGV4dC9ldmVudC1zdHJlYW0ifX0="},"chatgpt","claude","copy","view"],"display":"header"},"navigation":{"languages":[{"language":"en","href":""},{"language":"fr","href":"fr"},{"language":"ja","href":"ja"},{"language":"ko","href":"ko"}]}},"deployment":{"deploymentMetadataValue":{"subdomain":"wb-21fd5541","hasAudioNarration":false,"actualSubdomain":"wb-21fd5541","customDomains":["docs.wandb.ai"],"gitSource":{"type":"github","owner":"wandb","repo":"docs","deployBranch":"main","contentDirectory":"","isPrivate":false,"mountPath":"","label":""},"feedback":{"thumbs":true,"edits":true,"issues":true,"agentFeedback":true},"entitlements":{"AI_CHAT":{"status":"ENABLED"},"REMOVE_BRANDING":{"status":"ENABLED"},"CONTEXTUAL_FEEDBACK":{"status":"ENABLED"},"CODE_SNIPPET_FEEDBACK":{"status":"ENABLED"},"WEB_MCP":{"status":"ENABLED"},"NARRATION":{"status":"ENABLED"}},"buildId":"6abd16e4f7750a5210d47e77","clientVersion":"0.0.4271","searchFilterCounts":[{"language":"en","tag":"Products","count":420},{"language":"fr","tag":"Produits","count":420},{"language":"ja","tag":"プロダクト","count":420},{"language":"ko","tag":"제품","count":420},{"language":"en","tag":"Reference","count":397},{"language":"fr","tag":"Référence","count":397},{"language":"ja","tag":"リファレンス","count":397},{"language":"ko","tag":"레퍼런스","count":397},{"language":"en","tag":"W\u0026B Weave","count":356},{"language":"fr","tag":"W\u0026B Weave","count":356},{"language":"ja","tag":"W\u0026B Weave","count":356},{"language":"ko","tag":"W\u0026B Weave","count":356},{"language":"en","tag":"W\u0026B Models","count":333},{"language":"fr","tag":"W\u0026B Models","count":333},{"language":"ja","tag":"W\u0026B Models","count":333},{"language":"ko","tag":"W\u0026B Models","count":333},{"language":"en","tag":"Guides","count":259},{"language":"fr","tag":"Guides","count":259},{"language":"ja","tag":"ガイド","count":259},{"language":"ko","tag":"가이드","count":259},{"language":"en","tag":"Service API","count":113},{"language":"fr","tag":"API de service","count":113},{"language":"ja","tag":"サービス API","count":113},{"language":"ko","tag":"서비스 API","count":113},{"language":"en","tag":"Python","count":92},{"language":"fr","tag":"Python","count":92},{"language":"ja","tag":"Python","count":92},{"language":"ko","tag":"Python","count":92},{"language":"en","tag":"TypeScript SDK","count":88},{"language":"fr","tag":"SDK TypeScript","count":88},{"language":"ja","tag":"TypeScript SDK","count":88},{"language":"ko","tag":"TypeScript SDK","count":88},{"language":"en","tag":"Trace functions with Ops and Calls","count":64},{"language":"fr","tag":"Tracer des fonctions avec des Ops et des Calls","count":64},{"language":"ja","tag":"Ops と Calls を使用して関数をトレースする","count":64},{"language":"ko","tag":"Ops와 Calls로 함수 트레이스하기","count":64},{"language":"en","tag":"Docs Home","count":62},{"language":"fr","tag":"Accueil de la documentation","count":62},{"language":"ja","tag":"ドキュメント ホーム","count":62},{"language":"ko","tag":"문서 홈","count":62}],"searchSettings":{"maxResults":25}},"assistantConfig":{"enableAskAiButton":true,"enableAskAiSearchBar":false,"deflection":{"enabled":true,"email":"support@wandb.com"}},"basePath":"","hasChatPermissions":false,"hasWebMcpPermissions":true,"toggles":[{"name":"custom-rbac-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"rbac-expansion","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"site-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"full-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-dual-write","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"approve-review-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-metadata","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"sso-setup-deadline-banner","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-serve","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"live-agents","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"starter-pricing","enabled":true,"variant":{"name":"control","enabled":true,"payload":{"type":"string","value":"control"},"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"wildcard-email-auth","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"googleFontsInlined":true},"customCss":[{"filePath":"css/styles.css","content":":root{--background-light:251 250 248;--background-dark:26 29 36;--primary-foreground:217 13% 19%;--page-texture:url(\"/css/assets/light-dots.png\")}:root{--color-page-bg:#FBFAF8;--color-card-hover:#FFFFFF;--color-stroke:#DFE0E2;--color-sidebar-bg:#F9F7F3;--color-sidebar-hover:#F5EDE1;--color-card-rest:#FBFAF8;--color-card-active:#FFFFFF;--color-icon:#D4870D;--color-button:#FCBC32;--color-button-fg:#2B3038;--color-text-primary:#2B3038;--color-text-secondary:#4B535C}.dark{--color-page-bg:#1A1D24;--color-card-hover:#363C44;--color-stroke:#2B3038;--color-sidebar-bg:#20242B;--color-sidebar-hover:#2B3038;--color-card-rest:#1A1D24;--color-card-active:#2B3038;--color-icon:#D4870D;--color-button:#FCBC32;--color-text-primary:#E8E8E9;--color-text-secondary:#C5C7CC;--page-texture:url(\"/css/assets/dark-dots.png\")}:root{--font-serif:\"Source Serif 4\",\"Times New Roman\",serif;--font-sans:\"Source Sans 3\",Calibri,sans-serif;--font-weight-regular:400;--font-weight-medium:500;--font-weight-bold:650;--font-size-xs:0.75rem;--font-size-sm:0.9375rem;--font-size-md:1rem;--font-size-xm:1.125rem;--font-size-lg:1.5rem;--font-size-xl:2.5rem}body{font-family:var(--font-sans)}h1,h2,h3,h4,h5,h6{font-family:var(--font-serif);font-weight:var(--font-weight-medium)!important;color:var(--color-text-primary)!important}h1{font-size:var(--font-size-xl)}b,strong{font-weight:var(--font-weight-bold)!important;font-synthesis-weight:auto}.prose :where(h2):not(:where([class~=not-prose],[class~=not-prose] *)){margin-top:2.5rem}h2:is([id^=january-2],[id^=february-2],[id^=march-2],[id^=april-2],[id^=may-2],[id^=june-2],[id^=july-2],[id^=august-2],[id^=september-2],[id^=october-2],[id^=november-2],[id^=december-2]){margin-top:3rem}.banner-body,.banner-body *,.banner-title{color:#e8e8e9!important}body,html{background-color:var(--color-page-bg)!important;background-image:var(--page-texture)!important;background-repeat:no-repeat!important;background-size:500px auto!important;background-position:calc(100% + 375px) 200px!important;background-attachment:fixed!important}main{background-color:transparent!important}#background-color{background-color:transparent!important}span[style*=radial-gradient]{background:0 0!important}img.nav-logo{height:62px!important;min-height:62px!important;max-height:62px!important;width:233px!important;min-width:233px!important;max-width:233px!important;flex-shrink:0!important;flex-grow:0!important}a:has(>img.nav-logo){flex-shrink:0;min-width:233px}a.nav-tabs-item.font-medium:not([data-active=true]),button.nav-tabs-item:not(.text-primary)>div:first-child{color:var(--color-text-secondary)}button.nav-tabs-item.text-primary>div:first-child svg,button.nav-tabs-item:hover>div:first-child svg{color:var(--color-text-primary)!important}.nav-tabs-item{font-size:var(--font-size-md)}[data-active=true].hover\\:text-primary:hover,[data-active=true].text-primary{color:var(--color-text-primary);text-shadow:none;font-weight:var(--font-weight-bold)}button.nav-tabs-item.text-primary,button.nav-tabs-item.text-primary>div:first-child{color:var(--color-text-primary)!important;font-weight:var(--font-weight-bold)!important}.nav-tabs-item .bg-primary{background-color:var(--color-text-primary);height:2px}.nav-dropdown-item:has(.text-primary){background-color:var(--color-sidebar-hover)}.nav-dropdown-item .text-primary{color:var(--color-text-primary);font-weight:var(--font-weight-medium)}.nav-dropdown-item.group:hover .group-hover\\:text-primary{color:var(--color-text-primary)}#assistant-entry,#assistant-entry span,#localization-select-trigger,#localization-select-trigger span,#search-bar-entry,#search-bar-entry>div>div,#topbar-cta-button a,#topbar-cta-button a span,.topbar-right-container nav,.topbar-right-container nav a{font-size:var(--font-size-sm)}#topbar-cta-button a,#topbar-cta-button a span,#topbar-cta-button a svg{color:var(--color-button-fg)}#assistant-entry{background-color:var(--color-button)!important}#assistant-entry:hover{background-color:var(--color-button)!important;opacity:.9}#assistant-entry,#assistant-entry span,#assistant-entry svg{color:var(--color-button-fg)!important}#banner,#banner *{color:var(--color-button-fg)!important}button[aria-label=\"Dismiss banner\"] svg{stroke:var(--color-button-fg)!important}#sidebar,#sidebar-content{background-color:var(--color-sidebar-bg)}.sidebar-group-header,.sidebar-group-header>h5{font-family:var(--font-sans)!important;font-size:var(--font-size-md);font-weight:var(--font-weight-medium)!important}.sidebar-group a,.sidebar-group button{font-family:var(--font-sans);font-size:var(--font-size-sm);font-weight:var(--font-weight-medium)}#sidebar-content a.text-primary{color:var(--color-text-primary);background-color:var(--color-sidebar-hover);text-shadow:none}.nav-tag-pill-text{color:var(--color-text-secondary)}#sidebar-content .sidebar-group a .justify-end{flex:none}.toc-item[data-active=true]>a,.toc-item[data-active=true]>a:hover{color:var(--color-text-primary);border-color:var(--color-text-primary)}.colab-link,.github-source-link,.try-product-link{display:flex;align-items:center;justify-content:center;gap:12px;padding:8px 14px;border:1px solid color-mix(in srgb,var(--color-stroke) 70%,transparent);border-radius:12px;font-size:var(--font-size-md);font-weight:var(--font-weight-regular);font-family:var(--font-sans);color:var(--color-text-secondary);background-color:color-mix(in srgb,var(--color-card-hover) 50%,transparent);text-decoration:none;line-height:24px;white-space:nowrap;cursor:pointer;transition:all .2s ease;margin-bottom:1.5rem;width:fit-content;min-width:fit-content}.colab-link:hover,.github-source-link:hover,.try-product-link:hover{background-color:color-mix(in srgb,var(--color-card-hover) 80%,transparent);border-color:var(--color-stroke)}.dark .colab-link,.dark .github-source-link,.dark .try-product-link{color:var(--color-text-primary)}.source-link{float:right;display:inline-block;padding:2px 8px;border:1px solid color-mix(in srgb,var(--color-stroke) 70%,transparent);border-radius:6px;font-size:var(--font-size-xs);font-weight:var(--font-weight-medium);color:var(--color-text-secondary);background-color:color-mix(in srgb,var(--color-card-hover) 50%,transparent);text-decoration:none;transition:all .15s ease}.source-link:hover{background-color:color-mix(in srgb,var(--color-card-hover) 80%,transparent);border-color:var(--color-stroke);text-decoration:none}.dark .source-link{color:var(--color-text-primary)}button[type=submit].bg-primary.mint-text-white{color:var(--color-button-fg)!important}button[type=submit].bg-primary.mint-text-white [fill=white]{fill:currentColor!important}button[type=submit].bg-primary.mint-text-white [stroke=white]{stroke:currentColor!important}button[type=submit].bg-primary.mint-text-white svg{color:var(--color-button-fg)!important}div:has(>div>textarea.chat-assistant-input):not(#chat-assistant-sheet *){box-shadow:0 8px 24px #00000026}div:has(>.chat-assistant-floating-input){overflow:visible!important}textarea.chat-assistant-input{min-height:2.5rem!important;max-height:2.5rem!important}button.chat-assistant-send-button{background-color:var(--color-button)!important}button.chat-assistant-send-button svg{color:var(--color-button-fg)!important}.product-card{background-color:var(--color-card-rest);border:1px solid var(--color-stroke)}.product-card:hover{background-color:var(--color-card-active)}.product-card-title{font-size:var(--font-size-lg);font-weight:var(--font-weight-medium)!important;margin-top:0}.home-h3,.product-card-subtitle{font-family:var(--font-sans)!important;font-size:var(--font-size-xm);font-weight:var(--font-weight-regular);color:var(--color-text-primary)}.product-card-body{font-family:var(--font-sans);font-size:var(--font-size-sm);font-weight:var(--font-weight-regular);color:var(--color-text-secondary)}.card.block{background-color:var(--color-card-rest)!important;border-color:var(--color-stroke)!important}.card.block:hover{background-color:var(--color-card-active)!important;border-color:var(--color-stroke)!important}.card.block h1,.card.block h2,.card.block h3,.card.block h4,.card.block h5,.card.block h6{font-family:var(--font-sans);font-weight:var(--font-weight-bold)!important}#pagination>a{background-color:var(--color-card-rest);border-color:var(--color-stroke)}#pagination>a:hover{background-color:var(--color-card-active);border-color:var(--color-stroke)}[data-component-part=update-label]{color:var(--color-text-secondary)}[data-badge=true][data-color=orange]{--color-bg:color-mix(in srgb, var(--color-button) 20%, transparent);--color-text:var(--color-icon);--color-bg-disabled:color-mix(in srgb, var(--color-button) 8%, transparent);--color-text-disabled:color-mix(in srgb, var(--color-icon) 50%, transparent)}[data-badge=true] a,[data-badge=true] a:hover{text-decoration:none!important;border-bottom:0!important}.dark [data-badge=true][data-color=orange]{--color-text:var(--color-button);--color-text-disabled:color-mix(in srgb, var(--color-button) 50%, transparent)}"}]}
{"page":{"pagePath":"inference/response-settings/prefix-caching","slug":"/inference/response-settings/prefix-caching","pageMetadata":{"title":"Prefix caching","description":"Reduce latency for repeated prompts with prefix caching, and isolate cache reuse with cache_salt when needed.","href":"/inference/response-settings/prefix-caching"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Reduce latency for repeated prompts with prefix caching, and isolate cache reuse with cache_salt when needed.\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,code:`code`,li:`li`,p:`p`,pre:`pre`,span:`span`,ul:`ul`,..._provideComponents(),...props.components},{CodeBlock,Heading,Note,Tab,Tabs}=_components;if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!Note)_missingMdxReference(`Note`,true);if(!Tab)_missingMdxReference(`Tab`,true);if(!Tabs)_missingMdxReference(`Tabs`,true);return _jsxs(_Fragment,{children:[_jsx(_components.p,{children:`W\u0026B Inference uses prefix caching on supported hosted models to speed up repeated requests with identical prompt prefixes.`}),`\n`,_jsx(_components.p,{children:`When a request shares the same prompt prefix as an earlier request on the same backend, the model reuses the previously computed key-value (KV) cache instead of recomputing the entire prefix. This reduces latency for repeated prompts, long system prompts, and workloads with a stable shared prefix.`}),`\n`,_jsxs(_components.p,{children:[`Prefix caching is automatic on supported models, so you don’t need to enable it in your request. This page describes when prefix caching is most effective and how to control cache isolation with `,_jsx(_components.code,{children:`cache_salt`}),`.`]}),`\n`,_jsx(Heading,{level:`2`,id:`when-prefix-caching-helps`,children:`When prefix caching helps`}),`\n`,_jsx(_components.p,{children:`Prefix caching is most useful when you repeatedly send requests that share a long common prefix, such as:`}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`A large system prompt reused across many requests.`}),`\n`,_jsx(_components.li,{children:`A long shared document followed by different user questions.`}),`\n`,_jsx(_components.li,{children:`Repeated evaluation prompts with only small per-request changes.`}),`\n`,_jsx(_components.li,{children:`Multi-turn workloads where much of the conversation history stays the same.`}),`\n`]}),`\n`,_jsx(Heading,{level:`2`,id:`cache-isolation`,children:`Cache isolation`}),`\n`,_jsxs(_components.p,{children:[`In some environments, you might need to prevent cache reuse across different users or applications. The `,_jsx(_components.code,{children:`cache_salt`}),` parameter provides this control.`]}),`\n`,_jsx(_components.p,{children:`By default, requests with identical prompt prefixes may reuse cache on shared infrastructure when the backend allows it.`}),`\n`,_jsxs(_components.p,{children:[`To isolate cache reuse to a specific trust boundary, set the `,_jsx(_components.code,{children:`cache_salt`}),` request parameter. Requests only reuse prefix cache when both the prompt prefix and the `,_jsx(_components.code,{children:`cache_salt`}),` match.`]}),`\n`,_jsxs(_components.p,{children:[`Use `,_jsx(_components.code,{children:`cache_salt`}),` when you want cache reuse within a single user, tenant, session, or application boundary, but don’t want reuse across other callers.`]}),`\n`,_jsx(Heading,{level:`3`,id:`how-it-works`,children:`How it works`}),`\n`,_jsxs(_components.p,{children:[`The presence and value of `,_jsx(_components.code,{children:`cache_salt`}),` affect cache reuse as follows:`]}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[`Same prompt prefix, no `,_jsx(_components.code,{children:`cache_salt`}),`: cache may be reused across matching requests.`]}),`\n`,_jsxs(_components.li,{children:[`Same prompt prefix, same `,_jsx(_components.code,{children:`cache_salt`}),`: cache can be reused.`]}),`\n`,_jsxs(_components.li,{children:[`Same prompt prefix, different `,_jsx(_components.code,{children:`cache_salt`}),`: cache is isolated and not reused across salts.`]}),`\n`]}),`\n`,_jsx(Note,{children:_jsxs(_components.p,{children:[_jsx(_components.code,{children:`cache_salt`}),` must be a non-empty string when provided.`]})}),`\n`,_jsx(Heading,{level:`2`,id:`examples`,children:`Examples`}),`\n`,_jsxs(_components.p,{children:[`The following examples show how to send a chat completion request with `,_jsx(_components.code,{children:`cache_salt`}),` to isolate cache reuse.`]}),`\n`,_jsxs(Tabs,{children:[_jsx(Tab,{title:`Python`,id:`python`,children:_jsx(CodeBlock,{filename:``,numberOfLines:`25`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`25`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` openai`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`client `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` openai.OpenAI(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` base_url`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"https://api.inference.wandb.ai/v1\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` api_key`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"\u003cyour-api-key>\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`response `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` client.chat.completions.create(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` model`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"moonshotai/Kimi-K2.5\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` messages`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`[`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"role\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"system\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"content\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"You are a careful assistant that answers concisely.\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` },`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"role\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"user\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"content\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Summarize this document in one sentence: \u003clong shared prefix here>\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` },`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ],`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` extra_body`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"cache_salt\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"tenant-a-user-123-secret\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` },`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(response.choices[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`].message.content)`})]}),`\n`]})})})}),_jsx(Tab,{title:`Bash`,id:`bash`,children:_jsx(CodeBlock,{filename:``,numberOfLines:`11`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`11`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`curl`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` https://api.inference.wandb.ai/v1/chat/completions`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -H`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Content-Type: application/json\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -H`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Authorization: Bearer \u003cyour-api-key>\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -d`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` '{`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"model\": \"moonshotai/Kimi-K2.5\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"messages\": [`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` { \"role\": \"system\", \"content\": \"You are a careful assistant that answers concisely.\" },`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` { \"role\": \"user\", \"content\": \"Summarize this document in one sentence: \u003clong shared prefix here>\" }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ],`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"cache_salt\": \"tenant-a-user-123-secret\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` }'`})}),`\n`]})})})})]}),`\n`,_jsx(Heading,{level:`2`,id:`response-behavior`,children:`Response behavior`}),`\n`,_jsxs(_components.p,{children:[`To confirm that prefix caching is active for a request, check the response usage details. On some models, usage details may include cached token counts in `,_jsx(_components.code,{children:`usage.prompt_tokens_details.cached_tokens`}),` when prefix cache is reused. The availability of that field varies by model and backend.`]}),`\n`,_jsx(Heading,{level:`2`,id:`related-pages`,children:`Related pages`}),`\n`,_jsx(_components.p,{children:`The following pages cover related topics:`}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:_jsx(_components.a,{href:`/inference/api-reference/chat-completions`,children:`Chat Completions`})}),`\n`,_jsx(_components.li,{children:_jsx(_components.a,{href:`/inference/response-settings/streaming`,children:`Enable streaming responses`})}),`\n`,_jsx(_components.li,{children:_jsx(_components.a,{href:`/inference/response-settings/structured-output`,children:`Structured output`})}),`\n`,_jsx(_components.li,{children:_jsx(_components.a,{href:`/inference/response-settings/json-mode`,children:`JSON mode`})}),`\n`]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Prefix caching","description":"Reduce latency for repeated prompts with prefix caching, and isolate cache reuse with cache_salt when needed.","href":"/inference/response-settings/prefix-caching"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"When prefix caching helps","slug":"when-prefix-caching-helps","depth":2,"children":[]},{"title":"Cache isolation","slug":"cache-isolation","depth":2,"children":[{"title":"How it works","slug":"how-it-works","depth":3,"children":[]}]},{"title":"Examples","slug":"examples","depth":2,"children":[]},{"title":"Response behavior","slug":"response-behavior","depth":2,"children":[]},{"title":"Related pages","slug":"related-pages","depth":2,"children":[]}],"codeExamples":{}},"jsLinks":["/_mintlify/custom/wb-21fd5541/62b2917cd8d8869b/analytics.js","/_mintlify/custom/wb-21fd5541/1d706ba09a39aeb0/local-links-same-tab.js","/_mintlify/custom/wb-21fd5541/b496d1b047524c5a/dropdown-tabs-navigate.js","/_mintlify/custom/wb-21fd5541/ca910ce06377c00d/code-group-language-persist.js","/_mintlify/custom/wb-21fd5541/6e7fd8bf6d924ed7/kapa-widget.js"],"subdomain":"wb-21fd5541","siteName":"Weights \u0026 Biases Documentation","title":"Prefix caching - Weights \u0026 Biases Documentation","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{"en":{"href":"/inference/response-settings/prefix-caching","matchLevel":0},"fr":{"href":"/fr/inference/response-settings/prefix-caching","matchLevel":1},"ja":{"href":"/ja/inference/response-settings/prefix-caching","matchLevel":1},"ko":{"href":"/ko/inference/response-settings/prefix-caching","matchLevel":1}}},"lastModified":"2026-07-27T14:27:38.528Z","originalFileLocation":"inference/response-settings/prefix-caching.mdx","locale":"en"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Reduce latency for repeated prompts with prefix caching, and isolate cache reuse with cache_salt when needed.\">\u003cmeta data-mint-head name=\"application-name\" content=\"Weights \u0026amp; Biases Documentation\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/wb-21fd5541/5Z9xPfvPzDMWig23/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/wb-21fd5541/5Z9xPfvPzDMWig23/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/wb-21fd5541/5Z9xPfvPzDMWig23/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/wb-21fd5541/5Z9xPfvPzDMWig23/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/wb-21fd5541/5Z9xPfvPzDMWig23/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/wb-21fd5541/5Z9xPfvPzDMWig23/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/wb-21fd5541/5Z9xPfvPzDMWig23/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/wb-21fd5541/5Z9xPfvPzDMWig23/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/wb-21fd5541/5Z9xPfvPzDMWig23/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Prefix caching - Weights \u0026amp; Biases Documentation\">\u003cmeta data-mint-head property=\"og:description\" content=\"Reduce latency for repeated prompts with prefix caching, and isolate cache reuse with cache_salt when needed.\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Weights \u0026amp; Biases Documentation\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://docs.wandb.ai/inference/response-settings/prefix-caching\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://wb-21fd5541.mintlify.site/_mintlify/api/og/v2?division=Response+Settings\u0026amp;title=Prefix+caching\u0026amp;description=Reduce+latency+for+repeated+prompts+with+prefix+caching%2C+and+isolate+cache+reuse+with+cache_salt+when+needed.\u0026amp;theme=b2b0182ac5551ece3d862622\u0026amp;signature=67bbbc2679dbb825550443fa5dd221bc60f5da0a08ae9b13091be27ff52d4b50\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Prefix caching - Weights \u0026amp; Biases Documentation\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Reduce latency for repeated prompts with prefix caching, and isolate cache reuse with cache_salt when needed.\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://wb-21fd5541.mintlify.site/_mintlify/api/og/v2?division=Response+Settings\u0026amp;title=Prefix+caching\u0026amp;description=Reduce+latency+for+repeated+prompts+with+prefix+caching%2C+and+isolate+cache+reuse+with+cache_salt+when+needed.\u0026amp;theme=b2b0182ac5551ece3d862622\u0026amp;signature=67bbbc2679dbb825550443fa5dd221bc60f5da0a08ae9b13091be27ff52d4b50\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/wb-21fd5541/5Z9xPfvPzDMWig23/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Weights \u0026amp; Biases Documentation\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#FCBC32\">\u003clink data-mint-head rel=\"canonical\" href=\"https://docs.wandb.ai/inference/response-settings/prefix-caching\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/inference/response-settings/prefix-caching.md\">\u003clink data-mint-head rel=\"alternate\" hreflang=\"en\" href=\"https://docs.wandb.ai/inference/response-settings/prefix-caching\">\u003clink data-mint-head rel=\"alternate\" hreflang=\"fr\" href=\"https://docs.wandb.ai/fr/inference/response-settings/prefix-caching\">\u003clink data-mint-head rel=\"alternate\" hreflang=\"ja\" href=\"https://docs.wandb.ai/ja/inference/response-settings/prefix-caching\">\u003clink data-mint-head rel=\"alternate\" hreflang=\"ko\" href=\"https://docs.wandb.ai/ko/inference/response-settings/prefix-caching\">\u003clink data-mint-head rel=\"alternate\" hreflang=\"x-default\" href=\"https://docs.wandb.ai/inference/response-settings/prefix-caching\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://docs.wandb.ai/#organization\",\"name\":\"Weights \u0026 Biases Documentation\",\"url\":\"https://docs.wandb.ai\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/wb-21fd5541/C7PMWo9Jl9GZ2Zki/icons/Endorsed_primary_blackwhite.svg?fit=max\u0026auto=format\u0026n=C7PMWo9Jl9GZ2Zki\u0026q=85\u0026s=a9d999235518399090dd8ddd0a9e1e4a\"}},{\"@type\":\"WebSite\",\"@id\":\"https://docs.wandb.ai#website\",\"name\":\"Weights \u0026 Biases Documentation\",\"url\":\"https://docs.wandb.ai\",\"publisher\":{\"@id\":\"https://docs.wandb.ai/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://docs.wandb.ai/inference/response-settings/prefix-caching#webpage\",\"url\":\"https://docs.wandb.ai/inference/response-settings/prefix-caching\",\"name\":\"Prefix caching\",\"description\":\"Reduce latency for repeated prompts with prefix caching, and isolate cache reuse with cache_salt when needed.\",\"dateModified\":\"2026-07-27T14:27:38.528Z\",\"isPartOf\":{\"@id\":\"https://docs.wandb.ai#website\"},\"breadcrumb\":{\"@id\":\"https://docs.wandb.ai/inference/response-settings/prefix-caching#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://docs.wandb.ai/inference/response-settings/prefix-caching#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Response Settings\",\"item\":\"https://docs.wandb.ai/inference/response-settings/json-mode\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Prefix caching\",\"item\":\"https://docs.wandb.ai/inference/response-settings/prefix-caching\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://docs.wandb.ai/inference/response-settings/prefix-caching#article\",\"headline\":\"Prefix caching\",\"name\":\"Prefix caching\",\"description\":\"Reduce latency for repeated prompts with prefix caching, and isolate cache reuse with cache_salt when needed.\",\"url\":\"https://docs.wandb.ai/inference/response-settings/prefix-caching\",\"mainEntityOfPage\":{\"@id\":\"https://docs.wandb.ai/inference/response-settings/prefix-caching#webpage\"},\"dateModified\":\"2026-07-27T14:27:38.528Z\",\"publisher\":{\"@id\":\"https://docs.wandb.ai/#organization\"},\"isPartOf\":{\"@id\":\"https://docs.wandb.ai#website\"}}]}\u003c/script>","agentPointerHref":"/inference/response-settings/prefix-caching.md","siteName":"Weights \u0026 Biases Documentation","nav":{"global":null,"languages":[{"language":"en","tabs":[{"tab":"Products","menu":[{"item":"Docs Home","pages":[{"title":"Docs Home","href":""}]},{"item":"W\u0026B Models","pages":[{"title":"W\u0026B Models","href":"/models"}]},{"item":"W\u0026B Weave","pages":[{"title":"W\u0026B Weave","href":"/weave"}]},{"item":"Serverless Inference","pages":[{"title":"Serverless Inference","mode":"wide","href":"/inference"},{"title":"Prerequisites","href":"/inference/prerequisites"},{"title":"Available models","mode":"wide","href":"/inference/models"},{"title":"Model lifecycle","href":"/inference/lifecycle"},{"title":"Use Serverless LoRA Inference","href":"/inference/lora"},{"group":"Response Settings","pages":[{"title":"Enable JSON mode","href":"/inference/response-settings/json-mode"},{"title":"View reasoning information","href":"/inference/response-settings/reasoning"},{"title":"Enable streaming responses","description":"Enable streaming output with Serverless Inference to receive model responses incrementally.","href":"/inference/response-settings/streaming"},{"title":"Prefix caching","href":"/inference/response-settings/prefix-caching"},{"title":"Enable structured output","description":"How to configure structured output in Serverless Inference responses","href":"/inference/response-settings/structured-output"},{"title":"Call tools","href":"/inference/response-settings/tool-calling"}]},{"title":"Usage information and limits","href":"/inference/usage-limits"},{"group":"Tutorials","pages":[{"title":"Cline with Serverless Inference","href":"/inference/tutorials/integration-cline"},{"title":"Creating a fine-tuned LoRA","href":"/inference/tutorials/creating-lora"}]},{"group":"API Reference","pages":[{"title":"API Reference","url":"/inference/api-reference","href":"/inference/ref-link-api"}]},{"title":"Usage examples","href":"/inference/examples"},{"title":"UI guide","href":"/inference/ui-guide"},{"title":"Support: Inference","url":"/support/inference","href":"/inference/support-inference"}]},{"item":"Serverless Training","pages":[{"title":"Serverless Training","href":"/serverless-training"}]},{"item":"Serverless Sandboxes","pages":[{"title":"Serverless Sandboxes","href":"/sandboxes"}]}]},{"tab":"Get Started","pages":[{"title":"Get Started","href":"/get-started"}]},{"tab":"Reference","pages":[{"title":"Reference","href":"/reference"}]},{"tab":"Release Notes","pages":[{"title":"Release Notes","href":"/release-notes"}]},{"tab":"Support","menu":[{"item":"Support Home","pages":[{"title":"Support Home","href":"/support"}]},{"item":"W\u0026B Models","pages":[{"title":"W\u0026B Models","href":"/support/models"}]},{"item":"W\u0026B Weave","pages":[{"title":"W\u0026B Weave","href":"/support/weave"}]},{"item":"Serverless Inference","pages":[{"title":"Serverless Inference","href":"/support/inference"}]}],"pages":[{"title":"Support","href":"/support"}]},{"tab":"W\u0026B Launch","hidden":true,"pages":[{"title":"W\u0026B Launch","href":"/platform/launch"}]},{"tab":"Support: W\u0026B Models","hidden":true,"pages":[{"title":"Support: W\u0026B Models","href":"/support/models"}]},{"tab":"Support: W\u0026B Weave","hidden":true,"pages":[{"title":"Support: W\u0026B Weave","href":"/support/weave"}]},{"tab":"Support: W\u0026B Inference","hidden":true,"pages":[{"title":"Support: W\u0026B Inference","href":"/support/inference"}]}]},{"language":"fr","pages":[{"title":"fr","href":"/fr/inference/response-settings/prefix-caching"}]},{"language":"ja","pages":[{"title":"ja","href":"/ja/inference/response-settings/prefix-caching"}]},{"language":"ko","pages":[{"title":"ko","href":"/ko/inference/response-settings/prefix-caching"}]}]},"basePath":"","title":"Prefix caching - Weights \u0026 Biases Documentation","deploymentId":"dpl_Em2U4NQQ2N2DE3HPbPUJVZLnAtvc","buildId":"dpl_Em2U4NQQ2N2DE3HPbPUJVZLnAtvc","dynamics":["10e4515597"],"chunks":["BannerMarkdown-CVKwuln2.js","dist-CXMB3QQU.js","lib-DMeTj1gb.js","lib-B8HbPgcC.js","katex-Dbtb6Wk7.js","html-void-elements-mCSPHezU.js","marked.esm-CZWythHk.js"]}