Context budget and prompt cache

Every request Kallamo sends is checked before it leaves your computer. If it would not fit the model, it is not sent.

The limit

The limit for a request is the smaller of two values:

• the context window of the model, from its connection;
• the workspace’s Max API payload, in Configuration › Context & memory.

How a request is filled

1. The fixed part: the profile’s rules, always-on knowledge, attachments, and room reserved for the reply (Max reply tokens). If this alone does not fit, nothing else can.
2. The conversation: active messages, newest first, as many as fit.
3. Retrieval: what the search found, packed by priority into what is left, never less than 40% of the room.
4. A final check against the same limit, with a safety margin.

What does not fit is left out in this order of priority, most important kept first:

Priority	What
1	Worldbuild facts: sheets and relationships.
2	Files read in full.
3	Search hits, linked lore, passages smart search cited.
4	Everything else that was found.

A request that still does not fit is stopped before it reaches the provider. The error says how much came from instructions, retrieved context and history, so you know what to trim.

Token counts that learn

Kallamo counts tokens on your computer with one fixed tokenizer, and every model counts the same text a little differently. After each reply, Kallamo compares its count with the one the provider reports and learns the ratio for that connection and model. Limits are then adjusted, so a model that counts more never gets a request it will refuse.

Prompt cache

Many providers bill less for the part of a request they have already seen, as long as it starts exactly the same way. Kallamo lays every request out to take advantage of it:

• The stable part comes first: the profile’s instructions and voice, and the always-on knowledge you pinned.
• The moving part comes after: the story clock, retrieved context, attachments.

In Settings › Generation › Prompt cache:

Option	What it does
Use the prompt cache	Sends the unchanging part of every request first, so the provider can reuse it. Off sends requests exactly as older versions did.
In the system prompt	The moving part stays at the end of the system prompt, where it has always been.
Next to your message	The moving part goes after the history, so the history gets cached too: a bigger saving on long chats. The model reads context in a new place, so compare on one workspace first.

Kallamo asks for caching only where it pays, and picks how long an entry lives by how fast you are writing.

See the savings

With Token usage on in Settings › Diagnostics, each reply shows its tokens, and where the provider reports them, how many were read from the cache.

Streaming

Stream replies (Settings › Generation) shows the reply as it is written. It also avoids timeouts on slow local models. Providers without streaming fall back on their own.