AI Memory: Conversation History and Tokens

Key takeaways: Giving memory to a conversational AI relies on three mechanisms: a sliding window with priorities (system prompt + latest exchanges always included), a controlled token budget to manage costs (a 10-exchange conversation costs 5x more), and a fallback system between LLM models to ensure availability. This is the difference between a tool and a true assistant.

The first version of the AI in TAMSIV was amnesiac. "Add bread to my shopping list" worked perfectly. But following up with "Put it for tomorrow" failed miserably — the AI didn't know which task we were talking about. Each exchange started from scratch.

This problem is fundamental in any conversational AI application. LLMs have no native memory. With each request, the entire context must be sent back. And that changes everything — in terms of UX, cost, and architecture.

Here's how I gave memory to TAMSIV's AI.

Why don't LLMs have memory?

It's counter-intuitive when you use ChatGPT or Claude, but these models have no persistence between requests. Each API call is independent. The "memory" you perceive in a ChatGPT conversation is the application sending the complete history with each message.

Specifically, when you send your 5th message in a conversation, the API receives:

1. The system prompt (behavior instructions)
2. Message 1 + response 1
3. Message 2 + response 2
4. Message 3 + response 3
5. Message 4 + response 4
6. Your message 5

With each additional exchange, the payload grows. And with it, two problems: the context window (technical model limit) and the token cost (each token is billed).

This is what OpenAI documentation calls managing conversation state.

How does TAMSIV's sliding window work?

The classic solution is the sliding window: only the last N exchanges are kept. But a raw sliding window is too simplistic for an app like TAMSIV, where the AI needs to understand references to past actions.

I implemented a sliding window with priorities:

• Maximum priority: the system prompt (always included, never truncated)
• High priority: the last 2 exchanges (immediate context)
• High priority: function calls and their results (actions performed — task creation, memo modification, etc.)
• Medium priority: previous exchanges (3 to N-2)
• Low priority: old exchanges, summarized or deleted according to the token budget

Why are function calls prioritized? Because in TAMSIV, when the user says "Change the priority of the task we just created," the AI needs to know which task was created. This information is in the function_result of a previous exchange. Without it, the AI cannot resolve the pronoun "it."

How to manage the token budget without exploding costs?

This is the crux of the matter. A 10-exchange conversation can cost 5 times more than an isolated exchange, because the cumulative payload is sent back each time.

Concrete example with a model at $0.001 / 1000 tokens:

• Exchange 1: ~500 tokens (system prompt + message) = $0.0005
• Exchange 5: ~2500 tokens (history 1-4 + message 5) = $0.0025
• Exchange 10: ~5000 tokens (history 1-9 + message 10) = $0.005
• Total 10-exchange conversation: ~25,000 cumulative tokens = $0.025

For an app with thousands of users, these cents add up quickly. I put in place three safeguards:

1. Limit of 20 exchanges per conversation: beyond that, we recommend starting a new conversation. This avoids monster conversations with 100 exchanges.
2. Estimated token counter: before each call, the backend estimates the number of tokens in the complete payload. If it exceeds the budget, the oldest exchanges are truncated.
3. Model selection based on complexity: a simple message ("Add bread") uses a light model. A complex request ("Reorganize my weekly tasks by priority") uses a more powerful model.

How does the fallback between LLM models work?

In production, reliability is non-negotiable. If the main model (configured via OPENROUTER_MODEL) returns a 429 error (rate limit) or 503 (service unavailable), the user should not notice anything.

TAMSIV's backend implements an automatic fallback:

1. Call to the main model via OpenRouter
2. If 429/503 error → retry with OPENROUTER_FALLBACK_MODEL
3. The AlertService sends an email to the admin to report the fallback
4. The user receives their response normally — no perceptible degradation

In production, TAMSIV performs 2 to 3 fallbacks per week. It's minimal, but it happens. Without this mechanism, the user would see a generic error message — and probably leave the app.

The choice of OpenRouter as an LLM proxy (rather than directly calling the OpenAI or Anthropic API) is strategic: it allows changing models without modifying the code. If a new model comes out, you just need to change an environment variable. The complete voice pipeline remains identical.

What impact does memory have on user experience?

The difference is spectacular. Before conversation history, each exchange was independent. Afterward, the AI becomes a true assistant:

• "Change the priority of the task we just created" → the AI knows which task, and modifies it
• "Actually, put that to Friday" → the AI understands that "that" refers to the last event created
• "Also add a memo about that" → the AI picks up the conversation topic to create the memo
• "No, I said tomorrow, not the day after tomorrow" → the AI corrects by understanding the temporal reference

This is the difference between a voice form (repeat everything every time) and an assistant who listens and remembers. This is what makes TAMSIV's Dictaphone usable daily.

How does the system prompt influence the quality of responses?

The system prompt is the most important document in the application. It defines the AI's behavior: tone, response format, available function tools, constraints.

TAMSIV's system prompt includes:

• The persona: "You are the voice assistant for TAMSIV, a task and memo management application."
• Function tools: the 7 functions the AI can call (create_task, update_task, create_memo, update_memo, create_calendar_event, ask_clarification, end_conversation)
• Constraints: short responses (user listens via TTS), no markdown, no long lists
• User context: preferred language, time zone, plan (Free/Pro/Team)

A well-designed system prompt drastically reduces hallucinations and off-topic responses. It's an investment that pays off in every conversation.

How to manage multimodal conversations (voice + text)?

In TAMSIV, the user speaks, but the AI responds in voice (via OpenAI TTS) AND in text (displayed on screen). The history stores both forms:

• User side: the text transcribed by the STT (not raw audio — too heavy in tokens)
• AI side: the response text (sent to TTS and displayed)

The complete pipeline is: Audio → STT → Text → LLM (with history) → Text → TTS → Audio. The history only lives in the text layer, which greatly simplifies the architecture.

What alternatives to the sliding window exist?

The sliding window is not the only approach. Other strategies exist:

• Summarization: summarizing old exchanges into a condensed paragraph. Advantage: maximum compression. Disadvantage: loss of detail, and an additional LLM call for the summary.
• RAG (Retrieval Augmented Generation): storing history in a vector database and retrieving only relevant exchanges by semantic similarity. Powerful but heavy to implement.
• Explicit Memory: storing "facts" extracted from conversations (user preferences, recurring tasks) in a structured database. This is what ChatGPT does with its "Memory" feature.

For TAMSIV, the sliding window with priorities is the best compromise: simple to implement, token-efficient, and sufficient for conversations of 5 to 15 exchanges. If conversations became much longer, I would consider RAG.

How to implement conversation history in your own project?

If you're building an app with an LLM, here are the steps:

1. Store history server-side: never trust the client to maintain conversation state. The backend is the source of truth.
2. Implement a sliding window: start simple (keep the last N exchanges), then add priorities if necessary.
3. Budget tokens: count tokens before each call. Truncate intelligently if the budget is exceeded.
4. Add a fallback: at a minimum, a backup model in case of main model error.
5. Measure costs: log each call with the number of tokens used. This allows optimizing the system over time.

TAMSIV's admin dashboard displays conversation metrics in real-time: average number of exchanges, average cost per conversation, fallback rate. This data is essential for optimizing costs in production.

FAQ

How many exchanges does an average user make per conversation?

In TAMSIV, the average is 3 to 5 exchanges. The typical user says "Add a task for tomorrow: call the dentist," confirms the preview, and follows up with a modification ("Set it to high priority"). Long conversations (10+ exchanges) represent less than 10% of the volume.

Is the cost of conversation history significant?

Yes, it's the main expense item of the AI pipeline. Each additional exchange increases the cumulative cost. For TAMSIV, the average cost per conversation is about 0.01 to 0.03 EUR with the main model. The fallback model is generally cheaper, which partially offsets the extra costs.

Should history be stored in a database?

For TAMSIV, history lives in memory (backend side) during the WebSocket session. It is not persisted in the database — when the connection closes, the history is lost. This is a deliberate choice: conversations are short and transient. If you need to resume conversations later, you will need to persist them in the database.

OpenRouter vs directly calling LLM provider APIs?

OpenRouter acts as a proxy that unifies access to dozens of models (OpenAI, Anthropic, Google, etc.) via a single API. The advantage: changing models without modifying the code. The disadvantage: an additional layer (marginal latency). For an app like TAMSIV that needs flexibility and fallback, it's an obvious choice.

Does conversation memory work with Realtime mode?

TAMSIV has three WebSocket modes. In LiveWebSocket mode (the default mode), history is managed manually as described in this article. In Realtime mode (OpenAI Realtime API), history is managed natively by the API — it's a stateful bidirectional protocol. The legacy mode works in batch without persistence.