The Memory an AI Model Actually Has
Every time you send a message to an AI chatbot, it doesn’t truly “remember” your earlier messages the way a person does. Instead, the entire conversation so far — everything you typed, everything it replied — gets re-fed into the model as one long block of text each time you send a new message. That block has a hard size limit called the context window, measured in “tokens” (small chunks of text, roughly three-quarters of a word each). Once a conversation grows past that limit, the oldest parts simply fall off the back, and the model genuinely can no longer see them.
Why This Explains the “It Forgot What I Said” Problem
This is the real mechanism behind a frustrating pattern many people notice: an AI assistant that was tracking a project perfectly an hour ago suddenly seems to forget a name, a constraint, or an instruction you gave earlier in the same chat. It’s not being careless — the information has physically scrolled outside its context window and is no longer part of what it’s reading. A long conversation, a huge pasted document, or an extended back-and-forth with lots of code or data can fill up a context window far faster than people expect.
How Big Is a Context Window, Really?
Context window sizes vary widely by model and have grown enormously over the past few years, from a few thousand tokens in early chatbots to hundreds of thousands or more in current frontier models — enough to hold a lengthy book in a single conversation. But bigger isn’t automatically better in every way: even within a large context window, models can pay less attention to information buried in the middle of a very long conversation than to information near the beginning or end, a well-documented effect sometimes called “lost in the middle.”
Context Window vs. Actual Memory
It’s worth separating two different things people often conflate. A context window is temporary and resets with every new conversation unless the product you’re using has a separate, persistent memory feature built on top of it. Persistent memory systems work by saving specific facts to a database outside the conversation and re-inserting only the relevant ones into the context window later, rather than keeping the entire chat history alive forever. The context window itself has no permanent storage at all — it’s closer to short-term working memory than to a notebook.
How to Work With This Instead of Against It
Three habits make a real difference: restate critical facts or constraints periodically in long conversations rather than assuming they’re still “visible”; start a fresh conversation for a genuinely new task instead of bolting it onto an already-long one; and when a conversation is central to ongoing work, summarize the key decisions so far into a short recap message, which effectively refreshes the most important information near the top of the model’s attention. Understanding the context window turns a confusing glitch into a predictable, manageable limitation.
