Agent memory that learns open source software is moving past the demo trick of remembering a chat transcript. Hindsight, the open-source project from Vectorize at github.com/vectorize-io/hindsight, is built around a sharper idea: an agent should not just recall what was said, it should get better at its job as it accumulates experience. It has more than 54,000 stars and over 6,200 forks on GitHub, is written in Python, and is licensed under MIT — so teams can run it, study it, and build on it without a vendor deciding what their agents may remember.
This post is for engineering leads, product teams, and agency developers shipping agents that talk to customers, triage work, or write code across many sessions — and who keep hitting the same wall: every new session starts from zero. You will learn what Hindsight does, how it differs from plain RAG, how to run it in about ten minutes, and where its limits are.
Agent Memory That Learns Open Source: What Makes Hindsight Different
Most agent memory systems focus on recalling conversation history. That is useful, but it is not learning. Remembering that a user prefers concise answers is a fact lookup. Learning that concise answers followed by a worked example close more support tickets, then changing how the agent behaves next time, is a different capability — and it is the one Hindsight is designed for.
The project describes its approach as biomimetic: memory is organised more like human memory than like a flat document store. New information is sorted into world facts and experiences (things the agent itself did and what happened next). Over time, related facts consolidate into observations — deduplicated beliefs backed by evidence, with exact quotes and a proof count. Observations are refined rather than overwritten when new evidence arrives, so a belief can strengthen, weaken, or gain nuance without losing its history.
On top of that sit mental models and knowledge pages. A mental model is a standing answer to a question about a bank, such as what a user prefers or what the open risks on a project are. Define the question once; Hindsight writes the answer, stores it, and rewrites it in the background as the bank learns more. Reading one is a database read, not a retrieval plus a model call, so an agent can start a session with settled knowledge instead of rediscovering it every time. Knowledge pages are the same idea with the mechanics hidden: living documents the bank writes about itself, organised like a wiki and projectable onto disk as Markdown.
What Hindsight Can Do
Strip away the marketing and Hindsight is a memory server with three operations, a set of integrations, and a deployment story from laptop to managed cloud. The list below is taken from the project README and documentation.
- Three memory operations: retain, recall, and reflect, available through Python, Node.js, Go, CLI, and REST clients.
- Four-way retrieval: semantic vector search, BM25 keyword matching, graph traversal over entity and temporal links, and time-range filtering, merged with reciprocal rank fusion and a cross-encoder reranker.
- 60+ integrations: coding agents including Claude Code, Codex, Cursor, and GitHub Copilot; frameworks including LangGraph, CrewAI, and Pydantic AI; and no-code tools including n8n and Zapier — most needing no code changes.
- 25+ language-model providers: hosted providers such as OpenAI, Anthropic, Gemini, and DeepSeek, local options such as Ollama and llama.cpp, any OpenAI-compatible endpoint, and LiteLLM gateways. Existing ChatGPT Plus or Pro, Claude Pro or Max, Cursor, and GitHub Copilot subscriptions work without a separate API key.
- A two-line LLM wrapper: wrap an existing OpenAI or Anthropic client and Hindsight recalls memories before each call and retains the conversation afterwards.
- A built-in MCP server: every server exposes a Model Context Protocol endpoint per bank, so any MCP client can use retain, recall, and reflect as tools.
- Production operations: PostgreSQL with pgvector or Oracle AI Database storage, Prometheus metrics, an admin CLI, webhooks for memory lifecycle events, and hierarchical configuration down to per-bank settings.
The headline benchmark claim deserves a careful reading. The project reports state-of-the-art performance on LongMemEval, a widely used long-term memory benchmark, and says its results were independently reproduced by collaborators at Virginia Tech’s Sanghani Center and The Washington Post, with live results at benchmarks.hindsight.vectorize.io. Treat that as a strong signal, not a guarantee: benchmark leaders change, and your workload is not LongMemEval. The honest test is a small pilot on your own data, which the open licence makes cheap to run.
How It Works: Retain, Recall, Reflect
Everything in Hindsight flows through three operations. Understanding them is understanding the product.
Retain pushes new information into a memory bank. Behind the scenes it uses a language model to extract facts, temporal data, entities, and relationships, then normalises them into canonical entities, time series, and search indexes. A call looks like this:
client.retain(bank_id="my-bank", content="Alice got promoted to senior engineer", context="career update")
Recall retrieves memories, and is deliberately broader than a vector search. A query such as “What happened in June?” exercises the temporal path, while “What does Alice do?” exercises semantic and graph paths. Results from all four strategies are merged, reranked, and trimmed to a token budget, so the agent receives compact, relevant context rather than a dump of everything ever stored.
Reflect is the operation that justifies the word “learns”. Instead of answering from a lookup, reflect analyses existing memories to form new connections — the kind of question a human manager asks, such as which outreach messages earned replies and why, or which project risks need mitigating. The project’s examples include a sales agent reflecting on why certain messages worked and a support agent spotting questions customers keep asking that documentation does not answer. Reflect costs more than recall, so use it for periodic synthesis — nightly reviews, weekly reports, pre-meeting briefs — rather than on every turn.
Memories live in banks, the unit of isolation: one bank per user, customer, or project, with metadata restricting which memories a retrieval may touch. For multi-client agents, leakage between tenants is a data-protection incident, not a bug report — design bank boundaries before wiring up the wrapper, not after.
Getting Started in About Ten Minutes
The fastest honest evaluation is the Docker route. Start the server, point a client at it, retain one fact, recall it, and then test reflect on real conversations from your own work. Do not evaluate memory systems on toy data alone; toy data flatters every system.
- Start the server with Docker. The documented command runs the image from the GitHub Container Registry, exposes the API on port 8888 and the UI on port 9999, and persists data in a Docker volume. Set your language-model API key as an environment variable first.
- Install a client. Python users install the client package, Node.js users install the published npm package, and Go users fetch the client module. A CLI is also available via an install script from the documentation site.
- Retain, recall, reflect. Connect to the local server, retain real facts with context and timestamps, recall them with a natural question, then run reflect and judge whether the synthesis tells you anything new.
- Try the zero-code path. If you already have an agent built on the OpenAI or Anthropic SDK, wrap the client with the Hindsight LiteLLM wrapper, pass a bank ID, and watch memories accumulate automatically.
- Try the coding-agent path. The coding-agents installer wires a per-repository bank built from git history and past sessions into Claude Code, Codex, Cursor, and other CLI agents, with knowledge pages covering architecture, conventions, and in-flight work. This is where we see the most immediate payoff for agencies: the agent stops re-asking questions the last session already answered.
For a laptop-only experiment with no server at all, the project offers an embedded Python option that starts a server in-process. That is ideal for a spike; for anything shared, run the Docker server or the managed cloud offering, which promises usage-based pricing and a 99.9% uptime service level.
From Our Experience: Where This Pays Off, and Where It Does Not
After nine years and more than 300 projects, the pattern we keep seeing is that teams adopt agent tooling for the demo and abandon it for the maintenance. Memory is usually the reason: an agent that cannot remember decisions, preferences, and past failures forces humans to re-supervise every session, and supervision is the cost agent tooling was supposed to remove. Hindsight attacks the right problem, but it introduces costs of its own that a buyer should price honestly.
First, memory quality depends on extraction and bank discipline. Retain everything and you get a noisy bank that retrieves confidently wrong context; retain too little and the agent learns nothing. Budget time for bank hygiene, as you would for database maintenance. Second, every retain and reflect call spends model tokens, so memory has a running cost that scales with volume — measure tokens per retained conversation and set a recall budget per call. Third, the two-line wrapper is easy to install badly: automatic retention of every conversation, including sensitive ones, is a privacy decision, not a default. Decide what must never be retained — credentials, payment details, health information, anything a client has marked confidential — and enforce it before the wrapper goes live.
Where we would reach for it first: support and sales agents that handle repeat customers, coding agents working on long-lived codebases, and internal AI employees that own a recurring workflow. Where we would not: one-shot question answering, simple n8n-style workflows with no cross-session state (the project itself says it may be overkill there), and any workload where the data cannot legally be stored for learning. If your agent’s sessions are independent by design, a memory system is overhead, not leverage. Teams building persistent automation can talk to us through our n8n workflow automation service, and the wider landscape is tracked in our AI Tools Directory.
Frequently Asked Questions
Is Hindsight really open source?
Yes. The repository is licensed under MIT, the code is public on GitHub, and you can self-host it with Docker, pip, or Helm. A managed cloud offering exists for teams that prefer not to operate it, but the open-source server is the full system, not a crippled community edition.
How is Hindsight different from RAG?
RAG retrieves documents relevant to a query. Hindsight retrieves memories, consolidates them into evidence-backed observations, maintains standing mental models, and can reflect over memories to form new conclusions. RAG answers “what was said”; Hindsight aims to answer “what have we learned”.
Which language models does Hindsight work with?
More than 25 providers, including OpenAI, Anthropic, Gemini, Groq, DeepSeek, and local options such as Ollama and llama.cpp, plus any OpenAI-compatible endpoint and LiteLLM gateways covering more than 100 models. Existing ChatGPT, Claude, Cursor, and GitHub Copilot subscriptions can also be used without a separate API key.
Can I use Hindsight without running a server?
Yes, in two ways: the embedded Python option runs a server in-process with no separate deployment, and Hindsight Cloud is a managed option with usage-based pricing. For a quick experiment, embedded is fastest; for a shared team deployment, run the Docker server or use the cloud.
What should I not store in Hindsight?
Anything you would not store in a database with the same access controls: credentials, payment details, health information, and client-confidential material that has not been cleared for retention. Memory systems compound both value and risk, so set retention boundaries and per-bank isolation before enabling automatic retention.
Agent memory that learns is not a feature to bolt on at the end of a project; it is an architecture decision that shapes how your agents behave over months. Hindsight gives teams a credible open-source way to make that decision without surrendering their data to a closed platform — provided they treat bank design, retention policy, and running cost as seriously as the model choice. If you are building agents that should get smarter with every customer, ticket, and commit, this is worth a weekend pilot. And if you would rather have a team that has shipped this kind of system build the pilot with you, talk to MTD Technologies — we will tell you honestly whether memory is your bottleneck, or whether a simpler system would serve you better.