TL;DR
AI agents need structured memory that understands how customers, orders, products, and processes relate to each other. First-generation RAG was built on vector search - great for text similarity, but blind to relationships. Knowledge Graphs are the next step: a semantic layer on top of existing systems that gives agents real contextual understanding. Once built, any number of use cases can be layered on top.
Contents
- How has RAG evolved?
- What exactly is Agent Memory?
- What is an ontology - and why does it matter?
- What happens when agents don’t have structured memory?
- How do you build Agent Memory?
- Where is the long-term leverage?
- FAQ
How has RAG evolved?
In short: RAG hasn’t failed - it’s evolving. The first generation was purely vector-based. Today, retrieval also includes graphs, structured queries, and web search.
Retrieval Augmented Generation was an important breakthrough: instead of letting an LLM work only with its training corpus, you feed it relevant context from external sources at runtime. The first wave was almost exclusively vector-based - documents get converted into embeddings, stored in a vector database, and retrieved via semantic similarity.
This works surprisingly well as long as the task is text-based: “Find the section in our documentation that best matches this question.” For a support chatbot or internal knowledge search, that’s often sufficient.
But retrieval is a broader concept than vector search. RAG means: giving the model the right information at runtime. The source can be a vector database - or a web search, a SQL query against the ERP, an API call to the CRM, or a traversal through a Knowledge Graph.
Graph RAG is also RAG. The difference isn’t in the principle but in the type of retrieval. And that type makes a significant difference once the task goes beyond “find a similar text.”
A concrete example: An agent needs to verify whether an incoming order is complete. Vector search might find similar orders. But the agent needs to know: Is this customer approved? Do they have outstanding invoices? Is the referenced product available in this configuration? Who is the responsible account manager? These aren’t text similarity questions - they’re relationship questions.
What exactly is Agent Memory?
In short: A Knowledge Graph that serves as structured memory for AI agents - mapping not just content, but entities and their relationships to each other.
Agent Memory is built on a graph database, typically Neo4j or comparable systems. The core principle: there are nodes (entities like customers, orders, products, employees) and edges (relationships like “has ordered,” “is managed by,” “belongs to project”).
This sounds like what a relational database can do. The difference lies in the flexibility and in how queries work.
In a relational database, I define tables with fixed columns and connect them via foreign keys. If I want to map a new type of relationship - say “Employee X reviewed Project Y” - I need a new table or at least a schema change. And queries across multiple relationship levels (from customer to order to line item to product to supplier) quickly become complex and slow.
In a Knowledge Graph, a new relationship is simply a new edge. And traversal - “give me everything connected to this customer, up to three levels deep” - is exactly what graph databases are optimized for.
For an agent, this means: it can answer questions that span multiple systems and relationships without someone having to pre-define exactly which joins are needed. It navigates the graph the way an experienced employee draws on their knowledge of the company - associatively, context-aware, through connections.
What is an ontology - and why does it matter?
In short: The ontology defines what types of things exist in your company and how they relate - it’s the map that the Knowledge Graph is built from.
In practice, ontology means: What entity types exist in our business? Customer, order, product, employee, location, project. What relationship types exist? “Customer has order,” “order contains line item,” “line item references product,” “employee manages customer.”
This sounds trivial, but most companies have never made it explicit. Every system has its own implicit data model. In the ERP it’s called “debtor,” in the CRM “account,” in the project management software “client” - and they all mean the same customer, but none of the systems know that.
The ontology makes these implicit models explicit. It’s the shared language that says: “A customer is a customer, regardless of whether the ERP calls them a debtor.” In computer science, this is called a canonical representation.
In practice, this means: when an agent searches the Knowledge Graph for the customer “Mueller GmbH,” it finds everything - orders from the ERP, contact history from the CRM, project data from the PM tool, support tickets from the helpdesk. Not because all these systems are connected, but because the graph knows it’s the same entity.
The ontology doesn’t have to be perfect on day one. In practice, we start with the core entities - customers, orders, products - and expand iteratively. The graph is flexible enough to accommodate new entity types and relationships without rebuilding everything.
What happens when agents don’t have structured memory?
In short: Without structured memory, agents make assumptions, lose context, and produce results that look good on paper but don’t hold up in reality.
We see this in projects time and again. A few typical scenarios:
Duplicate work and contradictions. Two agents work on the same customer - one handles a proposal, another a support request. Without shared memory, neither knows about the other. The support agent recommends a workaround while the sales agent is offering a solution that would fix the underlying problem. The result: the customer gets an inconsistent picture.
Context loss across process boundaries. An agent takes an order and passes it to production. But it can’t convey that the customer had a complaint last time and explicitly values quality control this time around. That information exists - in a CRM notes field, in an email, maybe in the account manager’s head. But the agent can’t access it because it’s not in any structured format.
Hallucinated connections. Without clear data structure, an LLM starts inferring connections that don’t exist. “Customer A is probably interested in Product X because similar customers ordered it.” That might be true, but it might not. And when the basis is a vector search over unstructured documents, there’s no way to verify it. In a Knowledge Graph, every statement can be traced back to its source.
Scaling problems. The first chatbot works. So you build a second. And a third. Each has its own knowledge base, its own documents, its own vector database. After six months you have five isolated systems that don’t know about each other and partially contain contradictory information.
How do you build Agent Memory?
In short: Define the ontology, map existing data as a semantic layer, connect agents incrementally - without replacing source systems.
Develop the ontology
The first step isn’t technical but conceptual: How does work flow through our company? This can be worked out in a few focused workshops. We look at what entities exist, what relationships exist between them, and which attributes are relevant.
AI can accelerate this process. Feed a model sample orders, emails, project documentation, and it proposes entity types and relationships. This doesn’t replace domain expertise, but it gives you a fast first draft that you can validate with the right people.
Build the Knowledge Graph as a semantic layer
A critical point: the Knowledge Graph doesn’t replace ERP, CRM, or other systems. It’s a semantic layer on top - a machine-readable model of the company that connects concepts from different systems.
Data flows from source systems into the graph - via sync, events, or batch imports. The source systems remain the “source of truth” for their respective domains. The graph becomes the “source of context” - it knows how things connect.
This also means not every data field needs to be in the graph. The billing address stays in the ERP. The graph only records: “Customer A has Address B,” with a reference to the ERP for the details. Less is more here.
Connect agents incrementally
The third step is letting agents work against the graph. This works best incrementally: start with one use case, for example order processing. The agent learns to traverse the graph, identify missing information, and ask targeted questions.
Once that’s running, add the next use case - account management, reporting, project coordination. Each new agent benefits from what’s already in the graph while simultaneously enriching it.
Where is the long-term leverage?
In short: A company’s ontology rarely changes. Technologies change constantly. Investing in the data structure makes you independent of the next technology shift.
Many companies are currently investing in specific AI tools: a chatbot here, an automation there. This delivers short-term results, but it doesn’t create a foundation you can build on.
The Knowledge Graph flips this. It’s the stable layer on which everything else moves. Models get better - the graph remains. New agent frameworks arrive - the graph remains. How agents work will change dramatically over the next two years. How a company is structured will not.
In practice, this means: the effort for the first use case is the largest, because the ontology needs to be defined and the graph initially populated. The second use case goes faster. By the third, the investment starts paying off because the core structure is in place and new agents simply plug in.
That’s the fundamental difference from an approach where you build a separate solution for each use case. Instead of five isolated projects, you have one system that becomes more valuable with every new agent.
FAQ
Do I need to replace my ERP?
No. The Knowledge Graph works as a semantic layer on top of existing systems. ERP, CRM, and other tools remain the leading systems for their respective domains. The graph connects them but doesn’t replace them. Data flows into the graph via sync or events - and back out when needed.
How much effort is the initial build?
Less than most expect. The ontology can be developed in a few workshops. Populating the data can happen with AI assistance - a model extracts entities and relationships from existing data sources. The real effort isn’t technical but substantive: answering the question of how work flows through the company.
What’s the difference between a Knowledge Graph and a knowledge base?
A knowledge base (e.g., Confluence or a collection of documents in a vector database) stores content. A Knowledge Graph stores entities and relationships. The difference: from a knowledge base, I get text passages that are semantically similar to my question. From a Knowledge Graph, I get structured answers - “Customer A has three open orders, the largest is Order B with 12 line items, managed by Employee C.”
Can’t I just start with a chatbot?
You can, and for a clearly defined use case, that’s reasonable. The problem arises with scaling. Each chatbot needs its own knowledge base that needs to be maintained. By the second or third use case, you have redundant data, inconsistent answers, and growing maintenance overhead. The Knowledge Graph avoids this by creating a shared foundation for all agents.
Do I need special infrastructure for this?
Graph databases like Neo4j run in the cloud or on-premise. Resource requirements are moderate - for most mid-sized companies, a single instance is enough. The bigger question isn’t infrastructure but data modeling: Who defines the ontology, who maintains the data flows, who ensures quality?
Conclusion
Agent Memory is the foundation if AI in your business should go beyond a single chatbot. The Knowledge Graph as a semantic layer on top of existing systems gives agents the contextual knowledge they need to work effectively in complex business processes. The first step isn’t the technology - it’s the question: What entities exist in our business, how do they connect, and how does work flow through our company?
If you want to explore what Agent Memory could look like for your organization, we’d be happy to talk.
Based on an episode of the SNKI podcast “KI im B2B” with Manuel Zorzi and Michael Kirchberger: Listen now →