Building a Local AI Memory Repository

Ammar Najjar · Posted on October 21, 2026 · 9 min readaiproductivity

AI-generated image

In When AI Memory Doesn’t Follow You Between Tools, I described how I used Obsidian to share knowledge between ChatGPT, Claude Code, and local AI models. That approach worked, but retrieving relevant information and preserving new observations still depended on me initiating each step.

I wanted to reduce that effort while keeping control over the information I stored and shared. I started with a separate Markdown repository that agents with filesystem access could use as a common source of persistent context. Tools without direct access to that repository still needed a separate mechanism for receiving the information a conversation required.

The Case for a Local Repository

My main requirements were direct access to stored information and control over how it was shared. Keeping the repository on my machine lets me inspect, edit, and version its contents through Git. When an agent uses a hosted model, however, any context it supplies is processed outside my machine, so I still need to decide which information each environment can access.

I chose Markdown as the format because I already use it extensively in Obsidian and development workflows, and AI agents handle plain, human-readable files well. At this scale, plain files required no new tooling or conventions.

What matters is that the memory lives in one local place independent of any particular AI tool. The format is interchangeable; each environment still needs a mechanism to access the repository and load relevant information, but the records remain the same regardless of which tool reads them.

Separation of Concerns: Vault vs Memory

I already maintained a knowledge management system in Obsidian (Keeping Knowledge Useful), which made me ask why I should keep a second place for information.

The distinction comes from how I use the two systems.

The Obsidian vault contains detailed source material: meeting notes, technical decisions, reading notes, and observations from professional experiences. These records preserve original context and the details that may need revisiting later.

I designed the memory repository as a smaller, more general layer of information for AI tools to use across conversations. It includes relatively stable background information, working preferences, recurring observations, and context that may influence future discussions.

Sensitive source material stays out of the memory repository. When a task requires details from an original note, I make that material available selectively through the appropriate workflow.

This separation matters for privacy. Storing Markdown files locally controls storage, but not where contents are processed. A local model processes information on the machine; a hosted model receives whatever context the agent provides. The distinction determines which information each environment should use.

Organizing Stable and Evolving Information

I organized the repository around two directories because I wanted to distinguish relatively stable information from observations that would continue to evolve.

I put relatively stable information in context/: background, writing preferences, and working principles. These records provide consistent context across conversations, and I review changes deliberately.

I put information that develops through ongoing work in memory/: observations, recurring patterns, unresolved questions, and records replaced by newer information.

A simplified version of the structure:

my-ai-memory/
├── AGENTS.md
├── context/
│   ├── profile.md
│   └── principles.md
└── memory/
    ├── inbox.md
    ├── patterns.md
    └── superseded.md

The directory structure reflects how information is expected to be used and maintained. Each record’s status determines how confidently an agent should rely on it. A record in context/ is not automatically confirmed; a recent observation in memory/ is not necessarily unreliable. The instructions governing how agents use these records live separately in AGENTS.md.

A stable context file describing working environment and constraints:

# Working environment

- Prefer local tools that run on my hardware.
- Use version control for all tracked changes.
- Document configuration decisions alongside their reasoning.

When an agent loads this file, these preferences apply without needing to be repeated.

The evolving layer serves a different purpose. An inbox entry preserves an observation before its status is determined:

# Inbox

## Pattern: Debug logs expose infrastructure state
- **Confidence:** Low (single observation)
- **Source:** 2026-08-12 deployment discussion
- **Status:** Needs review

Observation: Container debug logs included internal IP addresses and service topology, revealing infrastructure details that should not appear in standard output.

This includes confidence level, source reference, and explicit status. Keeping it separate from established context allows revisiting the concern without treating an initial interpretation as a confirmed conclusion.

Records of superseded information remain documented with enough context to understand older references while making clear which information is current. Recurring patterns retain supporting observations and the uncertainty surrounding each conclusion.

Intended Workflow: From Conversation to Repository

The instructions in AGENTS.md described the following workflow. It represented what I expected agents to do when a conversation produced useful information.

Conversation ends
    │
    ▼
Agent reviews conversation for useful information
    │
    ▼
Decision: Preserve observation?
    │
    ├── No → nothing recorded
    └── Yes → Write to memory/inbox.md
                    │
                    ▼
            Confidence + source + status recorded
                    │
                    ▼
            (Optional) If stable, propose context/ update for review

The instructions cover capture, confidence marking, and the proposal of changes to stable context. The repository holds the records and the instructions define the process, but each step still depended on the agent choosing to follow it during an ordinary task.

Handling Conflicting Information

As the repository grows, different records can describe the same topic at different points in time. A preference may change, an earlier observation may prove incomplete, or a decision may be replaced.

The initial instructions included a precedence hierarchy: explicit instructions first, then recent evidence, then current recorded state, then older notes.

That hierarchy provided a simple starting point, but I began questioning how well it handled differences in confidence. A recent observation could still be tentative, while an older record might contain a confirmed decision, so the source and confidence of each record needed attention alongside its age.

I left the precedence hierarchy as written. The open question was how well recency, confidence, and authority would reconcile when they pointed in different directions. The repository can represent those differences, but it cannot enforce how an agent resolves them; that depends on the instructions the agent follows.

Connecting Agents to Memory

The repository is plain text, shared across environments, but a model does not automatically gain access to it. An agent needs filesystem access or another integration that can read the relevant files and supply their contents to the model.

In my local AI workflow, coding agents had filesystem access and could read the memory repository directly through standard filesystem operations. Claude Code accessed the repository files the same way.

ChatGPT presented a different case. Without assumed direct access to the repository, I supplied the relevant information manually, selecting and pasting from the Markdown files as a conversation required.

Each environment therefore needs its own mechanism to access the files and load shared instructions. The repository provides a common source of information; how each tool reaches it depends on that environment’s capabilities and configuration.

Memory Instructions via AGENTS.md

The shared instructions live in an AGENTS.md file at the repository root.

Different agents use different conventions for project instructions. Some recognize AGENTS.md; others require their own configuration or an explicit instruction to load it. The file serves as the shared location for memory guidance rather than assuming every agent follows it automatically.

The instructions cover how agents read and update memory. Before working on a topic, an agent should consult relevant context and check existing records to avoid unnecessary duplication. When recording an observation, uncertainty and sufficient source information should be preserved to understand where it came from.

Stable information is handled more carefully than evolving records. New observations are recorded in evolving memory; proposed changes to established context are flagged for review. These are behavioral instructions rather than enforced permissions. An agent can technically edit any file its filesystem access allows. Whether it respects the distinction depends on the instructions it follows.

Capturing new information follows the same pattern. When a conversation reaches a useful conclusion, the agent considers what should be preserved and records it in the appropriate place, including confidence markers and source references for new observations.

This approach describes a common memory workflow across agents while keeping the repository independent of any particular tool.

Limitations of an Instruction-Driven Approach

The repository solved several problems with memory tied to individual AI tools. Stored information is inspectable, mistakes are correctable, and changes are versioned through Git. Knowledge remains available independently of any specific agent or model, provided the environment was connected to the repository. The separation between established context and developing observations is clearer.

The remaining problem is keeping memory current and retrievable.

Three failure modes have emerged:

Silent non-capture. An agent completes a task without recording an observation that would have been useful later. The instructions ask it to review conversations and preserve relevant information, but that step depends on the agent recognizing the need and acting on it during ordinary execution.

Missed retrieval. Relevant information exists in the repository but is never loaded when starting a related task. Loading every file into every session would consume unnecessary context, so selecting the right information matters more as the collection grows.

No audit trail of misses. Both problems are difficult to notice during the conversation itself. The immediate response can be useful even when memory was not updated or relevant information was not retrieved. Gaps reveal themselves only when reviewing the files or returning to a topic later, with no record of what was missed.

Improving the instructions helped in some situations. The underlying process remains dependent on the conversational agent following those instructions during task execution. There is no separate mechanism to confirm that a conversation was checked for useful information, identify a missed update, or recover work that was not processed.

These gaps made me consider moving parts of memory maintenance outside the conversational agent’s normal workflow. I wanted a way to check whether useful information had been captured, to make missed processing visible, and to recover from interruptions. Retrieval needed similar attention, although determining which information should have been used in a particular session remained a separate challenge. An independent review or audit was one approach I began exploring for this.

Ongoing Work

The Markdown files gave me an inspectable, editable, and portable foundation for AI memory, and the instruction-driven approach held up in everyday use. What remained difficult was dependable capture and retrieval, and I began exploring ways to make those operations more explicit and observable so the repository stays useful as it grows. That work is ongoing, and the next article in this series examines the reliability problem.

If you maintain memory for your own AI tools, how do you decide what should be stored, and how much responsibility do you give the agent for keeping that information current?