> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/nullclaw/nullclaw/llms.txt
> Use this file to discover all available pages before exploring further.
# Memory
> Hybrid vector + FTS5 search with multiple backends
## Overview
NullClaw's memory system is a **custom-built hybrid search engine** with:
* **Zero external dependencies** (SQLite FTS5 + custom vector math)
* **Multiple backends**: SQLite, Markdown, Redis, PostgreSQL, LanceDB, API, in-memory LRU
* **Hybrid search**: Weighted merge of keyword (FTS5/BM25) + vector (cosine similarity)
* **Lifecycle management**: Automatic hygiene (archive/purge stale memories), snapshot export/import
* **Embedding providers**: OpenAI, Gemini, Voyage, Ollama, custom
## Memory Interface
```zig theme={null}
pub const Memory = struct {
ptr: *anyopaque,
vtable: *const VTable,
pub const VTable = struct {
name: *const fn(ptr: *anyopaque) []const u8,
store: *const fn(
ptr: *anyopaque,
key: []const u8,
content: []const u8,
category: MemoryCategory,
session_id: ?[]const u8,
) anyerror!void,
recall: *const fn(
ptr: *anyopaque,
allocator: std.mem.Allocator,
query: []const u8,
limit: usize,
session_id: ?[]const u8,
) anyerror![]MemoryEntry,
get: *const fn(
ptr: *anyopaque,
allocator: std.mem.Allocator,
key: []const u8,
) anyerror!?MemoryEntry,
list: *const fn(
ptr: *anyopaque,
allocator: std.mem.Allocator,
category: ?MemoryCategory,
session_id: ?[]const u8,
) anyerror![]MemoryEntry,
forget: *const fn(ptr: *anyopaque, key: []const u8) anyerror!bool,
count: *const fn(ptr: *anyopaque) anyerror!usize,
healthCheck: *const fn(ptr: *anyopaque) bool,
deinit: *const fn(ptr: *anyopaque) void,
};
};
```
## Memory Backends
| Backend | Storage | Search | Use Case |
| ---------------- | ------------------------- | ----------------------------- | ---------------------------- |
| **SQLite** | Single file | FTS5 + vector (in-table BLOB) | Default, embedded, portable |
| **Markdown** | `.md` files | Grep-like keyword search | Human-readable, git-friendly |
| **Redis** | Redis hash | Keyword prefix match | Distributed, fast, ephemeral |
| **PostgreSQL** | Postgres table + pgvector | Full-text + pgvector | Enterprise, scalable |
| **LanceDB** | LanceDB table | Native vector search | AI-first, columnar |
| **API** | Remote HTTP API | Backend-dependent | Centralized, multi-agent |
| **InMemory LRU** | Heap (transient) | Linear scan | Testing, ephemeral sessions |
| **None** | No-op (discard) | Always empty | Stateless mode |
## Hybrid Search Architecture
### Layer A: Primary Store
Persists structured knowledge:
```zig theme={null}
pub const MemoryEntry = struct {
id: []const u8,
key: []const u8,
content: []const u8,
category: MemoryCategory,
timestamp: []const u8,
session_id: ?[]const u8 = null,
score: ?f64 = null,
};
```
**Categories**:
* `core` — Long-term facts ("Alice prefers vim")
* `daily` — Today's context ("Meeting at 3pm")
* `conversation` — Session-specific ("User asked about X")
* Custom — Project-specific (e.g., `"project_alpha"`)
### Layer B: Retrieval Engine
Merges multiple search sources:
```mermaid theme={null}
flowchart LR
Query[User Query] --> Engine[Retrieval Engine]
Engine --> Primary[Primary Adapter]
Engine --> QMD[QMD Adapter]
Primary --> FTS[FTS5 Search]
Primary --> Vector[Vector Search]
QMD --> Files[Workspace .md Files]
FTS --> Merge[RRF Merge]
Vector --> Merge
Files --> Merge
Merge --> Results[Scored Results]
```
**Sources**:
* **Primary adapter**: Keyword (FTS5) + vector (cosine similarity)
* **QMD adapter**: Grep-based search on workspace `.md` files (IDENTITY.md, TOOLS.md, etc.)
**Merge strategy**: Reciprocal Rank Fusion (RRF) + temporal decay
### Layer C: Vector Plane
Optional vector search subsystem:
```mermaid theme={null}
flowchart TD
Store[memory.store] --> Embed[Embedding Provider]
Embed --> Vector[Vector Store]
Vector --> Index[Upsert Index]
Query[memory.recall] --> QueryEmbed[Embed Query]
QueryEmbed --> Search[Cosine Similarity]
Search --> Candidates[Top-K Results]
Candidates --> Merge[Hybrid Merge]
```
**Components**:
* **Embedding provider**: OpenAI, Gemini, Voyage, Ollama
* **Vector store**: SQLite (shared or sidecar), Qdrant, pgvector
* **Circuit breaker**: Auto-disable on repeated failures
* **Outbox**: Durable async sync queue (SQLite-backed)
### Layer D: Lifecycle
**Hygiene** (automatic cleanup):
* **Archive** stale memories after N days (moved to `archive.db`)
* **Purge** ancient memories after M days (deleted)
* **Conversation retention** (session-specific entries expire)
Configuration:
```json theme={null}
{
"memory": {
"lifecycle": {
"hygiene_enabled": true,
"archive_after_days": 30,
"purge_after_days": 90,
"conversation_retention_days": 7
}
}
}
```
**Snapshot** (export/import):
```bash theme={null}
# Export to workspace/snapshots/
nullclaw memory export
# Import from snapshot
nullclaw memory import snapshot_2026-03-01.json
```
**Response cache** (LLM deduplication):
```json theme={null}
{
"memory": {
"response_cache": {
"enabled": true,
"ttl_minutes": 60,
"max_entries": 10000
}
}
}
```
Cache key: `sha256(system_prompt + messages + model + temperature)`
## Configuration
### SQLite Backend (Default)
```json theme={null}
{
"memory": {
"backend": "sqlite",
"auto_save": true,
"search": {
"enabled": true,
"provider": "openai",
"model": "text-embedding-3-small",
"dimensions": 1536,
"query": {
"hybrid": {
"enabled": true,
"vector_weight": 0.7,
"keyword_weight": 0.3,
"rerank": false
}
},
"store": {
"kind": "auto" // sqlite_shared | sqlite_sidecar | qdrant | pgvector
}
}
}
}
```
**Store modes**:
* `auto` — sqlite\_shared if primary is SQLite, else sqlite\_sidecar
* `sqlite_shared` — Reuses primary backend's SQLite db (single file)
* `sqlite_sidecar` — Separate `vectors.db` file
* `qdrant` — Qdrant vector database (requires `qdrant_url`)
* `pgvector` — PostgreSQL pgvector extension (requires `postgres.url`)
### Embedding Providers
```json theme={null}
{
"memory": {
"search": {
"provider": "openai",
"model": "text-embedding-3-small",
"dimensions": 1536,
"fallback_provider": "gemini" // Optional fallback
}
}
}
```
Supported providers:
* **OpenAI**: `text-embedding-3-small`, `text-embedding-3-large`, `text-embedding-ada-002`
* **Gemini**: `text-embedding-004`
* **Voyage**: `voyage-3`, `voyage-3-lite`
* **Ollama**: Local models (e.g., `nomic-embed-text`)
API key resolution: `OPENAI_API_KEY`, `GEMINI_API_KEY`, `VOYAGE_API_KEY`, or config.
### QMD (Query Markdown Documents)
Search workspace markdown files:
```json theme={null}
{
"memory": {
"qmd": {
"enabled": true,
"include_default_memory": true, // Also search primary backend
"search_paths": ["workspace/docs/", "workspace/notes/"],
"file_patterns": ["*.md", "*.mdx"]
}
}
}
```
Searches: `IDENTITY.md`, `TOOLS.md`, `AGENTS.md`, `SOUL.md`, custom paths.
### Reliability
```json theme={null}
{
"memory": {
"reliability": {
"rollout_mode": "on", // on | off | canary | shadow
"fallback_policy": "degrade", // degrade | fail_fast
"circuit_breaker_failures": 5,
"circuit_breaker_cooldown_ms": 60000
},
"search": {
"sync": {
"mode": "best_effort", // best_effort | durable_outbox
"embed_max_retries": 3,
"vector_max_retries": 3
}
}
}
}
```
**Rollout modes**:
* `on` — Hybrid search always
* `off` — Keyword-only search always
* `canary` — Hybrid for N% of queries (config: `canary_percent`)
* `shadow` — Hybrid runs in background, serves keyword results (comparison)
**Fallback policies**:
* `degrade` — Fall back to keyword-only on vector failures
* `fail_fast` — Abort runtime init if vector plane fails
## Hybrid Search Flow
```mermaid theme={null}
flowchart TD
Query[User Query] --> Rollout{Rollout Mode}
Rollout -->|keyword_only| FTS[FTS5 Search]
Rollout -->|hybrid| Embed[Embed Query]
Rollout -->|shadow| Both[FTS + Embed]
Embed --> VectorSearch[Vector Search]
VectorSearch --> Candidates[Top-K Candidates]
FTS --> Keywords[Keyword Results]
Candidates --> RRF[RRF Merge]
Keywords --> RRF
Both --> Log[Log Comparison]
Log --> Keywords
RRF --> Decay[Temporal Decay]
Decay --> Final[Final Results]
```
### Search Pipeline Stages
1. **Query expansion** (optional): Generate synonyms/variations
2. **Adaptive analysis** (optional): Classify query intent (factual vs. semantic)
3. **Retrieval**: Fetch candidates from all sources
4. **RRF merge**: Combine results with reciprocal rank fusion
5. **Temporal decay**: Boost recent memories
6. **LLM rerank** (optional): Use LLM to reorder results
7. **MMR diversification** (optional): Maximal marginal relevance
Configuration:
```json theme={null}
{
"memory": {
"retrieval_stages": {
"query_expansion": { "enabled": false },
"adaptive": { "enabled": false },
"llm_rerank": { "enabled": false },
"mmr": { "enabled": false, "lambda": 0.5 }
}
}
}
```
## Memory Tools Integration
Agent tools interact with memory runtime:
### memory\_store
```json theme={null}
{
"key": "alice_preference_editor",
"content": "Alice prefers vim over emacs",
"category": "core"
}
```
Flow:
1. `memory.store(key, content, category, session_id)`
2. **Async vector sync** (if enabled):
* Embed content via embedding provider
* Upsert to vector store
* Errors logged, never block
### memory\_recall
```json theme={null}
{
"query": "What editor does Alice like?",
"limit": 5
}
```
Flow:
1. `memory_runtime.search(query, limit, session_id)`
2. **Rollout decision** (keyword-only vs. hybrid)
3. **Retrieval engine** merges sources
4. **Returns scored candidates**
## Vector Sync Modes
### Best-Effort (Default)
Immediate, non-blocking:
```zig theme={null}
memory_runtime.syncVectorAfterStore(allocator, key, content);
```
* Embeds content
* Upserts to vector store
* Errors logged, never propagated
* Circuit breaker disables on repeated failures
### Durable Outbox
Queued, retryable:
```json theme={null}
{
"memory": {
"search": {
"sync": {
"mode": "durable_outbox",
"embed_max_retries": 3,
"vector_max_retries": 3
}
}
}
}
```
Flow:
1. `store()` enqueues to SQLite outbox table
2. `drainOutbox()` called periodically (after each agent turn)
3. Retries on failure (exponential backoff)
4. Deletes from outbox on success
## Migration & Interop
### Import from OpenClaw
```bash theme={null}
nullclaw migrate openclaw --dry-run
nullclaw migrate openclaw
```
Migrates:
* Memory entries from `~/.openclaw/brain.db` (SQLite)
* Config from `~/.openclaw/config.json`
* Identity files from `~/.openclaw/workspace/`
### Export Snapshot
```bash theme={null}
nullclaw memory export
# → workspace/snapshots/snapshot_2026-03-01_123456.json
```
Snapshot includes:
* All memory entries (full content)
* Metadata (category, timestamp, session\_id)
* Excludes: autosave entries, bootstrap prompts
### Reindex Vectors
Rebuild vector index after embedding model change:
```bash theme={null}
nullclaw memory reindex
# → Re-embeds all entries, upserts to vector store
```
Reindexing can take minutes/hours for large memory sets and costs API credits (embedding API calls).
## Performance
| Metric | SQLite Backend | Qdrant Backend |
| -------------------- | -------------------------- | ----------------- |
| **Store** | \<1 ms | \<10 ms (network) |
| **Recall (keyword)** | \<5 ms (FTS5) | N/A |
| **Recall (hybrid)** | 50-200 ms (embed + search) | 100-300 ms |
| **Storage** | Single file | Separate service |
| **Scalability** | \~1M entries | \~100M+ entries |
SQLite backend is **fast enough** for most use cases (\<100K memories). Use Qdrant/pgvector for multi-agent deployments or massive datasets.
## Next Steps
Full memory configuration reference
Learn about memory tools