Memory

Overview

NullClaw’s memory system is a custom-built hybrid search engine with:

Zero external dependencies (SQLite FTS5 + custom vector math)
Multiple backends: SQLite, Markdown, Redis, PostgreSQL, LanceDB, API, in-memory LRU
Hybrid search: Weighted merge of keyword (FTS5/BM25) + vector (cosine similarity)
Lifecycle management: Automatic hygiene (archive/purge stale memories), snapshot export/import
Embedding providers: OpenAI, Gemini, Voyage, Ollama, custom

Memory Interface

pub const Memory = struct {
    ptr: *anyopaque,
    vtable: *const VTable,

    pub const VTable = struct {
        name: *const fn(ptr: *anyopaque) []const u8,
        
        store: *const fn(
            ptr: *anyopaque,
            key: []const u8,
            content: []const u8,
            category: MemoryCategory,
            session_id: ?[]const u8,
        ) anyerror!void,
        
        recall: *const fn(
            ptr: *anyopaque,
            allocator: std.mem.Allocator,
            query: []const u8,
            limit: usize,
            session_id: ?[]const u8,
        ) anyerror![]MemoryEntry,
        
        get: *const fn(
            ptr: *anyopaque,
            allocator: std.mem.Allocator,
            key: []const u8,
        ) anyerror!?MemoryEntry,
        
        list: *const fn(
            ptr: *anyopaque,
            allocator: std.mem.Allocator,
            category: ?MemoryCategory,
            session_id: ?[]const u8,
        ) anyerror![]MemoryEntry,
        
        forget: *const fn(ptr: *anyopaque, key: []const u8) anyerror!bool,
        count: *const fn(ptr: *anyopaque) anyerror!usize,
        healthCheck: *const fn(ptr: *anyopaque) bool,
        deinit: *const fn(ptr: *anyopaque) void,
    };
};

Memory Backends

Backend	Storage	Search	Use Case
SQLite	Single file	FTS5 + vector (in-table BLOB)	Default, embedded, portable
Markdown	`.md` files	Grep-like keyword search	Human-readable, git-friendly
Redis	Redis hash	Keyword prefix match	Distributed, fast, ephemeral
PostgreSQL	Postgres table + pgvector	Full-text + pgvector	Enterprise, scalable
LanceDB	LanceDB table	Native vector search	AI-first, columnar
API	Remote HTTP API	Backend-dependent	Centralized, multi-agent
InMemory LRU	Heap (transient)	Linear scan	Testing, ephemeral sessions
None	No-op (discard)	Always empty	Stateless mode

Hybrid Search Architecture

Layer A: Primary Store

Persists structured knowledge:

pub const MemoryEntry = struct {
    id: []const u8,
    key: []const u8,
    content: []const u8,
    category: MemoryCategory,
    timestamp: []const u8,
    session_id: ?[]const u8 = null,
    score: ?f64 = null,
};

Categories:

core — Long-term facts (“Alice prefers vim”)
daily — Today’s context (“Meeting at 3pm”)
conversation — Session-specific (“User asked about X”)
Custom — Project-specific (e.g., "project_alpha")

Layer B: Retrieval Engine

Merges multiple search sources: Sources:

Primary adapter: Keyword (FTS5) + vector (cosine similarity)
QMD adapter: Grep-based search on workspace .md files (IDENTITY.md, TOOLS.md, etc.)

Merge strategy: Reciprocal Rank Fusion (RRF) + temporal decay

Layer C: Vector Plane

Optional vector search subsystem: Components:

Embedding provider: OpenAI, Gemini, Voyage, Ollama
Vector store: SQLite (shared or sidecar), Qdrant, pgvector
Circuit breaker: Auto-disable on repeated failures
Outbox: Durable async sync queue (SQLite-backed)

Layer D: Lifecycle

Hygiene (automatic cleanup):

Archive stale memories after N days (moved to archive.db)
Purge ancient memories after M days (deleted)
Conversation retention (session-specific entries expire)

Configuration:

{
  "memory": {
    "lifecycle": {
      "hygiene_enabled": true,
      "archive_after_days": 30,
      "purge_after_days": 90,
      "conversation_retention_days": 7
    }
  }
}

Snapshot (export/import):

# Export to workspace/snapshots/
nullclaw memory export

# Import from snapshot
nullclaw memory import snapshot_2026-03-01.json

Response cache (LLM deduplication):

{
  "memory": {
    "response_cache": {
      "enabled": true,
      "ttl_minutes": 60,
      "max_entries": 10000
    }
  }
}

Cache key: sha256(system_prompt + messages + model + temperature)

Configuration

SQLite Backend (Default)

{
  "memory": {
    "backend": "sqlite",
    "auto_save": true,
    "search": {
      "enabled": true,
      "provider": "openai",
      "model": "text-embedding-3-small",
      "dimensions": 1536,
      "query": {
        "hybrid": {
          "enabled": true,
          "vector_weight": 0.7,
          "keyword_weight": 0.3,
          "rerank": false
        }
      },
      "store": {
        "kind": "auto"  // sqlite_shared | sqlite_sidecar | qdrant | pgvector
      }
    }
  }
}

Store modes:

auto — sqlite_shared if primary is SQLite, else sqlite_sidecar
sqlite_shared — Reuses primary backend’s SQLite db (single file)
sqlite_sidecar — Separate vectors.db file
qdrant — Qdrant vector database (requires qdrant_url)
pgvector — PostgreSQL pgvector extension (requires postgres.url)

Embedding Providers

{
  "memory": {
    "search": {
      "provider": "openai",
      "model": "text-embedding-3-small",
      "dimensions": 1536,
      "fallback_provider": "gemini"  // Optional fallback
    }
  }
}

Supported providers:

OpenAI: text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002
Gemini: text-embedding-004
Voyage: voyage-3, voyage-3-lite
Ollama: Local models (e.g., nomic-embed-text)

API key resolution: OPENAI_API_KEY, GEMINI_API_KEY, VOYAGE_API_KEY, or config.

QMD (Query Markdown Documents)

Search workspace markdown files:

{
  "memory": {
    "qmd": {
      "enabled": true,
      "include_default_memory": true,  // Also search primary backend
      "search_paths": ["workspace/docs/", "workspace/notes/"],
      "file_patterns": ["*.md", "*.mdx"]
    }
  }
}

Searches: IDENTITY.md, TOOLS.md, AGENTS.md, SOUL.md, custom paths.

Reliability

{
  "memory": {
    "reliability": {
      "rollout_mode": "on",  // on | off | canary | shadow
      "fallback_policy": "degrade",  // degrade | fail_fast
      "circuit_breaker_failures": 5,
      "circuit_breaker_cooldown_ms": 60000
    },
    "search": {
      "sync": {
        "mode": "best_effort",  // best_effort | durable_outbox
        "embed_max_retries": 3,
        "vector_max_retries": 3
      }
    }
  }
}

Rollout modes:

on — Hybrid search always
off — Keyword-only search always
canary — Hybrid for N% of queries (config: canary_percent)
shadow — Hybrid runs in background, serves keyword results (comparison)

Fallback policies:

degrade — Fall back to keyword-only on vector failures
fail_fast — Abort runtime init if vector plane fails

Hybrid Search Flow

Search Pipeline Stages

Query expansion (optional): Generate synonyms/variations
Adaptive analysis (optional): Classify query intent (factual vs. semantic)
Retrieval: Fetch candidates from all sources
RRF merge: Combine results with reciprocal rank fusion
Temporal decay: Boost recent memories
LLM rerank (optional): Use LLM to reorder results
MMR diversification (optional): Maximal marginal relevance

Configuration:

{
  "memory": {
    "retrieval_stages": {
      "query_expansion": { "enabled": false },
      "adaptive": { "enabled": false },
      "llm_rerank": { "enabled": false },
      "mmr": { "enabled": false, "lambda": 0.5 }
    }
  }
}

Memory Tools Integration

Agent tools interact with memory runtime:

memory_store

{
  "key": "alice_preference_editor",
  "content": "Alice prefers vim over emacs",
  "category": "core"
}

Flow:

memory.store(key, content, category, session_id)
Async vector sync (if enabled):
- Embed content via embedding provider
- Upsert to vector store
- Errors logged, never block

memory_recall

{
  "query": "What editor does Alice like?",
  "limit": 5
}

Flow:

memory_runtime.search(query, limit, session_id)
Rollout decision (keyword-only vs. hybrid)
Retrieval engine merges sources
Returns scored candidates

Vector Sync Modes

Best-Effort (Default)

Immediate, non-blocking:

memory_runtime.syncVectorAfterStore(allocator, key, content);

Embeds content
Upserts to vector store
Errors logged, never propagated
Circuit breaker disables on repeated failures

Durable Outbox

Queued, retryable:

{
  "memory": {
    "search": {
      "sync": {
        "mode": "durable_outbox",
        "embed_max_retries": 3,
        "vector_max_retries": 3
      }
    }
  }
}

Flow:

store() enqueues to SQLite outbox table
drainOutbox() called periodically (after each agent turn)
Retries on failure (exponential backoff)
Deletes from outbox on success

Migration & Interop

Import from OpenClaw

nullclaw migrate openclaw --dry-run
nullclaw migrate openclaw

Migrates:

Memory entries from ~/.openclaw/brain.db (SQLite)
Config from ~/.openclaw/config.json
Identity files from ~/.openclaw/workspace/

Export Snapshot

nullclaw memory export
# → workspace/snapshots/snapshot_2026-03-01_123456.json

Snapshot includes:

All memory entries (full content)
Metadata (category, timestamp, session_id)
Excludes: autosave entries, bootstrap prompts

Reindex Vectors

Rebuild vector index after embedding model change:

nullclaw memory reindex
# → Re-embeds all entries, upserts to vector store

Reindexing can take minutes/hours for large memory sets and costs API credits (embedding API calls).

Performance

Metric	SQLite Backend	Qdrant Backend
Store	<1 ms	<10 ms (network)
Recall (keyword)	<5 ms (FTS5)	N/A
Recall (hybrid)	50-200 ms (embed + search)	100-300 ms
Storage	Single file	Separate service
Scalability	~1M entries	~100M+ entries

SQLite backend is fast enough for most use cases (<100K memories). Use Qdrant/pgvector for multi-agent deployments or massive datasets.

Get Started

Core Concepts

Configuration

Deployment

Guides

Overview

Memory Interface

Memory Backends

Hybrid Search Architecture

Layer A: Primary Store

Layer B: Retrieval Engine

Layer C: Vector Plane

Layer D: Lifecycle

Configuration

SQLite Backend (Default)

Embedding Providers

QMD (Query Markdown Documents)

Reliability

Hybrid Search Flow

Search Pipeline Stages

Memory Tools Integration

memory_store

memory_recall

Vector Sync Modes

Best-Effort (Default)

Durable Outbox

Migration & Interop

Import from OpenClaw

Export Snapshot

Reindex Vectors

Performance

Next Steps

Configuration

Tools

​Overview

​Memory Interface

​Memory Backends

​Hybrid Search Architecture

​Layer A: Primary Store

​Layer B: Retrieval Engine

​Layer C: Vector Plane

​Layer D: Lifecycle

​Configuration

​SQLite Backend (Default)

​Embedding Providers

​QMD (Query Markdown Documents)

​Reliability

​Hybrid Search Flow

​Search Pipeline Stages

​Memory Tools Integration

​memory_store

​memory_recall

​Vector Sync Modes

​Best-Effort (Default)

​Durable Outbox

​Migration & Interop

​Import from OpenClaw

​Export Snapshot

​Reindex Vectors

​Performance

​Next Steps

Configuration

Tools

Overview

Memory Interface

Memory Backends

Hybrid Search Architecture

Layer A: Primary Store

Layer B: Retrieval Engine

Layer C: Vector Plane

Layer D: Lifecycle

Configuration

SQLite Backend (Default)

Embedding Providers

QMD (Query Markdown Documents)

Reliability

Hybrid Search Flow

Search Pipeline Stages

Memory Tools Integration

memory_store

memory_recall

Vector Sync Modes

Best-Effort (Default)

Durable Outbox

Migration & Interop

Import from OpenClaw

Export Snapshot

Reindex Vectors

Performance

Next Steps