Memory Backends

Configure SQLite, markdown, and vector search backends for NullClaw’s hybrid memory system

NullClaw features a hybrid memory system combining vector embeddings, keyword search (FTS5), and traditional storage backends - all with zero external dependencies.

Overview

NullClaw’s memory architecture:

Vector Search

Cosine similarity search on embeddings stored as BLOB

Keyword Search

FTS5 virtual tables with BM25 scoring

Hybrid Merge

Weighted combination of vector + keyword results

Memory Backends

SQLite (Default)

Full-featured backend with vector + FTS5 search:

config.json

{
  "memory": {
    "backend": "sqlite",
    "auto_save": true,
    "embedding_provider": "openai",
    "vector_weight": 0.7,
    "keyword_weight": 0.3
  }
}

Features:

• Vector embeddings as BLOB
• FTS5 full-text search
• Automatic archival & purge
• Snapshot export/import
• Transaction safety

Storage location: ~/.nullclaw/memory.db

Markdown (Simple)

Plain-text append-only storage:

config.json

{
  "memory": {
    "backend": "markdown",
    "auto_save": true,
    "profile": "markdown_only"
  }
}

Layout:

• workspace/MEMORY.md - Curated long-term memory
• workspace/memory/YYYY-MM-DD.md - Daily logs

Features:

• Human-readable
• Git-friendly
• Append-only (forget() is no-op)
• No search indexing

Use cases:

• Audit trails
• Simple deployments
• Text-first workflows

Vector Search Configuration

Embedding Providers

OpenAI

config.json

{
  "memory": {
    "search": {
      "enabled": true,
      "provider": "openai",
      "model": "text-embedding-3-small",
      "dimensions": 1536
    }
  },
  "models": {
    "providers": {
      "openai": {
        "api_key": "sk-..."
      }
    }
  }
}

Custom URL

config.json

{
  "memory": {
    "search": {
      "enabled": true,
      "provider": "custom",
      "custom_url": "https://your-embedding-api.com/v1/embeddings",
      "model": "your-model",
      "dimensions": 768
    }
  }
}

None (Disabled)

config.json

{
  "memory": {
    "search": {
      "enabled": false,
      "provider": "none"
    }
  }
}

Uses FTS5 keyword search only.

Embedding Models

Provider	Model	Dimensions	Cost
OpenAI	text-embedding-3-small	1536	Low
OpenAI	text-embedding-3-large	3072	Medium
OpenAI	text-embedding-ada-002	1536	Low
Custom	your-model	Varies	Varies

Vector Store Options

Auto (SQLite)

"search": {
  "store": {
    "kind": "auto"
  }
}

Stores embeddings as BLOB in SQLite.

Sidecar File

"search": {
  "store": {
    "kind": "sidecar",
    "sidecar_path": "~/.nullclaw/embeddings.bin"
  }
}

Separate binary file for embeddings.

Qdrant

"search": {
  "store": {
    "kind": "qdrant",
    "qdrant_url": "http://localhost:6333",
    "qdrant_api_key": "your-key",
    "qdrant_collection": "nullclaw_memories"
  }
}

External Qdrant vector database.

PostgreSQL pgvector

"search": {
  "store": {
    "kind": "pgvector",
    "pgvector_table": "memory_embeddings"
  }
},
"postgres": {
  "url": "postgresql://user:pass@localhost:5432/nullclaw"
}

PostgreSQL with pgvector extension.

Hybrid Search

Combine vector and keyword search:

config.json

{
  "memory": {
    "search": {
      "query": {
        "max_results": 6,
        "min_score": 0.0,
        "merge_strategy": "rrf",
        "rrf_k": 60,
        "hybrid": {
          "enabled": true,
          "vector_weight": 0.7,
          "text_weight": 0.3,
          "candidate_multiplier": 4
        }
      }
    }
  }
}

Merge Strategies

RRF (Reciprocal Rank Fusion)

"merge_strategy": "rrf",
"rrf_k": 60

Combines rankings from vector + keyword search. Best for balanced results.

Weighted

"merge_strategy": "weighted",
"hybrid": {
  "vector_weight": 0.7,
  "text_weight": 0.3
}

Linear combination of normalized scores.

Vector Only

"merge_strategy": "vector_only"

Semantic search only (ignores keyword results).

Keyword Only

"merge_strategy": "keyword_only"

BM25 full-text search only (ignores embeddings).

Advanced Query Options

config.json

{
  "memory": {
    "search": {
      "query": {
        "hybrid": {
          "mmr": {
            "enabled": true,
            "lambda": 0.7
          },
          "temporal_decay": {
            "enabled": true,
            "half_life_days": 30
          }
        }
      }
    }
  }
}

MMR (Maximal Marginal Relevance):

• Diversifies results to reduce redundancy
• lambda: 0.0 = max diversity, 1.0 = max relevance

Temporal Decay:

• Boosts recent memories
• half_life_days: Days until score halves

Chunking

Split long memories into smaller chunks:

config.json

{
  "memory": {
    "search": {
      "chunking": {
        "max_tokens": 512,
        "overlap": 64
      }
    }
  }
}

• max_tokens: Maximum tokens per chunk
• overlap: Overlapping tokens between chunks

Memory Lifecycle

Automatic Hygiene

config.json

{
  "memory": {
    "lifecycle": {
      "hygiene_enabled": true,
      "archive_after_days": 7,
      "purge_after_days": 30,
      "conversation_retention_days": 30
    }
  }
}

1. Archive

   After archive_after_days, memories are marked archived (not returned in searches).

2. Purge

   After purge_after_days, archived memories are permanently deleted.

3. Conversation Retention

   Full conversation logs retained for this many days.

Snapshots

Export/import full memory state:

config.json

{
  "memory": {
    "lifecycle": {
      "snapshot_enabled": true,
      "snapshot_on_hygiene": true
    }
  }
}

Export snapshot:

nullclaw memory export snapshot.json

Import snapshot:

nullclaw memory import snapshot.json

Performance Tuning

Response Caching

config.json

{
  "memory": {
    "response_cache": {
      "enabled": true,
      "ttl_minutes": 60,
      "max_entries": 5000
    }
  }
}

Caches search results for identical queries.

Embedding Sync

config.json

{
  "memory": {
    "search": {
      "sync": {
        "mode": "best_effort",
        "embed_timeout_ms": 15000,
        "vector_timeout_ms": 5000,
        "embed_max_retries": 2,
        "vector_max_retries": 2
      }
    }
  }
}

Modes:

• best_effort: Continue even if embedding fails
• strict: Fail if embedding fails
• async: Background embedding (non-blocking)

Query Optimization

config.json

{
  "memory": {
    "search": {
      "query": {
        "max_results": 6,
        "candidate_multiplier": 4
      },
      "cache": {
        "enabled": true,
        "max_entries": 10000
      }
    }
  }
}

• max_results: Final results returned
• candidate_multiplier: Fetch max_results * multiplier before reranking
• Cache frequently-accessed embeddings

Migration

From OpenClaw

# Dry run (preview changes)
nullclaw migrate openclaw --dry-run

# Execute migration
nullclaw migrate openclaw

# From custom path
nullclaw migrate openclaw --source /path/to/openclaw

Migrates:

• Memory entries
• Embeddings
• Config structure
• Session history

Between Backends

1. Export from Old Backend

   nullclaw memory export old_backend.json
   
2. Change Backend in Config

   "memory": { "backend": "sqlite" }
   
3. Import to New Backend

   nullclaw memory import old_backend.json
   

External Memory Engines

NullClaw supports pluggable memory backends:

Redis

config.json

{
  "memory": {
    "backend": "redis",
    "redis": {
      "host": "127.0.0.1",
      "port": 6379,
      "password": "",
      "db_index": 0,
      "key_prefix": "nullclaw",
      "ttl_seconds": 0
    }
  }
}

PostgreSQL

config.json

{
  "memory": {
    "backend": "postgres",
    "postgres": {
      "url": "postgresql://user:pass@localhost:5432/nullclaw",
      "schema": "public",
      "table": "memories",
      "connect_timeout_secs": 30
    }
  }
}

Requires PostgreSQL 12+ with pgvector extension for vector search.

API Backend

config.json

{
  "memory": {
    "backend": "api",
    "api": {
      "url": "http://localhost:8080",
      "api_key": "your-key",
      "timeout_ms": 10000,
      "namespace": ""
    }
  }
}

Connect to external memory service.

Troubleshooting

Embedding Failures

# Check embedding provider config
cat ~/.nullclaw/config.json | jq '.memory.search.provider'

# Test API key
curl https://api.openai.com/v1/embeddings \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{"input":"test","model":"text-embedding-3-small"}'

SQLite Locked

# Check for stale locks
lsof ~/.nullclaw/memory.db

# Kill stale processes
killall nullclaw

# Restart
nullclaw gateway

Poor Search Quality

Tune hybrid weights:

"vector_weight": 0.7,  // Increase for better semantic matching
"text_weight": 0.3     // Increase for better keyword matching

Enable MMR for diversity:

"mmr": { "enabled": true, "lambda": 0.7 }

High Memory Usage

Reduce cache size:

"cache": {
  "enabled": true,
  "max_entries": 1000  // Down from 10000
}

Enable aggressive hygiene:

"lifecycle": {
  "archive_after_days": 3,
  "purge_after_days": 7
}

Next Steps

Sandboxing

Configure security isolation

Hardware Integration

Connect Arduino, RPi, STM32