AI Skill — d4m

# d4m — What's different from MongoDB

## Architecture
- Single Rust binary, SQLite backend (one .db file per database)
- No mongod, no mongos, no config servers — just one process on one port
- WAL mode, no replica sets, no sharding
- TLS via --tls flag, uses tokio-rustls (pure Rust, no OpenSSL)
- Generate dev certs: make tls-certs (outputs ./certs/server.{crt,key})

## Authentication
- Default user always seeded: d4m / Secret2!2! on admin db, SCRAM-SHA-256, role root
- Configurable via D4M_ADMIN_USERNAME, D4M_ADMIN_PASSWORD, D4M_ADMIN_ROLE env vars
- Users are scoped to their creation DB → use ?authSource=admin in URI if user is in admin
- Default user is re-salted on every restart → client must reconnect fresh
- --no-auth flag skips auth check but SCRAM handshake still flows (tools like mongodump work without changes)

## Wire Protocol
- maxWireVersion: 21 (MongoDB 8.0+)
- OP_MSG, OP_QUERY (legacy), OP_GET_MORE, OP_KILL_CURSORS supported
- TLS via --tls flag, requires --tls-cert-file and --tls-key-file (defaults: ./certs/server.{crt,key})
- Uses tokio-rustls/rustls (pure Rust, no OpenSSL dependency)
- TLS handshake happens before any MongoDB wire bytes
- Non-TLS clients cannot connect to a TLS-enabled port (and vice versa)
- Compression: server responds with compression: [] (empty) regardless of client proposal — do NOT echo client's list back or drivers will send compressed messages that fail to parse
- Checksum flag (0x01) parsed and skipped; writer does not produce checksums
- hello/isMaster MUST echo clientConnectionId back or GUI tools break after reconnect
- Do NOT include topologyVersion in hello response — Node.js driver uses it to enable exhaustCommand (unsupported)

## Type Sensitivity (CRITICAL — silent wrong results)
- bson crate .as_i64()/.as_f64()/.as_str() only match exact BSON types:
  - Int32(5).as_i64() returns None, not Some(5)
  - RegularExpression("/pat/","").as_str() returns None
- d4m uses bson_to_i64()/bson_to_f64() helpers everywhere
- Driver sends 1 as Int32, 1.0 as Double
- $regex handles both {field: {$regex: "pat"}} and {field: {$regex: /pat/}} forms
- In $addFields/$set, eval_expression returning Bson::String("$$REMOVE") removes the field

## pymongo 4.17+ Gotcha
- C-level BSON encoder strips $-prefixed keys from documents inside arrays
- Breaks coll.aggregate() pipeline stages with $-prefixed field names
- Use mongosh or raw BSON for aggregate commands

## Not Implemented (will fail)
- Transactions, change streams, replica sets, sharding, sessions
- TTL/text/geo indexes, $where, mapReduce, server-side JavaScript
- $bucket, $bucketAuto, $graphLookup, $geoNear, $merge aggregation stages
- Causal consistency, snapshot reads

## Logs
- Hardcoded to ./logs/d4m-debug.log (auto-created)

## Cross-Field Text Search (two approaches)

# ✅ Preferred — $or with $regex (simplest, SQL pushdown):
# db.collection.aggregate([
#   { $match: { $or: fields.map(k => ({ [k]: { $regex: search, $options: "i" } })) } },
#   { $skip: skip },
#   { $limit: limit }
# ])

# ✅ Also works — $expr with $regexMatch/$convert:
# db.collection.aggregate([
#   { $match: { $expr: { $or: fields.map(k => ({
#       $regexMatch: { input: { $convert: { input: "$$k", to: "string", onNull: "", onError: "" } },
#                      regex: search, options: "i" }
#   })) } } },
#   { $skip: skip },
#   { $limit: limit }
# ])

# Node.js: $convert with $$field references can corrupt during BSON serialization.
# Prefer $or/$regex for Node.js.

## Pagination with Count (one pipeline round-trip)
# db.collection.aggregate([
#   { $match: ... },
#   { $facet: {
#       docs: [{ $skip: skip }, { $limit: limit }],
#       total: [{ $count: "count" }]
#   }}
# ])

## Field Caching
# Cache field names from a sampled document for ~60s to avoid findOne() per request.
# See server.js in https://github.com/anomalyco/d4m-client-example
Agent Skill
