High-Level API Architecture

The native high-level API is a header-driven C++ layer over AkkEngine. It does not create a second storage engine. It builds deterministic table prefixes, encodes structs with BinPack, maintains row-id metadata, and delegates persistence to the byte-oriented engine.

Runtime Parts

Part	Responsibility
AkkaraDB	Owns an engine::AkkEngine and exposes table<&T::id>() plus schema registration.
PackedTable<PrimaryKeyPtr>	Converts typed operations into engine keys, values, scans, and metadata writes.
BinPack	Encodes and decodes primary keys, entities, index values, refs, optional values, maps, and nested structs.
AKKARADB_ENTITY	Registers RefTraits and query proxy fields for a C++ struct.
Ref<T>	Stores a key, row id, or loaded entity and resolves through table bindings.
Schema	Registers table holders and installs hook-based foreign-key actions.

AkkaraDB::open() builds engine::AkkEngineOptions from the high-level options object. The wrapper does not hide the engine: db->engine() returns the underlying engine for low-level operations that are not surfaced by PackedTable.

Layering

Layer	What happens there
Application C++	Calls AkkaraDB, PackedTable, query expressions, refs, joins, and schema hooks.
Native high-level API	Encodes structs, builds table/index/metadata keys, manages row-id mappings, and runs hooks.
AkkEngine	Stores bytes, WAL records, memtable state, SSTables, blobs, VersionLog data, and optional server/cluster components.

The high-level API inherits the durability, compaction, blob, VersionLog, API server, and cluster behavior of the configured AkkEngine.

Table Namespace

Each table name is converted into an 8-byte prefix. Rows are stored under:

[tablePrefix][encodedPrimaryKey]

The table also owns metadata prefixes derived from the table name.

Prefix purpose	Used for
table rows	Encoded entity values.
idx:<field>	Secondary index entries for registered fields.
pk2row	Primary-key to stable row-id mapping.
row2pk	Stable row-id to current primary-key mapping.
nextrow	Row-id allocation counter.

Table names are therefore physical storage layout. Renaming a table points the API at a different key range.

Primary-Key Encoding

The primary key type is inferred from the member pointer passed to PackedTable.

auto users = db->table<&User::id>("users");

Numeric keys are encoded so lexicographic byte order matches numeric order. Signed integers flip the sign bit and then write big-endian bytes; unsigned integers are written big-endian. Other supported values use BinPack encoding. The encoded primary key is appended to the table prefix for row keys and to metadata prefixes for row-id mappings.

Index field encoding follows the same idea for ordered field types: integral values and floating-point values are converted to sortable bytes, Immutable<T> indexes the wrapped value, and Ref<T> indexes the referenced row id. Other field values use BinPack.

Write Path

put(entity) performs several steps around the underlying engine write:

1. Reset temporary arena-backed buffers.
2. Extract the primary key from entity.*PrimaryKeyPtr.
3. Build the table row key.
4. Load the previous entity when hooks or indexes need it.
5. Run update hooks and immutable-field checks for replacements.
6. Attach and flush dirty Ref<T> fields.
7. Validate registered foreign keys.
8. Encode the entity with BinPack.
9. Write the key/value through AkkEngine.
10. Allocate or reuse a stable row id.
11. Update row-id metadata.
12. Rebuild secondary index entries.

These are multiple engine operations around one logical table write. The high-level API does not add a multi-row transaction manager above AkkEngine.

Read Path

get(pk) builds the same namespaced key and asks AkkEngine for the value bytes. getInto(pk, out) decodes directly into an existing object.

After decode, the table attaches ref bindings and seals immutable fields. That means loaded Ref<T> fields can resolve through the table schema, and loaded Immutable<T> fields reject accidental mutation.

Scans

scanAll() builds the start/end range for the table prefix and walks engine scan results. Each returned key must still match the table prefix; the primary-key suffix is decoded into Entry::id and the value bytes are decoded into Entry::value.

scan(startPk, endPk) builds exact key boundaries from encoded primary keys. Because numeric keys are sortable, numeric range scans follow natural numeric order.

Secondary Indexes

index<&Field>() registers an IndexDef for a field and returns an Index<FieldPtr> object.

Index entries store enough information to find candidate primary-key bytes. Index::find(value) scans the index prefix for the encoded field value, extracts primary-key bytes from the index key, then loads the current entity by primary key.

Operation	Index maintenance
put() insert	Writes index entries after the entity is written.
put() replacement	Removes old index entries before writing new ones.
remove()	Removes old index entries before deleting row metadata.
updatePrimaryKey()	Writes new index entries and removes old primary-key entries.

The API does not backfill indexes automatically. Index registration affects future writes unless the application rewrites existing rows.

Query Model

The default C++ query API builds expression types through overloaded operators and field proxy objects.

profiles.query([](auto profile) {
    return profile.email == "a@example.test" && profile.age >= 18;
});

AKKARADB_QUERYABLE creates an akkaradbQueryProxy() overload for the entity. query() calls the predicate with that proxy and stores the resulting expression tree in QueryView.

Native builds may also run the optional akkara-query Clang plugin. Supported typed lambdas are rewritten to CompiledQueryDescriptor<T> bytecode descriptors. A bytecode query view evaluates directly against row bytes when all fields have raw readers, and otherwise decodes the entity and runs the same VM.

Supported expression families include:

Expression	Internal shape
==, !=, <, <=, >, >=	Compare<Op, L, R>
&&, `
in, notIn	List membership comparisons.
startsWith, contains, like	String-oriented comparisons.
isNull, isNotNull	Optional/null checks.
field<&Nested::x>()	Nested field path.
get(key) / mapGet(key)	Map lookup expression.

Query Planning

QueryView::begin() asks PackedTable to build a QueryPlan. The plan chooses between:

Source	Behavior
TABLE	Scan the table key range and evaluate the expression for each row.
INDEX	Scan one or more index ranges, load candidate rows by primary key, deduplicate when needed, then evaluate the full expression.

The planner can use registered indexes for suitable predicates. The returned rows are still filtered by the complete expression, so an index narrows the candidate set without changing query semantics.

The planner recursively searches AND expressions, scores usable indexed predicates, and chooses the stronger candidate source. It can create equality ranges, ordered numeric ranges, multiple IN ranges, null ranges for optional fields, prefix-index ranges for startsWith and simple like("prefix%"), or a full field-index scan for predicates such as !=, notIn, isNotNull, and contains. For OR, both sides must be indexable; when they are, the plan scans the union of their index ranges and deduplicates candidate primary keys. If either side of an OR is not indexable, the planner falls back to a table scan to avoid missing rows.

Bytecode where descriptors are composed into the existing descriptor with logical AND. Composed descriptors keep host-call and custom-opcode capture pointers per call or binding, so independently captured descriptors can be combined without forcing one shared capture pointer.

Row-Id Metadata

Each table keeps stable row ids separate from visible primary keys.

Metadata	Purpose
pk2row	Resolve a primary key to a stable row id.
row2pk	Resolve a row id back to the current primary key.
nextrow	Allocate the next row id.

updatePrimaryKey() rewrites the mapping while preserving the row id. This is why Ref<T> can survive primary-key changes.

References

Ref<T> can hold a known primary key, a known row id, a loaded value, or a dirty value. It resolves lazily through RefBinding<T>.

State	Meaning
key known	id() can return without table lookup.
row id known	rowId() can resolve through row-id metadata.
loaded	operator-> and operator* can return the entity.
dirty	The referenced entity should be flushed on owner write.
attached	A table binding is available for resolution.

When a table decodes an entity, it attaches ref bindings. When a table writes an entity, it flushes dirty refs before validating and encoding the owner.

Joins

Join helpers are typed read views, not a native join engine.

Join form	Execution shape
join<&RefField>(right)	Scans the left table and resolves refs through right-table row-id lookup.
join<&LeftField, &RightPk>(right)	Scans left and performs right primary-key lookups.
join<&LeftField, &RightField>(right)	Scans left and scans right for matching fields.

Each join row contains the left Entry and the decoded right entity. where(), first(), any(), count(), and toVector() are evaluated in the view.

Query bytecode does not traverse Ref<T> as a local row field. A predicate that crosses a ref uses normal C++ lazy resolution through host-call fallback in query(...), remains a decoded filter in where(...), or should be expressed through an explicit join(...).where(...) when both entities are part of the predicate.

Schema Hooks

AkkaraDB::Schema stores table holders keyed by C++ entity type. It also provides ref bindings so Ref<T> fields can resolve across registered tables.

foreignKey() installs behavior by adding hooks to source or target tables.

Hook area	Purpose
Source put()	Reject missing referenced targets.
Target remove()	Apply Restrict, Cascade, or SetNull.
Target primary-key update	Apply Restrict, Cascade, or SetNull for key movement.

Foreign-key actions scan source tables to find referencing rows. For large tables, use an explicit index or application-level lookup structure when reference checks need to be fast.

Target existence checks are more precise than the action scans: when a foreign key points at a non-primary target field, the target table registers an index for that field and uses indexed lookup for validation. Source-side delete/update actions still walk the source table and compare fields. For Ref<T> source fields, target lookup is row-id based and update actions are skipped because the row id remains stable across primary-key changes.

Consistency Boundary

The high-level API is intentionally thin over the engine. A single PackedTable operation may write the entity, row-id metadata, and index keys. Schema actions may write multiple rows. Joins may observe current state through scans and lookups.

For invariants that require all-or-nothing multi-row behavior, design an explicit coordination layer above the table API or keep the invariant inside one stored value.

Backup And Migration

Backups must keep table rows and table metadata together.

Data	Why
Engine files	Actual stored rows, WAL/SST/blob state, and low-level metadata.
Row-id mappings	Required for Ref<T>, getByRowId(), and updatePrimaryKey() stability.
Index entries	Required for Index::find() and indexed query planning.
VersionLog data	Required if the engine is configured for historical reads or rollback.

Migration tooling should treat table name, primary-key type, indexed fields, and struct encoding as storage layout decisions.

Because BinPack encodes aggregate fields in declaration order, adding, removing, reordering, or changing fields is not automatically a compatible schema change. Plan these as explicit rewrites or versioned entity formats.

When To Use Low-Level API

Drop down to engine::AkkEngine when you need exact byte layout control, a custom protocol/server layer, non-struct encoding, bulk storage surgery, or direct engine subsystem testing. Stay on the high-level API when typed structs, table-local indexes, query expressions, refs, and schema hooks are the main value.