Domain model

The entities Marginalia stores, how they relate, and the rules every entity follows: identity, ownership, soft delete, extended attributes and cleanup. Entities are in domain/model/ (base classes), domain/model/impl/ and domain/security/model/User.java. How the tables are created and changed is in Database & migrations.

Names in code and UI

Some entities kept their early names. The UI and this manual use the names on the right:

Overview

erDiagram
    User ||--o{ Manuscript : owns
    User ||--o{ Lorebook : owns
    User ||--o{ AI : owns
    User ||--o{ Protocol : owns
    User ||--o| UserSetting : has

    Manuscript }o--o| AI : "generates with"
    Manuscript }o--o| Protocol : "generates with"
    Manuscript }o--o| Lorebook : "uses"
    Manuscript ||--o{ ChatMessage : "story parts"
    Manuscript |o--o| ChatMessage : "activeLeaf"

    ChatMessage |o--o{ ChatMessage : "parent / children"
    ChatMessage |o--o| Summary : "summary"

    Lorebook ||--o{ LorebookEntry : entries
    Lorebook }o--o{ Lorebook : "subbooks"

    Tag ||--o{ TagRelation : ""
    TagRelation }o--|| Manuscript : "objectId + clazz"
    TagRelation }o--|| Lorebook : "objectId + clazz"
    TagRelation }o--|| LorebookEntry : "objectId + clazz"

Every entity except User has an owner, so each user has a separate set of books, lorebooks, providers, protocols and settings. The relationships between them never cross users: a book can only use the owner's provider, protocol and lorebook.

Base classes

classDiagram
    class BaseEntity {
        Long id
        String uuid
        boolean deleted
        Date creation
        Date modification
    }
    class OwnedEntity {
        User owner
    }
    class ExtendableEntity {
        byte[] extendedContent
        String _fulltext
        JsonObject attributes
    }
    BaseEntity <|-- User
    BaseEntity <|-- OwnedEntity
    OwnedEntity <|-- Resource
    OwnedEntity <|-- ExtendableEntity
    ExtendableEntity <|-- Manuscript
    ExtendableEntity <|-- ChatMessage
    ExtendableEntity <|-- Summary
    ExtendableEntity <|-- Lorebook
    ExtendableEntity <|-- LorebookEntry
    ExtendableEntity <|-- AI
    ExtendableEntity <|-- Protocol
    ExtendableEntity <|-- Tag
    ExtendableEntity <|-- TagRelation
    ExtendableEntity <|-- Setting
    AI <|-- OpenAICompatible
    Protocol <|-- ChatCompletionProtocol
    Setting <|-- AppSettings
    Setting <|-- UserSetting

BaseEntity - identity and soft delete

equals and hashCode use the id, so don't put unsaved entities into hash-based collections.

Soft delete. delete(entity, false) - the normal delete - only sets deleted = true. All finder queries skip deleted rows, so to the application the entity is gone, but the row stays and other rows can still point at it. The administrator's Cleanup page later purges deleted rows that nothing needs any more (see Cleanup references). delete(entity, true) removes the row right away; it is used where nothing can reference the row, e.g. replaced messages during a backup restore.

OwnedEntity - ownership

owner (column userId) is the user the entity belongs to; null for installation-wide rows (AppSettings). OwnedServiceImpl.save fills it with the current user when a new entity has no owner. Lookups that start from user input go through the owner-aware finders:

ExtendableEntity - extended attributes

Most of an entity's data is not in table columns but in one JSON document stored in the extendedContent blob. Two kinds of fields end up there:

• fields annotated @ExtendedAttribute (they are also @Transient, so Hibernate ignores them), and
• attributes, a free-form JsonObject for extensions. Its keys are stored with the prefix attributes_ next to the other fields.

{
  "instructions": "Mara finds the lighthouse logbook.",
  "response": "The logbook was heavier than she expected...",
  "modelUsed": "gpt-4.1",
  "request": "2026.10.08Z14:03:11.512",
  "attributes_reviewer": { "reviews": [ ... ] }
}

ExtendableEntityListener converts between the fields and the document:

• Load (@PostLoad): the JSON is parsed and the fields are set. Supported field types are String, numbers, Boolean, Date (ISO-8601 instant in UTC, e.g. 2026-10-09T08:15:30.123Z; values in the old yyyy.MM.dd'Z'HH:mm:ss.SSS format, written in the server's time zone, are still read), enums (by name) and JsonObject for inject = true.

• Types: the values are written with toString(), so a field can't be a collection. A list is kept as a JSON string in a String field with typed accessors, like ChatMessage.imageAttachments (getImages() / setImages(...), a JSON array of ImageAttachment(resource uuid, caption)).

• Save: JpaExtendableRepository.save serializes the fields into extendedContent before the entity is written.

Because the fields are @Transient, Hibernate's dirty checking doesn't see them. A change to an extended attribute is only stored by calling the service's save. Changing it on a managed entity and letting the transaction commit does nothing.

saveWithoutEvent writes extendedContent as it is, without serializing the fields first. It is used when extendedContent was set directly - copying the JSON of a backup into a restored entity - and must not be overwritten by the (empty) fields. The repository then reads the fields back from the JSON and rebuilds _fulltext from them.

Full-text column

Every ExtendableEntity table has a _fulltext text column (clob, @Lob String) next to extendedContent: the text of the entity that a search should find, in one place, even when the text is spread over columns and JSON. Fields annotated @Fulltextable (a column such as LorebookEntry.name or an @ExtendedAttribute) are collected by ExtendableEntityListener:

• serialize (called by every save) writes _fulltext right after extendedContent: the string value of each marked field (toString(), dates as ISO-8601), empty ones skipped, joined by a newline in declaration order. An entity without marked fields, or with all of them empty, has NULL.

• updateFulltext rebuilds only _fulltext, from the fields as they are. saveWithoutEvent uses it after reading the fields from the restored JSON, and FulltextMigration uses it to fill existing rows.

Full-text search

The only search over _fulltext so far is ChatMessageService.searchFulltext(manuscript, query), used by the search of the story tree. FulltextQuery.parse splits the query into words (double quotes make a phrase); a part matches when its text contains every word. In SQL (ChatMessageRepository.findFulltexts) each word is a _fulltext LIKE :pattern ESCAPE '\', built like the name filter of the book list (LikePatterns.fromWildcards): * becomes %, and \, _ and % are escaped so they are searched as typed. Only the id and _fulltext of the messages are read, never the whole entity. SQLite's LIKE ignores case only for A-Z, so every word is also tried in lower case, upper case and capitalized (FulltextQuery.likePatterns), which covers accented letters in the usual spellings; a word written in mixed case in the text (ŽlutÝ) is not found by a differently cased query. The snippet shown with a hit is cut from the text by FulltextQuery.snippet. The service checks that the book belongs to the current user.

Like extendedContent, the column is invisible to dirty checking of the fields, so it is only current after a service save. Marking a field of an entity that already has rows (or changing which fields are marked) leaves old rows out of date: add an app migration that calls updateFulltext for the entity, as FulltextMigration does (see Database & migrations).

Why most fields live in JSON:

• Adding or removing a field needs no database migration.
• Extensions can store data on any entity (getAttributes().add("myplugin", ...)) without their own tables.
• Book backups copy extendedContent as a whole, so extension data and new fields of books and parts travel with them (lorebook exports don't, see Backups and copies).

The price is that extended attributes can't be queried or indexed in SQL. Anything the application filters, sorts or joins on is a real column: relations, name, enabled, ordinal, published, lastOpened, tokenCount, wordCount...

Books and the story

Manuscript (book)

Tags of a book are TagRelations (see Tags).

ChatMessage (part) and the story tree

The story is a tree of parts. Each part points to its parent (null for the first part) and to its book (parentScript). The book's activeLeaf selects one path from the root - the branch the user sees.

flowchart LR
    P1["#1"] --> P2["#2"] --> P3["#3"] --> P4a["#4"] --> P5a["#5 ★ activeLeaf"]
    P3 --> P4b["#4 (swipe)"]
    P3 --> P4c["#4 (swipe)"] --> P5c["#5"]

Siblings are ordered by creation. The branch from the root to a leaf is read with a recursive SQL query (JpaChatMessageRepository.findBranchFromLeaf).

Template variables are stored when a part is generated (GenerateNewMessageStep): local ones (setvar), which follow the branch, in the new part's attributes, global ones (setglobalvar) in the book's attributes.

Summary

A summary belongs to one part and covers that part and all earlier parts back to the previous part with a summary (see Summaries). A meta summary (summaryType = META_SUMMARY) stands in for a range of summaries: it is stored on the part of the newest of them, in place of the summary that part had.

All of them are @ExtendedAttribute fields, so no migration was needed and they travel with backups.

A summary is owned by its part (@CleanupReference(OWNS)); branching a part copies its summary (all of the fields above).

The summary chain

The summaries of a branch are walked from the newest part to the root (SummaryService.collectBlocks; the generation, the invalidation check and the overview all use it):

• A part with a summary starts a block: the part and the parts after it (older), up to the next block.

• After a meta summary the summaries are skipped (they belong to its block) until the one whose uuid is its nextSummaryUuid, which starts the next block. The meta summary block's hash includes the text of the skipped summaries.

• The hash of a block is SHA-512 over, part by part: the text of the part, and the text of the summary of the part if it has one - except for the head of the block, whose own summary is what the hash checks (it didn't exist yet when the hash was made). A normal block has no other summaries in it, so its hash is that of the texts only.

Deleting a meta summary (SummaryService.removeSummary(part, unwind)) with unwind restores the summary from replacedSummary: it is deserialized into a new Summary. Because that summary can be a meta summary with its own replacedSummary, unwinding nests. A meta summary takes over the uuid of the summary it replaces (attachMetaSummary), and the restored summary gets it back, so the nextSummaryUuid of other meta summaries stays valid. Restoring a backup creates all summaries with new uuids, so ManuscriptServiceImpl.restoreMessages rewrites the nextSummaryUuid of the restored meta summaries (also inside replacedSummary) to the new ones.

Lorebooks

Lorebook

LorebookEntry

How entries are activated is described in Lorebooks and Generation pipeline.

Tags

Tag is a value (value) owned by a user. TagRelation (table t2e) attaches a tag to any entity:

Books, lorebooks and lorebook entries are tagged. TagRelationService creates and removes relations and finds the tags of an object (getTagsForObject(entity), getTagsForObject(entity, true) for negative ones). Because the relation is polymorphic, there is no foreign key from t2e to the tagged table; cleanup follows objectId / clazz through @CleanupReference(targetClassField = "clazz").

Inference providers and protocols

Both use joined inheritance: a base table with the common fields and one table per type. Today each has one type.

AI / OpenAICompatible (inference provider)

Protocol / ChatCompletionProtocol

Settings

Setting is one table (settings) for all settings classes, with the class name in DTYPE and in key. Almost all values are extended attributes.

SettingService.getOrCreate(UserSetting.class) returns the current user's settings, creating them on first use; getOrCreateApp(AppSettings.class) the installation's. Extensions keep their settings in the attributes of UserSetting (Reviewer and Side Query profiles do).

The prompt used for a book is resolved in three levels: the book's value (Manuscript.template...), else the user's default (UserSetting.masterTemplate...), else the built-in default (Defaults).

Users

User extends BaseEntity directly - it has no owner and no extended attributes.

The current user of a session is the session-scoped user bean (see Architecture), which holds a copy of the logged-in user's id, login and name - load the entity through UserService when you need more.

Resources

Resource (resources, an OwnedEntity) is an uploaded file; today the files are the images attached to parts (ChatMessage.getImages(), see Extended attributes).

Deleting a resource is a soft delete like any other; files are never deleted from disk.

Cleanup references

Soft-deleted rows are purged by CleanupService (Admin → Cleanup, administrators only: every method calls AdminGuard.requireAdmin()). Before purging, it builds a graph of all references between rows and decides what may go. Each reference has a policy (@CleanupReference, in domain/traits/):

JPA associations are found automatically from the Hibernate metamodel and default to STRONG. Plain id fields pointing to other entities ("soft references", including extended attributes) are only seen when annotated with target = SomeEntity.class or, for polymorphic ones, targetClassField = "clazz". On an entity class, field = "..." sets the policy of an inherited field (OwnedEntity makes owner OWNED_BY, so a deleted user is purged together with everything they own).

Trash

The same reference model drives the trash (TrashService, implemented by CleanupServiceImpl; Trash tab and Admin → Cleanup). It lists the soft-deleted rows of every root entity that is an OwnedEntity (Protocol stands for its subclasses) with narrow projections - the extendedContent blob is only read by getExtendedContent, which pretty-prints it - and restores them with a bulk UPDATE ... SET deleted = false, modification = now, so no entity listener runs and extended attributes stay untouched. Before it restores, it looks up the parents of the selected rows:

When a parent is deleted and not part of the same selection, nothing is restored and the result lists the blockers. A deleted owner counts as a parent, so the objects of a deleted user can't be restored. A restored ChatMessage does not get its children back: they were moved to its parent when it was deleted (deleteNodeAndMigrateChildren).

So a deleted provider that a book still uses shows up as blocked on the Cleanup page until the book is switched to another provider, while a deleted book takes its parts, their summaries and its tag relations with it.

Backups and copies

Book backups (BackupService) serialize a book as JSON: its columns, its extendedContent as is, its tags, all parts (with extendedContent and summaries) and references to its provider, protocol and lorebook by uuid and name. Restoring or cloning creates new rows (new ids and uuids for the parts) and copies extendedContent unchanged, so extended attributes and extension data come back exactly as they were. Linked entities are resolved among the current user's own entities, first by uuid, then by name.

Lorebooks are different: lorebook exports (LorebookServiceImpl.exportLorebook) and the lorebooks stored inside book backups list their fields one by one (name, payload, comment, filtering...), so the attributes of lorebooks and entries - extension data such as the Lorebook VCS history - are not included.

Keep this in mind when adding data: for books and parts anything in extendedContent is backed up automatically, while a new column has to be added to the backup and restore code (BackupServiceImpl, ManuscriptServiceImpl.restoreBackup / restoreMessages); a new lorebook or entry field has to be added to the export and import code in either case.

Adding a field or an entity

A field on an existing entity:

1. Decide where it lives. Prefer an extended attribute (@ExtendedAttribute @Transient) - no migration, backed up automatically. Use a column only when you need to query, sort or join on it.

2. For a column: add a Flyway migration (see Database & migrations); Hibernate validates the schema on start. Extend the backup / restore code if the value should survive a backup.

3. For a reference to another entity: think about its cleanup policy and annotate it if STRONG isn't right. Soft references (ids) must be annotated with target or targetClassField, otherwise cleanup may purge the referenced row.

4. Save it through the service's save (not saveWithoutEvent).

5. If its text should be found by a search, annotate it @Fulltextable (see Full-text column) and add an app migration that refills _fulltext of the existing rows.

A new entity:

1. Extend ExtendableEntity (or OwnedEntity if it never needs extended data) and annotate the references.

2. List the class in src/main/webapp/config/persistence.xml - unlisted classes are ignored.

3. Write a Flyway migration creating the table, its <table>_SEQ table and indexes.

4. Add a repository (JpaExtendableRepository subclass) and a service (ExtendableServiceImpl subclass), and define both beans in services-config.xml.

5. Add a CRUD test based on ExtendableCrudContract (see Testing).

6. Decide whether it belongs in book backups or lorebook exports.