Domain model

The entities Marginalia stores, how they relate, and the rules every entity follows: identity, ownership, soft delete, extended attributes and cleanup. Entities are in domain/model/ (base classes), domain/model/impl/ and domain/security/model/User.java. How the tables are created and changed is in Database & migrations.

Names in code and UI

Some entities kept their early names. The UI and this manual use the names on the right:

Overview

erDiagram
    User ||--o{ Manuscript : owns
    User ||--o{ Lorebook : owns
    User ||--o{ AI : owns
    AI ||--o{ Usage : "used by users"
    User ||--o{ Usage : owns
    User ||--o{ Protocol : owns
    User ||--o| UserSetting : has
    User ||--o{ Folder : owns
    Folder |o--o{ Folder : "parent / children"
    Folder |o--o{ Manuscript : contains

    Manuscript }o--o| AI : "generates with"
    Manuscript }o--o| Protocol : "generates with"
    Manuscript }o--o| Lorebook : "uses"
    Manuscript ||--o{ ChatMessage : "story parts"
    Manuscript |o--o| ChatMessage : "activeLeaf"

    ChatMessage |o--o{ ChatMessage : "parent / children"
    ChatMessage |o--o| Summary : "summary"

    Lorebook ||--o{ LorebookEntry : entries
    Lorebook }o--o{ Lorebook : "subbooks"

    Tag ||--o{ TagRelation : ""
    TagRelation }o--|| Manuscript : "objectId + clazz"
    TagRelation }o--|| Lorebook : "objectId + clazz"
    TagRelation }o--|| LorebookEntry : "objectId + clazz"
    TagRelation }o--|| Folder : "objectId + clazz"

Every entity except User has an owner, so each user has a separate set of books, lorebooks, providers, protocols and settings. The relationships between them never cross users: a book can only use the owner's provider, protocol and lorebook.

Base classes

classDiagram
    class BaseEntity {
        Long id
        String uuid
        boolean deleted
        Date creation
        Date modification
    }
    class OwnedEntity {
        User owner
    }
    class ExtendableEntity {
        byte[] extendedContent
        String _fulltext
        JsonObject attributes
    }
    BaseEntity <|-- User
    BaseEntity <|-- OwnedEntity
    OwnedEntity <|-- Resource
    OwnedEntity <|-- ExtendableEntity
    ExtendableEntity <|-- Manuscript
    ExtendableEntity <|-- ChatMessage
    ExtendableEntity <|-- Summary
    ExtendableEntity <|-- Lorebook
    ExtendableEntity <|-- LorebookEntry
    OwnedEntity <|-- Usage
    OwnedEntity <|-- ManuscriptStatistic
    OwnedEntity <|-- ActivityInterval
    OwnedEntity <|-- ManuscriptSize
    ExtendableEntity <|-- AI
    ExtendableEntity <|-- Protocol
    ExtendableEntity <|-- Tag
    ExtendableEntity <|-- TagRelation
    ExtendableEntity <|-- Setting
    AI <|-- OpenAICompatible
    AI <|-- ExtensionAI
    Protocol <|-- ChatCompletionProtocol
    Setting <|-- AppSettings
    Setting <|-- UserSetting

BaseEntity - identity and soft delete

equals and hashCode use the id, so don't put unsaved entities into hash-based collections.

Soft delete. delete(entity, false) - the normal delete - only sets deleted = true. All finder queries skip deleted rows, so to the application the entity is gone, but the row stays and other rows can still point at it. The administrator's Cleanup page later purges deleted rows that nothing needs any more (see Cleanup references). delete(entity, true) removes the row right away; it is used where nothing can reference the row, e.g. replaced messages during a backup restore.

A soft delete takes the dependent rows with it. The services whose entities own other rows override delete and, when it is not a hard delete, soft delete those rows first (the OWNED_BY / OWNS rows of the cleanup references), so the trash and the cleanup see the whole group deleted:

A book has thousands of messages, so the messages and summaries are soft deleted with one bulk UPDATE per kind (softDeleteByManuscript, softDeleteByIds; deleted = true, modification = now): the rows are not loaded, no entity listener or event runs and extendedContent stays as it is. The rest goes through the services' delete. Objects that only reference the deleted one (a book using a deleted provider or lorebook, a lorebook that includes the deleted one as a subbook) are not deleted. A hard delete does not cascade. Code that adds an entity owned by another one has to add it to the cascade, or the owned rows stay live while their owner is in the trash.

OwnedEntity - ownership

owner (column userId) is the user the entity belongs to; null for installation-wide rows (AppSettings). OwnedServiceImpl.save fills it with the current user when a new entity has no owner. Lookups that start from user input go through the owner-aware finders:

ExtendableEntity - extended attributes

Most of an entity's data is not in table columns but in one JSON document stored in the extendedContent blob. Two kinds of fields end up there:

• fields annotated @ExtendedAttribute (they are also @Transient, so Hibernate ignores them), and
• attributes, a free-form JsonObject for extensions. Its keys are stored with the prefix attributes_ next to the other fields.

{
  "instructions": "Mara finds the lighthouse logbook.",
  "response": "The logbook was heavier than she expected...",
  "modelUsed": "gpt-4.1",
  "request": "2026.10.08Z14:03:11.512",
  "attributes_reviewer": { "reviews": [ ... ] }
}

ExtendableEntityListener converts between the fields and the document:

• Load (@PostLoad): the JSON is parsed and the fields are set. Supported field types are String, numbers, Boolean, Date (ISO-8601 instant in UTC, e.g. 2026-10-09T08:15:30.123Z; values in the old yyyy.MM.dd'Z'HH:mm:ss.SSS format, written in the server's time zone, are still read), enums (by name) and any type for inject = true: a JsonObject has its keys stored flat as <injectPrefix>_<key>, any other type is converted with Gson (object-shaped results are flattened the same way, lists and scalars are stored under the plain injectPrefix key).

• Types: the values are written with toString(), so a field can't be a collection. A list is kept as a JSON string in a String field with typed accessors, like ChatMessage.imageAttachments (getImages() / setImages(...), a JSON array of ImageAttachment(resource uuid, caption, anchor, wrap, width): the 3x3 ImageAnchor of the image (null: after the text, centered), whether the text flows around it (beside it at the left or the right, in two columns on both sides of one in the middle) and its width in percent of the text (null: its size, 30 for one with text on both sides); old values without the last three keep the look they had, and the viewer and the exports place the images with PartLayout).

• Save: JpaExtendableRepository.save serializes the fields into extendedContent before the entity is written.

Compression. The document is mostly text, and an entity keeps it next to the fields made from it, so ExtendableEntity stores it gzipped when it is 256 bytes or more: setExtendedContent compresses and getExtendedContent gives the plain JSON, so only the field of the column is compressed and the code that works with the bytes doesn't change. A row stored plain (before app version 6) still reads - a gzip stream starts with 1f 8b, which no JSON or text does - and is compressed when it is saved again; ExtendedContentMigration does it for all of them once. What reads the column without the entity (SELECT e.extendedContent, the trash viewer) must call ExtendableEntity.plain(bytes).

Because the fields are @Transient, Hibernate's dirty checking doesn't see them. A change to an extended attribute is only stored by calling the service's save. Changing it on a managed entity and letting the transaction commit does nothing.

saveWithoutEvent writes extendedContent as it is, without serializing the fields first. It is used when extendedContent was set directly - copying the JSON of a backup into a restored entity - and must not be overwritten by the (empty) fields. The repository then reads the fields back from the JSON and rebuilds _fulltext from them.

Full-text column

Every ExtendableEntity table has a _fulltext text column (clob, @Lob String) next to extendedContent: the text of the entity that a search should find, in one place, even when the text is spread over columns and JSON. Fields annotated @Fulltextable (a column such as LorebookEntry.name or an @ExtendedAttribute) are collected by ExtendableEntityListener:

• serialize (called by every save) writes _fulltext right after extendedContent: the string value of each marked field (toString(), dates as ISO-8601), empty ones skipped, joined by a newline in declaration order. An entity without marked fields, or with all of them empty, has NULL.

• updateFulltext rebuilds only _fulltext, from the fields as they are. saveWithoutEvent uses it after reading the fields from the restored JSON, and FulltextMigration uses it to fill existing rows.

Full-text search

The searches over _fulltext are ChatMessageService.searchFulltext(manuscript, query), used by the search of the story tree. FulltextQuery.parse splits the query into words (double quotes make a phrase); a part matches when its text contains every word. In SQL (ChatMessageRepository.findFulltexts) each word is a _fulltext LIKE :pattern ESCAPE '\', built like the name filter of the book list (LikePatterns.fromWildcards): * becomes %, and \, _ and % are escaped so they are searched as typed. Only the id and _fulltext of the messages are read, never the whole entity. SQLite's LIKE ignores case only for A-Z, so every word is also tried in lower case, upper case and capitalized (FulltextQuery.likePatterns), which covers accented letters in the usual spellings; a word written in mixed case in the text (ŽlutÝ) is not found by a differently cased query. The snippet shown with a hit is cut from the text by FulltextQuery.snippet. The service checks that the book belongs to the current user. The advanced search of the book list uses the same LIKEs over all the candidate books at once (ChatMessageRepository.findFulltextHits, ids only) and, unless it searches all branches, keeps the books whose hit lies on the branch of the active part (findActiveBranchIds, a walk up the parent links).

Like extendedContent, the column is invisible to dirty checking of the fields, so it is only current after a service save. Marking a field of an entity that already has rows (or changing which fields are marked) leaves old rows out of date: add an app migration that calls updateFulltext for the entity, as FulltextMigration does (see Database & migrations).

Why most fields live in JSON:

• Adding or removing a field needs no database migration.
• Extensions can store data on any entity (getAttributes().add("myplugin", ...)) without their own tables.
• Book backups copy extendedContent as a whole, so extension data and new fields of books and parts travel with them; lorebook exports and the lorebooks in a backup carry the attributes of lorebooks and entries (see Backups and copies).

The price is that extended attributes can't be queried or indexed in SQL. Anything the application filters, sorts or joins on is a real column: relations, name, enabled, ordinal, published, lastOpened, tokenCount, wordCount...

Books and the story

Manuscript (book)

Tags of a book are TagRelations (see Tags).

Folder

A group of books, nested through parent (a column, OWNED_BY for the cleanup), with a name (column), a description and the same twelve text settings and the language as a book, all extended attributes where null means inherit; tags like a book. Described in Folders. Migration V21.

ChatMessage (part) and the story tree

The story is a tree of parts. Each part points to its parent (null for the first part) and to its book (parentScript). The book's activeLeaf selects one path from the root - the branch the user sees.

flowchart LR
    P1["#1"] --> P2["#2"] --> P3["#3"] --> P4a["#4"] --> P5a["#5 ★ activeLeaf"]
    P3 --> P4b["#4 (swipe)"]
    P3 --> P4c["#4 (swipe)"] --> P5c["#5"]

Siblings are ordered by creation. The branch from the root to a leaf is read with a recursive SQL query (JpaChatMessageRepository.findBranchFromLeaf).

Template variables are stored when a part is generated (GenerateNewMessageStep): local ones (setvar), which follow the branch, in the new part's attributes, global ones (setglobalvar) in the book's attributes.

MessagePrompt

What a part was written from: the built prompt (the payload sent to the model as JSON, 100-200 KB for a long story) and the lore that was in it. It is a row of its own (message_prompts, one per part at most, owned by the user of the part) and not a part of ChatMessage, because the parts of a branch are loaded together and a story view holds them as long as it is open: with the prompt in the part, a book of 50 parts weighed tens of megabytes per load. The parts have only builtPromptTokens; whoever needs the text asks MessagePromptService:

The texts are stored deflated (byte[] columns, the accessors of the entity take and give plain text). Deleting a part deletes its prompt with it (softDeleteForParts, hardDeleteForParts, softDeleteForManuscript, like the annotations), restoring the part from the trash brings it back, and a permanent delete takes it (CleanupServiceImpl.OWNED_ON_DELETE).

Backups. A backup of a book still has the prompt in the extendedContent of the message (builtPrompt, backgroundLore), as it did before the row existed, so backups of all versions have the same format: ManuscriptServiceImpl.createBackup puts the prompt back there and restoreMessages takes it out again and stores the row. The data was moved by ExtendedContentMigration (app version 6, see Database).

Summary

A summary belongs to one part and covers that part and all earlier parts back to the previous part with a summary (see Summaries). A meta summary (summaryType = META_SUMMARY) stands in for a range of summaries: it is stored on the part of the newest of them, in place of the summary that part had.

All of them are @ExtendedAttribute fields, so no migration was needed and they travel with backups.

A summary is owned by its part (@CleanupReference(OWNS)); branching a part copies its summary (all of the fields above).

The summary chain

The summaries of a branch are walked from the newest part to the root (SummaryService.collectBlocks; the generation, the invalidation check and the overview all use it):

• A part with a summary starts a block: the part and the parts after it (older), up to the next block.

• After a meta summary the summaries are skipped (they belong to its block) until the one whose uuid is its nextSummaryUuid, which starts the next block. The meta summary block's hash includes the text of the skipped summaries.

• The hash of a block is SHA-512 over, part by part: the text of the part, and the text of the summary of the part if it has one - except for the head of the block, whose own summary is what the hash checks (it didn't exist yet when the hash was made). A normal block has no other summaries in it, so its hash is that of the texts only.

Deleting a meta summary (SummaryService.removeSummary(part, unwind)) with unwind restores the summary from replacedSummary: it is deserialized into a new Summary. Because that summary can be a meta summary with its own replacedSummary, unwinding nests. A meta summary takes over the uuid of the summary it replaces (attachMetaSummary), and the restored summary gets it back, so the nextSummaryUuid of other meta summaries stays valid. Restoring a backup creates all summaries with new uuids, so ManuscriptServiceImpl.restoreMessages rewrites the nextSummaryUuid of the restored meta summaries (also inside replacedSummary) to the new ones.

Annotation

A note of the author, a model, an agent or an extension about a passage of a part (see Annotations). A real table (annotations, migration V19: the story view, the panel and the cleanup ask by part and status), the rest is extended content. part is @CleanupReference(OWNED_BY): the part takes its annotations to the trash and the purge; the column part_id is also mapped as partId (read only), so code without a session (the commands, the UI lists) can tell the part without loading the lazy proxy.

The text carries no marker. A marker in the stored text would leak into prompts, summaries, exports, search and extensions, so the ranges are kept apart and moved through every change of the text. The pieces:

• utils.TextAlignment aligns an old and a new text token by token (TextTokens: white space, words, single characters; longest common subsequence after the common start and end are cut off, 4 million cells at most, an insertion or deletion block is slid to the latest place it can sit, so "hours. Then" to "hours. Very much. Then" keeps the full stop with the old sentence) and refines the gaps by letters (a typo in a word is the same word). mapStart / mapEnd move a range like the markers of a word processor: an edit before moves it, inside grows or shrinks it, text typed exactly at the start or the end is outside; isPartlyRetained is false when no letter of the range survived (it is gone), isApproximate when the texts were too different to align (then a range that is not inside a retained stretch is handled by the quote).

• utils.QuoteFinder finds a quote in a text: letter for letter (several places: the context around it and the hint decide), then ignoring case and white space, then the passage with the most words of the quote in order (70 %). The same finder serves the AI annotate, the commands and the selection of the browser.

• AnnotationService.shift(book, part, oldText, newText) is called after every edit of the text by the author: ChatMessageServiceImpl.storeResponse (so updateResponse, the redraft, the part edit commands, the agent merge of a text), the search and replace (ManuscriptReplaceServiceImpl.apply) and the agent merge that keeps the counts (AgentMergeServiceImpl.storeResponse). A text written another way needs its own call. mergeIntoPrevious calls moveToPrevious with the offset of the later text in the joined one, branch calls copyForBranch (a swipe is a new text and gets none), a generation of the same part (REGENERATE) calls relocate in CleanupStep (quote search only, the text is not an edit of the old one). An orphaned annotation whose quote is back (an undo) is found again by shift.

• ChatMessageService.delete soft deletes the annotations of the part (and of the parts below it), a hard delete removes them first (hardDeleteForParts: the foreign key), the delete of a book soft deletes them in bulk; the cleanup knows Annotation as owned by a part (OWNED_ON_DELETE).

Backups. A message of the backup JSON has annotations: an array of {uuid, status, source, start, end, extendedContent} (AnnotationService.marshalForManuscript makes them for the whole book with one query, restoreForPart makes them again, with new uuids; the overload with a Collection<UuidPair> also reports ANNOTATION pairs, the uuid in the backup and the new one, for the working copy of an agent). Everything that copies a book through the backup form (the clone, the restore, the working copy of an agent) copies them.

Lorebooks

Lorebook

LorebookEntry

How entries are activated is described in Lorebooks and Generation pipeline.

Tags

Tag is a value (value) owned by a user. TagRelation (table t2e) attaches a tag to any entity:

Books, lorebooks and lorebook entries are tagged. TagRelationService creates and removes relations and finds the tags of an object (getTagsForObject(entity), getTagsForObject(entity, true) for negative ones). Because the relation is polymorphic, there is no foreign key from t2e to the tagged table; cleanup follows objectId / clazz through @CleanupReference(targetClassField = "clazz").

Inference providers and protocols

Both use joined inheritance: a base table with the common fields and one table per type. Protocol has one type; AI has two, OpenAICompatible and ExtensionAI.

AI / OpenAICompatible / ExtensionAI (inference provider)

Usage (tokens used with an AI)

One row per AI and user (owner), unique on (ai_id, userId): a personal AI has one, a shared AI one for everybody who uses it. Counts what the user used, and since the last reset for the user's limit on the AI (below).

All are real columns: they are added up in SQL (UsageRepository.add, UPDATE ... SET x = x + :n), not read and saved, so parallel requests don't lose counts. UsageService.record(ai, prefill, generated) is the way to count: it adds to the row of the current user, creating it on first use (under a lock held until the commit, so two first uses can't both insert), and never throws. findAllForAI(ai) lists every user's usage for the owner of the AI or an administrator; findAllUsages() lists all of them, administrators only (the Usage per user dialog of the Models tab). See Inference.

Limits of the users of a shared AI

AI.limits is an @ExtendedAttribute string holding a JSON object: the AILimit of each user under the user's uuid (strategy LimitStrategy UNLIMITED / TOKEN_LIMIT / NOT_AVAILABLE, resetStrategy ResetStrategy DAILY / WEEKLY / MONTHLY, and prefillTokens, generatedTokens, totalTokens, each null for unlimited). AILimits reads and writes it; a user without an entry is not limited. These are only the settings and they are set by the owner (an administrator). What a user used is not in this JSON on purpose: it changes with every request, from the thread of any user, and AIService.save is for the owner and would lose parallel updates, so it is in the two period* columns of the user's Usage, added up in SQL like the other counters.

The reset is lazy. ResetStrategy.periodStart(now) is the start of the calendar day / week (from Monday) / month in the server's time zone. UsageService.add (under record) of a user with a TOKEN_LIMIT first runs UsageRepository.resetPeriod (zeroes the period counters when lastUsed is before the period start) and then adds; for a user without one it zeroes them. AILimitService.getStatus does not write: it reports 0 used when lastUsed is before the period start. LimitStatus.block(promptTokens) is the first reached of the sent, generated and total limits, or NOT_AVAILABLE. The prompt is known before the request, so a request is refused when the sent (or total) counter is at its limit or the estimated prompt (approximatePromptTokens, our count) would take it over; the generated counter is only compared with its limit, so the last response can go over - unless AILimit.hardGeneratedLimit is set: then the generated counter plus the response the request reserves (TokenLimits.responseTokens(ai, protocol), the max_completion_tokens it sends; 0 when there is none) has to fit into the generated limit. The hard flag does not apply to the total limit. AILimit.validate() rejects negative numbers and, when all three are set, sent plus generated below the total (the total could never be reached).

Writing statistics (ManuscriptStatistic, ActivityInterval, ManuscriptSize)

What happened to a book, day by day (design and plan: TODO.USAGE.MD; not to be confused with Usage, which counts what a user used of an AI and knows no book). Three OwnedEntitys of the owner of the book. The book is a plain bookUuid column, not a reference, and none of them has a @CleanupReference to it: the trash and the purge of the book leave the rows, the user clears them (ManuscriptStatisticService.clearBook / clearAll, a hard delete of all three tables). So bookName and folderPath are copied into every row, to show a book that is gone. The day is a LocalDate stored as ISO text (LocalDateConverter), the day of the server's time zone (ManuscriptStatisticService.today(); a test sets the clock of ManuscriptStatisticServiceImpl).

ManuscriptStatisticService.count(book, [agent,] StatisticCounts) is the way to count: it resolves the owner from the book (no session needed, it runs on worker threads), reads the folder path from the folder chain (FolderService.findChain), updates the row of the day and makes it on the first count; it never throws, a failure to count is logged and must not fail the work. The writers go one at a time (a read-write transaction takes the SQLite write lock when it begins, see ReadWriteRoutingDataSource), so nobody inserts the row between the update and the insert. The reads take a StatisticFilter (days, StatisticSide ALL / AUTHOR / AGENTS, optionally a book) and only see the rows of the current user: total, sumByDay, sumByBook (with the BookStatistic.BookState of every book: PRESENT, TRASHED for a book in the trash, PURGED for one that is gone) and sumByFolder.

Name and path. The rows hold a copy of the name and the folder path of the book (bookName, folderPath) of the time they were written, and they are kept equal in all the rows of a book of all three tables. ManuscriptStatisticService.refreshBook(book) does it for a book (every ManuscriptServiceImpl.save of an existing book), refreshBooks(books, folder) for books that were moved into a folder (moveToFolder is a bulk update), and refreshFolders() for every book of the user that is not in the trash, after a folder was renamed, moved or deleted (its content moves up) or an Organise move: it reads the names and folders of the books and folders straight from the database (findPlacements), so it sees what a bulk update did. The updates are guarded to write only the rows that differ. A book in the trash, and what deleteWithContent of a folder takes with it, keeps its last name and path; deleting a book changes nothing. noteBook(book, path) is the safety net, called by the writers of the rows (count, ActivityIntervalService.start, ManuscriptSizeService.set): when the name or path differs from what the last writer told and from the stored book, the rows are corrected. Clones (duplicate, branch into a new book, cloneFromBackup), the imports, restoreBackup and the agent's working copy book nothing (their versions are DUPLICATE or flagged uncounted), and a backup carries no statistics.

Size. ManuscriptSizeService.touch(book) only notes that the book may have changed; one worker thread measures it 2 s after the last touch (so a burst of edits, the end of a generation or an import is one measurement; touchNow when a window is closed). It loads the book by id, skips an agent's hidden working copy and a book in the trash, adds up what the parts have saved (getTotalWordCount, getTotalTokenCount, getActiveBranchSizes: no tokenizer is called, so changing the model changes no stored number) and sets the row of today (nothing is written when nothing differs). The first row of a book that was made under tracking (touch(book, true)) is a normal row, so the growth from nothing is the first day's; any other book without a row gets a baseline row, which stays baseline for the rest of its day. growth(book, from, to) is the size at the end of the range minus the size before it (the latest earlier row, or 0 for a book born in the range, or the baseline row the book was first seen with); a day without a row carries the size before it, and a shrinking book has negative growth. The touches are in the choke points: ChatMessageServiceImpl.save / delete, ManuscriptServiceImpl.save (the active leaf moves with it), cloneFromBackup, restoreBackup, MessageRevisionServiceImpl.append, the restore and the permanent delete of the trash, and the open and close of a book window.

Time. The stretches are written only by the ActivityTracker (see Services): the registrations of a key (kind, book, side) open together are one row, and the owner of a row is the owner of the book, read from it, never the current user. A row is cut at midnight of the server's zone (two rows that meet at 00:00:00.000). Every time that is shown goes through TimeUnion: the stretches sorted by start, swept, the overlapping and touching ones merged, so five agents running for 24 hours on five books are 24 hours in total and 24 hours each, and the four kinds are four numbers that are never added (ActivityTimes: editing happens while the book is open). ActivityIntervalService.times(filter) is the union over the filter (the side ALL puts the author's and the agents' stretches together before the union), timesByDay the union per day, timesByBook the own time of each book (they overlap, so they are not parts of the total).

Showing it. StatisticReportService.report(filter) reads the three tables for a filter and returns one StatisticReport: the counts and the time of the range, the days that have anything (DayLine), the size on every day (SizeSeries.build: a day without a row has the size of the last row before it, the growth of a day is the sum of the books' changes, a book first seen as a baseline has no growth that day, a book born in the range grows from nothing, a day with another active leaf is marked), the growth of the range (SizeSeries.perBook, the same rules as ManuscriptSizeService.growth) and, for all the books, a line per book (BookLine: the counts of sumByBook, the book's own time of timesByBook, its growth; a book that only has time or a size row is named from its size rows or the book). The windows (StatisticsView, see User interface) and the commands only format it. StatisticRange (TODAY, YESTERDAY, LAST_7_DAYS, LAST_30_DAYS, THIS_MONTH, ALL_TIME, CUSTOM) resolves the presets to two days against the date of the server.

Rebuilt history. StatisticsBackfillMigration (app 8 → 9, see Database & migrations) writes the rows of the books that existed before the statistics, for the days before the day of the upgrade, with reconstructed true; the live counting is the only writer from that day on, and a book that has rows is not touched. StatisticsReconstruction holds the rules (the live ones, StatisticCounts.ofVersion): the first version of a part is a part created and its words written or generated, a later one is an edit whose change of the word count is written / generated or removed; a copy, a saved version, an import and a repair count nothing; a version of an agent, and the merge of its work, go to the agents' side; a part that was deleted is a delete with its words removed on the day of its last change (not for the parts of a book in the trash); a part without versions is rebuilt from the part itself. The tokens are an estimate: the tokens of the model's versions and the prompt tokens the part saved, on the day of its last generation. The size of the book at the end of each day on which a part changed (a row only when it differs from the day before) is the sum of the newest versions of the parts that existed and were not deleted, whole and along the branch of today's active leaf. The time is not rebuilt: UserSetting.statisticsSince (an extended attribute, an ISO date) is the day it is recorded from, StatisticReport.timeSince carries it and the windows show a dash for the days before it and leave them out of the chart; a rebuilt day is marked ~ and the notes under the totals say how many there are (DayLine.reconstructed). A size row measured live after a rebuilt one is not a baseline: its growth is against the rebuilt size.

Days worked on, streaks and the goal. Nothing of it is stored. StatisticReportService.activity(bookUuid) reads the author's side (counts per day, the EDIT stretches as a union per day, the last change of a count row or the end of the last stretch as the last activity) and returns a WritingActivity: the days that were worked on, which is WritingActivity.worked (words written, an edit, a part created or editing time; the agents' side, generated words alone and opening a book are not a day), the current and longest streak (today not yet worked on does not break the current one) and the shade of a day (level: by the goal when there is one, else by the busiest day shown; a day whose time was not recorded is only marked). The daily goal is UserSetting.dailyGoalWords / dailyGoalMinutes (extended attributes, no migration; the words the author wrote themselves and the minutes of editing, over all the books), read as a WritingGoal only for the request of all the books; a day reaches it when every goal that is set is reached, and currentGoalStreak / longestGoalStreak count those days.

Protocol / ChatCompletionProtocol

Settings

Setting is one table (settings) for all settings classes, with the class name in DTYPE and in key. Almost all values are extended attributes.

SettingService.getOrCreate(UserSetting.class) returns the current user's settings, creating them on first use; getOrCreateApp(AppSettings.class) the installation's. A function that only reads uses findOrBlank(cls, user) instead (a missing setting is a blank one that is not stored, so a read-only transaction does not write: creating the row there fails with SQLITE_BUSY when another connection wrote in between; find returns null). Extensions keep their settings in the attributes of UserSetting (Reviewer and Side Query profiles do).

The prompt used for a book is resolved in four levels: the book's value (Manuscript.template...), else the nearest folder above it that sets one, else the user's default (UserSetting.masterTemplate...), else the built-in default (Defaults).

Users

User extends BaseEntity directly - it has no owner and no extended attributes.

The current user of a session is the session-scoped user bean (see Architecture), which holds a copy of the logged-in user's id, login and name - load the entity through UserService when you need more.

Permission (what a user was allowed)

Permission (domain.security.model, with PermissionType in domain.security.permissions; table permissions, V15) extends ExtendableEntity. There is at most one live row per holder and PermissionType; a user without one does not have the permission.

Agents

An agent is an LLM that works on one manuscript on its own and calls commands as its tools; what it does is a run. Both extend ExtendableEntity. The services are in Services.

Agent and AgentRun (agents)

Agent (table agents, V16) is the definition.

AgentRun (table agent_runs, V16) is one execution, the whole job.

The columns status, statusReason, the counters and finishedAt are not updatable in the mapping: they are changed only by the SQL updates of AgentRunRepository (through AgentRunService), so a save of a stale copy of the run (what a runner thread or the UI holds) never writes them back. Their setters work for a new run only. The run folder on the disk is derived from the uuid (<application folder>/agent-runs/<uuid>), no path is stored. One partial unique index keeps at most one run per agent that is not over, see Database.

AgentMessage (table agent_messages, V16) is a row of the transcript of a run: run (column run_id, OWNED_BY), ordinal (the place in the transcript) and the LLMChatMessage as JSON in the extended attribute message (toChatMessage/setChatMessage). Rows are only appended; the system prompt is not stored. The answer of the model also carries the extended attribute contextTokens (the tokens of the request and the answer, AgentMessageService.appendAnswer); and reasoning (what the provider sent as the reasoning of the round, kept beside the answer for the console and the download, never sent back to the model); the last contextTokens is the current context of the run (getContextTokens, shown by the console), kept on the row and not on the run because an extended attribute of a row that is never updated needs no SQL update and no migration. The column thread (V18) is the handle of the sub-agent whose conversation the row belongs to (sub1...), null for the agent's own; the ordinal counts per thread and the queries of the agent's transcript read thread IS NULL. The first row of a conversation has the extended attributes threadTask and threadCall (the id of the spawn_agent call), the row that ends it threadEnd (DONE, FAILED, CANCELLED); a conversation without one was only paused. See Sub-agents.

AgentRunPair (table agent_run_pairs, V17) is not a BaseEntity: a row says which item of the working copy of a run is which item of the original, and a book of thousands of parts is thousands of rows that are only looked up by the run, so there is no uuid, no soft delete, no extended content and no owner. Columns: runId (run_id, the id of the run, not a reference), kind (AgentRunPairKind: MESSAGE, SUMMARY, LOREBOOK, ENTRY), copyUuid (unique in the run), originalUuid (null for an item the agent made) and handle (unique in the run, null until the agent is given one). The cleanup does not see it, so the rows are deleted by AgentRunPairService.deleteForRun when the run lets go of its copy. Reach them through the service, which checks that the run is the user's.

A soft deleted manuscript takes its agent with it, and a soft deleted agent its runs (ManuscriptService.delete, AgentService.delete); a run that is deleted while it is not over is discarded first. All three are in the trash and restored together.

Besides the rows an agent has files: its workspace (<application folder>/agent-workspaces/<agent uuid>, notes that stay between the runs) and, for each run, a folder (agent-runs/<run uuid>). They are not rows and not in the backups of the database; the cleanup, deleting for good and cloning an agent take care of them, see Files of an agent.

Resources

Resource (resources, an OwnedEntity) is an uploaded file; today the files are the images attached to parts (ChatMessage.getImages(), see Extended attributes).

Deleting a resource is a soft delete like any other; files are never deleted from disk.

Cleanup references

Soft-deleted rows are purged by CleanupService (Admin → Cleanup, administrators only: every method calls AdminGuard.requireAdmin()). Before purging, it builds a graph of all references between rows and decides what may go. Each reference has a policy (@CleanupReference, in domain/traits/):

JPA associations are found automatically from the Hibernate metamodel and default to STRONG. Plain id fields pointing to other entities ("soft references", including extended attributes) are only seen when annotated with target = SomeEntity.class or, for polymorphic ones, targetClassField = "clazz". On an entity class, field = "..." sets the policy of an inherited field (OwnedEntity makes owner OWNED_BY, so a deleted user is purged together with everything they own).

Trash

The same reference model drives the trash (TrashService, implemented by CleanupServiceImpl; Trash tab and Admin → Cleanup). It lists the soft-deleted rows of every root entity that is an OwnedEntity (Protocol stands for its subclasses) with narrow projections - the extendedContent blob is only read by getExtendedContent, which pretty-prints it - and restores them with a bulk UPDATE ... SET deleted = false, modification = now, so no entity listener runs and extended attributes stay untouched. Before it restores, it looks up the parents of the selected rows:

When a parent is deleted and not part of the same selection, nothing is restored and the result lists the blockers. A deleted owner counts as a parent, so the objects of a deleted user can't be restored.

The restore is the reverse of the soft delete cascade: the selection is first extended with the deleted rows it owns (OWNED_BY rows pointing at a selected row, rows a selected row OWNS, repeatedly), so restoring a book brings back its parts, summaries and tags, and RestoreResult.getRestored() counts them too. It can't tell a row that was deleted together with its owner from one deleted on its own earlier, so the latter comes back as well. A restored ChatMessage deleted by deleteNodeAndMigrateChildren does not get its children back: they were moved to its parent when it was deleted.

TrashService.deletePermanently removes deleted rows for good, for the Delete Permanently button of TrashGrid. It is limited to getPermanentlyDeletableTypes() (ChatMessage, LorebookEntry; anything else raises IllegalArgumentException) so a user can't purge a whole book, and checks ownership like restore. It runs the purge of the cleanup (executePurge) on the selection: the rows that go with it are only Summary and TagRelation rows, a summary something else uses stays, and a block by a CleanupContributor (or any other row that would have to go) makes the whole call return DeleteResult.getBlocked() and roll back. Before that, moveChildrenOfMessages moves the children of the selected messages - deleted ones too, which the editor's deleteNodeAndMigrateChildren never sees - and the activeLeaf of books to the nearest ancestor that is not selected, with bulk updates, so removing a chain of parts keeps the parts below it connected.

So a deleted provider that a book still uses shows up as blocked on the Cleanup page until the book is switched to another provider, while a deleted book takes its parts, their summaries and its tag relations with it.

Because the soft delete deletes the owned rows too, they are purge roots of their own. A root whose owner (OWNED_BY, directly or through other rows) is blocked is held back: it stays, is not listed as blocked itself and is not counted as purgeable, so a blocked lorebook that is still restorable keeps its entries.

Backups and copies

Book backups (BackupService) serialize a book as JSON: its columns, its extendedContent as is, its tags, all parts (with extendedContent and summaries) and references to its provider, protocol and lorebook by uuid and name. The backup also names the folders the book was in (the optional key folder, written by takeBackup, read by BackupFolders; see Folders). Restoring or cloning creates new rows (new ids and uuids for the parts) and copies extendedContent unchanged, so extended attributes and extension data come back exactly as they were. Linked entities are resolved among the current user's own entities, first by uuid, then by name.

Lorebooks list their fields one by one (name, payload, comment, filtering...) in LorebookService.marshalLorebooks, which the lorebook export (exportLorebook), the lorebooks inside a book backup and the copy of the lorebooks for an agent share. Since 1.4.0 it adds the attributes of lorebooks and entries under attributes (when there are any), and the import (unmarshalContent, importLorebooks) sets them on the new lorebooks and entries, so extension data such as the Lorebook VCS history travels too. A field that is a real column, or an @ExtendedAttribute field that is not listed there, does not (add it to the marshalling). Imported entries have new uuids (the format has none), so data that refers to an entry by its uuid does not find it.

Keep this in mind when adding data: for books and parts anything in extendedContent is backed up automatically, while a new column has to be added to the backup and restore code (BackupServiceImpl, ManuscriptServiceImpl.restoreBackup / restoreMessages); a new column or @ExtendedAttribute field of a lorebook or entry has to be added to marshalLorebooks and unmarshalContent, data in the attributes goes along by itself.

Adding a field or an entity

A field on an existing entity:

1. Decide where it lives. Prefer an extended attribute (@ExtendedAttribute @Transient) - no migration, backed up automatically. Use a column only when you need to query, sort or join on it.

2. For a column: add a Flyway migration (see Database & migrations); Hibernate validates the schema on start. Extend the backup / restore code if the value should survive a backup.

3. For a reference to another entity: think about its cleanup policy and annotate it if STRONG isn't right. Soft references (ids) must be annotated with target or targetClassField, otherwise cleanup may purge the referenced row.

4. Save it through the service's save (not saveWithoutEvent).

5. If its text should be found by a search, annotate it @Fulltextable (see Full-text column) and add an app migration that refills _fulltext of the existing rows.

A new entity:

1. Extend ExtendableEntity (or OwnedEntity if it never needs extended data) and annotate the references.

2. List the class in src/main/webapp/config/persistence.xml - unlisted classes are ignored.

3. Write a Flyway migration creating the table, its <table>_SEQ table and indexes.

4. Add a repository (JpaExtendableRepository subclass) and a service (ExtendableServiceImpl subclass), and define both beans in services-config.xml.

5. Add a CRUD test based on ExtendableCrudContract (see Testing).

6. Decide whether it belongs in book backups or lorebook exports.