Extended attributes

Plugins can't add tables or columns - there is no way to run their own Flyway migrations. Instead, every data entity carries a JSON object where a plugin can keep whatever it needs: ExtendableEntity.attributes. It is saved with the entity, loaded with it, and travels with it into backups.

Where attributes are

All data entities extend ExtendableEntity (Domain model):

User, Resource, Usage and AgentRunPair are not extendable.

JsonObject attributes = entity.getAttributes();       // never null after loading; a new entity starts with {}

Storing and reading data

Use one top-level key per plugin, named after the plugin's package, and put a JSON tree of your own model classes under it. The bundled plugins all follow this pattern:

public class ReviewerSettings {
    public static final String KEY = "com.github.enerccio.marginalia.extensions.reviewer";
    ...
}

public ReviewData getReviewData(ChatMessage message) {
    JsonObject attrs = message.getAttributes();
    if (attrs != null && attrs.has(ReviewerSettings.KEY)) {
        return gson.fromJson(attrs.get(ReviewerSettings.KEY), ReviewData.class);
    }
    return null;
}

public void saveReviewData(ChatMessage message, ReviewData data) throws Exception {
    JsonObject attrs = message.getAttributes();
    if (attrs == null) {
        attrs = new JsonObject();
        message.setAttributes(attrs);
    }
    attrs.add(ReviewerSettings.KEY, gson.toJsonTree(data));
    chatMessageService.save(message);
}

Rules:

• Save through the entity's service - save(...), not saveWithoutEvent(...). See How attributes are stored for why.

• Load before you change. Entities held by the UI are detached copies, and the one you got from a decorator may be older than the database. Reload it (service.find(entity) or find(id)), change the attributes, save it, and use the returned object from then on. Saving an old copy overwrites changes made since - to the plugin's data and to the rest of the entity. Lorebook VCS's saveVCSData is a good template:

  Lorebook fresh = lorebookService.find(lorebook);
  fresh.getAttributes().add(LorebookVCSData.KEY, gson.toJsonTree(data));
  Lorebook saved = lorebookService.save(fresh);

  The same goes for the plugin's own data: when several components edit parts of one JSON tree, each change should re-read the stored tree, change only its part and save it (Lorebook VCS's updateEntryData), not save a copy it loaded earlier.

• Version your data. There are no migrations; if the format changes, read old formats too (a version field in your JSON tree makes this easy) and never fail on data you don't understand.

• Keep it small. The whole JSON object is serialized on every save of the entity, and some entities are saved often - a story part is saved up to every 250 ms while it's generated. Megabytes of history in a part's attributes slow generation down. Lorebook VCS stores the history once per lorebook, not per entry, for this reason.

Settings

• Per user: settingService.getOrCreate(UserSetting.class) returns the current user's settings; store under your key, save with settingService.save(userSetting). To only read them (in a read-only transaction) use findOrBlank(UserSetting.class, user), which does not create the row. To add the settings to the Settings screen, see Extending the UI → Settings panels.

• Whole installation: AppSettings the same way, only administrators should change it.

Keys used by the application

The application itself uses attributes too. Don't use these keys:

How attributes are stored

flowchart LR
    subgraph Entity["Entity in memory"]
        A["attributes<br/>{ &quot;com.example.x&quot;: {...} }"]
        F["@ExtendedAttribute fields<br/>e.g. ChatMessage.sceneSetting"]
    end
    Col[("extendedContent<br/>BLOB column")]
    Entity -->|"JpaExtendableRepository.save<br/>ExtendableEntityListener.serialize"| Col
    Col -->|"@PostLoad<br/>ExtendableEntityListener.deserialize"| Entity

attributes is not a column. The entity has one extendedContent column (bytes, UTF-8 JSON), and ExtendableEntityListener converts between the two:

• On save (JpaExtendableRepository.save) it builds one JSON object: each @ExtendedAttribute field of the entity under its field name, as a string, and each key of attributes with the prefix attributes_.

• On load (@PostLoad) and after every save it fills the fields and attributes back from that JSON.

The column is gzipped when the content is 256 bytes or more (ExtendableEntity.setExtendedContent; getExtendedContent returns the plain JSON). Rows saved before that are still read and are compressed when the entity is saved again (the application migration ExtendedContentMigration does it for all of them). Code that reads the column with SQL gets bytes that are not JSON.

The column of a story part looks like this:

{
  "sceneSetting": "The lighthouse at night",
  "response": "The lamp turned...",
  "attributes_templateVariables": { "mood": "grim" },
  "attributes_com.github.enerccio.marginalia.extensions.reviewer": { "reviews": [ ... ], "current": 0 }
}

Two things follow from this:

• saveWithoutEvent (used by backup restore, which writes extendedContent directly) does not serialize: changes to attributes saved that way are lost. Always use save.

• The _fulltext column that is written next to extendedContent is built only from the application's @Fulltextable fields (see Domain model). A plugin's attributes are not in it and can't be added to it.

• @ExtendedAttribute is the application's own way of adding fields to an entity without a migration - many fields of ChatMessage, Manuscript, Protocol and the settings are stored like this. It works for String, the numeric types, boolean, Date and enums, stored as strings (dates as ISO-8601 instants in UTC). @ExtendedAttribute(inject = true, injectPrefix = ...) on a JsonObject field spreads its keys with that prefix, which is how attributes itself is declared; a field of any other type (POJO with nested objects and lists, a list) is converted with Gson, an object spread the same way, a list or scalar stored under the plain prefix. This is for application code: a plugin can't add fields to an entity class, it uses attributes.

Where the data goes

Plan for the cases where your data does not follow: data that describes a part's text (a review) may be stale after Regenerate; data stored on a lorebook comes back after an export and import, but not attached to the old entries (their uuids are new).

References to other entities

If your data refers to another entity, store its uuid (or id). The Cleanup page of the administration purges soft-deleted rows (Domain model), and it knows only the references the application declares (@CleanupReference) - ids inside a plugin's attributes are invisible to it. Two consequences:

• A deleted entity your data points to can be purged. Always handle a reference that resolves to nothing or to a deleted entity. Reviewer falls back to the book's provider when the configured one is missing or deleted:

  AI configuredAi = aiService.find(setting.getSelectedAiId());
  if (configuredAi != null && !configuredAi.isDeleted()) {
      return configuredAi;
  }
  return aiService.find(manuscript.getAi());

• If a reference must keep its target alive, or must be cleared when the target is purged, register a CleanupService.CleanupContributor:

  cleanupContributor = new CleanupService.CleanupContributor() {
      @Override
      public void collectReferences(Set<CleanupService.EntityKey> candidates, CleanupService.BlockSink sink) {
          // during Analyze / Purge: block candidates the plugin still references
          for (Long aiId : idsOfProvidersMyPluginUses()) {
              CleanupService.EntityKey key = new CleanupService.EntityKey(AI.class, aiId);
              if (candidates.contains(key)) {
                  sink.block(key, "used by My Plugin");      // shown as the reason on the Cleanup page
              }
          }
      }
  
      @Override
      public void beforePurge(Set<CleanupService.EntityKey> purged) {
          // in the purge transaction, before the rows are removed: drop references to them
      }
  };
  cleanupService.registerContributor(cleanupContributor);    // onExtensionLoad
  cleanupService.unregisterContributor(cleanupContributor);  // onExtensionUnload

  EntityKey uses the root entity class of a hierarchy (AI, not OpenAICompatible). Cleanup runs for the whole installation, not for one user - don't use currentUser in a contributor.

When resolving anything that came from data the user can edit or import, use findForUser(uuid) so that one user's data can't point a plugin at another user's entity.