Testing

Marginalia's tests are integration tests first: most of them start the real Spring XML configuration against a throwaway SQLite database and talk to a fake OpenAI-compatible server. They need no network, no model and no API key, and they run in CI on every push and pull request. This page explains the test infrastructure and how to add tests.

All test code is in marginalia/src/test/java/com/github/enerccio/marginalia/; paths below are relative to it.

Running the tests

From marginalia/:

mvn test                                     # all tests
mvn test -Dtest=LorebookActivationTest       # one class
mvn test -Dtest='Backup*Test'                # a pattern
mvn test -Dtest='MacroRenderingTest#name'    # one method

Surefire is configured in pom.xml with:

How long mvn test takes (about 3,000 tests), with the machine, as the time depends on it. Add a row for yours: the Total time line of Maven, once with -Dtest.forks=1 and once as it is.

After a mvn install or mvn package that compiled with plain javac, mvn test may skip the AspectJ compile of the classes that did not change and leave them unwoven: a @Configurable class (a command, a UI class, an anonymous subclass in a test such as SearchReplaceCommandsTest) then has null in its @Autowired fields. touch the source of the failing test, or build with clean; it is not a bug of the code.

A class can run after any other in its JVM, and the classes of a JVM share its database, so a test must not count on what the database holds (see The shared database): create data with uniqueName and compare with what was there before.

Reports are written to target/surefire-reports/; the CI job uploads them as an artifact when a test fails.

In the IDE

Tests run from IntelliJ IDEA like any JUnit test, with two conditions:

• the classes must be compiled with AspectJ (see Building from source), otherwise UI classes and @Configurable beans miss their dependencies;

• add -XX:+EnableDynamicAgentLoading -Djdk.attach.allowAttachSelf=true to the VM options of the JUnit run configuration template if you run RuntimeInstrumentationTest.

Without marginalia.test.home the tests fall back to target/test-home relative to the working directory, so run them with marginalia/ as the working directory (IntelliJ's default for the module).

Libraries

There is no mocking library. Services are tested through the real Spring beans and the real database; the only fake is the model server.

What is covered

The tests added in 1.4.0 check, besides what the names say: GenerationOptionsTest (a generation with another model and protocol than the book's, the book untouched), PermissionServiceTest and PermissionsDialogTest (the permissions and their window), LazyContextMenuTest (a menu filled when it opens) and LazyBranchTest (which parts of a long story the window has loaded while it slides), TextDiffTest and WordDiffTest (the sentence and the word diff of the review), RoundStatsCollectorTest (what a streamed answer says about how it went: finish reason, cache, reasoning, llama.cpp timings), AgentCleanupTest (an agent, its runs and the working copy are deleted, restored and purged together) and ConcurrentTransactionsTest (read-write transactions take the SQLite write lock, read-only ones do not wait for them). test/TestImages makes small pictures and test/AgenticRequestAttributes the request context of a run, for the tests of the agents.

Not covered by automated tests: the Vaadin UI (apart from LLMSnapshotTest, HelpAssistantWidgetTest, ResourcesPartTest, BranchSearchTest, WorkspaceStateTest, AIDialogExtensionTest, TabDockTest, LorebookPreviewTest and InvalidExtensionsWarningTest), the bundled plugins (apart from the unit tests of google-genai and anthropic), and the desktop launcher (the CI smoke test only checks that the packaged app starts).

Test bases

classDiagram
    MarginaliaTestBase <|-- GenerationTestBase
    MarginaliaTestBase <|-- OwnedCrudContract
    OwnedCrudContract <|-- ExtendableCrudContract
    MarginaliaTestBase <|-- BackupTestBase
    GenerationTestBase <|-- LorebookActivationTest
    ExtendableCrudContract <|-- ProtocolCrudTest
    TemplateTestBase <|-- MacroRenderingTest
    class MarginaliaTestBase {
        +llm
        +createUser()
        +login()
        +createAI()
    }
    class GenerationTestBase {
        +ai
        +protocol
        +newManuscript()
        +generate()
        +onEvent()
    }
    class TemplateTestBase {
        +context
        +data
        +render()
    }

TemplateTestBase doesn't start Spring. Pick the lightest base that works:

MarginaliaTestBase

test/MarginaliaTestBase starts the application context from src/test/resources/META-INF/spring/test-application-config.xml. That file imports the production application-config.xml and adds TestContextPostProcessor, which

• removes the OSGi framework (OsgiServiceImpl) and the instrumentation agent (RuntimeInstrumentationInitializer),
• points Configuration to a new folder target/test-home/ctx-<random>/ instead of ~/.marginalia,
• makes TestUserService the user service: it hashes passwords with one round of SHA-256 instead of PBKDF2 (65536 rounds), which was close to half of the time the tests spent, every test creates a user. authenticate works as usual; PasswordHashTest keeps the real PBKDF2 hash covered.

Everything else is the production wiring: the services, Hibernate, Flyway migrations, the session-scoped current user. The base class gives you:

Each test method runs in its own mock HTTP request and session (@SpringJUnitWebConfig), so session-scoped beans start empty: call login() before using a service that needs the current user.

The shared database

Spring caches the application context, and with it the database, for the whole test run. All test classes that use MarginaliaTestBase share one SQLite database. Write tests so that they don't depend on what else is in it:

• create the data a test needs in the test, with uniqueName(...) for anything looked up by name;
• assert on the entities you created (contains, doesNotContain), not on totals (hasSize);
• operations that work on the whole database (cleanup, database backups) must only assert on their own data - see CleanupServiceTest.

When a test really needs an empty database or changes global state, mark it with @DirtiesContext. The next test then gets a new context with a new folder and database:

@Test
@DirtiesContext(methodMode = DirtiesContext.MethodMode.BEFORE_METHOD)   // starts on an empty database
void lastAdminIsProtected() throws Exception { ... }

@DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_CLASS)        // changes the shared backup schedule
class DatabaseBackupScheduleTest extends MarginaliaTestBase { ... }

The first context of a JVM costs about 20 seconds (Spring, Hibernate, Flyway; measured as the start of one test class on the first machine of the table, a context built later in a warm JVM should cost less), so use it sparingly.

The application folder

Each context gets its own folder under target/test-home/ with the database, data, backup and extension folders. They are not deleted after the run, so you can open a test database with any SQLite tool after a failure. mvn clean or deleting target/test-home removes them.

GenerationTestBase

test/GenerationTestBase runs real generations through every pipeline step (see Generation pipeline). Before each test it logs in a new user, creates an AI pointing to llm, saves a chat-completion protocol and sets the mock's fallback answer to "generated".

test/GenerationRun is the GenerationListener that stands in for the story editor. It records the streamed response and reasoning, errors (getErrors()) and simple errors (getSimpleErrors(), the messages the UI shows in a notification), warnings (getWarnings()), the created part (getMessage()), and the outcome (COMPLETED or CANCELLED). Questions the pipeline asks the user (askQuestion) are answered yes.

class MyGenerationTest extends GenerationTestBase {

    @Test
    void instructionsReachTheModel() throws Exception {
        Manuscript book = manuscriptService.save(newManuscript());
        llm.reply("The lighthouse was dark.");

        GenerationRun run = generate(book, "The keeper climbs the stairs.");

        assertThat(run.await(TIMEOUT)).isEqualTo(GenerationRun.Outcome.COMPLETED);
        assertThat(run.getErrors()).isEmpty();
        assertThat(run.getResponse()).isEqualTo("The lighthouse was dark.");
        assertThat(llm.lastCompletionRequest().getLastMessage().content())
                .contains("The keeper climbs the stairs.");
    }
}

To look at the pipeline state in the middle of a generation, capture it in an event listener. LorebookActivationTest reads the activated entries this way:

AtomicReference<List<LorebookEntry>> activated = new AtomicReference<>();
onEvent(Events.AFTER_PROCESS_LOREBOOK, book,
        e -> activated.set(e.getProperty(GenerationProperties.ACTIVATED_LOREBOOK_ENTRIES)));
generate(book, "...");

Listeners are global (StoryGenerationService.addEventListener); onEvent filters by book and always calls chain.next(), so a failing assertion inside the action doesn't stop the generation. Assert after generate returns, not inside the listener - an exception in the listener only ends up in the log.

CRUD contracts

Every entity that belongs to a user has a CRUD test built from crud/OwnedCrudContract. The contract contains the tests; a subclass only says how to make and change an entity:

The inherited tests check identity and owner on create, find by id / uuid / for user, update (creation date kept, modification date moved), soft delete (hidden from all find*ForUser and findAll* methods), hard delete, isolation between users, and that saving as another user keeps the owner.

Entities extending ExtendableEntity use crud/ExtendableCrudContract, which adds the persistence of plugin attributes (getAttributes()): nested JSON, changes and removals, unrelated updates, and saveWithoutEvent (used by restore). Put at least one @ExtendedAttribute field into newEntity() and modify(), so the inherited tests also cover the extended fields. crud/ProtocolCrudTest is a short, complete example.

When you add an entity (see Database & migrations), add its <Entity>CrudTest.

TemplateTestBase

templates/TemplateTestBase creates a TemplateServiceImpl directly, without Spring, with a deterministic context: the clock is fixed at Sunday, 15 March 2026, 14:30:45 UTC, random macros use a seeded Random, and the template data describes a scene in the middle of a story (POV Alice, characters Alice, Bob, Carol...). One data object is shared by all render(...) calls of a test, the way lorebook entries of one generation share their variables.

MacroLorebookFixtureTest renders src/test/resources/macro-test.json, a SillyTavern lorebook whose entries document their expected output. When you change macro behaviour, update the fixture and the expectations together.

The fake LLM server

test/llm/MockLLMServer is a small OpenAI-compatible server on 127.0.0.1 with a random port, built on the JDK's HttpServer. One shared server lives for the whole test JVM (MockLLMServer.shared()); each test gets its own scenario with its own base URL, http://127.0.0.1:<port>/<scenario-id>/v1. A provider created with createAI() points to the test's scenario, so tests never see each other's requests or answers.

It answers:

Programming answers

Chat completions are answered in this order:

1. queued responses - llm.enqueue(...) or llm.reply("text"), one per request;
2. the responder function - llm.respondWith(request -> ...), if set and it returns a response;
3. the fallback - llm.fallback(...);
4. otherwise HTTP 500 no response programmed.

MockLLMResponse builds one answer:

MockLLMResponse.text("Hello world")                        // one content chunk
MockLLMResponse.chunks("Hel", "lo").chunkDelay(ofMillis(50))
MockLLMResponse.split(longText, 20)                        // chunks of 20 characters
MockLLMResponse.text("answer").withReasoning("thinking")   // reasoning chunks first
MockLLMResponse.text("answer").withReasoning("...").reasoningField("reasoning")  // instead of reasoning_content
MockLLMResponse.text("cut").finishReason("length")
MockLLMResponse.error(429, "slow down")                    // OpenAI-style error body
MockLLMResponse.chunks("a", "b", "c").disconnectAfter(2)   // stream ends abruptly
MockLLMResponse.text("answer").usage(120, 8)               // token counts in the last chunk (only if asked for)
MockLLMResponse.text("late").waitFor(latch)                // held until the latch is released
MockLLMResponse.toolCall("add", "{\"a\":1}")               // the model calls a function; .andToolCall(...) adds one

waitFor(latch) is the tool for "while generating" states: the request is received, the pipeline waits for the stream, and the test can stop the generation or inspect the state before releasing the latch.

Inspecting requests

Every request, including model listing, is recorded:

For code that calls an InferenceService directly (without the pipeline), test/InferenceCollector is a callback that pulls the whole stream and waits for the end (it also records the onFunctionCall reports).

Expected errors in the log

Tests of failure paths make the code log errors on purpose. To keep them out of the test output (where they look like real failures) and to assert on them, capture the logger with test/ExpectedLog:

try (ExpectedLog log = ExpectedLog.capture(ExtensionServiceImpl.class)) {   // or capture("extension.my-plugin")
    ...
    assertThat(log.errors()).anyMatch(m -> m.contains("broken"));
}

While the capture is open, the logger's events are recorded and not passed to the console; errors(), warnings() and atLeast(level) return the messages, entries() the full events with their exceptions.

Instrumentation tests

The instruct tests check the bytecode that makes @Extendable methods extensible (see Plugin development):

• ExtendableMethodVisitorTest transforms fixture classes from instruct/fixture/ with instruct/Instrumenter - the same transformation the agent applies, without an agent - and checks that the result verifies, keeps the class schema, behaves like the original and exposes arguments and locals to decorators.

• RuntimeInstrumentationTest installs the real agent into the test JVM and checks classes loaded both before and after the installation (fixture/agent/).

• ExtensionVerifierTest runs the extension verification on the extensions of fixture/verify/VerifyFixtures (decorators that are valid, ask for things that don't exist, or can't be followed) against the decorated VerifyTarget. Their class files are packed into a JAR on the fly and served by a stand-in Bundle. Decorators must be anonymous classes with the context calls inside them, as in a real extension - a decorator that delegates to a helper class would not be checked.

• PublicAPISignatureTest checks the @PublicAPI of every instrumented method of every @Extendable class of the application against the compiled classes (target/classes). It reads them with ByteBuddy's type pool and the local variable table (PublicAPISignatures), so the application classes are not loaded or initialized, and it needs the classes compiled by ajc like the application. A failure names the method and prints the annotation the method needs.

The fixtures are compiled with the same -parameters and preserveAllLocals options as the application; if you change the compiler settings in pom.xml, these tests tell you whether extensions still see names of arguments and locals.

Checklist for a new test

• Name it <Subject>Test and put it in the package of the area it tests; Surefire runs every *Test class.

• Extend the lightest base that works (Test bases).

• Create your own data with unique names; don't depend on what other tests left in the shared database.

• Don't call real models or other network services - program llm instead.

• Assert with AssertJ, with .as("...") where the failure message would otherwise be unclear.

• Capture expected error logs with ExpectedLog.

• A new entity gets a CRUD contract test; a new migration is covered by FlywayMigrationTest automatically (it runs all migrations and validates the schema against the entities).

• A bug fix comes with a test that fails without the fix, where the area is testable without the UI.

• Run mvn test before opening a pull request - CI runs the same command.