Methods

The Game is aFootIngesting the adventures of Sherlock Holmes

There is an episode of Star Trek: The Next Generationin which Data, the android, spends his off-duty hours on the holodeck playing Sherlock Holmes. The ship's computer has read the whole canon and rebuilds Victorian London around him. It is a lovely piece of science fiction, and it raises a question worth taking seriously. Could you build that now? Could you step into 221B Baker Street and work a case alongside Holmes?

The tempting answer is that large language models will have already read the works of Arthur Conan Doyle, and if you were to ask one about the canon it would answer fluently. Fluency, though, is not the same as reliability. The same model that quotes The Final Problem will also invent, with identical confidence, a scene that never happened, because it has read the stories but cannot point at them.

Before anything like the holodeck is possible, we need a system where every claim traces back to the sentence that supports it. This post is about building that substrate.

“Data! Data! Data! I can't make bricks without clay.” Holmes, The Copper Beeches

01Getting the clay

Ingesting the whole canon

The raw material, at least, is easy to come by. The Holmes stories are in the public domain, and Project Gutenberg hosts clean transcriptions of all of them. Our corpus is 0 words across 0 works.

Four novels and thirteen short stories. A novel runs five to eight times the length of a story, a split that resurfaces later when we measure extraction density.

The harder question is what to do with all that text. There are three obvious ways to hand a corpus to a language model, but none of them fit our needs.

What all three are missing is addressability, a way to refer to a specific sentence in a specific position and build on top of that reference. So the plan became to build the address space first and layer meaning on top of it afterwards.

02Architecture

A three-layer approach

The system is arranged as three layers, and keeping their jobs separate turns out to matter more than anything else in the design.

The lexical graph layer holds the text exactly as Conan Doyle wrote it, with a stable identifier for every sentence. It is the ground truth that everything above it points back to.

The entity extraction layer holds what the text describes: the people, places and objects of the stories, and the relationships between them.

The character-state layer holds perspective. What does Watson know at this point in the story? Who has heard about the note, and who is still in the dark? Crucially, this layer is computed at query time from the layer beneath it. The graph itself records only what happened. What a character believes about what happened is derived on demand, which is what will later let us model characters who are mistaken, deceived, or simply out of the room.

03The lexical graph

An address space made of sentences

A lexical graph represents text as nodes and edges, breaking it up into a hierarchical tree structure that can be traversed. In our graph, sentences are the atomic nodes. Each carries a stable ID and its global position in the corpus. Above the sentences sit paragraphs, above those the books, and at the root, the author.

This nesting gives every sentence its own address. A path like final-problem/section_1/sentence_3 points to one exact line of the canon, and everything built in the layers above cites these addresses.

Once every sentence has an address, entities can attach to it

Every sentence now has a stable ID and a position in the full collective works, which gives entity extraction something solid to reference. We can tag that Holmes finds an item in one paragraph and the same item is discarded in a later paragraph. When an entity is found, it attaches to a sentence ID, even when they come from entirely different documents ingested months apart. The map extends indefinitely, It joins the same traversable structure with no re-indexing of the existing corpus and no retraining of anything.

The chain along the bottom is the lexical layer. The pills above it are extracted entities, and the dotted lines are mention edges tying each entity back into a sentence. Hover an entity to isolate its citations.

Where this shows up outside fiction

The pattern generalises well beyond a detective canon.

• In legal discovery, every clause that mentions a counterparty or a defined term becomes traversable across thousands of otherwise unrelated contracts.
• In financial research, a sentence from an earnings call can link to the line in the annual report it corroborates or contradicts, across filings that share nothing else.
• In clinical records, a patient's medications and conditions become addressable per note, per visit and per provider, forming one timeline even when the notes come from different systems.
• In recruitment, a line in a candidate's CV claiming a skill can link to the sentence in a reference letter or interview note that backs it up or contradicts it, across documents that were never written to align with each other.

Querying the text directly

With the lexical graph in place, the canon becomes something you can query by position, by entity, or by paragraph. Here are four examples, each running against the ingested data.

-- MENTIONED_IN ties an entity to a section and carries the
-- sentence ids it was found in. Unnest those to get the sentences back.
SELECT m.story, s.id AS sentence_id, s.props->>'text' AS text
  FROM edges m
  JOIN nodes s
    ON s.story = m.story
   AND s.id = ANY (ARRAY(SELECT jsonb_array_elements_text(m.props->'sentenceIds')))
 WHERE m.rel_type = 'MENTIONED_IN'
   AND m.from_id IN ()
 ORDER BY m.story, s.pos;

04Entity extraction

From free text to structured facts

Now that we have an addressable namespace, we can run each paragraph through an entity extraction pipeline: an LLM reads the text and returns the entities present, the events that occur, any state changes, and who witnessed what. Extraction targets a fixed list of entity types we model character, location, object, case, document, and organisationrather than open-ended tagging. It's the same technique that turns a CV into a candidate profile or an invoice into line items, used here to turn Victorian prose into database join keys that connect text to a graph.

This is also where determinism ends. Splitting sentences involved no judgement, but extraction relies on the model's reading, so every event it returns carries source_nodes, the sentence IDs it was drawn from, keeping every claim in the graph traceable back to the text that supports it.

Two choices make this stricter than typical entity recognition. First, state changes are anchored to the event that caused them, not to a timestamp. Every state edge stores valid_from as the ID of the event that started it, and the model never has to say when it ends. Instead, the next event that moves the same entity closes the previous edge automatically, so an object can pass through several containers over the course of a story and each move stays correctly ordered relative to the others, with no wall-clock time required. Second, speech is captured as an event in its own right, since the fact that a character said something is objective, while whether what they said is true is a separate question, held for the layer above.

Model choice was a chunk-size problem as much as a quality one. On A Case of Identity (5,000 words) as a test bed, gpt-5-nano at 600-word chunks gave 311 events but 8 duplicate cases; gpt-5-mini at the same size cut that to 132 events and 3 duplicates, and 3,000-word chunks dropped duplicates to 2 at the cost of recall (88 events). Weaker models want smaller chunks; stronger ones can take bigger chunks and resolve coreference themselves. gpt-5-miniat 600 words won on balance and ran the full canon: 0 calls over 0 words, atgpt-5-mini's $0.25 / $2.00 per million input/output tokens, putting the full ingest at somewhere around $1–2.

Watch the graph assemble

Below is the full extraction for The Final Problem, in the project's graph viewer. Scrub the timeline and the graph builds as the story reaches each point. Entities appear as they are introduced, events fire, and edges form between participants. Speech arrives as a COMMUNICATES event, which is the separation that will later let the character-state layer derive a false belief without the graph ever storing one. The perspective dropdown is a preview of that layer. Pick a character, and the graph narrows to what they witnessed or were told.

Open the full graph explorer → all seventeen works, with the perspective filter and the live SQL.

Where this shows up outside fiction

• In accounts payable, a scanned supplier invoice becomes a structured record of supplier, line items and VAT, ready for a Making Tax Digital submission to HMRC, with each field still pointing back at the line on the document it came from.
• In property conveyancing, a solicitor's search results and title deeds become structured records of covenants, easements and charges, each one traceable back to the clause in the document that raised it.
• In financial compliance, names and directorships pulled from Companies House filings and client correspondence are checked against the OFSI sanctions list as structured records, rather than by re-reading the prose.

Here is the same passage from earlier, seen three ways. The lexical graph, the entity extraction on top of it, and reverse citation, where you pick a fact and watch the sentences that support it light up.

05Yield

From 0,000 words to 0,000 events

The pipeline is a distillation. The corpus shrinks at every step, from words to sentences to entities to events, and what survives is the structured residue the rest of the system runs on. Each row below is the canon-wide total at that layer.

Yield varies by work

Normalised by word count, the short stories are denser than the novels. A story has no room to breathe, so nearly every paragraph introduces someone or moves something. The novels dilute, and their long flashback halves, narrated far from Baker Street, are noticeably thin on extractable entities.

06Cleaning up across works

Reconciling entities across seventeen works

Extraction runs one work at a time, so the model only ever sees a local view. Across the canon that produces two opposite problems. Sometimes one person accumulates several labels, and the variants need merging into a single node. And sometimes one label is shared by several different things, which must be kept apart.

The canon offers a perfect specimen of the second problem, because it contains three Hudsons: Mrs Hudson, the housekeeper at Baker Street; Hudson the seaman, who drives the blackmail plot of The Gloria Scott; and Hudson Street, an address in The Crooked Man. They share a surname, but they are three distinct entities, and one of them is not even a person.

Cross-work deduplication runs after per-work extraction. Alias rules for the major recurring characters are hand-curated, while the long tail of one-off clients and witnesses is resolved by label similarity combined with type equality, so a street can never merge with a seaman.

07What you can now ask

Four questions, answered from the graph

Everything below runs live against the same canon-wide data used throughout this post. The full portfolio of query primitives, including aggregates, lives on /analysis.