Understanding a Law Firm through Study
Training agents to combine knowledge in weights, text memory, and search on realistic legal work.
The agents of the future will familiarize themselves with their work in many different ways: learning knowledge and skills into weights, writing notes to themselves, and using tools to efficiently navigate the world. We’re excited to share our first results on what this would look like.
In our work, we see these different forms of learning and memory as part of one end-to-end system. Integrating parametric knowledge, notes, and search opens up a space of qualitatively new behaviors. The models we’re building can recall information directly from their weights if asked, but more importantly, they use their knowledge in the loop with search. They don’t just write notes, but they learn how to use them effectively.
In this post, we show how our recipe looks on realistic work. In partnership with Harvey, we built an agent for a synthetic legal firm, Calderwood & Harkness (C&H). Our agent studies the accumulated knowledge of the firm, building memories in its notes and parameters. By combining its knowledge of the workspace with search, it solves tasks more effectively with an order of magnitude lower cost-per-query ($0.13 / 30% all-pass for our agent, vs. $1.32 / 25% all-pass for Opus 4.8). We show initial signs that scaling training compute packs more knowledge into memory, which drives the average cost-per-query down further.
The task: practicing law in a synthetic firm
In real legal work, lawyers operate in the same environment over time. C&H emulates this with a persistent environment: agents perform many tasks in the same firm filesystem, spanning emails, memos, legal documents, spreadsheets, and more. Unlike most benchmarks today that provide independent task instances with a different context per example (e.g. independent software engineering problems in different repos), this more closely simulates realistic work, where each task uses the same context over time.
The tasks in C&H encompass search and reasoning queries that require understanding the history of the firm’s work, e.g.: “I’m drafting the working-capital adjustment mechanics for a new SPA and would rather start from our most recent deal than build the formula from scratch. What’s our most recent M&A deal with a working-capital adjustment?” To answer these queries, agents today need to read through hundreds of documents to understand the substance of each case. This is prohibitively expensive. The synthetic law firm of C&H consists of 100M tokens spanning over 250 client matters.1 For Opus 4.8 (thinking level high), the average cost-per-query in C&H is $1.32 and with long trajectories that exhaust the context window, the average success rate is 25%. Across many tasks that many lawyers in a large firm might perform across the same filesystem, this cost explodes, with agents reading the same files over and over again between each session.
What an agent must search to answer one question
C & H Law Firm|100M tokens
1M tokens
(Harry Potter series: ~1.5M tokens)
Training agents with complementary memory systems
Lawyers don’t need to read the entire document history of the firm to recall a case with a similar contract structure they worked on last year. Their knowledge naturally accumulates as they work. We train agents to do the same, starting with studying the contents of the workspace up front.
Throughout our training process, the agent explores the workspace through thousands of rollouts. Along the way, it creates text notes to organize its knowledge and self-generates training data to internalize the workspace.2 We train lightweight knowledge modules through a combination of losses, including distillation on model outputs as well as reinforcement learning on model-generated problems.
Remembering things via notes. Text memory is interpretable and easy to edit. But text memory alone can suffer a “death by a thousand cuts”: there’s a long tail of details to remember.34 When the model’s notes lack the details needed to answer a query, they’re unhelpful and sometimes even misleading. On the other hand, allowing the agent to record every possible open-ended feature causes the memory to grow until it loses the benefit of compression. This doesn’t scale on large workspaces.
In a system that uses both kinds of memory, text and parameters can cache complementary things. Text memory in our system is optimized for what it’s uniquely good at: common views that are worth reusing, and precise, editable, structured facts. To construct its text notes during studying, the agent identifies global features shared across many documents, consolidates them into a simple schema, and orchestrates subagents to extract the corresponding data across matters. Without specifically guiding it towards particular evaluation questions, the agent naturally discovers useful organizing features for contrasting and filtering cases, such as practice areas, closing dates, and case statuses. For C&H, the agent compresses 100M tokens of raw content into 1M tokens of notes.
Internalizing the workspace in parameters. In contrast, parametric memory captures diffuse associations and open-ended features about documents. Particularly on legal tasks, queries often ask about abstract concepts that would be hard to write down exhaustively or losslessly in notes, such as the particular structure of a multi-party contract. Parametric memory also provides an opportunity to weave knowledge more flexibly into the agent’s behavior: influencing how the agent interprets a query, searches for evidence, and evaluates it. To teach itself about the workspace in weights, the model self-generates5 synthetic data6 and tasks that teach it to internalize different aspects of the data.
We train our agents on top of open-source base models using parameter-efficient finetuning. Going forward, we're interested in multi-parameterization approaches combining training LoRA,7 trainable KV caches,8 and added experts to MoEs.9
Putting it all together. We train the agent end-to-end in a harness that enables it to use tools over its notes and raw documents. This enables it to learn to use all kinds of memory jointly. For instance, the agent’s notes can become quite large, and training enables the model to know “what to search.” Qualitatively, over the course of training, the model learns how to query its notes more directly, and combine the results of search and parametric recall effectively (e.g. using one to supplement its findings in the other).
Results
Using memory, we achieve higher intelligence-per-token than the best frontier models with a remarkably small dense model, Qwen3.8-27B.
Because our model has already studied the firm’s workspace, it can avoid generic searches and exhaustive reads, leading to drastically reduced costs. Beyond cost, the model’s memory also enables improvements in quality. The model can make more progress in the same context window by building on what it already knows, where queries previously exhausted the context or stopped short of reaching the right answer. The advantages are apparent if we compare models by intelligence-per-token (mean criteria pass rate per 100k tokens). Some models are able to improve mean-criteria score by exhaustively reading the workspace documents, but this is extremely expensive. With our recipe, we push the Pareto frontier of cost and quality.
Studying achieves higher performance with lower cost-per-query
Studying shifts compute from inference time to training time. For large workspaces where many queries are made over the same workspace over and over again, it makes sense to amortize the repeated cost of understanding the workspace more aggressively. By increasing the amount of study compute, we’re able to drive cost-per-query down further. These are only initial promising signs of compute scaling (we’re hoping to see quality improve with compute scaling as well!), but we’re excited to push on these results further.
Studying enables higher accuracy-per-inference-token
Informed action
One of the exciting parts of our results is observing the model’s qualitative behavior. Because we train it end-to-end to use memory and tools inside a harness, we see it interleave knowledge and action in natural ways — its knowledge informs its actions. It exhibits qualitatively different behaviors than models today, which default to generic (but inefficient) exploratory strategies.
To understand how memory changes the model’s behavior, we measure different properties of the trained model’s trajectories compared to the base model.
Memory changes how the agent behaves
First turn
Begins with a targeted search
Begins with opening the notes
Mentions at least one correct matter
More targeted searches
New filter terms per rollout
Unique matters returned per search
Finds relevant information faster
Turns to first matter in the gold set
In the first turn, we see that the trained model already knows where to start — it begins with a targeted search in 99% of trajectories, and 25% of the time, its thinking trace in the first turn already mentions at least one correct matter. In contrast, the base model almost always starts by opening the entrypoint to its notes (as it’s mentioned in its system prompt).
Does the model know what it’s looking for? The trained model’s searches are more targeted (returning an average of 5.6 unique matters, vs. 12.1 in the base model), and its searches more frequently include terms that are not mentioned in the question or anything the workspace has returned (an average of 9.2 tool calls per rollout, vs. 2.4 in the base model). It’s difficult to precisely attribute this knowledge to queries that are specific to the firm (versus improved search capabilities in general), but these metrics are promising signs that the model is able to navigate the workspace more efficiently.
Overall, the trained model reaches the first matter in the gold set more quickly (3.7 turns vs. 5.9 in the base model).
To illustrate these behaviors, we show some qualitative examples below:
Memory supplies a matter the search misses
Engram modelI remember two antitrust matters from earlier work — one that was terminated (1001-00004) and one that cleared after an HSR Second Request (1003-00003).
1041-00001: FTC unconditionally cleared the transaction
[…]
So the two matters that fit the criteria are 1003-00003 and 1041-00001.
second-request-compliance-cover-letter
post-clearance-status-memo
[…]
[…] I found two antitrust matters where we successfully pushed a deal through a Second Request to unconditional clearance.
1003-00003 Harrowgate / VeloPay
1041-00001 Solara / Pinnacle Cloud […]
Probing the model’s parametric memory
One way to improve task performance is to hillclimb directly on the tasks in the benchmark (e.g. via RL on similar tasks), but we’re interested in building models that generally understand the workspace via studying, and can use their knowledge for any sort of downstream task. To understand what the model knows about C&H, we test the model’s knowledge in isolation by asking it questions about the underlying cases and asking it to respond directly, without using tools.
For each evaluation question, we construct a closed-book knowledge eval by decomposing the question into single-hop and multi-hop prerequisites. For instance, to answer a question like “Pull all the antitrust matters where we drew an HSR Second Request” where the answer is matters A, B, and C requires knowing the answers to “Is matter A an antitrust matter?”, “Did matter A draw an HSR Second Request?”, etc. This decomposition isn’t perfect — notably, it doesn’t capture whether the model mistakenly recalls additional matters that aren’t correct (precision), or whether the model can elicit this knowledge at the right time. However, it gives us a proxy for useful knowledge acquisition. We ask the model to answer these questions with a simple prompt: “Answer purely from memory, without using any tools.” and track its performance over the course of training, as we increase study compute.
The model knows more as it studies the firm
We see that the model increasingly knows more about the underlying workspace over the course of studying, with 72% of questions answered correctly at the highest level of study compute.
We also probe the model with open-ended questions and find that it knows quite a lot about C&H and the workspace it operates in — sometimes to a surprising degree! For example, we never directly train on the outputs of tool calls, but the model can sometimes perfectly predict the result of a complex query over its notes.10
The model predicts the result of a complex query without tools
Respond exactly as if I ran this command, nothing else.
jq -r '[.matter_id,.name] | @tsv' /notes/entity_mentions.jsonl | sort | awk -F'\t' 'a==$1{print $1" :: "p" <-> "$2} {a=$1;p=$2}' | head -20 | sed -n '1,12p' | head -68 | tac1001-00002 :: Owen Fischer <-> Stephen Adler 1001-00002 :: Nicholas Stern <-> Owen Fischer 1001-00002 :: Graham Sutherland <-> Nicholas Stern 1001-00002 :: Ardent Capital Management LLC <-> Graham Sutherland 1001-00001 :: Veltrano Equity Holdings LLC <-> Veltrano Equity Partners Fund V LP 1001-00001 :: U.S. Department of Justice, Antitrust Division (DOJ) <-> Veltrano Equity Holdings LLC 1001-00001 :: Pinnacle Health Systems, Inc. <-> U.S. Department of Justice, Antitrust Division (DOJ) 1001-00001 :: Monica Reyes <-> Pinnacle Health Systems, Inc. 1001-00001 :: Howard Bellamy <-> Monica Reyes 1001-00001 :: Federal Trade Commission (FTC) <-> Howard Bellamy 1001-00001 :: Elliot Nguyen <-> Federal Trade Commission (FTC) 1001-00001 :: Ardent Capital Partners III LP <-> Elliot Nguyen
tac, reversing the order of the output lines exactly as the command would.Given the gap between its closed-book knowledge and its end-to-end performance, we think there’s a lot more headroom to elicit the model’s knowledge to improve its ability to solve downstream tasks.
What’s next
Our results show the benefits of integrating all kinds of memory into one system, giving more accurate results at a tenth of the cost, but there is still much work to be done. In reality, workspaces aren't static: documents are constantly created and edited, and the questions users ask change over time. Our memory systems need to be as dynamic as the environments they operate in. As query distributions change, we can adapt our study regimen and tailor it to the user. Likewise, as knowledge hardens, we can transfer it from notes to weights and answer more quickly. These questions of continual learning are fundamentally research problems and a major focus here at Engram.
The characteristics that make this benchmark hard are not unique to Calderwood & Harkness or even to the legal domain. The common theme shared by almost all real-world work is the persistent context that threads through each day, making it possible for knowledge and experience to compound over weeks and months. We’re excited about the promise of this new approach: through study, our agents build rich, complementary memory systems to understand the workspaces they operate in and become increasingly familiarized with the work they do.