Contributing legal tasks

We are collecting realistic legal work tasks written by practising lawyers. Each task is a piece of work a junior lawyer might be given by a partner or client: a set of matter documents, a short instruction, a model answer, and a marking scheme. Frontier AI models attempt every task, and an automated grader marks each answer against the marking scheme. We accept tasks that a careful lawyer can answer well but strong AI models still get materially wrong.

The sample in albion-house-lease-break-sample.zip is a complete task in the required format: an English-law commercial lease break. Unzip it to see every part.

What a task contains

your-task-name/
  instruction.md      the short instruction the lawyer receives
  documents/          the matter documents (emails, agreements, letters, spreadsheets, notes)
  reference/          your model answer, one file per requested output
  task.json           title, practice area, jurisdiction, outputs and the marking criteria

Zip the folder and upload the zip on the submission site.

instruction.md

A short, natural instruction of the kind a partner or client actually gives. The detailed brief usually belongs in an email inside documents/, as it would in practice. Name the output file the lawyer should save, for example "Save your advice as Northmere_break_advice.md".

documents/

5 to 30 documents in their native formats: .eml, .docx, .pdf, .xlsx or .pptx. Every file must open.
Realistic file names, as they would appear in a matter file: Lease - Albion House (executed) 25.04.2022.pdf, not doc_03.pdf.
Include the legal materials needed to answer: short extracts or summaries of the relevant statutes, rules or cases, in your own words with brief quotations. The task must be answerable from the documents alone, without outside research.
Everything must be fictional. Use invented people, companies and addresses, and no real client information.

reference/

The answer you would expect from a strong senior associate. It must satisfy at least 95% of your own criteria. Write it as real work product: for example an advice memo with a heading, a summary of advice, the analysis with clause and statutory references, and next steps with dates.

task.json

Copy the structure from the sample. The fields are:

Field	Content
`task_id`	A short lowercase name, such as `kl-commercial-lease-renewal`
`title`	A one-line title
`practice_area`	For example "Corporate", "Real estate", "Employment", "Construction", "Disputes"
`jurisdiction`	For example "Malaysia" or "England and Wales"
`task_type`	`analyze`, `draft`, `review` or `research`
`instructions`	The same text as `instruction.md`
`outputs`	Each requested output, with a `name` (the file name without extension) and a one-line `description`
`reference_files`	The file names in `documents/`
`criteria`	The marking scheme (below)

Writing the criteria

Each criterion is one pass or fail check on the answer. Aim for 40 to 70 criteria.

{
  "id": "C07",
  "rule": {
    "question": "Does the advice state that the break notice must be served on the current landlord, Harrowgate Estates Ltd, at the address in its letter of 3 September 2026, because clause 23.2 requires service at the landlord's last notified address?",
    "pass": "States that service must be on Harrowgate at the notified address, or equivalent. Equivalent wording and reasoning are acceptable.",
    "fail": "Advises service on the original landlord or at the address in the lease, or does not address where the notice must be served."
  },
  "deliverable": ["Northmere_break_advice.md"]
}

Rules that every criterion must follow:

One point per criterion. Test a single fact, conclusion, step or drafting requirement.
Objective. Two lawyers would agree on the result. Do not test style, tone or overall quality.
Legally correct and supported by the documents. Cite the clause, section or document the point rests on.
Fair. Accept equivalent wording and any other reasonable correct approach. Do not fail an answer for a small slip or because a point appears in a different place.
Positive. Write each criterion as something the answer should do.
No overlap. Two criteria must not test the same point.

What makes a good task

The tasks we value most are the ones where strong AI models make the mistakes a careful lawyer would not. In our experience these come from:

Several instruments read together. A later variation, side letter or consent changes the effect of the main agreement, and the answer differs item by item.
Exact deadlines and amounts, where the documents contain a tempting but wrong trigger date or figure.
Consequences the instruction does not mention, such as an adjacent statutory regime, what survives termination, set-off, interest, tax or insolvency effects.
Limits on authority. The client says what may and may not be done today, and the answer must still be useful within those limits.
A plausible but wrong position taken by the other side or by the client's own team, which the answer must deal with correctly.
A draft to correct. An unagreed working draft of a notice, letter or agreement that must be turned into clean, correct operative text.

Avoid making a task hard through ambiguity, missing facts or trick wording. A qualified lawyer must agree that every criterion is fair.

How submissions are assessed

After you upload, the site:

checks that your reference answer passes at least 95% of your criteria;
checks that an empty answer passes almost none;
has several frontier AI models attempt the task, and grades each attempt against your criteria.

You will see your reference score and how the models scored. A task is accepted when it is fair, realistic and still hard for the models. We may suggest revisions to criteria that need clarifying.

Originality and confidentiality

Submit only your own original work. Do not copy published precedents, textbooks or other copyrighted material beyond brief quotations, and do not include any real client or personal data. By submitting, you confirm that the task is original and fictional, and that you may license it to us.

Back to the upload form · README.md