An LLM that can’t invent a quote

ActionPoints takes a meeting transcript and turns it into action points you can push to Linear. A language model (Claude) reads the transcript and proposes the action points; a person reviews them; the accepted ones become tasks. I built the first version solo in a twelve-hour sprint in July 2026 and refined it over the following week. It’s Elixir and Phoenix LiveView on PostgreSQL, deployed to Fly.io, and the source is public.

It was my first real foray into building on an LLM. The lesson I took away is simple to say: anything the product promises has to be enforced by code, because a prompt is a request, not a guarantee.

Quotes are checked, not trusted

The product’s value is trust. Each action point shows the quote from the meeting that supports it. And a model asked to pull out action points will, now and again, produce a confident quote that nobody said.

You can tell the model not to do that. It will mostly listen. “Mostly” is not good enough when one invented quote is enough for people to stop trusting the whole product.

So a proposed quote is kept only if it appears word for word in the transcript, after normalising whitespace. Anything else is thrown away before it reaches the screen.

It’s worth being precise about what that promises: no quote shown to a user is invented. It doesn’t promise the model picked a good quote. But it’s a guarantee a test can enforce, which a prompt never is.

The model classifies, the code calculates

The first version asked the model for due dates as ISO dates, resolving things like “by Wednesday” against the meeting. The headline case, “have it ready for review by Wednesday”, came back with no due date at all. Calendar arithmetic is exactly what language models are unreliable at.

So the job was split along what each side is good at. The model reports what kind of timing it heard: an absolute date, a weekday, a relative day, the end of a span, a duration, or something vague. Plain, tested application code turns that into a date, using the meeting date.

Some useful consequences fell out of that:

• “Vague” has nowhere to put a date, so the model can’t produce a date for “soon” even if it wanted to.
• The prompt never contains today’s date, so it’s static. It can be cached, and test fixtures give the same answer whenever they run.
• Every date rule is ordinary code with unit tests. Said on Saturday 31 October, “end of the week” resolves to Friday 6 November, and “end of the month” deliberately resolves to nothing, because that month’s working days are already over.

A person decides on every item

Nothing goes to Linear without review. Pushing automatically was rejected early: the model will sometimes turn “we should probably look into that” into a task, and unfiltered output landing in a colleague’s workspace reads as spam.

Review started as a list of cards with everything accepted by default. That turned out to be the wrong default: eight tasks could be created in someone’s workspace having been read by nobody. Review became a walk through one action point at a time, with a new “undecided” state that can’t be pushed. The order follows the dependencies the model found: blockers before the tasks they block, and subtasks before their parent. You make the hardest call last, when you know the most.

Review can correct what the model extracted, such as a title, a date or who it’s for. It can’t invent structure the meeting didn’t contain. That keeps the tool honest about what it is: a record of the meeting, not a planning tool.

Test the model separately, and expect it to wobble

The ordinary test suite never touches the network. Every external service (the model, Linear, Stripe) sits behind an interface with a fake used in tests, so the suite of 539 tests is fast and deterministic.

But the prompt is exactly the thing a refactor can quietly break, so there’s a separate evaluation suite: thirteen fixture transcripts with known answers, run against the real API. It’s excluded from the normal test run, so routine runs never spend money, and it’s run deliberately whenever the prompt changes.

The first version called the model once per fixture, and a clean run failed somewhere about half the time. That’s how you train people to ignore a test. Now each fixture is sampled three times and has to be right in the majority. A real regression is wrong in nearly every sample, so it still fails.

What I’d take into the next one

1. Write down what the product promises, then ask which of those promises the code can enforce.
2. Give the model the judgement calls and give the code the arithmetic.
3. Keep a person in the loop wherever the output lands somewhere that matters.
4. Test the model like the unreliable component it is: separately, deliberately, and with enough samples to tell a regression from a bad day.

The decision records in the repo go into more detail on each of these.