Here's a question from our test set. A caller rings a multi-trade maintenance company:
"Hi, I had a plumber round last week and there's a problem with what he did, it's still leaking. Who do I contact about that?"
Our AI receptionist does what we built it to do. It searches the company's website for context and hands the closest matches to the language model. The closest match, according to our production embeddings, was a blog post called How to Find a Reliable Plumber.
Which, if you think about it, is about the worst page you could hand an LLM while it's on the phone with one of the company's unhappy customers.
That's one of 39 questions where our search put the wrong page first, in a test we ran at the start of October. This post is about what we tried instead: a model that can't write a word, picking pages off a list. It put the right page first 88% of the time against our production search's 70%. It handed our LLM the wrong page a tenth as often. And, the bit I didn't see coming, it knew when the website didn't have the answer.
How our receptionist reads a website today
Sayora answers the phone for small businesses in the UK: plumbers, estate agents, electricians, bars. Most calls are routine. Bookings, messages, "are you open on Saturday?". The interesting ones are questions about the business itself. Is the valuation free? Do you fit water softeners? Can I bring the dog?
The answers are usually on the business's own website, so today we fetch them the standard way:
For a small site, we paste the homepage (or up to eight pages) straight into the prompt.
For anything bigger, we cut every page into 1,200-character chunks with 200 characters of overlap, and embed them with OpenAI's text-embedding-3-large at 768 dimensions.
On every caller turn, we embed what they said (plus a version with the turn before, for follow-ups) and inject the closest chunks scoring above a cosine similarity of 0.36.
It's the standard RAG recipe, and it works well enough that we shipped it. It also always bugged me.
Websites are already organised
A small business's website isn't a pile of documents. Someone who knows the business has already organised it into pages, and each page is about one thing: boiler servicing, fees, the team, the Christmas menu. Shredding those pages into confetti and trusting cosine similarity to put the meaning back together always felt backwards.
So we tried the obvious alternative:
Keep pages whole, and write each one a short description.
Give the receptionist a tool. When it hits something it doesn't know, it says "let me check that for you" and calls it.
The tool asks a model a multiple-choice question: which of these pages answers what the caller just said? "None of these" is one of the options.
The best page or two go to the LLM. If the answer is "none", the receptionist says it doesn't know and passes the question on.
One lookup from the test, on an estate agent's 74-page site.
The model doing the picking is Jev, a decision model TypeSafe launched on 15 September. It doesn't generate text. You send it some state and some typed questions, and it sends back a choice with a probability for every option. A choice question takes up to 255 options and a request up to 64k tokens. It costs $0.042 per million input tokens, and output is free. On a 100-page site, that's about a twentieth of a cent per lookup.
(New to Jev? I've written a plain-English introduction: What is Jev good for?)
The shape fits the job almost suspiciously well. The options are the pages, the state is what the caller said, and "none of these" is just another option. That last part turned out to matter more than the ranking.
How we tested it
We took four of our customers' websites and removed the pages a page picker should never see. Individual property listings went, because a feed already covers those. So did near-identical "electrician in [town]" landing pages, because a coverage list answers "do you cover X?" better than a choice between 31 copies of the same page. That left:
a bar and restaurant, 22 pages
an estate agent, 74 pages
an electrician, 75 pages
a multi-trade maintenance company, 232 pages
Claude Haiku 4.5 read every page once and wrote a description: the title, the path, a 30-word factual summary and a few questions a caller might ask. About 100 tokens a page.
Separately, Claude Opus 5.5 read the page text (never the descriptions) and wrote questions the way callers actually talk. That gave us 168 questions: 128 the sites answer and 40 they don't. A quarter of the answerable ones also got a follow-up form ("and how much is that?") after two turns of conversation.
Then nine pickers had a go at every question:
Jev: one choice question over every page on the site, plus "none"
Jev with thinner descriptions: title and summary only, and title only
Embeddings shortlist, then Jev: the top 40 pages by embedding, then Jev picks
GPT-6 Luna: the same list, asked for up to three page ids, or none
Embeddings on page descriptions: the same descriptions Jev read, embedded
Embeddings on chunks: our production retrieval, exactly as it runs today
BM25: keyword search, on the descriptions and on the page text
Judging was pooled. For each question, every page that any picker ranked in its top three went to a judge (Claude Opus 5.5), which read the actual page text and decided whether a receptionist could answer from it. That's 1,668 judgements. No picker was graded against pages another picker chose. When the judge found an answer to a question we'd written as unanswerable, or nothing for one we'd written as answerable, the question switched sides. That happened six times.
A picker "hands over" its first two pages, unless it says none. The intervals below are a paired bootstrap over questions.
The result: 88% against 70%
Share of the 128 answerable questions where the picker's first page was one the judge said answers it.
Jev put a right page first on 88% of the answerable questions, and in its top three on 95%. Our production search managed 70%. That's 18 points, with a 95% interval of 11 to 26. Or, if you prefer headline maths, 26% better.
The fairer fight is embeddings over the exact descriptions Jev read. That narrows the gap to 14 points (7 to 21), still well clear of zero. Keyword search came last, at 53% to 57%.
Jev's lead over each alternative on the same 128 questions. Intervals that cross zero mean no clear difference.
The part that surprised us: knowing when to stop
Ranking is only half the job. On a phone call, the worse mistake is handing the LLM a page that doesn't answer the question, because a fluent model will often build an answer out of it anyway. That's how a receptionist ends up quoting a guide's average prices as if they were the business's own.
A wrong page handed over means the picker returned pages and neither of the first two answers the question.
At our production cut-off of 0.36, chunk search handed over a wrong page on 33% of all questions. It said the website didn't have the answer for just 9 of the 40 questions the sites can't answer. Jev handed over a wrong page 3% of the time and said "not on the website" for 38 of the 40.
The obvious objection is that we should just raise the cut-off. So I swept it. Even tuning it on these same questions, which flatters it, the best chunk search managed 73% answered right end to end, and still handed over a wrong page 15% of the time. Jev, with nothing tuned, managed 90% and 3%. Jev's "none" isn't a threshold on a similarity score. It's an option the model weighs against every page, and it shows.
That "none" is useful for more than safety, too. Every time it fires, we've found something callers ask that the website doesn't cover, which is exactly what a business owner wants to hear about.
Why embeddings lose on websites
I read every question where chunk search put a wrong page first. Of the 39, 23 went to a generic guide or blog post. "Rewiring a house in the UK" for "will I get a certificate?". "How to find a reliable plumber" for the leak complaint. A long advice article shares more words with a caller's question than the business's own short page does, so it wins the cosine contest.
Jev reads the list more like a person would. When it was wrong, it was usually wrong between near twins ("Certified electrician" instead of "Part P certificate"), and only one of its 16 wrong first pages was a guide.
Six questions from the test. Questions and page names are lightly edited to keep the businesses anonymous.
The last receipt is Jev's real weakness. It reads descriptions literally. The plumbing installation page does mention water softeners, but its description didn't, so Jev said "none" and meant it. Chunk search found the page because it reads the page text. Key cutting on a fees page and EV chargers on an electrical installation page failed the same way. The fix is on our side: descriptions have to name the specific things callers ask about.
Couldn't an LLM just pick?
Fair question. It's the first one I got asked. We gave GPT-6 Luna the same list and asked for up to three page ids, or none. OpenAI's new Decisions API is reportedly built on a tuned GPT-6 Luna. We don't have access to it yet, so this stands in for its judgement, not its speed.
Luna tied Jev on picking: 86% right first against Jev's 88%, a difference of 2 points with an interval of −4 to +7. But it was slower, at 0.66 s median and 1.27 s at p95 (reasoning effort low), against Jev's 0.28 s and 0.42 s. It was also less willing to say "none". It caught 80% of the unanswerable questions and handed over a wrong page 8% of the time.
When Luna missed, it reached for something plausible. Asked about a kids' menu the bar doesn't have, it handed over the food menu, with full confidence. On a phone call, that's the receptionist reading the adult menu out to a parent and implying there's something for the kids.
Embeddings still earn their keep
Some of the best numbers in the whole test came from using both. Embeddings shortlist 40 pages, then Jev picks among them: 88% right first, in 0.25 s. On the 232-page site, a shortlist of 100 got 95% right first and never handed over a wrong page.
Embeddings are good at "somewhere around here". Jev is good at "this one". You don't have to choose.
How many pages can it take?
A choice question tops out at 255 options, so we checked whether that bites. Of the 449 business websites in our customer records, 353 publish a sitemap. The median has 14 pages, and 95% have 255 or fewer. Small businesses, it turns out, have small websites. Who knew.
Then we pushed it. For each question we built option lists of 16 up to 232 pages, filling them with the most similar pages first (from the same section of the site), so every extra option was a plausible distractor.
Each list holds the right pages plus lookalikes from the same section. Jev picked from the list; embeddings ranked the same list.
Accuracy drifts down as the list fills with lookalikes. On the biggest site, Jev went from 100% at 16 options to 92% at 232, and the right page stayed in its top three every time. Embeddings over the same lists fell from 89% to 74%. The electrician's site was the hardest for both, with dozens of near-duplicate service pages.
Median time per Jev request against its input tokens, from a laptop in the UK.
Time follows tokens, not options or questions. A 16-page list (about 2,000 tokens) took 0.23 s and a 232-page list (24,000 tokens) took 0.43 s. Across everything we measured, latency fits roughly 0.23 s plus 7 ms per thousand input tokens.
Past 254 pages you need more than one question, so we tried that on a site I know very well: The Podcast Host, with 1,253 pages. We packed a section question plus one page question per section into a single request: 553 described pages and 63,000 tokens, just under the cap, answered in 0.63 s. The extra questions cost no extra time. Only the tokens did.
The descriptions do the heavy lifting
Answered right end to end: a right page handed over, or "none" when the site doesn't say.
Jev only knows what the description tells it. Page titles alone cost 9 points on the right page first and 16 points end to end. Title plus a one-line summary did as well as the full description with example questions, so the example questions can go. Writing the summaries cost about $0.002 a page with Claude Haiku, once per page and again only when a page changes.
Follow-ups come free
On the phone, people don't repeat themselves. "And how much is that?" only makes sense with the two turns before it. We gave Jev those turns as state and the caller's last words as the question. It put the right page first 91% of the time, exactly the same as when we handed it a cleanly rewritten question. Our production search, which already widens its query for follow-ups, dropped from 78% to 72%.
So the tool can take the caller's actual words. Nothing needs rewriting before it's called.
The hard part was the crawler
Picking the page was the easy bit. Reading real small-business websites broke something different almost every time.
Two sites didn't make it into the test at all:
A used car dealer's host answers honest crawlers with an empty "202 Accepted" on every inner page. Our production scanner gets the same, which is why that customer's receptionist only knows their homepage. We don't dress our crawler up as a browser to get past bot protection, so the way in is the owner allow-listing us.
An electrician's site was made with an AI site builder that draws everything in the browser. A plain fetch gets an empty shell.
And two that did:
The estate agent's page-builder site fooled our main-content extractor, which returned the cookie banner as the content of the valuation page, the team page and every "selling in [town]" page. Falling back to the page body minus every line repeated across the site recovered 49 of its 91 pages.
The Podcast Host's CDN started answering 429 after a few minutes of six parallel requests. At one request every 1.2 seconds (its robots.txt asks search engines for one a second), all 1,253 pages came through clean in about 25 minutes.
What we're not claiming
The questions aren't real calls. A model wrote them from page text. They sound like callers, but real call transcripts are the next test.
One model judged relevance. We checked five of its verdicts by hand where Jev and chunk search disagreed, and it was right all five times. The rest are unchecked.
The sample is small. About 35 questions per site, so a single site's numbers can swing 15 points either way. Trust the pooled numbers and their intervals.
One run, one place. Every picker ran once, from a laptop in the UK.
Our chunk search isn't the best embedding setup possible. A reranker, hybrid search or smarter chunking would close some of the gap. We tested what we actually run, plus embeddings over the same descriptions Jev read.
Jev is three weeks old. If we ship this, the embeddings shortlist stays in as the fallback.
The big site is half done. We wanted The Podcast Host's 1,253 pages in the accuracy table. We ran out of API credit halfway through writing its page descriptions. Classic.
What's next
Real caller questions from transcripts, then a test on live calls. If it holds up, our receptionist reads a business's whole website instead of its homepage, the owner chooses which parts it can use, and every "not on the website" becomes a question for the owner to answer. We'd also like to put OpenAI's Decisions API through the same test once we can get in.
If you've tried decision models for retrieval, or you think we've missed something obvious, I'd love to hear about it.
The numbers in full
| Picker | Right page first | Top 3 | Answered right | Wrong page handed over | Says "not on site" | p50 | p95 |
|---|---|---|---|---|---|---|---|
| Jev, every page | 88% | 95% | 90% | 3% | 95% | 0.28 s | 0.42 s |
| Jev, title and summary | 86% | 95% | 88% | 4% | 95% | 0.27 s | 0.38 s |
| Jev, title only | 78% | 90% | 74% | 7% | 88% | 0.25 s | 0.33 s |
| Embeddings shortlist 40, then Jev | 88% | 93% | 89% | 5% | 93% | 0.25 s | 0.36 s |
| GPT-6 Luna picks from the list | 86% | 92% | 88% | 8% | 80% | 0.66 s | 1.27 s |
| Embeddings on descriptions | 73% | 87% | 69% | 27% | 30% | 0.16 s | 0.23 s |
| Embeddings on chunks (production) | 70% | 88% | 65% | 33% | 23% | 0.16 s | 0.23 s |
| BM25 on descriptions | 57% | 77% | – | – | – | <1 ms | <1 ms |
| BM25 on page text | 53% | 72% | – | – | – | <1 ms | <1 ms |
"Right page first" and "Top 3" are out of the 128 answerable questions. "Answered right" covers all 168: a right page among the two handed over, or "none" when the site doesn't say. "Says not on site" is out of the 40 the sites don't answer. The embedding latencies are one query embedding plus an in-memory search.
| Method | Detail |
|---|---|
| Websites | Four small businesses, 22 to 232 pages after removing listings and town landing pages. We've left them unnamed. |
| Questions | 168: 128 the sites answer, 40 they don't. 32 also asked as a follow-up after two turns. |
| Descriptions | Claude Haiku 4.5, one per page, about 100 tokens: title, path, a 30-word summary, three example questions. |
| Judging | Claude Opus 5.5 read the page text of every page any picker ranked in its top three: 1,668 judgements. |
| Jev | jev-1.13.0. One choice question over the pages plus "none", with the caller's words as state. |
| Embeddings | text-embedding-3-large at 768 dimensions. Production chunks are 1,200 characters with 200 of overlap; the cut-off is 0.36. |
| GPT-6 Luna | Through the chat API on Azure, reasoning effort low, asked for up to three page ids or none. |
| Handed over | The first two pages a picker returns, unless it says none. |
| Intervals | Paired bootstrap over questions, 4,000 resamples. |
| Cost | 1,449 Jev requests, 10.6 million input tokens: about $0.45. |
