Yurumi: the 56 seconds nobody was measuring
yurumi·

Yurumi: the 56 seconds nobody was measuring

The right answer arriving too late

Yurumi has a question endpoint: you send a question in plain language, it searches the project’s memory and answers only from what the sources actually say. When it does not know, it answers NOT FOUND. It does not invent.

On Monday I measured that endpoint in production and the number was 56 seconds. Then 53. Before that, 40 and 48. The answer, when it arrived, was frequently NOT FOUND.

The tools I use on this project live in the skills store — the first one is free.

The detail that stung: there is a deterministic mode, no LLM involved, that answers the same question in 1.8s and 6.1s — and it was more accurate. The expensive, slow, heavily layered version was losing to the dumb one. That is not a model problem. It is a time-architecture problem.

What the clock showed

I measured both routes side by side, same question, same database:

  • online, with the LLM in the loop: 40s, 48s, later 56s and 53s
  • deterministic, no LLM: 1.8s and 6.1s, with a correct answer

Four independent causes surfaced. None of them was “the model is slow”. All of them were wasted work — and all of them could be measured in isolation, which is precisely what made the fix possible.

Cause 1: downloading the whole graph to find one needle

The graph relation lookup needed the edges touching a given entity. The implementation downloaded the entire edge collection — 143,624 points, 864 pagination calls, 16.4 seconds per search — and filtered in Python afterwards.

Filtering on the client what the server already knows how to filter. This pattern does not look wrong while the collection is small; it only shows up when the data grows. And when it shows up, it appears as “the endpoint is slow”, never as “this query is wrong”.

The fix moved the filter to the server, exact match on the two fields that matter, with a keyword index built on those fields. The full scan stayed as a fallback, used only when the filter finds nothing.

# before: fetch everything, filter in memory
points = fetch_entire_collection(edges)
related = [p for p in points if entity in (p["source"], p["target"])]

# after: the server filters; the client receives only what matters
filter_ = Filter(must=[FieldCondition(key="source", match=MatchValue(value=entity))])
points, _ = client.scroll(collection, scroll_filter=filter_, limit=20, with_payload=True)

Measured result: 16.4s became 0.09 to 0.37s. That is not an incremental optimisation, it is the difference between a query and a download.

Cause 2: empty success treated as an answer

The LLM provider chain had a wrong implicit contract: if the provider responded, the answer exists. But reasoning models return a null content with the text in the reasoning field, especially when the token ceiling is low.

The code received empty content and treated it as the final answer. That aborted the chain at the first provider — and the endpoint silently fell back to the deterministic path, having burned the whole call time just to discard the result.

The correction is small and obvious once seen: empty content is not success, it is a failure of that provider. Move on to the next one.

Before and after: the call returned an empty string in 1.07s; it now returns a real answer in 2.5s. A second and a half slower, and infinitely more useful.

Cause 3: two LLM calls to confirm the obvious

The reasoning loop spent two LLM calls per iteration — a judge scoring the result and a validator checking the judge. Together: 10.9 seconds. To confirm a result that already had a high score and real word overlap with the question.

A fast path existed and went unused: when the best result already satisfies both criteria, answer directly. The second test question dropped from 93s to 3.5s on that alone.

Worth being precise: this is not removing validation. It is not paying for validation on the case that is already proven. The slow path still exists for low confidence.

Cause 4: three reformulations to land on the same emptiness

When the first iteration returned nothing, the loop fired three LLM-authored query reformulations — roughly 93 seconds — to end up returning the same NOT FOUND as the first attempt.

Three model calls to confirm absence. The fix was to try one reduced query, without the LLM, before any expensive reformulation. The first test question went from 93s to 12s.

The time budget that was only checked where it did not matter

There was a configured time budget — 12 seconds by default. It was checked at the top of each loop iteration. The problem is that the first iteration alone took over 40 seconds: the ceiling was inspected after it had already been blown, so it never cut anything.

A limit checked in the wrong place is decoration. It became an absolute deadline, consulted inside the model call, shrinking each provider’s timeout to whatever remains of the budget. Blown: return the best result already in hand, with the score it has — and never claim to be grounded without a real source.

What changes

Final production measurement: 11.9s and 3.5s, against 56s and 53s before. The search, answer, retrieval and graph suite stayed green — 458 tests.

One decision was deliberately left out: the reranker stays off. Enabling it with the environment’s GPU runtime pushed the same questions to 200s and 156s. That is not a quality gain worth losing two minutes over; it goes to a separate analysis, outside the critical path.

The transferable lesson is not about this project. It is about method: four causes, all independent, all measurable in isolation. Had I treated “the endpoint is slow” as a single problem and touched whatever seemed most likely, I would have swapped the model — and stayed at 50 seconds with a bigger bill.

Latency is a sum, not a symptom. You can only cut what you measure per stage.

Next step

Re-evaluate the reranker outside the critical path — in batch, offline, with its own measurement — and start tracking the endpoint’s p95 continuously. The next slowdown should not be discovered through a complaint.

Related entries

Same project, same ground — the stories connect.

Next step

If this was useful, there is more: I write about the same ground in the other projects, and the store holds the tools I use here.

No spam, one-click unsubscribe. I am also here: Skills store · Portfolio