Aethis Labs

Helm: choosing an agent's next tool without generating text

Abstract

When an AI agent decides what to do next, it usually writes out the name of the tool it wants, piece by piece. Helm takes a different route. It reads the task, the calls made so far, and the available tools once, and scores every tool. Because it never writes text, it cannot call a tool that doesn't exist.

§1 The problem

Agents pick tools by writing the name out word by word, the same way they write a sentence. 1 Strictly, token by token. A token is a short piece of text, often part of a word, so a name like get_weather_forecast takes several of them.

Each piece is another trip through the model before the agent can act, so the choice is slow. And because the model is writing freely, it sometimes produces a name that looks right but is not in the toolbox at all.

Writing the name

    Helm

      Sketch. The task from Figure 1, decided two ways. Under writing the name, the name comes out one piece at a time; most generated names are valid, and every other pass shows the rare case where the name is invented. That half is schematic. Under Helm, all seven tools are scored in one pass, using the real output from Figure 1.

      §2 How Helm works

      Helm puts the task, the calls made so far, and each tool's name and description into one prompt, and reads it in a single pass. From that pass it takes one vector that sums up the situation, and one vector for each tool, drawn from the words of that tool's name. It compares the situation with each tool, which gives every tool a score, and sorts the tools by score. 2 The model is Qwen2.5-1.5B-Instruct with a small LoRA adapter and a scoring head. The prob values are a softmax over the tools in one request: a ranking signal, not a calibrated probability.

      Nothing is generated. The only possible answers are the tools you sent.

      Task, calls so far, and tools are read in one pass, producing a situation vector and one vector per tool, which are compared to produce a ranking. task calls so far tools one pass over the prompt situation one per tool compare rank
      Diagram. One read of the prompt gives a vector for the situation and one for each tool. Comparing them gives the scores.

      §3 An example

      Figure 1 is a real request. The agent has already searched for flights, so the useful next step is the weather forecast, and that is what Helm ranks first. 3 Finish, the action for stopping and answering, is added to the list automatically when it is left out, because the model was trained with it present.

      Request

      POST /rank
task          I am flying from San Francisco to Tokyo next
              Friday. Find me a flight and tell me what the
              weather will be like when I land.
calls so far  search_flights  SFO to HND, 2026-10-30
tools         search_flights, get_weather_forecast,
              convert_currency, book_hotel,
              get_exchange_rates, translate_text

      Response

      rank tool prob
      1 get_weather_forecast 0.768
      2 search_flights 0.080
      3 Finish 0.048
      4 get_exchange_rates 0.044
      5 book_hotel 0.044
      6 convert_currency 0.013
      7 translate_text 0.003

      latency 184 ms (Apple Silicon)

      Figure 1. A request to Helm and its ranked response.

      §4 Results

      We compared Helm with the usual approach: the same model, fine-tuned with the same budget, writing out the tool name. Helm never invents a name, decides 2.6 times faster, and misses the right tool from its top five 3.7 times less often. On the very first pick it is 2.6 points behind. 4 ToolBench, Qwen2.5-1.5B, mean of 3 runs. Top-1 accuracy is on par with the baseline; Helm's advantages are reliability and speed.

      Helm Generate the name Difference
      Invented tool names 0% 0.9% none invented
      Time per decision 247 ms 632 ms 2.6x faster
      Correct tool in top 5 97.5% 90.7% +6.8 points
      Correct first pick 64.0% 66.6% 2.6 points lower

      Top 5 misses fall from 9.3% to 2.5%, 3.7 times fewer. Speed is 632 ms divided by 247 ms.

      The first-pick numbers are close: 64.0% against 66.6%, a gap of 2.6 points. The absolute level reflects the small 1.5B model both systems share, so the comparison is like for like. What Helm adds is that its mistakes stay inside the toolbox and close to the top, and that it gets there faster.

      §5 Limits

      1. Helm ranks tool names only, not arguments. Your agent still writes the call.
      2. Each request takes 2 to 10 tools.
      3. It was trained on RapidAPI-style tools, and has not been tested on other kinds of toolsets.

      §6 Access

      Helm is in private preview. To try it, write to yash@aethislabs.com.

      The code is at github.com/aethislabs/aethis-helm.