KEPLIN Docs

Tools and batches

Letting the model use the script's functions with ai.run, and processing many requests in the background with ai.batch.

Two operations for work bigger than a single question: ai.run lets the model use the script's functions while it answers, and ai.batch processes many requests in the background.

Tools (ai.run)

With ai.run, the model can ask for the script's functions while it answers: query a database, calculate a value or call an API. The SDK runs the function in the script, returns the result to the model and repeats until the model gives the final answer.

from api_manager import ai, db


def stock(produto: str) -> int:
    """Unidades em armazém de um produto, pelo nome."""
    linha = db("Dados CRM").query_one("select quantidade from stock where nome = ?", [produto])
    return linha["quantidade"] if linha else 0


def preco(produto: str) -> float:
    """Preço unitário de um produto, em euros."""
    return db("Dados CRM").query_one("select preco from produtos where nome = ?", [produto])["preco"]


def main(input):
    r = ai.run(input["args"]["pergunta"], tools=[stock, preco])
    return {"resposta": r["text"], "passos": r["steps"]}
  • A tool is a Python function. Its name, docstring and the type annotations of its arguments are what the model reads to decide when to use it, so describe the function well in the docstring.
  • The types that become JSON Schema are str, int, float, bool, list and dict. An argument with a default value is optional.
  • To control the arguments' schema, pass a dict instead of the function: {"name": "stock", "description": "...", "parameters": {...}, "function": stock}, with parameters in JSON Schema.
  • The result is the one from ai.chat, plus steps: the list of tools that ran, with the arguments and the result of each one.
  • An exception in a tool does not stop the task: it goes to the model as text («Erro: …»), and the model decides what to do.
  • max_steps (10 by default) is the maximum number of requests to the model. If the model keeps asking for tools after that, ai.run raises ai.Error with the code max_steps.

Warning

The model chooses the arguments. A tool that changes data (deleting, sending an email, saving) runs with whatever the model decides: validate the arguments inside the function, and do not give the model tools that cannot run unsupervised.

ai.run takes the same parameters as ai.chat: system, messages, model, max_tokens, temperature and files.

Batches (ai.batch)

A batch processes many text requests in the background. The provider has up to 24 hours to answer. It suits work that is not urgent: summarising the week's support requests, classifying thousands of records or translating a catalogue.

A batch happens in two steps. One script submits the requests and keeps the id:

from api_manager import ai, db


def main(input):
    crm = db("Dados CRM")
    pedidos = crm.query("select id, texto from pedidos where resumo is null")
    lote = ai.batch.submit(
        [{"id": str(p["id"]), "prompt": "Resume numa frase: " + p["texto"]} for p in pedidos],
        model="texto",
    )
    crm.query("insert into lotes (id, estado) values (?, 'em curso')", [lote["id"]])
    return lote  # {"id": "lote-...", "status": "in_progress", "count": 120}

Later, another script (for example, scheduled every hour) checks the status and reads the results:

from api_manager import ai, db


def main(input):
    crm = db("Dados CRM")
    for linha in crm.query("select id from lotes where estado = 'em curso'"):
        estado = ai.batch.get(linha["id"])
        if estado["status"] == "in_progress":
            continue
        if estado["status"] == "ended":
            for r in ai.batch.results(linha["id"]):
                if r["ok"]:
                    crm.query("update pedidos set resumo = ? where id = ?", [r["text"], int(r["id"])])
        crm.query("update lotes set estado = ? where id = ?", [estado["status"], linha["id"]])
    return {"ok": True}
Function What it does
ai.batch.submit(pedidos, model=None, system=None, max_tokens=None, temperature=None, schema=None) Submits the batch. Each request is a text, or a dict with id, prompt (or messages) and, optionally, system. Returns id, status and count.
ai.batch.get(id) The status (in_progress, ended, failed, cancelled or expired) and the counts (total, succeeded, failed and processing).
ai.batch.results(id) The results of a finished batch: a list with id, ok, text, data (with schema), error and usage.
ai.batch.cancel(id) Cancels the batch and returns the status.
  • A batch has up to 10 000 requests. The id values use letters, digits, hyphens and underscores, up to 64 characters, without repeating. Without id, the requests are numbered from 1.
  • With schema, each result is validated as in ai.json: the ones that do not match the schema come with ok false and the reason in error.
  • OpenAI, Anthropic and Google through the Gemini API do batches. With other providers, ai.batch.submit gives the not_supported error.
  • A batch id only works in the app that submitted it.

Frequently asked questions

Can a tool call an external service? Yes. A tool is a Python function like any other, and it can use http, db or workflow.

How long does a batch take? It depends on the provider and its load, and it can take up to 24 hours. A batch that does not finish within that time ends as expired.

Can I use tools in a batch? No. A batch makes text and JSON requests; for tools, use ai.run.