Tools and batches
Letting the model use the script's functions with ai.run, and processing many requests in the background with ai.batch.
Two operations for work bigger than a single question: ai.run lets the
model use the script's functions while it answers, and ai.batch processes
many requests in the background.
Tools (ai.run)
With ai.run, the model can ask for the script's functions while it answers:
query a database, calculate a value or call an API. The SDK runs the function
in the script, returns the result to the model and repeats until the model
gives the final answer.
from api_manager import ai, db
def stock(produto: str) -> int:
"""Unidades em armazém de um produto, pelo nome."""
linha = db("Dados CRM").query_one("select quantidade from stock where nome = ?", [produto])
return linha["quantidade"] if linha else 0
def preco(produto: str) -> float:
"""Preço unitário de um produto, em euros."""
return db("Dados CRM").query_one("select preco from produtos where nome = ?", [produto])["preco"]
def main(input):
r = ai.run(input["args"]["pergunta"], tools=[stock, preco])
return {"resposta": r["text"], "passos": r["steps"]}
- A tool is a Python function. Its name, docstring and the type annotations of its arguments are what the model reads to decide when to use it, so describe the function well in the docstring.
- The types that become JSON Schema are
str,int,float,bool,listanddict. An argument with a default value is optional. - To control the arguments' schema, pass a dict instead of the function:
{"name": "stock", "description": "...", "parameters": {...}, "function": stock}, withparametersin JSON Schema. - The result is the one from
ai.chat, plussteps: the list of tools that ran, with the arguments and the result of each one. - An exception in a tool does not stop the task: it goes to the model as text («Erro: …»), and the model decides what to do.
max_steps(10 by default) is the maximum number of requests to the model. If the model keeps asking for tools after that,ai.runraisesai.Errorwith the codemax_steps.
Warning
The model chooses the arguments. A tool that changes data (deleting, sending an email, saving) runs with whatever the model decides: validate the arguments inside the function, and do not give the model tools that cannot run unsupervised.
ai.run takes the same parameters as ai.chat: system, messages,
model, max_tokens, temperature and files.
Batches (ai.batch)
A batch processes many text requests in the background. The provider has up to 24 hours to answer. It suits work that is not urgent: summarising the week's support requests, classifying thousands of records or translating a catalogue.
A batch happens in two steps. One script submits the requests and keeps the id:
from api_manager import ai, db
def main(input):
crm = db("Dados CRM")
pedidos = crm.query("select id, texto from pedidos where resumo is null")
lote = ai.batch.submit(
[{"id": str(p["id"]), "prompt": "Resume numa frase: " + p["texto"]} for p in pedidos],
model="texto",
)
crm.query("insert into lotes (id, estado) values (?, 'em curso')", [lote["id"]])
return lote # {"id": "lote-...", "status": "in_progress", "count": 120}
Later, another script (for example, scheduled every hour) checks the status and reads the results:
from api_manager import ai, db
def main(input):
crm = db("Dados CRM")
for linha in crm.query("select id from lotes where estado = 'em curso'"):
estado = ai.batch.get(linha["id"])
if estado["status"] == "in_progress":
continue
if estado["status"] == "ended":
for r in ai.batch.results(linha["id"]):
if r["ok"]:
crm.query("update pedidos set resumo = ? where id = ?", [r["text"], int(r["id"])])
crm.query("update lotes set estado = ? where id = ?", [estado["status"], linha["id"]])
return {"ok": True}
| Function | What it does |
|---|---|
ai.batch.submit(pedidos, model=None, system=None, max_tokens=None, temperature=None, schema=None) |
Submits the batch. Each request is a text, or a dict with id, prompt (or messages) and, optionally, system. Returns id, status and count. |
ai.batch.get(id) |
The status (in_progress, ended, failed, cancelled or expired) and the counts (total, succeeded, failed and processing). |
ai.batch.results(id) |
The results of a finished batch: a list with id, ok, text, data (with schema), error and usage. |
ai.batch.cancel(id) |
Cancels the batch and returns the status. |
- A batch has up to 10 000 requests. The
idvalues use letters, digits, hyphens and underscores, up to 64 characters, without repeating. Withoutid, the requests are numbered from 1. - With
schema, each result is validated as inai.json: the ones that do not match the schema come withokfalse and the reason inerror. - OpenAI, Anthropic and Google through the Gemini API do batches. With other
providers,
ai.batch.submitgives thenot_supportederror. - A batch id only works in the app that submitted it.
Frequently asked questions
Can a tool call an external service? Yes. A tool is a Python function
like any other, and it can use http, db or workflow.
How long does a batch take? It depends on the provider and its load, and
it can take up to 24 hours. A batch that does not finish within that time
ends as expired.
Can I use tools in a batch? No. A batch makes text and JSON requests; for
tools, use ai.run.