LLMs

Persistent application agents can use Agent(llm_factory=...) to resolve a provider lazily for each request. The factory reads frozen settings through current_request_context() and returns an LLM or model specification; native tool calling, streaming and session history use the normal inference loop. See private request context and model selection.

ProtoLink's LLM package starts with a simple promise: choose where a model runs, then use the same application-facing contract to talk to it. That contract can stop after one text response, stream text as it arrives, or continue into a controlled Agent loop where the model requests tools and delegates work while ProtoLink remains responsible for execution.

from protolink import Agent, create_llm

model = create_llm("mock", default_response="Hello")
agent = Agent(name="helper", llm=model)
print(agent.sync.invoke("Say hello"))  # Hello

The mock backend needs no credentials or server. Replace it with a provider alias or configured adapter through llm=. The adapter handles model communication; the Agent owns tool execution, policy and task lifecycle.

The useful mental model is a progression:

1. Choose a backend: hosted API, model server, in-process local model, or deterministic mock.
2. Create one adapter with create_llm() or a concrete provider class.
3. Pick the interaction level: chat() for one direct response, infer() for controlled multi-step work, or the lower-level call*() methods when implementing an adapter.
4. Let an Agent own runtime concerns such as tool selection, delegation, policy, cancellation, state and events.
5. Add operational features when needed: persistent history, explicit compaction, context reporting, cost estimates and run budgets.

Model runtime moduleLLM

Choose a model backend, make direct or streamed calls and graduate to a controlled inference loop without rewriting the rest of the application.

protolink.llms

Provider-neutralDirect and streamed textTool-aware inferenceAgent integrationOptional operations

ChooseUse a hosted API, your own model server, an in-process GGUF model, or a deterministic mock.create_llm()

ConverseAsk for one complete response or consume text chunks as they arrive.LLM.chat()

ActRun the typed action loop used by Agent for tools and delegation.LLM.infer()

OperateManage history and add compaction, metrics, context visibility and budgets when the application needs them.LLM.compact_history()

Start with the interaction you need

Most application code should begin with either chat() or an Agent. The other methods exist so provider adapters and advanced integrations can participate in the same runtime.

• One complete response - llm.chat("...")
  Adds the user message and performs one synchronous provider call.

• Visible incremental text - llm.chat("...", streaming=True)
  Returns an asynchronous iterator of text chunks.

• Tools, delegation, policy, retries, budgets and multiple steps - Agent(..., llm=llm)
  The Agent prepares the runtime and invokes LLM.infer() with the appropriate services.

• A custom controlled runtime - await llm.infer(...)
  Advanced integration path where the caller prepares the prompt, history, tools, callbacks and runtime context that an Agent normally supplies.

• A provider adapter or diagnostic integration - call(), call_stream() and call_action()
  Works at the provider boundary with explicit ConversationHistory and normalized actions.

chat() and infer() are intentionally different. A chat call asks the model for text. Inference asks the model for one typed decision at a time: finish with text, call a local tool, or delegate to another Agent. ProtoLink validates that decision and performs the side effect outside the model.

The common application path

Use chat() for a small standalone model interaction. Use an Agent when model output can cause work to happen. Reach for call() and call_stream() primarily when extending or diagnosing a provider adapter.

Provider-neutral scope

The current common contract is text-oriented: chat inputs, canonical history content, complete responses and stream chunks are strings. Provider support does not imply that every provider-specific feature is wrapped. Use the provider SDK directly for capabilities outside this surface, such as file-upload APIs, image or audio generation, embeddings, fine-tuning, batch administration, or model management.

Choose where the model runs

ProtoLink groups model backends into hosted APIs, model servers and in-process local runtimes. A deterministic testing adapter follows the same contract without making a model request.

[ API ] [ Server ] [ Local ]

• API - calls a remote provider and normally requires an API key:

  • OpenAILLM: OpenAI Responses API, including native function tools and streamed tool-call events.
  • AnthropicLLM: Anthropic Messages API, including native tool_use blocks.
  • GeminiLLM: Google GenAI API, including native function declarations.
  • DeepSeekLLM: DeepSeek Chat Completions API, with optional native tools.
  • GrokLLM: xAI Chat Completions API, with optional native tools.
  • HuggingFaceLLM: Hugging Face Inference API for direct and streamed text, with JSON-action inference.

• Server - connects to a model server that you run locally or remotely:

  • OllamaLLM: connects to an Ollama /api/chat endpoint.
  • LlamaCPPServerLLM: connects directly to a llama-server.
  • LMStudioLLM: connects to LM Studio's OpenAI-compatible server.
  • VLLMLLM: connects to vLLM's OpenAI-compatible server.
  • OpenAICompatibleLLM: connects to a service exposing /v1/chat/completions and /v1/models.

• Local - runs the model inside the Python process:

  • LlamaCPPLocalLLM: loads a local GGUF file through llama-cpp-python.

For deterministic tests and offline examples, MockLLM can return fixed responses, response sequences, or callback-generated responses. It is the fastest way to test Agent behavior without credentials, network access, or nondeterministic model output.

You can also use a third-party LLM client directly when your application only needs that client's raw API. ProtoLink's wrappers become valuable when you want provider-neutral history, streaming, typed actions, Agent integration, compaction, events, metrics, or budgets.

How to choose

• Choose a hosted API when you want a managed model and accept provider credentials, network latency and usage billing.
• Choose a server adapter when the model is exposed by Ollama, llama-server, LM Studio, vLLM, or another OpenAI-compatible endpoint that you control.
• Choose LlamaCPPLocalLLM when the GGUF model should run inside the Python process and the process can afford model-loading and inference resources.
• Choose MockLLM for tests, examples and runtime development.

Native tool support is an adapter-and-model capability, not a requirement for ProtoLink inference. OpenAI, Anthropic and Gemini use provider-native tool structures. DeepSeek and Grok enable native tools by default but can fall back to the portable JSON action protocol. Self-hosted and local adapters use that JSON path by default unless a known-compatible model/server is explicitly opted into native tool calling.

Install, configure and make the first call

Configuration varies by backend, but the first successful model interaction follows one continuous path.

1. Install the relevant extras:

   # Choose your provider
   uv add "protolink[openai]"

Choosing LLM extras

Each provider has a focused extra. Combine it with a transport as needed, for example uv add "protolink[http,openai]". The existing protolink[llms] bundle still installs every integration, including local llama.cpp bindings. Core ProtoLink requires only Pydantic; mock needs no extra.

Provider alias	Installation extra	Direct dependency
openai	protolink[openai]	openai
anthropic	protolink[anthropic]	anthropic
gemini	protolink[gemini]	google-genai
huggingface	protolink[huggingface]	huggingface-hub
deepseek	protolink[deepseek]	openai
grok	protolink[grok]	httpx2
ollama	protolink[ollama]	httpx2
openai-compatible	protolink[openai-compatible]	httpx2
lmstudio	protolink[lmstudio]	httpx2
vllm	protolink[vllm]	httpx2
llama.cpp-server	protolink[llama-cpp-server]	httpx2
llama.cpp-local	protolink[llama-cpp-local]	llama-cpp-python

2. Construct the LLM through the lazy factory:

   from protolink import create_llm
   
   llm = create_llm(
       "openai",
       model="gpt-4o-mini",
       # api_key is normally read from OPENAI_API_KEY
   )

   Import a concrete adapter when provider-specific configuration makes the code clearer:

   from protolink.llms.api import OpenAILLM
   
   llm = OpenAILLM(model="gpt-4o-mini")

   The factory lazily imports only the selected adapter and its optional SDK. Construction itself is not deferred: concrete adapters currently validate during initialization and may perform network, server-health, model-loading, or filesystem work.

API keys

Never commit API keys to version control. Read them from environment variables or a secure secrets manager.

3. Make a direct call:

   response = llm.chat("Explain what this service does in two sentences.")
   print(response)

   As an alternative, use a fresh or explicitly managed history and consume the asynchronous iterator returned by streaming chat:

   streaming_llm = create_llm("openai", model="gpt-4o-mini")
   
   async for chunk in streaming_llm.chat(
       "Draft a short welcome message.",
       streaming=True,
   ):
       print(chunk, end="", flush=True)

4. Pass the same LLM to an Agent when the model should participate in a controlled runtime:

   from protolink.agents import Agent
   from protolink.models import AgentCard
   
   agent_card = AgentCard(
       url="http://localhost:8020",
       name="llm_agent",
       description="Agent backed by an LLM",
   )
   
   agent = Agent(card=agent_card, transport="http", llm=llm)

For local and server-style LLMs (LlamaCPPLocalLLM, LlamaCPPServerLLM, OllamaLLM, LMStudioLLM, VLLMLLM and OpenAICompatibleLLM), configuration additionally includes a model-file path or server URL. The individual class entries below describe the resolution order for those values.

Generation parameters are deliberately provider-specific. model_params is forwarded to the selected SDK or server; ProtoLink does not translate names such as max_tokens, max_output_tokens, or provider-specific thinking controls into one synthetic schema.

Direct chat history

chat() appends the user message before calling the provider, but it does not append the returned assistant text. That small behavior keeps the base method predictable across streaming and non-streaming adapters. Manage ConversationHistory explicitly for a standalone multi-turn script, or let an Agent own task-local and persistent conversation history.

From text generation to controlled inference

Direct generation ends when the provider returns text. Controlled inference can continue because the model is treated as a planner rather than an executor.

Each inference step follows the same story:

1. ProtoLink adds the user query and prepares the current system prompt, history, tools and discovered Agents.
2. The provider adapter obtains one decision, either through native function/tool calling or the portable JSON action protocol.
3. ProtoLink validates the decision as FinalAction, ToolCallAction, or AgentCallAction.
4. A final action returns user-facing text. A tool or Agent action passes through authorization and is executed by the runtime.
5. The result becomes a new observation in history and the model receives another step only when more work is necessary.

answer = await agent.invoke(
    "Summarize the available information.",
    session_id="customer-42",
)
print(answer)

The Agent rebuilds the system prompt for the current tool set, action mode, discovered Agents, flow position and application instructions before it invokes LLM.infer(). It also supplies the policy authorizer, cancellation token, run context, budget policy, event callback and isolated history.

Calling infer() directly

Direct inference is available for custom runtimes, but passing a tools dictionary does not itself rebuild the LLM's system prompt. A direct caller must call build_system_prompt() with the matching tool and Agent descriptions and must provide any callbacks, authorization, cancellation and context it needs. For normal tool use, prefer Agent.add_tool() or @agent.tool followed by await agent.invoke(...).

Streaming means two different things

• chat(..., streaming=True) returns an async iterator because the chunks are the result.
• infer(streaming=True) still returns one final Part. Intermediate model text is emitted as llm_chunk events through the inference event callback while the runtime waits for a complete, validated action.

This distinction lets an Agent show live progress without dispatching a partial or malformed tool request.

Stream into your application

RunHandle starts an Agent task and exposes live, typed events. No network transport is required for an embedded Agent:

import asyncio

from protolink import Agent, RunHandle, Task, create_llm


async def main():
    agent = Agent(
        {
            "name": "assistant",
            "description": "Embedded assistant",
            "url": "runtime://assistant",
            "capabilities": {"streaming": True},
        },
        llm=create_llm("ollama", base_url="http://localhost:11434", model="gemma4:e4b"),
        verbosity=0,
    )
    handle = RunHandle.start(agent, Task.create_infer(prompt="Explain streaming briefly."))
    try:
        async for event in handle.events():
            kind = event.payload.get("llm_event_type")
            if kind == "llm_chunk":
                print(event.payload["content"], end="", flush=True)
            elif kind == "llm_final":
                print("\nComplete answer:", event.payload["content"])
        result = await handle.result()
        print("Task status:", result.status)
    finally:
        await handle.cancel()  # No-op when the task has already completed.


asyncio.run(main())

llm_chunk carries each text fragment immediately. In default JSON-action mode, those fragments are raw action JSON, so a UI should show them as generation progress and use llm_final for the complete answer. With a model configured for native tools (supports_tool_calling=True on Ollama and other server adapters), text chunks are ordinary model text; tool arguments are accumulated separately and dispatched only after a complete action is validated. A tool-only response can have no text chunks.

llm_final remains the model's completion event. Continue reading until the terminal task status to receive artifacts and the final task state. handle.result() returns the completed result; it does not provide incremental output. Each handle.events() subscriber receives recorded history followed by new events. Closing a subscriber leaves the shared run active; use await handle.cancel() when the application wants to stop generation.

Streaming lifecycle and provider behavior

Ollama, llama.cpp server, vLLM, LM Studio and generic OpenAI-compatible streams use asynchronous HTTP reads. Requests preserve custom headers, HTTPS URLs and base-path prefixes. Responses and their request-scoped clients close on completion, cancellation, callback failure, or provider errors. Ollama's done and SSE's [DONE] markers end the response without waiting for the server to close its connection. Invalid JSON and provider error frames fail the stream rather than silently discarding output.

OpenAI, Anthropic, Gemini, DeepSeek, Hugging Face and local llama.cpp use their existing synchronous SDK iterators on a dedicated worker per stream. Opening, reading and closing happen on that worker; text callbacks run on the application's event loop. Only the next item is requested, so the adapter does not accumulate a background queue. Cancellation stops the async consumer promptly, but Python cannot interrupt an SDK read or model evaluation already executing: the worker closes the iterator after that operation returns or times out. Grok already uses asynchronous HTTP streaming.

For direct streams, close the iterator if the consumer stops early:

from contextlib import aclosing

async with aclosing(llm.chat("Hello", streaming=True)) as chunks:
    async for chunk in chunks:
        print(chunk, end="", flush=True)

Custom adapters must also avoid synchronous network reads inside async def methods. Awaiting an unbounded queue's put() does not guarantee that event consumers can run. call_action_stream() must await each text callback and assemble a complete action before returning it. HTTPX2 retains the HTTPX async streaming API; see also the Ollama streaming protocol reference.

Tools and delegated Agents stay behind a runtime boundary

Provider-native adapters expose tool schemas using the provider's function-calling format. Other adapters describe the same choices in the system prompt and parse one JSON action. Both routes converge on the same typed action models before anything executes.

The runtime then applies the protections that a raw model call does not provide: unknown-tool checks, policy and approval decisions, cancellation, duplicate-action detection, parse-error feedback, transient provider retries, per-run budgets and a ten-step inference limit.

The detailed controlled inference and tool-use chapter below explains action acquisition, recovery, prompts and delegation. The LLM.infer() reference documents every integration hook.

Conversation history inside and outside an Agent

An LLM instance still exposes llm.history for direct usage and backward-compatible introspection. When the same LLM is plugged into an Agent, ProtoLink binds a task-local ConversationHistory around each run so concurrent tasks do not interleave messages on one shared mutable history object.

For stateless agents, each task receives a fresh history seeded by the compiled system prompt. After normal completion, llm.history points at a copy of that task history for debugging and simple scripts. A failed turn normally stays isolated; if an action_result receipt proves that a side effect completed, ProtoLink retains the history containing that observation so a retry does not behave as though the action never ran.

For persistent conversation state, enable state=["conversation"]. The Agent loads the requested session_id, serializes concurrent tasks for that same session with an async lock, saves normally completed history back to state and exposes a copy as llm.history. Failed history is saved only when a new action_result receipt proves that the turn completed a tool or delegation side effect.

from protolink import Agent, AgentCard, RunContext, Task, create_llm

agent = Agent(
    AgentCard(
        name="assistant",
        description="Assistant",
        url="runtime://assistant",
    ),
    llm=create_llm("mock", default_response="ok"),
    state=["conversation"],
)

task = Task.create_infer(prompt="remember this")
RunContext(session_id="customer-42").attach_to_task(task)
await agent.execute_task(task)

Direct llm.infer(...) calls are unchanged: they use the LLM's default history unless you explicitly call llm.use_history(history).

Model routing and fallback

RoutedLLM chooses a configured adapter for each logical inference step and provides bounded fallback for transient request failures. Use it through the ordinary Agent llm= parameter; tools, policy, budgets and durable execution remain owned by the Agent.

from protolink import Agent, RoutedLLM
from protolink.llms import MockLLM

models = RoutedLLM(
    {
        "fast": MockLLM(default_response="A quick response"),
        "reasoning": MockLLM(default_response="A detailed response"),
    }
)
agent = Agent(name="helper", llm=models)
print(agent.sync.invoke("Hello"))

The first configured key is the default. Values can be configured LLM objects or normal provider:model aliases. To route explicitly, supply a synchronous selector returning a configured key:

from protolink import RoutedLLM


def choose_model(history):
    latest_user = next(
        (message["content"] for message in reversed(history.messages) if message["role"] == "user"),
        "",
    )
    return "reasoning" if "analyze" in latest_user.lower() else "fast"


models = RoutedLLM(
    {"fast": fast_adapter, "reasoning": reasoning_adapter},
    selector=choose_model,
    fallbacks=["reasoning"],
    retries_per_model=0,
)

fast_adapter and reasoning_adapter are configured application models. The selector receives a copied canonical history after hooks and context preparation. It runs once per step, before physical request attempts. It can use the latest question or accumulated evidence; it should be pure and deterministic for recoverable execution. Unknown keys fail visibly. Nested routers are rejected.

Fallback keys are tried in declared order, excluding the already selected key. retries_per_model defaults to zero; increasing it permits the normal transient retry/backoff behavior before moving to the next candidate. Every attempted request consumes the same root model-call/input budget. Fallback does not add another logical step or reset token, runtime or tool limits.

Only transient provider/connection failures qualify. Invalid credentials, validation failures, policy denials and cancellation do not switch providers. If streamed content has already been exposed, the request fails visibly without retry or fallback; consumers must handle the partial stream as incomplete. Parsing invalid action text uses normal inference correction rather than silently retrying the same request through another provider. Tool operations execute after a successful parsed action and are never replayed by routing.

The router inherits the default inference loop and uses ProtoLink's portable JSON action prompt and observations. It delegates plain text/stream requests to adapters instead of mixing provider-native tool IDs across models. Applications requiring native provider features can use a concrete adapter directly. Response metadata includes the selected key, provider, model and total request attempts for the step. Metrics and token budgets remain observable; the portable path may rely on estimated usage rather than provider-native response metadata. Configure conservative context/cost profiles on the router for the set of possible models, or set an explicit ContextPolicy.max_tokens.

Model identities, declared fallback order and non-secret configuration hashes serialize as routing descriptors. Reconnect executable adapters and the selector with Agent.from_dict(saved, llm=RoutedLLM(...)); credentials and callbacks are not restored from descriptors. Changes to declared adapter parameters affect the durable contract. Increment execution_version when selector or other executable behavior changes. Construction makes no provider calls; validate_connection() explicitly checks configured adapters.

History compaction

For automatic preparation on every inference step, see Context management. It reserves output space, prunes complete old turns, clears older acknowledged observations and offloads large results to scoped retrieval artifacts. The explicit compaction API below remains useful when the application chooses a summary strategy or a specific compaction boundary.

Every LLM wrapper owns a modular HistoryCompactor at llm.compactor. Its compact() method mutates the current ConversationHistory in place and returns a HistoryCompactionResult with before/after message and estimated-token counts. LLM.compact_history() remains as a convenient facade, so direct usage stays concise.

# Fastest: keep the system prompt and 19 newest messages.
report = llm.compact_history("recent", max_messages=20)

# Local and budget-aware: keep a recent suffix near 8,000 estimated tokens.
report = llm.compact_history(
    "tokens",
    max_tokens=8_000,
    preserve_recent=6,
)

# Highest context fidelity: summarize old turns, preserve the newest 8 verbatim.
report = llm.compact_history(
    "summary",
    preserve_recent=8,
    summary_max_tokens=600,
)

print(report.to_dict())

# Equivalent component-oriented API:
report = llm.compactor.compact("tokens", max_tokens=8_000)

Three strategies cover different cost and fidelity needs:

Strategy	Model calls	Behavior	Best for
recent	0	Keeps the leading system prompt and newest max_messages messages.	A simple, fast sliding window.
tokens	0	Keeps the newest chronological suffix near max_tokens, using tiktoken when installed or the built-in estimate otherwise.	Deterministic context-budget control.
summary	1	Replaces older turns with a model-generated system summary and keeps preserve_recent turns verbatim.	Retaining decisions and constraints from long sessions.

The tokens limit is deliberately soft when the leading system prompt plus protected recent messages already exceed the budget: ProtoLink preserves those messages instead of silently removing the active request. The summary strategy makes its model call with a temporary history. The live history is changed only after a non-empty summary is returned, so a provider failure leaves it untouched.

Agent-requested compaction

Agent-requested compaction is a control-plane request, not a model tool and not a task part. Call the Agent method directly or use the client request spec. The compaction capability is never appended to the model prompt and is never exposed through provider-native or JSON tool calling, which keeps the prompt smaller and friendlier to very small models.

from protolink import HistoryCompactionRequest

report = await agent.compact_history(
    HistoryCompactionRequest(
        strategy="summary",
        preserve_recent=8,
        summary_max_tokens=600,
        session_id="customer-42",
    )
)

For remote agents, use the client spec-backed convenience method:

from protolink.client import AgentClient

client = AgentClient("runtime", url="runtime://client")
report = await client.compact_history(
    "runtime://agent",
    strategy="summary",
    preserve_recent=8,
    summary_max_tokens=600,
    session_id="customer-42",
)

The remote path is POST /llm/history/compact, represented by AgentClient.COMPACT_HISTORY_REQUEST and an EndpointSpec registered by AgentServer. When state=["conversation"] is enabled and session_id is supplied, the Agent loads the session history before compaction and saves the compacted history afterward. The runtime action is still evaluated through the policy boundary with the llm.history.compact capability, so applications can allow, deny, or require approval for context loss.

Compaction is explicit

ProtoLink does not compact history automatically based on an arbitrary context threshold. Applications can call llm.compact_history() for local use, agent.compact_history() inside an Agent process, or AgentClient.compact_history() over a transport. Natural-language requests such as “please compact your context” are application intent; convert them into a control-plane request before calling the Agent if you want deterministic behavior.

How the LLM package is organized

The public facade is protolink.llms.base.LLM. Internally, the base class owns provider-neutral orchestration: history binding, metrics, budgets, retries, tool execution, Agent delegation and final response handling. The strict action parser lives in protolink.llms.parsing, where raw model text becomes one validated LLMAction and narrow fallback shorthands are repaired only when the target tool or Agent is unambiguous.

Provider adapters keep request and stream handling in their own modules, then return typed results to the shared inference loop:

• protolink.llms.api contains hosted-provider adapters.
• protolink.llms.server contains HTTP model-server adapters.
• protolink.llms.local contains the in-process llama.cpp adapter.
• protolink.llms.mock_client provides deterministic testing behavior.

The rest of the package is partitioned by responsibility:

• factory.py lazily resolves provider names and constructs adapters.
• history.py defines canonical messages and provider-neutral conversation history.
• actions.py, parsing.py and tool_calling.py define typed runtime actions and translate provider-native function calls into them.
• context.py builds a context manifest for each inference step.
• compaction.py owns explicit history reduction and summary compaction.
• metrics.py normalizes usage, estimates missing token counts and calculates application-supplied costs.
• prompts/ keeps JSON-action and provider-native prompt families separate so the model never receives conflicting tool instructions.
• serialization.py provides JSON-safe conversion for history and runtime payloads.
• _deps.py loads optional provider SDKs only when their adapter is selected.

This separation keeps the everyday API small while allowing native tool-calling providers and JSON-fallback models to participate in the same controlled runtime.

Observe and constrain model work

Profiles, context manifests, call metrics and run budgets matter when a working model integration becomes an operated system. They are intentionally optional and do not need to be configured before the first call.

Model metadata

LLMModelProfile describes the deployment information ProtoLink cannot safely hardcode: context-window size, application-supplied input and output prices, tokenizer metadata and descriptive capability flags. A profile does not change the provider request or enable a feature in the selected model.

from protolink import LLMModelProfile, create_llm

llm = create_llm(
    "openai-compatible",
    model="my-model",
    metrics_profile=LLMModelProfile(
        context_window=128_000,
        input_cost_per_million=1.0,  # example value; use current provider pricing
        output_cost_per_million=5.0,  # example value; use current provider pricing
        supports_tools=True,
        supports_streaming=True,
        supports_json_schema=True,
        tokenizer="cl100k_base",
    ),
)

You can also configure metrics after construction:

llm.configure_metrics(
    context_window=128_000,
    input_cost_per_million=1.25,
    output_cost_per_million=10.0,
)

Provider-reported token usage is used when the SDK response includes it. Otherwise ProtoLink estimates token counts locally. If tiktoken is installed through protolink[metrics], ProtoLink resolves an encoder from the active model name and falls back to cl100k_base; without it, ProtoLink uses a lightweight character heuristic. The profile's tokenizer field is currently descriptive metadata and does not select the estimator. Prices, model limits and capabilities change over time, so LLMModelProfile is application-owned metadata rather than a hardcoded billing catalog.

Context and call events

LLM wrappers can emit pre-call context manifests plus per-call latency, token usage, context-window pressure and estimated cost through the existing infer() event stream and telemetry hooks. Observing these events does not change the request payload sent to the provider.

Before each model call, ProtoLink emits a provider-neutral context_prepared event:

{
    "type": "context_prepared",
    "step": 1,
    "manifest": {
        "run_id": "run_123",
        "agent_name": "researcher",
        "system_tokens": 900,
        "tool_prompt_tokens": 300,
        "history_tokens": 2200,
        "user_tokens": 120,
        "total_estimated_tokens": 3520,
        "context_window": 128000,
    },
}

When an event_callback or telemetry backend is attached, each model call inside the inference loop can also emit:

{
    "type": "llm_call_metrics",
    "step": 1,
    "provider": "openai-compatible",
    "model": "my-model",
    "latency_ms": 842.37,
    "usage": {
        "input_tokens": 1200,
        "output_tokens": 180,
        "estimated": False,
    },
    "context": {
        "used_tokens": 1200,
        "window_tokens": 128000,
        "used_percent": 0.938,
    },
    "cost": {
        "input_cost": 0.0012,
        "output_cost": 0.0009,
        "total_cost": 0.0021,
    },
}

This is especially useful for CLIs, dashboards and budget-aware agents that want to show context pressure or session cost while a multi-step tool loop is running.

Run-budget enforcement

Model profiles are observational metadata. Run budgets are the separate enforcement mechanism.

If a RunContext carries a RunBudget, LLM.infer() enforces it through the default BudgetEnforcer. When an Agent executes a task, one task-local enforcer is shared by every infer part and explicit tool-call part, so counters do not reset between parts. A nested task gets its own scope and direct LLM.infer() calls still create an independent enforcer unless the advanced caller supplies budget_enforcer= explicitly.

Pre-call limits such as max_llm_calls and max_input_tokens are checked before every physical provider attempt, including transient retries. max_tool_calls applies before explicit and model-selected tools execute; max_output_tokens is checked after provider usage or local estimates are available. Provider runtime is checked again after each request, including a final response. A tool or delegated call can already have committed a side effect when it returns, so ProtoLink records and injects that result before cancellation or runtime enforcement stops the next step. Warnings appear as budget_warning events and hard denials appear as budget_exceeded events.

LLM API reference

The rest of this page is the detailed contract. It starts with construction and the base methods, then follows controlled inference, prompts, concrete providers, related objects, examples and failure handling.

Provider switching in action

The same application code works across providers. Keep provider choice in configuration, construct exactly one adapter and leave the calling code unchanged. chat() is the high-level convenience method for direct text generation; internally it selects call() or call_stream().

from protolink import create_llm

# Choose one deployment in application configuration.
provider = "ollama"
provider_options = {
    "openai": {
        "model": "gpt-4o-mini",
    },
    "anthropic": {
        "model": "claude-sonnet-4-20250514",
    },
    "ollama": {
        "model": "qwen3",
        "base_url": "http://localhost:11434",
    },
    "lmstudio": {
        "model": "local-model",
        "base_url": "http://localhost:1234/v1",
    },
    "vllm": {
        "model": "Qwen/Qwen3-8B",
        "base_url": "http://localhost:8000/v1",
    },
}

llm = create_llm(provider, **provider_options[provider])

# The calling code stays the same.
response = llm.chat("Hello! How are you?")
print(response)

LLM hierarchy

• LLM - abstract base class with the common runtime behavior.
• APILLM - base for API-hosted adapters.
• ServerLLM - base for HTTP model servers.
• LocalLLM - base for in-process local runtimes.
• Concrete implementations - OpenAILLM, AnthropicLLM, GeminiLLM, DeepSeekLLM, GrokLLM, HuggingFaceLLM, OllamaLLM, LlamaCPPServerLLM, LMStudioLLM, VLLMLLM, OpenAICompatibleLLM, LlamaCPPLocalLLM and MockLLM.

Each callable below keeps the scikit-learn-style layout: exact signature, explanation, separately labeled parameters and defaults, return values, raised errors, notes and focused examples.

create_llm

functionprotolink.create_llm

source↗

create_llm(
  provider: str | LLMProvider,
  **kwargs,
) -> LLM

Create an LLM adapter without importing the selected provider until it is needed. Both create_llm("ollama:qwen3:4b") and create_llm("ollama", model="qwen3:4b") select the same model. Only the first colon separates provider and model; case, paths and additional colons are kept. Do not combine an inline model with model=. Agent(name="helper", llm="mock") and Agent(name="helper", llm="ollama:qwen3:4b") use this same factory.

Parameters

providerstr | LLMProviderrequired

Provider alias or provider:model string. Supported aliases are anthropic, deepseek, gemini, grok, huggingface, llama.cpp-local, llama.cpp-server, lmstudio, mock, ollama, openai, openai-compatible and vllm.

**kwargsAny

Forwarded to the selected adapter constructor. Runtime keywords are also recognized: metrics_profile configures model metrics after construction, metrics_enabled enables or disables their emission, tool_calling selects auto, native or json action acquisition and max_parse_failures sets the validated consecutive action-parse failure limit, while max_tool_corrections bounds tool-request correction. These runtime options are not forwarded to provider generation parameters.

Returns

llmLLM

An initialized concrete adapter for the requested provider.

Raises

ValueError

Raised when the provider string is unknown or empty, an inline model is empty or also supplied through model=, max_parse_failures or max_tool_corrections is outside its supported range, tool_calling is invalid, or native mode is requested on an adapter without a native implementation.

TypeError

Raised when max_parse_failures or max_tool_corrections is not an integer. Boolean values are not accepted as integers for these options.

ImportError

Raised when the selected adapter requires an optional dependency that is not installed.

provider or constructor error

Credential, model-path, URL and client-construction errors from the selected adapter are not hidden.

Current behavior

Pass provider names as strings. Although the annotation includes LLMProvider, enum instances are not currently normalized correctly by the factory.

Examples

from protolink import LLMModelProfile, create_llm

llm = create_llm(
    "openai-compatible",
    base_url="http://localhost:1234/v1",
    model="local-model",
    metrics_profile=LLMModelProfile(context_window=32_768),
    max_parse_failures=4,
)

max_parse_failures defaults to 3 and accepts integers from 1 through 10. Keep it as a top-level factory argument. Do not place it in model_params, whose entries are provider generation options.

max_tool_corrections independently bounds unknown-tool, argument-schema and explicit no-change rejection feedback. It defaults to 2 corrective turns and accepts integers from 0 through 10. Pass it to create_llm() or assign llm.max_tool_corrections. Successful actions do not reset this counter.

Base LLM contract

The LLM class defines the common interface that every implementation follows. Concrete adapters are deliberately thin at the runtime boundary: they translate ConversationHistory into a provider request, translate responses and streams back into ProtoLink values and validate connectivity. The base class supplies the shared behavior around those calls.

Application code normally constructs a provider through create_llm() and calls chat() for direct text generation. Agent uses the more powerful infer() path, which adds typed actions, tool execution, delegation, policy checks, budgets, events, retries and bounded iteration.

The core surface falls into four groups:

• Provider invocation: call() and call_stream() are implemented by concrete adapters.
• Direct conversation: chat() appends a user message and selects the blocking or streaming provider path.
• Structured action acquisition: call_action() and call_action_stream() normalize native tool calls or JSON output into LLMActionResult.
• Runtime orchestration: infer() repeatedly validates and dispatches those actions until the model returns a final answer.

History performance

ConversationHistory uses a collections.deque internally. Prepending or replacing the system prompt is an O(1) operation and trimming older turns avoids repeated list reallocation on hot agent paths.

Abstract base class

Do not instantiate LLM directly. Use a concrete implementation such as OpenAILLM, AnthropicLLM, OllamaLLM, or MockLLM, normally through create_llm().

LLM

abstract classprotolink.llms.base.LLM

source↗

class LLM(
  model: str,
  model_params: dict[str, Any],
  *,
  reasoning: Literal["none", "low", "medium", "high"] = "none",
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

Abstract provider-neutral model contract. Subclasses implement text generation and connection validation; the base class owns history binding, typed action parsing, inference orchestration, compaction and metrics.

Parameters

modelstrrequired

Provider-specific model identifier or local model path.

model_paramsdict[str, Any]required

Generation parameters passed to the adapter. The base class verifies only that later assignments remain dictionaries; the downstream SDK or server validates individual keys and values.

reasoning"none" | "low" | "medium" | "high"default: "none"

Selects the reasoning instruction block included when the base system prompt is built. This argument is mainly for custom subclasses; current concrete provider constructors do not expose it.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime action acquisition policy. Omit it to retain the adapter's existing default, including its legacy capability flag. Explicit modes override that flag; auto uses cached capability evidence.

Attributes

modelstr

Resolved model identifier.

model_paramsdict[str, Any]

Mutable provider generation parameters.

historyConversationHistory

Default history for direct calls, or the task-local history currently bound by use_history().

has_active_historybool

Whether the current execution context has a task-local history binding.

compactorHistoryCompactor

Component that mutates the live history using recent-message, token-budget, or summary compaction.

system_promptstr

Prompt last built or assigned on the adapter.

metrics_profileLLMModelProfile | None

Optional application-owned context-window and cost metadata.

metrics_enabledbool

Whether inference may emit metrics when an observer is attached.

max_parse_failuresint

Maximum consecutive JSON-action parsing or validation failures allowed during one inference run. Defaults to 3 and accepts values from 1 through the ten-step inference ceiling. This runtime control is deliberately separate from provider generation parameters.

max_tool_correctionsint

Permitted corrective turns for unknown tools, invalid argument schemas and explicit no-change tool rejections. Defaults to 2; accepts 0 through 10. Success does not reset the counter. Durable checkpoints preserve it. Configure through the factory or this property, outside generation parameters.

tool_calling"auto" | "native" | "json" | None

Configured runtime protocol policy. Changes apply to subsequent inferences; active and durable continuations retain the selected protocol. See action acquisition.

syncSyncLLM

Blocking wrapper for infer(). Do not use it inside an active event loop.

Module constant

The ten-step inference limit is protolink.llms.base.MAX_INFER_STEPS. It is a module constant, not an LLM class attribute.

LLM.chat

methodprotolink.llms.base.LLM.chat

source↗

chat(
  user_query: str,
  *,
  streaming: bool = False,
) -> str | AsyncIterator[str]

Add one user message to the active history, then make a direct provider call.

Parameters

user_querystrrequired

User text appended to the current ConversationHistory.

streamingbooldefault: False

When False, return the complete provider response. When True, return an asynchronous iterator of text chunks.

Returns

responsestr

Complete model text when streaming=False.

chunksAsyncIterator[str]

Asynchronous text stream when streaming=True.

History behavior

chat() appends the user message, but it does not append the returned assistant text. Use Agent state or manage ConversationHistory explicitly when you need durable multi-turn history.

Examples

answer = llm.chat("Give me a title for this report.")

async for chunk in llm.chat("Draft the introduction.", streaming=True):
    print(chunk, end="")

LLM.call

abstract methodprotolink.llms.base.LLM.call

source↗

call(
  history: ConversationHistory,
) -> str

Generate one complete text response from a conversation history. Concrete adapters translate the history into their provider’s request format.

Parameters

historyConversationHistoryrequired

Ordered system, user, assistant and tool messages sent to the model.

Returns

responsestr

Raw text extracted from the provider response.

Raises

NotImplementedError

Raised by the abstract base implementation.

provider error

Concrete adapters generally propagate authentication, HTTP, SDK and model errors from the provider call.

LLM.call_stream

abstract methodprotolink.llms.base.LLM.call_stream

source↗

call_stream(
  history: ConversationHistory,
) -> AsyncIterator[str]

Start a streaming provider call and yield incremental text chunks.

Parameters

historyConversationHistoryrequired

Conversation sent to the provider.

Yields

chunkstr

The next incremental piece of model output.

Consumption

Consume the returned object with async for. Do not await call_stream() itself.

LLM.call_action

methodprotolink.llms.base.LLM.call_action

source↗

call_action(
  history: ConversationHistory,
  *,
  tools: dict[str, BaseTool],
  agent_callback_available: bool = False,
  agent_cards: list[Any] | None = None,
) -> LLMActionResult

Acquire one validated runtime action. The base implementation parses a JSON action from text; native-capable adapters override it and normalize provider tool calls into the same result type.

Parameters

historyConversationHistoryrequired

Conversation for the current inference step.

toolsdict[str, BaseTool]required

Tool names mapped to executable tool objects. Native adapters turn this mapping into provider tool declarations.

agent_callback_availablebooldefault: False

Whether the runtime can dispatch agent-delegation actions.

agent_cardslist[Any] | Nonedefault: None

Discovered agents available for native declarations and unambiguous fallback repair.

Returns

resultLLMActionResult

A validated FinalAction, ToolCallAction, or AgentCallAction, together with the raw response and provider metadata.

Raises

ValueError

Raised when direct fallback parsing cannot produce a valid action.

LLM.call_action_stream

async methodprotolink.llms.base.LLM.call_action_stream

source↗

async call_action_stream(
  history: ConversationHistory,
  *,
  tools: dict[str, BaseTool],
  agent_callback_available: bool = False,
  agent_cards: list[Any] | None = None,
  chunk_callback: Callable[[str], Awaitable[None]] | None = None,
) -> LLMActionResult

Streaming counterpart to call_action(). The fallback implementation forwards text chunks to the observer, buffers the complete response and validates exactly one action after the stream ends.

Parameters

historyConversationHistoryrequired

Conversation for the current inference step.

toolsdict[str, BaseTool]required

Tools available to the model.

agent_callback_availablebooldefault: False

Whether agent delegation can be dispatched.

agent_cardslist[Any] | Nonedefault: None

Discovered agents available for delegation.

chunk_callbackCallable[[str], Awaitable[None]] | Nonedefault: None

Async observer invoked for each emitted text chunk.

Returns

resultLLMActionResult

One complete validated runtime action. Partial JSON fragments are never dispatched.

Raises

ValueError

Raised when an empty or malformed fallback stream cannot be parsed as an action.

Controlled inference and tool use

What inference means in ProtoLink

infer() is the cornerstone of ProtoLink's agent runtime. A normal model call asks for text and returns text. Inference instead treats the model as a decision-maker inside a controlled loop: the model declares one typed action, ProtoLink validates and performs that action, the result is returned to the model as an observation and the cycle continues until the model can produce a final answer.

This distinction is important. The LLM never executes Python code, invokes a remote agent, grants its own approval, or decides whether a budget may be exceeded. It can only request a final, tool_call, or agent_call action. ProtoLink remains the executor and policy boundary.

In normal applications, Agent prepares the prompt and calls infer() automatically. Direct calls remain available for advanced integrations that already have tools, callbacks, runtime policy and history under their own control.

infer() enables a model to:

1. Make tool calls - request external functions with structured arguments.
2. Delegate to agents - pass work to a discovered specialized agent.
3. Observe results - receive tool or agent output in conversation history.
4. Self-correct - revise malformed actions, wrong arguments, or unavailable targets.
5. Generate a final response - return user-facing content only after the required work is complete.

The complete inference cycle

An inference run can contain several model turns. With the ordinary Agent API, the application supplies the model, tools and optional specialists; the Agent wires them into this cycle:

1. Prepare the model input. Combine the task, instructions, conversation history, available tools and agent cards. Configured context policies and hooks prepare each turn. JSON or native tool calling is selected for the run.
2. Ask for one decision. The model proposes final, tool_call or agent_call. Both provider modes normalize into the same typed actions; streamed fragments are collected before validation or dispatch.
3. Validate the proposal. Check action shape, target identity, inference support, tool name and argument schema. A correctable mistake becomes feedback in the current conversation. No tool or child task executes for a rejected pre-dispatch proposal. The same model chooses its next action; ProtoLink does not guess a replacement request.
4. Authorize and execute. A valid executable request passes cancellation, policy, approval and budget checks. ProtoLink then invokes the local tool or delegates the child task. Duplicate completed requests receive guidance to use their existing observations instead of executing again.
5. Return an observation. Feed the result back to the calling model, then repeat from the updated history. An explicit ToolRejectedError also becomes feedback when the tool guarantees that its rejection made no changes. Ordinary execution exceptions remain failures.
6. Finish, pause or stop. A final action returns the answer. Exhausted limits, cancellation, permission denial or execution failure stop the run. With durable execution, approval or input checkpoints can pause it and later resume with saved history and committed results.

The model decides what to request; the runtime decides whether and how that request can execute. Correcting a proposal does not restart the task, clear completed observations or replay earlier actions. Delegated infer work may use the specialist's own model, but correction feedback returns to the model that made the rejected request.

A run with two corrected requests

Suppose coder owns read_file and write_file but has no LLM, while planner has an LLM and no tools. One possible run to add docstrings looks like this; the actions shown are equivalent in JSON and native modes:

Model decision	Runtime handling	What the next model turn receives
coder.read_file("utils.py") through agent_call / tool_call	Validate, authorize and read the file once.	The code as a completed observation.
Ask planner to execute a tool named tool_call	Reject the unadvertised tool before dispatch.	Available tools and guidance to use infer with a prompt.
Ask coder to infer a docstring plan	Reject unsupported inference before dispatch.	Coder's advertised tools and guidance to choose another action or inference-capable agent.
Ask planner to infer a plan using the code already read	Validate, authorize and delegate the prompt.	The Planner's response.
coder.write_file(...) through agent_call / tool_call	Validate, authorize and execute the write.	The write result.
final with a summary	End the inference loop.	The application receives the final answer.

The read observation stays available throughout both corrections. Rejected requests are recorded separately from completed execution. max_parse_failures bounds consecutive malformed action proposals; max_tool_corrections bounds tool and delegation request rejections across the run. Physical provider retries are a separate mechanism for transient request failures. Every path still consumes the applicable shared budgets and inference-step limit. See safety guardrails for the exact limits and failure contracts.

How action acquisition works

Every provider ultimately returns the same LLMActionResult, but it can acquire that action in one of two ways:

Mode	Used by	Model instruction	Runtime behavior
JSON action mode	Default for local/small models and providers without reliable native tools	Return one JSON object such as {"type":"tool_call","tool":"search","args":{"q":"..."}}	call_action() or the fallback call_action_stream() parses text, validates it with Pydantic and returns a typed action.
Native action mode	OpenAI, Anthropic, Gemini and opted-in tool-capable servers	Use the provider's function or tool interface	The adapter sends real tool declarations, receives provider-native tool events and normalizes them into the same typed action models.

Choose the protocol once on the LLM. The Agent prepares matching instructions and uses the same mode throughout the current inference, including correction attempts:

from protolink import Agent, create_llm

llm = create_llm(
    "ollama:qwen3:4b",
    base_url="http://localhost:11434",
    tool_calling="auto",
)
agent = Agent(name="helper", llm=llm)
print(agent.sync.invoke("Explain what an agent harness does."))

Selecting native tools or JSON actions

tool_calling is ProtoLink runtime configuration, so keep it outside model_params. The factory and concrete adapter constructors accept the same option; existing instances can use llm.tool_calling = "json".

Setting	Behavior
Omitted / None	Preserves the adapter's current default and supports_tool_calling flag. Existing applications make no discovery requests.
"auto"	Uses provider capability evidence. Advertised native support selects native tools; unsupported or unknown configurations use JSON actions.
"native"	Explicitly sends provider-native tool declarations. The adapter must implement native actions; provider rejection remains visible.
"json"	Explicitly uses the portable JSON action protocol, including on hosted adapters that normally use native tools.

Ollama obtains model-specific evidence through POST /api/show and checks the exact model's tools capability. ProtoLink reuses the result for five minutes on that adapter, keyed by provider, endpoint, model identifier, generation configuration and request headers. Both supported and unknown results are cached. Discovery has a five-second request timeout and does not load a model, generate a test completion or invoke application tools. Missing capabilities, failed requests and malformed metadata select JSON conservatively. Direct tool tasks do not need model capability discovery.

OpenAI, Anthropic, Gemini, DeepSeek and Grok use their adapters' declared native API contracts for automatic selection. The generic OpenAI-compatible, LM Studio, vLLM and llama.cpp adapters do not have a reliable portable metadata contract for the selected model and server configuration, so automatic selection uses JSON there. Set "native" when the application knows its model, template and server tool parser are configured appropriately. Hugging Face, MockLLM and RoutedLLM use JSON; requesting native mode on an adapter without native action support raises ValueError.

Capability metadata advertises a feature; it does not measure tool-choice or argument accuracy. It also does not prove that the inference backend can enforce JSON or a JSON Schema. Ollama's structured-output fallback remains a separate mechanism with its own explicit-format contract.

For more control:

llm.tool_calling = "json"  # Explicit portable mode for the next inference.
llm.tool_calling = "auto"  # Return to cached automatic selection.
llm.refresh_capabilities()  # Recheck after replacing a model/template or updating the server.
print(llm.resolve_tool_calling())  # "native" or "json"; auto may perform a metadata request.

Refreshing capabilities or changing the policy does not switch an active inference's protocol. Cached evidence is process-local and is not serialized; Agent configuration and durable execution contracts retain the configured policy. Durable checkpoints also retain the selected protocol so a resumed run uses requests compatible with its saved history, even when fresh capability metadata is temporarily unavailable. Custom adapters can implement discover_tool_calling() -> bool | None to supply metadata evidence while leaving cache and selection policy in the shared engine. Native implementations declare _native_tool_calling_available = True and normalize native events through call_action() and call_action_stream().

The offline correction example runs the real action adapter against scripted responses in JSON and automatic native mode:

python examples/action_correction.py

It demonstrates an invalid wrapper being corrected after a completed tool, with the tool executing once.

The rest of the loop is identical:

1. Prompt selection: the Agent builds either the portable JSON prompt or the native-tool prompt. Streaming uses native instructions only when the adapter supports native streamed actions.
2. Context preparation: the current query, history, tools, discovered agents, flow context and runtime manifest are assembled.
3. Action acquisition: call_action() or call_action_stream() obtains one model decision.
4. Validation: the decision becomes a FinalAction, ToolCallAction, or AgentCallAction. Raw provider objects never reach the dispatcher.
5. Policy and budget checks: cancellation, authorization, capabilities, approvals and remaining run budget are evaluated before side effects.
6. Dispatch: ProtoLink executes a local tool, delegates to another agent, or accepts the final response.
7. Observation injection: tool and agent results are appended through the provider-specific or provider-neutral history path.
8. Iteration: the process repeats until final is produced or a guardrail stops the run.

What streaming JSON means

Streaming JSON does not mean ProtoLink dispatches incomplete JSON fragments. The model streams ordinary text chunks that eventually form one complete action object. ProtoLink can forward those chunks to observers, but it buffers and validates the complete object before executing anything.

Small-model support

Ollama, llama.cpp, LM Studio, vLLM and generic OpenAI-compatible servers retain their default JSON action mode. Use tool_calling="auto" for advertised Ollama support, or tool_calling="native" for a known tool-capable configuration. The existing supports_tool_calling=True flag remains supported when tool_calling is omitted.

Inference-loop safety guardrails

The inference loop includes multiple layers of protection against unreliable model output and runaway execution.

1. Deduplication detection

The runtime tracks a sliding window of successfully completed side-effect signatures. If the model requests an identical tool or agent action with identical arguments, ProtoLink:

• does not execute the duplicate;
• injects corrective guidance into history; and
• asks the model to use the existing observation, choose a different action, or finish.

Failed validation or execution does not poison the window, delegated infer signatures include the complete prompt and final actions are returned immediately even when their text repeats. This prevents repeated side effects without suppressing a valid answer.

You have already performed this action. The result is in your context.
Proceed with the task: produce a final response or choose a different action.

2. Action parsing and the parse-failure circuit breaker

The parser is a security and interoperability boundary between untrusted model output and the runtime dispatcher. It does not execute a tool or contact an Agent while trying to understand malformed text. It first produces one typed FinalAction, ToolCallAction, or AgentCallAction; only that validated action can continue to policy, budget and dispatch checks.

Provider-native tool calls and portable JSON actions enter this boundary differently:

• A native-capable adapter translates the provider's structured tool event into the same Pydantic action models.
• A JSON-mode adapter passes the complete model text through the shared prompt-fallback parser.

The JSON path follows a deliberately conservative pipeline:

1. Decode the untouched whole response. Valid JSON is never rewritten first. This preserves literal text such as "<think>keep this</think>" inside a valid final response.
2. Recover syntax-only wrappers when whole-response decoding fails. ProtoLink can unwrap a complete JSON code fence, ignore one complete leading <think> or <thought> block, remove trailing commas outside JSON strings, or extract exactly one balanced object embedded in surrounding prose. The balanced scanner tracks nesting, quoted strings and escapes in one pass.
3. Normalize only deterministic response shapes. A structured value placed in FinalAction.content can be serialized losslessly as JSON text. A non-empty application object that omits the outer action envelope can become final content only when it contains no ProtoLink action-envelope fields. Existing legacy tool-call shorthands are repaired only when the tool and optional Agent target are unambiguous and their arguments are literal data.
4. Validate the complete action with Pydantic. Required fields, discriminated action types, forbidden extra fields, delegated-action combinations and content types are checked before the infer loop sees the result.

Malformed action wrappers and native answer text

An outer tool_call, tool_calls, function_call, function_calls or agent_call field is recognizable action protocol structure. ProtoLink does not accept such a wrapper as an ordinary application answer or guess how to execute it. For example, this response fails action validation:

{"tool_call":{"name":"explorer","arguments":{"query":"docs"}}}

The correction loop requests the canonical shape, such as {"type":"tool_call","tool":"explorer","args":{"query":"docs"}}, or the provider's native tool interface. The same validation failure reaches the loop in JSON and native modes, including after a complete stream. Native adapters do not turn a recognizable invalid action into a successful final answer.

Native responses may still contain plain text, ordinary application JSON and code examples. Standalone action objects, optionally in one JSON fence or after a complete reasoning block, enter strict action parsing. Prose explaining an example and Python code fences remain answer text; the native text normalizer never searches those explanations for an executable action. In JSON mode, put examples and application objects using reserved action fields inside explicit final.content:

{"type":"final","content":{"example":{"tool_call":{"name":"explorer"}}}}

Correction continues inside the current inference with its existing history, completed observations, consecutive parse limit and shared run budgets. Invalid action text consumes model-call and step budgets; its output tokens are counted using provider usage when available and local estimates otherwise. A correction does not restart the task or replay completed tools. Recent duplicate requests continue to use the existing deduplication guardrail. Streaming chunks can already have reached observers, so consumers must treat them as provisional until a validated action or final event.

Common small-model variations therefore have explicit outcomes:

Model output	Parser behavior
{"type":"final","content":{"answer":42}}	Serializes the object to JSON text and returns a FinalAction.
{"answer":42,"sources":["doc-1"]}	Wraps the application object as final JSON content because no action-envelope field is present.
```json {"type":"final","content":"done"} ```	Unwraps the complete fence and validates the action.
Result: {"type":"final","content":"done"}	Extracts and validates the single balanced object.
{"type":"final","content":"done",}	Removes the unambiguous trailing comma outside strings, then validates.
A complete leading <think>...</think> followed by one action	Ignores the reasoning wrapper only after untouched JSON decoding failed.

The parser intentionally refuses cases where repair would require choosing or inventing meaning:

• two or more valid top-level JSON objects;
• an incomplete leading reasoning wrapper;
• an empty object or a non-object action payload;
• an explicit unknown action type;
• a tool- or Agent-shaped object with a missing action type;
• a missing tool, target, prompt, or argument that cannot be inferred uniquely; or
• an action found only inside a private reasoning wrapper with no public action after it.

Raw responses and parsed payloads use deterministic, 2,000-character head-and-tail previews in diagnostics. This keeps an oversized valid or invalid response from flooding logs and telemetry while retaining enough beginning/end context to diagnose truncation. For a decoded action that fails schema validation, correction history receives the concise field-level feedback and detected outer action type, not another copy of the parsed-payload preview.

Correction attempts

Action parsing uses the limit described here. Once a valid tool_call reaches dispatch, unknown tools, invalid argument schemas and explicit ToolRejectedError use a separate max_tool_corrections allowance. Both remain in the current inference, preserve completed observations and consume existing budgets. See recoverable tool validation for structured rejection feedback.

Parsing and action-schema failures are recoverable inside infer(). ProtoLink emits an llm_parse_error event, retains bounded diagnostics for observability and adds a correction message to conversation history. When JSON decoding succeeded, the parser carries the decoded outer type separately from the rendered error. The retry can therefore distinguish a malformed agent_call, malformed tool_call, invalid final, unknown action type, or object with no recognizable outer type without scraping its own diagnostic text.

If the configured correction limit is exhausted, infer() raises InferParseError. It identifies the failed inference step and number of consecutive attempts, gives a short explanation, keeps the final parser error and preserves the exact final model response. An empty Ollama response is reported as <empty> rather than appearing as a blank diagnostic.

The correction also uses the current runtime capabilities. If the model selected agent_call but this inference has no delegation callback, ProtoLink says that the action cannot be dispatched and does not show Agent-call examples. If local tools or delegation are available, their canonical action shapes are included. ProtoLink does not silently convert an explicit malformed side-effect request into final content.

Your previous response was not a dispatchable ProtoLink action.
Validation feedback:
Action validation failed. Field-level errors:
  - Field 'agent_call -> action': Input should be 'tool_call' or 'infer'

The response selected `agent_call`, but no agent delegation route is available
for this inference, so that action cannot be dispatched.
Return exactly one JSON object using a currently dispatchable action:
- `final`: {"type":"final","content":"..."}

That is representative rather than a fixed template: field details and available alternatives depend on the rejected payload, local tools and delegation callback. A structurally valid tool_call or agent_call that reaches the dispatcher but names an unavailable capability receives similarly explicit correction. It is not counted as a parse failure because its action envelope was valid; the overall inference-step limit still prevents an endless correction loop.

max_parse_failures controls the consecutive failure circuit breaker. It defaults to 3, accepts integers from 1 through 10 and resets after a successfully validated action:

from protolink import create_llm

llm = create_llm(
    "ollama",
    base_url="http://localhost:11434",
    model="qwen3",
    max_parse_failures=4,
)

Pass this option directly to create_llm(), or assign llm.max_parse_failures after construction. Do not put it in model_params: those values are provider generation settings and are forwarded to the selected backend, while the parse limit is ProtoLink runtime configuration.

The parse limit counts consecutive model action proposals that fail JSON decoding or action validation. It does not configure transient HTTP/provider retries and it does not validate an application-specific schema inside final text. Those are separate boundaries whose retry budgets can multiply. Raising the limit can help a smaller model self-correct, but repeating a deterministic shape error is better handled by a lossless normalization or a clearer prompt than by unlimited retries.

from protolink import InferParseError

try:
    result = await llm.infer(query="Review the evidence.", tools={})
except InferParseError as exc:
    print(f"Failed after {exc.attempts} invalid proposals at step {exc.step}")
    print(exc.explanation)
    print(f"Last model response: {exc.raw_response!r}")
    print(f"Parser detail: {exc.last_error}")

raw_response is the exact final provider text: "" means the provider returned no content, while None means an adapter could not expose the text. The rendered exception keeps the existing bounded diagnostic preview, but the exact attribute is not truncated. Treat it as potentially sensitive model output when logging or exporting it.

The action envelope is not the application schema

FinalAction.content is user-facing text. ProtoLink validates the outer action but cannot know whether that text must also satisfy a domain-specific JSON schema. If an application expects a decision, invoice, ballot, or other structured result, it should validate part.content separately and apply only domain-safe recovery:

from pydantic import BaseModel, Field


class Decision(BaseModel):
    label: str
    confidence: float = Field(ge=0, le=1)


part = await llm.infer(
    query="Return the requested decision as JSON in final content.",
    tools={},
)
decision = Decision.model_validate_json(part.content)

Do not infer missing high-impact fields from prose merely to make validation pass. A harmless public statement may permit a text fallback; a payment instruction, tool target, categorical decision, or authorization result normally should fail closed and request a new structured response.

Parsing validates action shape, not semantic truth or authorization

A hallucinated tool call can be perfectly valid JSON with the correct field types. Parsing proves that the requested action is unambiguous and structurally valid; it does not prove that the action is factually correct, appropriate, or authorized. Keep tools narrowly allowlisted, use strict argument schemas and capability policies, require approval for important operations, enforce budgets and idempotency and validate external identifiers at the execution boundary.

3. Self-correcting error recovery

Many model mistakes are observations, not immediate application failures:

Error type	Runtime response
Unknown or unavailable tool	Lists tools that are actually available, or states that this inference has none.
Missing required fields	Returns field-level validation details.
Wrong tool arguments	Explains the callable mismatch and asks the model to check the input schema.
Agent not found	Rejects a target absent from the current advertised catalog and lists available agents.
Unknown delegated tool	Lists the target's advertised tools; a reasoning agent receives guidance to use infer with a prompt.
Wrong delegated tool arguments	Validates the advertised input schema before contacting the target.
Unsupported delegated inference	Rejects a target declaring supports_inference=False and suggests its tools or another agent.
Direct URL or delegation cycle	Rejects the unsafe target and asks for another advertised agent or action.
Agent delegation unavailable	States that no delegation route exists and suggests only dispatchable alternatives.
Invalid action type	Reports the detected type and lists only action kinds available in the current inference.
Explicit ToolRejectedError	Returns structured feedback when the rejection guarantees no changes.

Tool arguments are validated and conservatively coerced before authorization, dispatch or tool-budget consumption. These model mistakes normally remain inside infer() so the model can correct them. An ordinary exception raised by the tool body, including TypeError, remains an execution failure and is not relabeled as an argument mismatch.

Delegation capability checks

Delegation has two explicit modes: infer asks an agent to handle a prompt, while tool_call selects a tool advertised by that agent. tool_call is an action kind, not a built-in tool named tool_call. A Planner with an LLM and no tools should receive:

{"type":"agent_call","agent":"planner","action":"infer","prompt":"Add docstrings to the following code: ..."}

If the model instead selects planner.tool_call, ProtoLink rejects that unsupported tool before dispatch and returns an agent_result containing structured agent_rejection feedback. The next model turn receives the target name, available tools, error code and suggestion to use infer. The runtime does not reinterpret the action, move arguments into a guessed prompt or contact the Planner with an invalid tool task.

This is automatic for ordinary Agents. Registry targets and configured local subagents use the same check. A supplied agent_cards catalog also enables it for direct LLM.infer() integrations; cards may be AgentCard objects or native serialized card dictionaries. Exact names, skill IDs and advertised argument schemas define the available requests. Empty input schemas add no constraints. Known registry cards are checked again at dispatch using the normal discovery cache, so a changed catalog can reject a request before sending it. Discovery data never grants permission: existing policy and approvals still authorize every valid operation.

No new API or setting is needed. max_tool_corrections covers local tool validation, delegation capability checks, unavailable routes and explicit no-change rejections together. It defaults to two corrective turns; successful work does not replenish it. Rejected requests consume their existing model/step/token budgets, preserve observations of completed work and are not remembered as successfully executed actions. Pre-dispatch rejection does not reserve a tool call or create a child task. Once a request reaches a receiver, its reserved budgets remain counted even if it returns a no-change rejection. See the runnable offline delegation-correction example.

Inference requests use capabilities.supports_inference: False rejects infer before dispatch with the code unsupported_inference, while True supports both LLM agents and custom prompt handlers. Ordinary Agents advertise this automatically from their configured LLM; a temporary connection failure does not change that request contract. Custom handlers can declare support in their card without installing a model. See custom prompt handlers and the delegation pipeline.

Missing or None support remains unknown for legacy/external cards and undeclared custom handlers; those requests are left to the target for compatibility. has_llm supplies only a positive correction hint when support is unknown. An explicit False takes precedence over that hint. Custom callbacks without a catalog can still use agent_cards=None; their own dispatch layer remains responsible for capability validation.

After dispatch, automatic recovery is deliberately narrow: a failed native Task must contain exactly one requested tool call and one matching tool_output.error with status="rejected" and changes_made=false, and no completed execution evidence. The parent receives that rejection as correction feedback. Failed inference, mixed requests, partial execution, untyped legacy errors, input-required responses and lost connections remain failures. The runtime never classifies them by matching error text. Ordinary callback exceptions, including ValueError, are terminal; custom callbacks use ToolRejectedError only when they guarantee no changes.

agent_call_rejected normalizes to action.rejected, with warning severity, correction counters and a no-change summary. It records an action_rejection artifact rather than successful execution evidence. Durable local child calls retain explicit rejection receipts and the same correction counter, avoiding unresolved execution state when the child has guaranteed that its rejected tool request made no changes.

When an Agent-owned inference tool or delegation succeeds, the Agent also attaches and immediately snapshots an Artifact(kind="action_result") receipt keyed by the runtime action_id. The receipt records completion and correlation metadata but deliberately omits the internal result, which remains in private LLM history. If a later step fails, exceeds budget, or is canceled, the Task and optional RunStore snapshot still show what already happened without exposing model-only observations or inviting an unsafe blind retry. Non-JSON or circular results receive a bounded serialization fallback in that private history so the completion observation cannot be lost.

4. Physical retry safety

Transient provider failures use exponential backoff and jitter. Each physical attempt consumes the same LLM-call and input-token budget as an initial request and produces structured attempt metadata. Non-streaming calls may retry up to the configured retry limit; streaming calls may retry only before the first output chunk is exposed, preventing duplicated or discontinuous client output.

5. Bounded execution

By default, MAX_INFER_STEPS limits an inference loop to ten model decisions. If the model never produces a final action, the method raises RuntimeError with diagnostic context instead of looping indefinitely.

Agent(yield_after_steps=N, durability=...) opts into cooperative work slices. The low-level infer(..., yield_after_steps=N) parameter requires an active durable execution scope; configure it on the Agent for ordinary application use. At a settled boundary, it checkpoints a RunHandoff and returns input-required. Correction attempts count as model steps too. Explicit resume_task() grants the next slice and retains cumulative steps, observations, deduplication history and correction allowances. If RunBudget.max_steps is supplied with this option, it becomes the finite total ceiling and may exceed ten; otherwise ten remains the total ceiling across slices. Other hard budgets and cancellation still stop the run. See worker execution.

Observers do not control execution

Inference event_callback failures are logged and the observer is disabled for the remainder of that infer call. A telemetry or UI exporter cannot turn an otherwise successful provider or tool operation into an application failure.

Debugging inference loops

If “maximum inference steps exceeded” appears frequently:

1. make the completion condition explicit in the system prompt;
2. split a broad task into smaller agent or flow steps;
3. improve tool names, descriptions and input schemas;
4. observe normalized inference events to inspect each decision; and
5. verify that tool observations give the model enough information to finish.

Tool-call handling

Tool use has two separate phases:

1. Action acquisition: the adapter obtains one model decision and returns LLMActionResult.
2. Observation injection: after ProtoLink executes the tool, the adapter adds the result to history so the model can continue.

def call_action(...) -> LLMActionResult:
    """Return one validated action for the current inference step."""

async def call_action_stream(...) -> LLMActionResult:
    """Return one validated action from a streaming inference step."""

def _inject_tool_call(
    self,
    *,
    tool_name: str,
    tool_args: dict,
    tool_result: Any,
) -> None:
    """Inject the runtime observation after a tool has executed."""

The base implementation asks for JSON, validates it and injects a provider-neutral observation. Native adapters override action acquisition and observation injection where their API requires provider-specific tool-call IDs or message roles, but all paths converge before runtime dispatch.

Provider-specific action modes

The table shows existing defaults. Explicit tool_calling settings override those defaults as described above.

Provider	Non-streaming infer()	Streaming infer()	Notes
OpenAI	Native Responses function tools	Native streamed function-call events	Parallel tool calls are disabled so the runtime receives one action at a time.
Anthropic	Native tool_use blocks	Native streamed input_json_delta tool input	System instructions come from the task-local history; parallel tool calls are rejected.
Gemini	Native function declarations	Native streamed function-call parts	Function-call parts are normalized into ProtoLink actions.
DeepSeek	Native Chat Completions tools when supports_tool_calling=True	Native streamed tool deltas when enabled	Set the flag to False to force JSON action mode.
Grok	Native Chat Completions tools when supports_tool_calling=True	Native streamed tool deltas when enabled	Set the flag to False to force JSON action mode.
Ollama	JSON by default; native tools when explicitly enabled	JSON by default; native tool events when explicitly enabled	Keeps small/local model behavior simple unless tool support is known to work.
llama.cpp server/local	JSON by default; native tools when explicitly enabled	JSON by default; native tool events when explicitly enabled	Reliability depends on the selected model and chat template.
LM Studio / vLLM / OpenAI-compatible	JSON by default; native tools when explicitly enabled	JSON by default; native tool events when explicitly enabled	Dedicated subclasses provide conventional defaults and provider identity; the generic adapter covers LocalAI and compatible custom servers.
Hugging Face	JSON action fallback	Streamed JSON action text	The complete response is validated before dispatch.

Prompt selection

ProtoLink uses two prompt families:

• JSON prompt: describes final, tool_call and agent_call objects. It is the compatibility path for small and local models.
• Native prompt: tells the model to use the provider tool interface. It intentionally omits JSON action examples so native providers do not receive two conflicting tool protocols.

For streaming inference, prompt selection follows the adapter capability:

action_mode = "native" if llm.supports_native_action_stream else "json"

Native tool calls are not ordinary text. A provider may stream text deltas, function-argument fragments, or SDK objects. call_action_stream() hides those shapes and returns one complete typed action to the shared loop.

Agent delegation tools

Native providers receive synthetic delegation tools only when both conditions are true:

• the current runtime can dispatch agent calls; and
• discovered agent cards provide valid targets.

This avoids advertising a callable delegation interface that cannot succeed. JSON action mode can still produce an agent_call; if delegation is unavailable, the runtime injects corrective feedback instead of performing a side effect.

Design rationale

The layered design keeps the runtime strict without making every provider use the same wire protocol:

• LLM.infer() dispatches one typed action at a time.
• Provider adapters own provider-specific request and stream parsing.
• Small and local models keep a compact JSON protocol by default.
• On the Agent-prepared path, native providers use their real tool API without receiving JSON tool instructions.
• Every path converges on FinalAction, ToolCallAction and AgentCallAction.
• Policy, approval, cancellation, budgets, retries, events and execution remain outside the model.

Inference example

from protolink import Agent, AgentCard, create_llm

agent = Agent(
    AgentCard(
        name="weather-assistant",
        description="Answers weather questions",
        url="runtime://weather-assistant",
    ),
    transport="runtime",
    llm=create_llm("openai", model="gpt-4o-mini"),
)


@agent.tool(name="weather", description="Return the weather for a location")
async def weather(location: str) -> str:
    return f"The weather in {location} is sunny."


answer = await agent.invoke("What's the weather in Tokyo?")
print(answer)

In JSON action mode, the intermediate model response must be exactly one supported object:

{
  "type": "tool_call",
  "tool": "weather",
  "args": {"location": "Tokyo"}
}

After observing the tool result, the model completes with:

{
  "type": "final",
  "content": "The weather in Tokyo is sunny."
}

LLM.infer

async methodprotolink.llms.base.LLM.infer

source↗

async infer(
  *,
  query: str,
  tools: dict[str, BaseTool],
  agent_callback: Callable[[str, str, dict[str, Any]], Awaitable[Any]] | None = None,
  agent_cards: list[Any] | None = None,
  streaming: bool = False,
  event_callback: Callable[[dict[str, Any]], Awaitable[None]] | None = None,
  event_metrics: bool | None = None,
  expose_tool_results: bool = False,
  action_authorizer: Callable[[RunAction], Awaitable[ActionAuthorization]] | None = None,
  cancellation_token: CancellationToken | None = None,
  run_context: RunContext | dict[str, Any] | None = None,
  budget_policy: BudgetPolicy | None = None,
  budget_enforcer: BudgetEnforcer | None = None,
  context_policy: ContextPolicy | None = None,
  hooks: tuple[AgentHooks, ...] = (),
  tool_prompt_builder: Callable[[dict[str, BaseTool]], None] | None = None,
) -> Part

Run the controlled multi-step inference loop used by Agent. The model declares typed intent; ProtoLink validates and executes tools or agent calls, feeds observations back to the model and stops on a final action or safety limit.

Configure context policy and lifecycle hooks on the Agent for integrated scoped retrieval, roster prompt regeneration and durable recovery. Direct LLM.infer() callers can pass the corresponding extension objects, but must supply an artifact scope/retrieval tool when observation offloading is needed. The Agent supplies tool_prompt_builder to regenerate its compiled tool prompt after roster filtering; it is a runtime integration callback rather than an application configuration field.

Parameters

querystrrequired

User task added to the active conversation history.

toolsdict[str, BaseTool]required

Executable tools available during this run.

agent_callbackCallable[[str, str, dict[str, Any]], Awaitable[Any]] | Nonedefault: None

Async dispatcher for delegated tool_call and infer actions. Without it, attempted delegation receives bounded correction feedback. Explicit ToolRejectedError reports a guaranteed no-change rejection; ordinary callback exceptions remain terminal.

agent_cardslist[Any] | Nonedefault: None

Advertised targets exposed to the model and checked before dispatch, including tool IDs and input schemas. Accepts AgentCard objects or native serialized mappings. None preserves callback integrations without an authoritative catalog. See delegation capability checks.

streamingbooldefault: False

Acquire each model action from the streaming adapter path.

event_callbackCallable[[dict[str, Any]], Awaitable[None]] | Nonedefault: None

Async observer for normalized chunks, actions, tool events, delegation events, budget decisions, errors and final output. The observer is non-authoritative; its first exception is logged and disables further callbacks for this infer call.

event_metricsbool | Nonedefault: None

Controls whether an attached event callback activates optional per-call metrics. Direct callers retain the existing default behavior; Agent sets this to false when its callback exists only to maintain internal completion receipts.

expose_tool_resultsbooldefault: False

Emit a JSON-safe snapshot of the actual tool return before hooks or context policies transform model history. The Agent forwards its corresponding opt-in and continues to sanitize knowledge passages. See completed tool evidence for visibility rules.

action_authorizerCallable[[RunAction], Awaitable[ActionAuthorization]] | Nonedefault: None

Policy boundary invoked after validation and before any tool or delegated-agent side effect.

cancellation_tokenCancellationToken | Nonedefault: None

Live token checked before provider calls and action dispatch.

run_contextRunContext | dict[str, Any] | Nonedefault: None

Correlation, session and budget context for the run.

budget_policyBudgetPolicy | Nonedefault: None

Allow, warn, or deny policy used by the built-in budget enforcer.

budget_enforcerBudgetEnforcer | Nonedefault: None

Existing stateful enforcer shared across several inference or tool operations. Omit it for an independent direct call; Agent supplies its task-local enforcer when the override accepts this keyword.

Returns

partPart

A part with type infer_output and the final user-facing content.

Raises

ToolCorrectionLimitError

Raised when another tool rejection exceeds max_tool_corrections. Exposes the target tool, rejected-request count, configured limit and final ToolRejectedError feedback.

InferParseError

Raised when the configured number of consecutive model responses cannot be decoded or validated as ProtoLink actions. Exposes attempts, step, explanation, raw_response and last_error.

RuntimeError

Raised after ten steps without a final action, or when an unrecoverable provider, tool, or delegated-agent failure is wrapped by the loop.

BudgetExceededError

Raised when the configured run budget denies further work.

ApprovalRequiredError | ActionDeniedError

Raised when runtime authorization requires approval or denies an action.

asyncio.CancelledError

Raised when the run is cancelled.

Self-correction

Unknown tools, malformed actions, missing agents and argument mismatches are normally reported back to the model for correction. Explicit ToolRejectedError also returns feedback when the tool guarantees no changes. Tool validation uses max_tool_corrections; action parsing uses max_parse_failures. Ordinary tool exceptions abort execution. See tool rejection for the complete contract.

Direct-call responsibility

Agent normally rebuilds the system prompt before calling this method. A custom runtime that invokes infer() directly must prepare matching tool and Agent descriptions with build_system_prompt(); supplying tools here only provides executables to the loop.

Override compatibility

The budget_enforcer parameter is optional. When an Agent uses a custom LLM.infer() override, it supplies the shared enforcer only if the override declares that keyword or accepts arbitrary keyword arguments.

Examples

# Safe direct use when no tools or delegated Agents need to be advertised.
result = await llm.infer(
    query="Summarize the supplied context.",
    tools={},
)

print(result.content)

LLM.sync.infer

methodprotolink.llms.base.SyncLLM.infer

source↗

llm.sync.infer(
  *,
  query: str,
  tools: dict[str, BaseTool],
  agent_callback: Callable | None = None,
  agent_cards: list[Any] | None = None,
  streaming: bool = False,
  event_callback: Callable | None = None,
) -> Part

Blocking wrapper around LLM.infer() for scripts and synchronous command-line programs.

Parameters

querystrrequired

User task passed to the asynchronous inference loop.

toolsdict[str, BaseTool]required

Tools available during inference.

agent_callbackCallable | Nonedefault: None

Optional async agent dispatcher.

agent_cardslist[Any] | Nonedefault: None

Discovered agents available for delegation.

streamingbooldefault: False

Use the underlying streaming action path.

event_callbackCallable | Nonedefault: None

Optional async event observer.

Returns

partPart

Final infer_output part.

Event loops

This wrapper uses asyncio.run(). Do not call it from FastAPI handlers, asynchronous notebook cells, or any other active event loop; await llm.infer() there instead.

LLM.use_history

context managerprotolink.llms.base.LLM.use_history

source↗

use_history(
  history: ConversationHistory,
) -> ContextManager[ConversationHistory]

Temporarily bind a conversation history to the current execution context. The binding uses contextvars, so concurrent asyncio tasks can share one LLM instance without sharing mutable messages.

Parameters

historyConversationHistoryrequired

History returned by llm.history inside the context.

Yields

historyConversationHistory

The same object supplied by the caller.

Raises

TypeError

Raised when history is not a ConversationHistory.

Examples

from protolink.llms.history import ConversationHistory

customer_history = ConversationHistory("You support account customer-42.")

with llm.use_history(customer_history):
    response = llm.chat("What do you remember about this account?")

LLM.compact_history

methodprotolink.llms.base.LLM.compact_history

source↗

compact_history(
  strategy: Literal["recent", "tokens", "summary"] = "recent",
  *,
  max_messages: int = 20,
  max_tokens: int = 4000,
  preserve_recent: int = 6,
  summary_max_tokens: int = 512,
) -> HistoryCompactionResult

Compact the active history in place. recent and tokens are local operations; summary makes one isolated synchronous model call before replacing older messages.

Parameters

strategy"recent" | "tokens" | "summary"default: "recent"

recent retains a message window, tokens retains the newest suffix near a soft estimated-token ceiling and summary replaces older messages with model-generated durable context.

max_messagesintdefault: 20

Maximum retained messages for the recent strategy, including a leading system prompt.

max_tokensintdefault: 4000

Soft estimated-token ceiling for the tokens strategy. Protected messages may exceed it.

preserve_recentintdefault: 6

Newest non-system messages protected by tokens and summary.

summary_max_tokensintdefault: 512

Requested maximum length of a generated summary.

Returns

reportHistoryCompactionResult

Before/after message counts, estimated token counts, removed-message count, selected strategy and whether a summary was created.

Raises

ValueError

Raised for an unknown strategy, invalid limits, or an empty summary result.

provider error

Errors from the isolated summary call propagate when strategy="summary".

Equivalent API

llm.compactor.compact(...) has the same signature and behavior.

Examples

report = llm.compact_history("recent", max_messages=20)
report = llm.compact_history("tokens", max_tokens=8_000, preserve_recent=6)
report = llm.compact_history("summary", preserve_recent=8, summary_max_tokens=600)

print(report.to_dict())

LLM.configure_metrics

methodprotolink.llms.base.LLM.configure_metrics

source↗

configure_metrics(
  profile: LLMModelProfile | dict[str, Any] | None = None,
  *,
  context_window: int | None = None,
  input_cost_per_million: float | None = None,
  output_cost_per_million: float | None = None,
  currency: str = "USD",
  enabled: bool = True,
) -> LLM

Attach application-owned model limits and prices used for observational inference metrics.

Parameters

profileLLMModelProfile | dict[str, Any] | Nonedefault: None

Complete profile object or equivalent mapping. When supplied, it takes precedence and the individual context/cost arguments are ignored.

context_windowint | Nonedefault: None

Model context window used to calculate pressure and remaining-token estimates.

input_cost_per_millionfloat | Nonedefault: None

Input-token price per one million tokens.

output_cost_per_millionfloat | Nonedefault: None

Output-token price per one million tokens.

currencystrdefault: "USD"

Currency label attached to calculated costs.

enabledbooldefault: True

Whether inference may emit metrics events.

Returns

selfLLM

The same LLM instance for fluent configuration.

Observational only

Metrics do not change provider payloads, retry behavior, or responses. They are emitted by infer() when an event observer or telemetry backend is attached.

Prompt architecture

ProtoLink keeps its prompt families in protolink/llms/prompts and chooses between them according to the action-acquisition mode. The runtime deliberately avoids giving a model two tool-calling contracts at once.

The system-prompt blueprint

LLM.build_system_prompt() assembles the prompt used by infer(). In JSON mode it describes the portable action objects. In native mode it tells the model to use the provider tool interface and leaves function-call syntax to the backend SDK or API.

By default, building a prompt resets history to the new system message. Passing persist=True updates the system message while retaining the rest of the current conversation, which is essential for long-lived sessions.

The final prompt is composed from:

1. Base instructions

   • Define the model's role inside a deterministic runtime.
   • Make it clear that the model requests actions rather than pretending to execute them.
   • Select BASE_SYSTEM_PROMPT for JSON mode or NATIVE_SYSTEM_PROMPT for native mode.

2. Tool instructions

   • JSON mode injects the available tool descriptions and the tool_call object format.
   • Native mode adds a short instruction to use the provider tool interface; concrete schemas travel in the provider request.

3. Agent capabilities

   • JSON mode describes the agent_call object and discovered targets.
   • Native mode exposes synthetic delegation functions only when dispatch is available and agent cards exist.

4. Flow context

   • Pipelines, routers and graphs can inject topology-aware instructions for the current step.
   • The model receives only the semantic context it needs for the active flow position.

5. Application instructions

   • Your domain-specific prompt, such as “You are a coding assistant.”
   • Appended to the shared runtime rules unless override_system_prompt=True.

Tool and discovered-Agent metadata is serialized as deterministic valid JSON with stable ordering and explicit capabilities. The surrounding prompt labels those descriptions, schemas and examples as untrusted data rather than executable instructions. This avoids Python-repr syntax and brace-escaping artifacts while keeping prompt caching and smaller-model parsing predictable.

Reasoning versus execution

When infer() runs, the prompt makes the LLM a reasoning and action-selection engine while ProtoLink remains the executor:

1. Input: the model receives the task, history, tools, agents and relevant flow context.
2. Selection: it chooses final, tool_call, or agent_call.
3. Structured output: it returns JSON or uses the provider-native tool channel.
4. Validation: ProtoLink converts the result into a typed action.
5. Execution: the runtime applies policy and performs the actual Python call or agent dispatch.
6. Observation: the result is returned to the model for its next decision.

{
  "type": "tool_call",
  "tool": "get_weather",
  "args": {"location": "Geneva"}
}

This separation of reasoning from execution is what allows one inference loop to support hosted APIs, local servers and in-process models without handing runtime authority to untrusted model output.

LLM.build_system_prompt

methodprotolink.llms.base.LLM.build_system_prompt

source↗

build_system_prompt(
  user_instructions: str | None = None,
  agent_cards: str | None = None,
  tools: str | None = None,
  *,
  action_mode: Literal["json", "native"] | None = None,
  flow_instructions: str | None = None,
  override_system_prompt: bool = False,
  persist: bool = False,
  agent_name: str | None = None,
) -> str

Build and store the complete runtime system prompt from base instructions, action mode, tools, discovered agents, flow context and application instructions.

Parameters

user_instructionsstr | Nonedefault: None

Application instructions appended to the shared runtime prompt, or used as the complete prompt when override_system_prompt=True.

agent_cardsstr | Nonedefault: None

Serialized discovered-agent descriptions used for delegation guidance.

toolsstr | Nonedefault: None

Serialized tool descriptions used by the selected action prompt.

action_mode"json" | "native" | Nonedefault: None

Explicit action acquisition mode. When omitted, the adapter’s uses_native_action_prompt property selects the mode.

flow_instructionsstr | Nonedefault: None

Optional pipeline, router, or graph context.

override_system_promptbooldefault: False

Replace the shared runtime template with user_instructions.

persistbooldefault: False

Preserve non-system history while updating the system message. The default clears history and resets it to the newly built system prompt.

agent_namestr | Nonedefault: None

Current registered agent name used to prohibit self-delegation.

Returns

promptstr

The assembled prompt, also stored on llm.system_prompt.

History mutation

With persist=False, this method removes existing conversation turns. Use persist=True to retain them.

LLM.set_system_prompt

methodprotolink.llms.base.LLM.set_system_prompt

source↗

set_system_prompt(
  system_prompt: str,
) -> None

Assign a new value to llm.system_prompt.

Parameters

system_promptstrrequired

Replacement prompt string.

Returns

NoneNone

This method mutates the adapter.

Important

This setter does not rewrite the system message already stored in llm.history. Use build_system_prompt(..., override_system_prompt=True) when history and prompt state must be updated together.

LLM.validate_connection

abstract methodprotolink.llms.base.LLM.validate_connection

source↗

validate_connection() -> bool

Check whether the provider client, server, or local model can respond.

Returns

connectedbool

True when adapter-specific validation succeeds; otherwise usually False.

Construction

Current concrete adapters already call validation during initialization. Most implementations catch validation failures, log them and return False rather than failing construction.

API providers

Hosted-provider adapters read credentials from their conventional environment variable when api_key is omitted. They all implement direct call(), call_stream() and validate_connection() methods and inherit chat(), history management, compaction, metrics and the controlled infer() loop.

OpenAI, Anthropic and Gemini always acquire actions through their provider-native function interface. DeepSeek and Grok use native Chat Completions tools by default but can be forced into portable JSON mode. Hugging Face supports direct and streamed text and uses the portable JSON-action protocol for inference.

• OpenAI - OpenAILLM, default model gpt-4o-mini, credential OPENAI_API_KEY.
• Anthropic - AnthropicLLM, default model claude-sonnet-4-20250514, credential ANTHROPIC_API_KEY.
• Google Gemini - GeminiLLM, default model gemini-3-flash-preview, credential GEMINI_API_KEY.
• DeepSeek - DeepSeekLLM, default model deepseek-chat, credential DEEPSEEK_API_KEY.
• Grok - GrokLLM, default model grok-4-latest, credential XAI_API_KEY or GROK_API_KEY.
• Hugging Face - HuggingFaceLLM, explicit model recommended, credential HF_API_TOKEN.

OpenAILLM

classprotolink.llms.api.OpenAILLM

source↗

class OpenAILLM(
  *,
  api_key: str | None = None,
  model: str | None = None,
  model_params: dict[str, Any] | None = None,
  base_url: str | None = None,
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

OpenAI Responses API adapter with native function tools and native streamed tool-call events. Use it for the official OpenAI service or for a custom base_url that implements the Responses API, not merely Chat Completions.

Direct calls translate ConversationHistory into Responses input and extract text from the returned response. In inference mode, real function declarations are sent to the provider, parallel tool calls are disabled and returned function calls are normalized into ProtoLink actions before the runtime executes them. Streaming follows the same contract while forwarding text deltas and buffering function arguments until they form one complete action.

Failed or incomplete responses and stream error events raise RuntimeError, including after partial text has already been emitted. Native responses containing multiple tool calls are rejected instead of selecting one.

Parameters

api_keystr | Nonedefault: None

OpenAI API key. Falls back to OPENAI_API_KEY.

modelstr | Nonedefault: None

Model identifier. None resolves to gpt-4o-mini.

model_paramsdict[str, Any] | Nonedefault: None

Values merged over temperature=1.0, top_p=1.0, top_logprobs=None and truncation="disabled".

base_urlstr | Nonedefault: None

Optional OpenAI client base URL. The endpoint must implement the Responses API; use OpenAICompatibleLLM for Chat Completions-only servers.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Raises

ImportError

Raised when the OpenAI SDK is not installed.

OpenAI client error

Missing credentials and request errors originate from the SDK.

Examples

from protolink.llms.api import OpenAILLM

llm = OpenAILLM(
    model="gpt-4o-mini",
    model_params={"temperature": 0.3, "max_output_tokens": 800},
)

AnthropicLLM

classprotolink.llms.api.AnthropicLLM

source↗

class AnthropicLLM(
  *,
  api_key: str | None = None,
  model: str | None = None,
  model_params: dict[str, Any] | None = None,
  base_url: str | None = None,
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

Anthropic Messages API adapter with native tool_use actions and streamed tool-input deltas. The adapter derives both the separated system prompt and conversational messages from the supplied task-local ConversationHistory, converts ProtoLink tools to Anthropic tool schemas and keeps the provider's tool-use identifier in action metadata.

After ProtoLink executes a requested tool, the adapter uses that identifier to inject the observation in the shape expected by the Messages API. Both streaming and non-streaming inference therefore share the same public LLMActionResult even though Anthropic's wire representation differs from OpenAI's. The runtime accepts one action per step and rejects multiple parallel tool_use blocks instead of silently selecting or merging them.

Parameters

api_keystr | Nonedefault: None

Anthropic API key. Falls back to ANTHROPIC_API_KEY.

modelstr | Nonedefault: None

Claude model identifier. None resolves to claude-sonnet-4-20250514.

model_paramsdict[str, Any] | Nonedefault: None

Values merged over max_tokens=1024. Sampling uses the selected model's defaults unless explicitly configured. For compatibility with Anthropic SDK v1, explicit temperature, top_p and top_k values are forwarded through extra_body for models that support them. Matching keys already supplied in extra_body take precedence.

base_urlstr | Nonedefault: None

Optional Anthropic-compatible API base URL passed to the SDK.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Examples

from protolink.llms.api import AnthropicLLM

llm = AnthropicLLM(
    model="claude-sonnet-4-20250514",
    model_params={"max_tokens": 2048},
)

GeminiLLM

classprotolink.llms.api.GeminiLLM

source↗

class GeminiLLM(
  *,
  api_key: str | None = None,
  model: str | None = None,
  model_params: dict[str, Any] | None = None,
  base_url: str | None = None,
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

Google GenAI adapter with native function declarations and native streamed actions. It converts conversation messages and tool schemas into Google GenAI content, generation configuration and function declarations.

Function-call parts are normalized into ToolCallAction or AgentCallAction before dispatch. Text-only responses become final actions, so application and Agent code sees the same result types used by every other provider.

Parameters

api_keystr | Nonedefault: None

Google API key. Falls back to GEMINI_API_KEY.

modelstr | Nonedefault: None

Gemini model identifier. None resolves to gemini-3-flash-preview.

model_paramsdict[str, Any] | Nonedefault: None

Generation configuration merged over temperature=1.0 and top_p=1.0.

base_urlstr | Nonedefault: None

Custom Gemini API root used for text and native-tool requests, including streaming. Omit it to use the SDK's default endpoint.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Examples

from protolink.llms.api import GeminiLLM

llm = GeminiLLM(model="gemini-3-flash-preview")

DeepSeekLLM

classprotolink.llms.api.DeepSeekLLM

source↗

class DeepSeekLLM(
  *,
  api_key: str | None = None,
  model: str | None = None,
  model_params: dict[str, Any] | None = None,
  base_url: str | None = "https://api.deepseek.com",
  supports_tool_calling: bool = True,
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

DeepSeek Chat Completions adapter implemented through the OpenAI SDK with DeepSeek's API root. It supports ordinary text calls, incremental content streams, native Chat Completions tool calls and streamed tool-argument deltas.

Native action acquisition is enabled by default. Set supports_tool_calling=False when the selected model behaves more reliably with ProtoLink's JSON action prompt; the surrounding inference loop, tool execution and return types remain unchanged.

Parameters

api_keystr | Nonedefault: None

DeepSeek API key. Falls back to DEEPSEEK_API_KEY.

modelstr | Nonedefault: None

Model identifier. None resolves to deepseek-chat.

model_paramsdict[str, Any] | Nonedefault: None

Values merged over temperature=1.0 and top_p=1.0.

base_urlstr | Nonedefault: "https://api.deepseek.com"

DeepSeek-compatible API root.

supports_tool_callingbooldefault: True

Use native Chat Completions tools. Set to False to force the portable JSON action fallback.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Examples

from protolink.llms.api import DeepSeekLLM

llm = DeepSeekLLM(model="deepseek-chat")

GrokLLM

classprotolink.llms.api.GrokLLM

source↗

class GrokLLM(
  *,
  api_key: str | None = None,
  model: str | None = None,
  model_params: dict[str, Any] | None = None,
  base_url: str | None = None,
  supports_tool_calling: bool = True,
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

xAI Chat Completions adapter using direct synchronous and asynchronous HTTP clients. It builds OpenAI-style message and tool payloads, parses content or tool calls and normalizes usage metadata when the response includes it.

Native tools and streamed tool deltas are enabled by default. Disable supports_tool_calling to use the portable JSON action protocol with a model or endpoint that cannot reliably follow the native function format.

Parameters

api_keystr | Nonedefault: None

xAI key. Falls back to XAI_API_KEY, then GROK_API_KEY.

modelstr | Nonedefault: None

Model identifier. None resolves to grok-4-latest.

model_paramsdict[str, Any] | Nonedefault: None

Values merged over temperature=1.0.

base_urlstr | Nonedefault: None

API root. None resolves to https://api.x.ai/v1.

supports_tool_callingbooldefault: True

Use native tool calls and streamed tool deltas. Set to False for JSON action mode.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Examples

from protolink.llms.api import GrokLLM

llm = GrokLLM(model="grok-4-latest")

HuggingFaceLLM

classprotolink.llms.api.HuggingFaceLLM

source↗

class HuggingFaceLLM(
  *,
  api_key: str | None = None,
  model: str | None = None,
  model_params: dict[str, Any] | None = None,
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

Hugging Face Inference API adapter for direct and streamed chat completions. It is useful when a hosted Hub model is available through the inference service and you want that model behind the same LLM interface.

Pass an explicit Hub model identifier. The adapter streams text deltas and uses JSON-action fallback for Agent inference; it does not send native tool declarations. The selected model must support chat completions and follow the JSON action prompt. Only part of model_params is forwarded by call() and call_stream().

Parameters

api_keystr | Nonedefault: None

Hugging Face token. Falls back to HF_API_TOKEN.

modelstr | Nonedefault: None

Hub model identifier. The effective built-in default is an empty string, so pass an explicit model for normal use.

model_paramsdict[str, Any] | Nonedefault: None

Values merged over max_new_tokens=512, temperature=1.0, top_p=1.0 and repetition_penalty=1.0. The current call() path forwards only temperature.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Streaming

chat(..., streaming=True) yields text deltas. Agent inference forwards the same deltas as llm_chunk events and parses one JSON action after the stream finishes.

Examples

from protolink.llms.api import HuggingFaceLLM

llm = HuggingFaceLLM(model="your-org/your-chat-model")

Server providers

Server adapters connect to a model process over HTTP, whether that process runs on the same machine or on remote infrastructure. The server owns model loading and hardware resources; the ProtoLink adapter owns history serialization, request construction, streaming, action normalization, connection validation and integration with the shared inference loop.

All server adapters inherit from ServerLLM. Their common configuration consists of a server URL, a model identifier understood by that server, optional generation parameters and the tool_calling runtime policy. The existing supports_tool_calling flag remains available. See action acquisition for automatic discovery, explicit overrides and provider-specific capability limits.

The inherited model_params property can be replaced with a dictionary, set_system_prompt() updates the adapter's prompt value and each concrete provider implements call(), call_stream() and validate_connection() for its endpoint.

OllamaLLM

classprotolink.llms.server.OllamaLLM

source↗

class OllamaLLM(
  *,
  base_url: str | None = None,
  headers: dict[str, str] | None = None,
  model: str | None = None,
  model_params: dict[str, Any] | None = None,
  supports_tool_calling: bool = False,
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

Client for Ollama's /api/chat endpoint. It serializes ConversationHistory into Ollama messages and supports ordinary responses, streamed chunks, usage normalization and optional native tool events.

JSON action mode remains the default. Use tool_calling="auto" to select native tools when the exact Ollama model advertises support, or choose "native" / "json" explicitly. Direct chat() and call() usage does not use this action-acquisition policy. The legacy supports_tool_calling flag applies when the new option is omitted.

The adapter uses Ollama's documented /api/chat endpoint. JSON action mode normally requests format="json" to constrain the server's output. Some runners, including MLX configurations, reject structured output with HTTP 501 and structured output is unavailable. For that specific rejection of the default format, ProtoLink retries once without the format field and remembers the limitation for the same server/model on that adapter. The inference prompt and action validation remain in place, so tools and delegation still use the standard action contract. This applies to ordinary and streaming calls; errors inside an accepted stream propagate without a format fallback.

Use model_params={"format": None} to omit server formatting from the outset. An explicit "json" or JSON schema is a required formatting contract and is never silently dropped. format, think and keep_alive are sent as top-level Ollama request controls; generation settings such as temperature, num_ctx and num_predict remain in options. The existing max_tokens alias maps to num_predict.

Parameters

base_urlstr | Nonedefault: None

Ollama server root. Falls back to OLLAMA_URL; if neither is supplied, construction raises ValueError.

headersdict[str, str] | Nonedefault: None

Headers forwarded to chat requests. When omitted, OLLAMA_API_KEY supplies a bearer token if set. Pass an explicit dictionary to override that environment-based authentication.

modelstr | Nonedefault: None

Ollama model name. None resolves to gemma4:e4b.

model_paramsdict[str, Any] | Nonedefault: None

Values merged over temperature=1.0, num_predict=8192 and num_ctx=8192. The format, think and keep_alive keys configure top-level request controls; other values configure generation options.

supports_tool_callingbooldefault: False

Opt into native Ollama tools. The default uses JSON action mode.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Raises

ValueError

Raised for a missing base URL, unsupported URL scheme, missing hostname, or unavailable client during a call.

RuntimeError

Raised for non-success Ollama responses or malformed response payloads.

Examples

from protolink.llms.server import OllamaLLM

llm = OllamaLLM(
    base_url="http://localhost:11434",
    model="qwen3",
)

For an installed Gemma MLX model, keep the default action mode and optionally disable thinking output:

from protolink import Agent

agent = Agent(
    name="helper",
    llm=OllamaLLM(
        base_url="http://127.0.0.1:11434",
        model="gemma4:12b-mlx",
        model_params={"think": False},
    ),
)
print(agent.sync.invoke("Say hello in one sentence."))

Use the exact tag installed on your server. The default-format fallback is triggered by the server response, rather than inferred from a model's name or family. Native tool support and structured-output support are independent capabilities.

LlamaCPPServerLLM

classprotolink.llms.server.LlamaCPPServerLLM

source↗

class LlamaCPPServerLLM(
  *,
  base_url: str | None = None,
  headers: dict[str, str] | None = None,
  model: str | None = None,
  model_params: dict[str, Any] | None = None,
  supports_tool_calling: bool = False,
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

Direct client for a llama-server OpenAI-style Chat Completions endpoint. It talks to the server over HTTP without loading a model in the ProtoLink process, making it suitable when model lifecycle and hardware allocation belong to a separate service.

The adapter supports direct and streamed text calls. Native tool declarations are opt-in because correctness depends on the loaded model, chat template and server build; JSON actions remain the compatibility default.

Parameters

base_urlstr | Nonedefault: None

Server root. Resolution order is the argument, LLAMACPP_SERVER_URL, then http://localhost:8080.

headersdict[str, str] | Nonedefault: None

Extra HTTP headers. Content-Type: application/json is added when absent.

modelstr | Nonedefault: None

Server model identifier. None resolves to gemma4:e4b.

model_paramsdict[str, Any] | Nonedefault: None

Values merged over temperature=1.0.

supports_tool_callingbooldefault: False

Opt into native Chat Completions tools for a compatible model/template.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Examples

from protolink.llms.server import LlamaCPPServerLLM

llm = LlamaCPPServerLLM(base_url="http://localhost:8080")

OpenAICompatibleLLM

classprotolink.llms.server.OpenAICompatibleLLM

source↗

class OpenAICompatibleLLM(
  *,
  base_url: str | None = None,
  api_key: str | None = None,
  headers: dict[str, str] | None = None,
  model: str | None = None,
  model_params: dict[str, Any] | None = None,
  supports_tool_calling: bool = False,
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

Generic client for servers exposing /v1/chat/completions and /v1/models, including LocalAI and compatible custom services. Use it when the endpoint follows the Chat Completions protocol but is not the official OpenAI Responses API. Prefer VLLMLLM or LMStudioLLM when their conventional URL, credential environment variables and provider identity are useful.

It supports custom headers, optional bearer authentication, direct and streamed content and opt-in native tools. The default JSON response format makes the adapter well suited to ProtoLink's portable action protocol, while supports_tool_calling=True switches action acquisition to provider-style tool payloads.

Parameters

base_urlstr | Nonedefault: None

Server root. Falls back to OPENAI_COMPATIBLE_BASE_URL, then http://localhost:1234/v1.

api_keystr | Nonedefault: None

Optional bearer token. Falls back to OPENAI_COMPATIBLE_API_KEY.

headersdict[str, str] | Nonedefault: None

Extra request headers merged with the JSON defaults and authorization header.

modelstr | Nonedefault: None

Model id passed to the server. None resolves to local-model.

model_paramsdict[str, Any] | Nonedefault: None

Values merged over temperature=1.0. Direct text calls also add response_format={"type": "json_object"} unless you override it.

supports_tool_callingbooldefault: False

Enable native Chat Completions tool payloads for a compatible server/model pair.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Examples

from protolink.llms.server import OpenAICompatibleLLM

llm = OpenAICompatibleLLM(
    base_url="http://localhost:8000/v1",
    model="Qwen/Qwen3-8B",
)

VLLMLLM

classprotolink.llms.server.VLLMLLM

source↗

class VLLMLLM(
  *,
  model: str,
  base_url: str | None = None,
  api_key: str | None = None,
  headers: dict[str, str] | None = None,
  model_params: dict[str, Any] | None = None,
  supports_tool_calling: bool = False,
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

Convenience specialization of OpenAICompatibleLLM for a separately managed vLLM server. It inherits the compatible adapter's direct and streamed Chat Completions requests, portable JSON action fallback, usage normalization, connection validation, custom headers, optional bearer authentication and opt-in native tools. The adapter itself does not install or import the vllm Python package.

The model is required because it must match the identifier accepted by the running server: normally the model passed to vllm serve, or a name configured with vLLM's --served-model-name option. The subclass supplies vLLM's conventional port and provider-specific environment variables and reports vllm in events and metrics.

Parameters

modelstrrequired

Model identifier exposed by the vLLM server. It must match the served model name.

base_urlstr | Nonedefault: None

Resolution order is the argument, VLLM_URL, then http://localhost:8000/v1.

api_keystr | Nonedefault: None

Optional bearer token. Falls back to VLLM_API_KEY; no placeholder credential is added when both are absent.

headersdict[str, str] | Nonedefault: None

Extra request headers merged with the inherited JSON defaults and optional authorization header.

model_paramsdict[str, Any] | Nonedefault: None

Generation parameters forwarded to vLLM and merged over temperature=1.0.

supports_tool_callingbooldefault: False

Opt into native Chat Completions tools only after the vLLM server and selected model are configured for automatic tool choice.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Native tool setup

The default portable JSON action mode needs no vLLM tool parser. Before setting supports_tool_calling=True, start vLLM with --enable-auto-tool-choice and a model-appropriate --tool-call-parser and ensure the selected model and chat template support tools.

Examples

from protolink.llms.server import VLLMLLM

# Start the model server separately: vllm serve Qwen/Qwen3-8B
llm = VLLMLLM(model="Qwen/Qwen3-8B")

LMStudioLLM

classprotolink.llms.server.LMStudioLLM

source↗

class LMStudioLLM(
  *,
  base_url: str | None = None,
  api_key: str | None = None,
  headers: dict[str, str] | None = None,
  model: str | None = None,
  model_params: dict[str, Any] | None = None,
  supports_tool_calling: bool = False,
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

Convenience specialization of OpenAICompatibleLLM for LM Studio. It keeps the complete compatible-server behavior while supplying LM Studio's conventional URL, credential fallback and provider identity.

Use the generic parent class when you want environment variables and labels that are not tied to LM Studio. Use this subclass when local development should work with LM Studio's normal defaults and appear as lmstudio in events and metrics.

Parameters

base_urlstr | Nonedefault: None

Resolution order is the argument, LMSTUDIO_URL, then http://localhost:1234/v1.

api_keystr | Nonedefault: None

Optional token. Falls back to LMSTUDIO_API_KEY, then the local placeholder lm-studio.

headersdict[str, str] | Nonedefault: None

Extra request headers.

modelstr | Nonedefault: None

Model id selected in LM Studio. Inherited default: local-model.

model_paramsdict[str, Any] | Nonedefault: None

Generation parameters forwarded to the compatible server.

supports_tool_callingbooldefault: False

Opt into native tools when the selected model supports them.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Examples

from protolink.llms.server import LMStudioLLM

llm = LMStudioLLM(model="local-model")

Local provider

Local adapters run inference inside the Python host rather than transmitting prompts to a server. This offers complete control over model files and data movement, but it also makes the application responsible for compatible native libraries, model loading, memory use, acceleration and process stability.

LlamaCPPLocalLLM

classprotolink.llms.local.llamacpp_client.LlamaCPPLocalLLM

source↗

class LlamaCPPLocalLLM(
  *,
  model: str,
  model_params: dict[str, Any] | None = None,
  supports_tool_calling: bool = False,
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

In-process llama-cpp-python adapter for a local GGUF model file. Unlike LlamaCPPServerLLM, it loads the model in the current Python process, so model initialization time, native-library installation, memory use and hardware configuration belong to the application.

The class implements complete and streamed chat-completion calls. Native tools are opt-in and depend on the loaded model and chat handler; JSON action mode is the safer default for portable inference.

Parameters

modelstrrequired

Path to the local model file.

model_paramsdict[str, Any] | Nonedefault: None

Values merged over temperature=0.8 and max_tokens=8192.

supports_tool_callingbooldefault: False

Opt into native llama.cpp tools when the loaded model and chat handler support them.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Raises

ImportError

Raised when llama-cpp-python is not installed.

FileNotFoundError

Raised when model does not exist.

Recommended construction

The local package currently has no convenience init export. Prefer create_llm("llama.cpp-local", model="..."), or import the class from the full path shown above.

Examples

from protolink import create_llm

llm = create_llm(
    "llama.cpp-local",
    model="/models/qwen3-8b.gguf",
)

Testing provider

MockLLM

classprotolink.llms.MockLLM

source↗

class MockLLM(
  model: str = "mock-gpt",
  model_params: dict[str, Any] | None = None,
  *,
  mock_responses: dict[str, Any] | None = None,
  sequential_responses: list[Any] | None = None,
  response_callback: Callable[[ConversationHistory, str], Any] | None = None,
  default_response: str = "Unprocessed generic mock response",
  tool_calling: Literal["auto", "native", "json"] | None = None,
)

Dependency-free deterministic adapter for tests, examples and offline runtime development. It implements the same LLM contract without network access, credentials, model files, or nondeterministic generation.

Responses are selected in a predictable priority order: a custom callback can inspect the full history, sequential responses can model multi-step action loops, keyword mappings can match prompts and default_response handles everything else. This makes MockLLM suitable for testing Agent behavior, tool dispatch, parsing, history isolation and failure paths rather than only simple chat.

Parameters

modelstrdefault: "mock-gpt"

Identifier reported by the mock adapter.

model_paramsdict[str, Any] | Nonedefault: None

Optional generation metadata retained on the instance.

mock_responsesdict[str, Any] | Nonedefault: None

Keyword-based response mapping. Nested mappings can first match system-prompt text and then the latest user message.

sequential_responseslist[Any] | Nonedefault: None

Responses consumed in order across calls.

response_callbackCallable[[ConversationHistory, str], Any] | Nonedefault: None

Custom callback receiving the full history and system prompt.

default_responsestrdefault: "Unprocessed generic mock response"

Fallback returned when no callback, sequential response, or mapping matches.

tool_calling"auto" | "native" | "json" | Nonedefault: None

Runtime protocol selection. Omit it to preserve this adapter's default; use auto for cached capability evidence, or native / json for an explicit override. See action acquisition for provider behavior.

Examples

from protolink import create_llm

llm = create_llm(
    "mock",
    sequential_responses=[
        {"type": "tool_call", "tool": "search", "args": {"query": "ProtoLink"}},
        {"type": "final", "content": "Done"},
    ],
)

Related objects

LLMModelProfile

dataclassprotolink.LLMModelProfile

source↗

class LLMModelProfile(
  context_window: int | None = None,
  input_cost_per_million: float | None = None,
  output_cost_per_million: float | None = None,
  currency: str = "USD",
  provider: str | None = None,
  model: str | None = None,
  supports_tools: bool | None = None,
  supports_streaming: bool | None = None,
  supports_json_schema: bool | None = None,
  tokenizer: str | None = None,
  metadata: dict[str, Any] = {},
)

Immutable application-owned metadata used to calculate context pressure and estimated cost. It describes a deployment rather than configuring the provider request: changing this profile cannot enable tools, streaming, JSON schema support, or a larger model context window.

ProtoLink does not maintain a provider pricing catalog because limits and prices change independently of the library. Supply values from the provider contract used by your application and update them on your own release schedule. The class does not range-check those supplied values.

Parameters

context_windowint | Nonedefault: None

Total model context window in tokens.

input_cost_per_millionfloat | Nonedefault: None

Input price per one million tokens.

output_cost_per_millionfloat | Nonedefault: None

Output price per one million tokens.

currencystrdefault: "USD"

Currency label for estimates.

providerstr | Nonedefault: None

Optional provider label.

modelstr | Nonedefault: None

Optional model label.

supports_toolsbool | Nonedefault: None

Descriptive tool support flag.

supports_streamingbool | Nonedefault: None

Descriptive streaming support flag.

supports_json_schemabool | Nonedefault: None

Descriptive JSON-schema support flag.

tokenizerstr | Nonedefault: None

Optional tokenizer name used by application metadata.

metadatadict[str, Any]default: {}

Additional application-defined metadata.

Usage examples

Basic chat

from protolink import create_llm

llm = create_llm("openai", model="gpt-4o-mini")

response = llm.chat("Hello, how are you?")
print(response)

Choose streaming as a separate interaction with a fresh or explicitly managed history:

streaming_llm = create_llm("openai", model="gpt-4o-mini")

async for chunk in streaming_llm.chat(
    "Draft a short welcome message.",
    streaming=True,
):
    print(chunk, end="", flush=True)

chat() is deliberately small: it appends the user message and makes one provider call. It does not run tools, delegate to agents, apply the multi-step safety loop, or append the returned assistant text. Use infer() through an Agent when those runtime capabilities are needed.

Advanced inference with tools

import asyncio

from protolink import Agent, AgentCard, create_llm


async def main():
    agent = Agent(
        AgentCard(
            name="calculator",
            description="Performs checked calculations",
            url="runtime://calculator",
        ),
        transport="runtime",
        llm=create_llm("openai", model="gpt-4o-mini"),
    )

    @agent.tool(name="multiply", description="Multiply two numbers")
    async def multiply(left: float, right: float) -> float:
        return left * right

    answer = await agent.invoke("What is 15 multiplied by 8?")
    print(f"Final answer: {answer}")


asyncio.run(main())

The Agent prepares the tool prompt and executable mapping, history binding, discovered-Agent context, policy boundary, cancellation token, run context and budget configuration before invoking the LLM loop.

Updating parameters and prompts

llm.model_params = {
    "temperature": 0.7,
    "max_output_tokens": 500,
}

llm.set_system_prompt("You are a helpful coding assistant.")

Generation-parameter names are provider-specific. The base class requires a dictionary but does not translate keys such as max_tokens and max_output_tokens; the selected SDK or server decides which values are valid.

set_system_prompt() changes the adapter attribute only. To rebuild the runtime prompt and update the system message in history, use build_system_prompt(). Pass persist=True when existing turns must be retained.

Connection validation

if llm.validate_connection():
    print("LLM is reachable.")
else:
    print("LLM validation failed.")

Concrete constructors currently perform their own validation during initialization. Calling the method explicitly is still useful for health checks and diagnostics after a server, credential, network, or model state may have changed.

Error handling

LLM failures can originate at several different boundaries:

• Authentication errors: a provider rejects a missing, expired, or invalid credential.
• Connection errors: a server is unavailable, a URL is invalid, or the network call fails.
• Model errors: a model identifier is unknown, unavailable, or incompatible with the requested feature.
• Parameter errors: the downstream SDK or server rejects generation settings.
• Action errors: a model emits malformed or invalid structured intent.
• Execution errors: a selected tool, delegated agent, authorization policy, cancellation token, or run budget stops the loop.
• Parse guardrail errors: repeated parse failures produce InferParseError.
• Tool-correction guardrail errors: rejected tool requests exhaust max_tool_corrections and produce ToolCorrectionLimitError.
• Step guardrail errors: the ten-step safety limit produces RuntimeError.

Recoverable action mistakes are normally injected back into history so the model can self-correct. Application-level exception handling should focus on provider failures and runtime boundaries that cannot be repaired inside the loop.

A direct call_action() or call_action_stream() invocation raises ValueError when its one response cannot become a valid action. infer() catches that validation failure, emits llm_parse_error, requests a correction and raises InferParseError only when max_parse_failures consecutive proposals have failed. Inspect the final field-level diagnostic before increasing the limit: repeated syntax drift may benefit from another attempt, while an unavailable tool, ambiguous action, or application-schema mismatch needs a prompt, capability, or application-layer fix.

import asyncio

from protolink import (
    ActionDeniedError,
    ApprovalRequiredError,
    BudgetExceededError,
    InferParseError,
    create_llm,
)


async def safe_inference():
    llm = create_llm("openai", model="gpt-4o-mini")

    try:
        result = await llm.infer(
            query="Summarize the available information.",
            tools={},
        )
        print(f"Success: {result.content}")
    except BudgetExceededError as exc:
        print(f"Budget stopped the run: {exc}")
    except (ApprovalRequiredError, ActionDeniedError) as exc:
        print(f"Policy stopped the action: {exc}")
    except InferParseError as exc:
        print(f"Model action parsing failed: {exc.explanation}")
        print(f"Last model response: {exc.raw_response!r}")
    except asyncio.CancelledError:
        print("The run was cancelled.")
        raise
    except RuntimeError as exc:
        print(f"Inference failed: {exc}")
    except Exception as exc:
        print(f"Provider or configuration error: {exc}")


asyncio.run(safe_inference())

An in-process RuntimeTransport request still raises TransportRemoteError at the client boundary, while preserving the typed infer error as its cause:

from protolink import InferParseError, TransportRemoteError

try:
    task = await client.send_infer_task("Review the evidence.", agent_url)
except TransportRemoteError as exc:
    if isinstance(exc.__cause__, InferParseError):
        print(exc.__cause__.explanation)
        print(f"Last model response: {exc.__cause__.raw_response!r}")
    raise

Remote network transports serialize failures rather than sharing Python exception objects, so this direct cause access is specific to the in-process runtime transport.

Type aliases

The public type aliases describe provider-neutral model metadata:

from typing import Literal, TypeAlias

LLMType: TypeAlias = Literal["api", "local", "server"]

LLMProvider: TypeAlias = Literal[
    "openai",
    "anthropic",
    "gemini",
    "deepseek",
    "grok",
    "huggingface",
    "llama.cpp-local",
    "llama.cpp-server",
    "lmstudio",
    "mock",
    "ollama",
    "openai-compatible",
    "vllm",
]

ReasoningLevel: TypeAlias = Literal["none", "low", "medium", "high"]

The factory also defines an LLMProvider enum with the same provider values. For current factory behavior, pass the string value such as "openai" or "ollama".

Migration guide

When migrating code written against earlier ProtoLink model wrappers:

1. Replace generate_response() with chat().
2. Replace generate_stream_response() with chat(..., streaming=True).
3. Use Agent.invoke() for normal tool calling, delegation, policy and multi-step execution.
4. Use LLM.infer() directly only when implementing the surrounding prompt and runtime preparation yourself.
5. Await Agent or inference calls; direct non-streaming chat() remains synchronous.
6. Expect a string from chat() and Agent.invoke(), or a Part with type infer_output from direct infer().

# Earlier style
# response = llm.generate_response(messages)
# print(response.content)

# Direct text generation
response = llm.chat("Hello, how are you?")
print(response)

# Controlled Agent inference
answer = await agent.invoke("What's the weather?")
print(answer)

See also

• LLM examples for larger provider and tool-call examples.
• Agents for how Agent binds state and invokes LLM.infer().
• State for persistent conversation sessions.
• Runtime for cancellation, policy, approvals, budgets and event recording.