crp.providers¶

Auto-generated reference for the crp.providers subpackage.

`providers`¶

`crp.providers` ¶

LLM provider adapters - OpenAI, Anthropic, Ollama, llama.cpp, custom.

All adapters implement :class:LLMProvider and can be passed directly to crp.Client(provider=...).

Usage::

from crp.providers import OpenAIAdapter, OllamaAdapter

provider = OpenAIAdapter(model="gpt-4o")
# or
provider = OllamaAdapter(model="llama3.1")

`LLMProvider` ¶

Bases: ABC

Abstract interface for LLM backends.

Implementations: OpenAIAdapter, AnthropicAdapter, LlamaCppAdapter, CustomProvider.

Subclasses must implement the three core abstract methods: generate_chat, count_tokens, and context_window_size. The remaining methods provide optional metadata, cost estimates, and tool or streaming support.

`max_output_tokens` `property` ¶

Optional provider-reported max output tokens.

Returns:

Type	Description
`int \| None`	Maximum output tokens, or None if not known.

`model_name` `property` ¶

Human-readable model identifier for diagnostics.

Returns:

Type	Description
`str`	Class name by default; subclasses may override with a specific
`str`	model identifier.

`is_thinking_model` `property` ¶

Return True if the model produces reasoning_content alongside output.

Thinking models (qwen3, deepseek-r1, o1, etc.) spend a significant portion of tokens on internal reasoning, requiring a larger generation reserve to avoid starving the final content output.

Returns:

Type	Description
`bool`	True when the provider represents a thinking model.

`generate_chat(messages, **kwargs)` `abstractmethod` ¶

Generate text completion from a message array.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, str]]`	List of chat messages, e.g. `[{"role": "system", "content": "..."}, ...]`.	required
`**kwargs`	`object`	Provider-specific overrides (max_tokens, temperature, etc.)	`{}`

Returns:

Type	Description
`tuple[str, str]`	`(output_text, finish_reason)` where `finish_reason` is one of: - `"length"` - model hit generation reserve limit (physical wall) - `"stop"` - model produced natural EOS - `"error"` - provider returned an error

`count_tokens(text)` `abstractmethod` ¶

Count tokens in text using this provider's exact tokenizer.

MUST return accurate per-model counts - no estimates.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text to tokenise.	required

Returns:

Type	Description
`int`	Number of tokens in `text`.

`context_window_size()` `abstractmethod` ¶

Return the model's physical context window size in tokens.

E.g. 128_000 for Claude 3 Opus, 8_192 for GPT-3.5.

Returns:

Type	Description
`int`	Context window size in tokens.

`cost_per_1k_tokens()` ¶

Return (input_cost_per_1k, output_cost_per_1k) in USD.

Returns:

Type	Description
`float`	`(input_cost, output_cost)` per 1,000 tokens. Defaults to
`float`	`(0.0, 0.0)` for local models or unknown pricing.

`supports_tools()` ¶

Return True if this provider supports function/tool calling.

Override in subclasses that implement generate_chat_with_tools(). The orchestrator uses this to decide between push (envelope) and pull (tool-mediated) context relay.

Returns:

Type	Description
`bool`	True when tool-mediated dispatch is supported.

`generate_chat_with_tools(messages, tools, **kwargs)` ¶

Generate with tool/function calling support.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, object]]`	Chat messages (may include tool role messages).	required
`tools`	`list[dict[str, object]]`	Tool definitions in OpenAI-compatible format.	required
`**kwargs`	`object`	Provider-specific overrides.	`{}`

Returns:

Name	Type	Description
	`str`	`(output_text, finish_reason, tool_calls, raw_assistant_message)`
`where`	`str`
	`list[dict[str, object]] \| None`	`output_text`: Generated text content (may be empty if tool_calls)
	`dict[str, object] \| None`	`finish_reason`: "stop", "length", "tool_calls", or "error"
	`tuple[str, str, list[dict[str, object]] \| None, dict[str, object] \| None]`	`tool_calls`: List of tool call dicts if finish_reason=="tool_calls"
	`tuple[str, str, list[dict[str, object]] \| None, dict[str, object] \| None]`	`raw_assistant_message`: The full assistant message dict (for appending to conversation history in the tool loop)

Raises:

Type	Description
`NotImplementedError`	By default; subclasses must override this method when `supports_tools()` returns True.

`generate_chat_stream(messages, **kwargs)` ¶

Stream token chunks from the LLM.

Yields individual token chunks as strings. The return value (accessible via StopIteration.value) is the finish_reason ("stop" or "length").

Default implementation falls back to generate_chat() and yields the full output as a single chunk. Override in subclasses for real streaming.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, str]]`	Chat messages.	required
`**kwargs`	`object`	Provider-specific overrides.	`{}`

Yields:

Type	Description
`str`	Token chunks as strings.

Returns:

Type	Description
`str`	The finish reason string.

`CustomProvider` ¶

Bases: LLMProvider

User-supplied LLM backend.

Example::

provider = CustomProvider(
    generate_fn=my_generate,
    count_tokens_fn=my_tokenizer,
    context_size=8192,
)

Parameters:

Name	Type	Description	Default
`generate_fn`	`Callable[[list[dict[str, str]]], tuple[str, str]]`	Callable accepting a message list and returning `(output_text, finish_reason)`.	required
`count_tokens_fn`	`Callable[[str], int]`	Callable returning the token count for a string.	required
`context_size`	`int`	Model context window size in tokens.	required
`name`	`str`	Human-readable provider name.	`'custom'`
`max_output`	`int \| None`	Optional maximum output token count.	`None`

`max_output_tokens` `property` ¶

Return the configured maximum output tokens, if any.

`model_name` `property` ¶

Return the configured provider name.

`generate_chat(messages, **kwargs)` ¶

Generate a completion using the user-supplied function.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, str]]`	Chat messages.	required
`**kwargs`	`Any`	Ignored by this adapter; accepted for API compatibility.	`{}`

Returns:

Type	Description
`tuple[str, str]`	`(output_text, finish_reason)` from `generate_fn`.

`count_tokens(text)` ¶

Count tokens using the user-supplied tokenizer.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text to tokenise.	required

Returns:

Type	Description
`int`	Token count.

`context_window_size()` ¶

Return the configured context window size.

Returns:

Type	Description
`int`	Context window size in tokens.

`DetectedModel` `dataclass` ¶

A model discovered on a local runtime, with its capabilities.

`is_loaded` `property` ¶

Return whether this object is loaded.

`context_utilisation` `property` ¶

Loaded window as a fraction of the model's maximum (0.0–1.0).

0.03 here means the runtime allocated only 3 % of what the model can actually handle - a strong signal that aggressive context management (CKF, continuation, windowing) is required.

`to_provider()` ¶

Return a CRP provider adapter configured for this detected model.

Returns None if the runtime kind is not recognised or the model is not an LLM.

`to_dict()` ¶

Serialize the detected model to a dict.

`DetectedRuntime` `dataclass` ¶

A local runtime endpoint and the models it serves.

`to_dict()` ¶

Serialize the detected runtime to a dict.

`DiscoveryReport` `dataclass` ¶

The result of probing all local runtimes.

`reachable_runtimes` `property` ¶

Return the reachable runtimes.

`models` `property` ¶

Return the models.

`loaded_models` `property` ¶

Return the loaded models.

`any_reachable` `property` ¶

Return whether the any reachable condition holds.

`primary_model()` ¶

Best candidate to dispatch to: a loaded LLM, else any LLM.

`to_dict()` ¶

Serialize the full discovery report to a dict.

`ModelState` ¶

Bases: str, Enum

Whether the model is resident in memory and ready to serve.

`RuntimeKind` ¶

Bases: str, Enum

The local inference runtime serving a model.

`LlamaCppAdapter` ¶

Bases: LLMProvider

llama.cpp adapter - local inference or HTTP server.

Parameters:

Name	Type	Description	Default
`model_path`	`str \| None`	Path to a GGUF model file (Python binding mode).	`None`
`server_url`	`str \| None`	Base URL for llama.cpp's HTTP server (e.g. "http://localhost:8080"). If provided, model_path is ignored.	`None`
`context_size`	`int`	Context window size in tokens (default: 4096).	`4096`
`max_tokens`	`int`	Max output tokens per generation (default: 2048).	`2048`
`n_gpu_layers`	`int`	GPU layers for Python binding (default: -1 = all).	`-1`
`n_threads`	`int \| None`	CPU threads for Python binding (default: os.cpu_count()).	`None`

`max_output_tokens` `property` ¶

Return the max output tokens.

`model_name` `property` ¶

Return the model name.

`generate_chat(messages, **kwargs)` ¶

Generate via Python binding or HTTP server.

`count_tokens(text)` ¶

Count tokens via llama.cpp's tokenizer or heuristic fallback.

`context_window_size()` ¶

Return the current context window count.

Returns:

Type	Description
`int`	`int`.

`supports_tools()` ¶

llama.cpp supports OpenAI-compatible tool calling.

In HTTP-server mode this depends on the server exposing /v1/chat/completions with tool support. In Python-binding mode it depends on the underlying llama-cpp-python version and model. CRP advertises support and lets the runtime fail if the model or server does not implement it.

`generate_chat_with_tools(messages, tools, **kwargs)` ¶

Generate with llama.cpp tool/function calling.

Returns (text, finish_reason, tool_calls, raw_assistant_message).

`OllamaAdapter` ¶

Bases: LLMProvider

Ollama REST API adapter for local model inference.

Parameters:

Name	Type	Description	Default
`model`	`str`	Model name (e.g. "llama3.1", "mistral", "gemma2").	`'llama3.1'`
`base_url`	`str \| None`	Ollama API base URL. Defaults to `OLLAMA_HOST` env var or `http://localhost:11434`.	`None`
`context_size`	`int \| None`	Override context window size (tokens).	`None`
`max_tokens`	`int`	Max output tokens per request (default: 2048).	`2048`
`timeout`	`float`	HTTP timeout in seconds (default: 300 - local models can be slow on CPU).	`300.0`

`max_output_tokens` `property` ¶

Return the max output tokens.

`model_name` `property` ¶

Return the model name.

`generate_chat(messages, **kwargs)` ¶

Call Ollama /api/chat endpoint with retry on transient failures.

Returns (output_text, finish_reason).

`count_tokens(text)` ¶

Estimate token count (Ollama doesn't expose tokenizer directly).

Uses a conservative ~3.5 chars/token estimate for most models. For exact counts, use the LlamaCppAdapter with model_path instead.

`context_window_size()` ¶

Return the current context window count.

Returns:

Type	Description
`int`	`int`.

`supports_tools()` ¶

Ollama supports OpenAI-compatible tool calling from 0.3.0+.

Actual availability depends on the model (e.g. llama3.1, qwen2.5, mistral). CRP advertises support here and lets the server fail gracefully if the loaded model is not tool-capable.

`generate_chat_with_tools(messages, tools, **kwargs)` ¶

Generate with Ollama tool/function calling.

Returns (text, finish_reason, tool_calls, raw_assistant_message). Finish reason is "tool_calls" when the model emits tool calls.

`OpenAIAdapter` ¶

Bases: LLMProvider

OpenAI chat completions adapter.

Works with OpenAI API and any OpenAI-compatible server (LM Studio, vLLM, llama.cpp server, Ollama OpenAI compat, TGI, etc.).

Model capabilities are auto-discovered via 3-layer resolution: 1. Exact match against known OpenAI models 2. Prefix match against 50+ open-source model families 3. Server-side probing (for vLLM, Ollama-compat endpoints) 4. Conservative fallback (8K context) - safe for unknown models

Parameters:

Name	Type	Description	Default
`model`	`str`	Model name (e.g. "gpt-4o", "qwen3-4b", "llama3.1").	`'gpt-4o'`
`api_key`	`str \| None`	API key. Defaults to `OPENAI_API_KEY` env var.	`None`
`base_url`	`str \| None`	Override API base URL (for LM Studio, vLLM, etc.).	`None`
`context_size`	`int \| None`	Override auto-discovered context window (tokens).	`None`
`max_tokens`	`int \| None`	Override auto-discovered max output tokens per request.	`None`
`timeout`	`float`	HTTP timeout in seconds (default: 120).	`120.0`

`max_output_tokens` `property` ¶

Return the max output tokens.

`model_name` `property` ¶

Return the model name.

`is_thinking_model` `property` ¶

Detect if the current model is a thinking/reasoning model.

`generate_chat(messages, **kwargs)` ¶

Call OpenAI chat completions API with retry on transient failures.

Handles "thinking" models (Qwen3, DeepSeek-R1, o1, etc.) that split output into reasoning_content + content fields. CRP extracts the final content and preserves the full reasoning for downstream extraction.

Returns (output_text, finish_reason).

`count_tokens(text)` ¶

Count tokens using tiktoken (exact) or fallback heuristic.

`context_window_size()` ¶

Return the current context window count.

Returns:

Type	Description
`int`	`int`.

`cost_per_1k_tokens()` ¶

OpenAI pricing per 1K tokens (USD) - updated 2025-Q2.

`supports_tools()` ¶

OpenAI and compatible servers support function/tool calling.

`generate_chat_with_tools(messages, tools, **kwargs)` ¶

Generate with OpenAI tool/function calling.

Returns (text, finish_reason, tool_calls, raw_assistant_message). When the model wants to call tools, finish_reason="tool_calls" and the tool_calls list contains structured call requests. The raw_assistant_message is the full message dict for appending to conversation history (required by the OpenAI tool protocol).

`generate_chat_stream(messages, **kwargs)` ¶

Stream token chunks from OpenAI.

Yields individual token deltas. Return value is finish_reason.

`discover_local_llms(endpoints=None, *, timeout=2.0)` ¶

Probe local LLM runtimes and report every model and its capabilities.

Parameters:

Name	Type	Description	Default
`endpoints`	`tuple[_Endpoint, ...] \| None`	Override the set of endpoints to probe. Defaults to the standard LM Studio / Ollama / llama.cpp / vLLM ports.	`None`
`timeout`	`float`	Per-request timeout in seconds. Discovery is best-effort - unreachable runtimes are reported as `reachable=False` and never raise.	`2.0`

Returns:

Name	Type	Description
`A`	`DiscoveryReport`	class:`DiscoveryReport`.

`providers.anthropic`¶

`crp.providers.anthropic` ¶

Anthropic adapter - Claude 3/3.5/4 families (§6.1).

Requires anthropic>=0.25 (pip install crprotocol[full]).

Usage::

from crp.providers.anthropic import AnthropicAdapter

provider = AnthropicAdapter(model="claude-sonnet-4-20250514")
output, reason = provider.generate_chat([
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "Hello!"},
])

`AnthropicAdapter` ¶

Bases: LLMProvider

Anthropic Claude chat adapter.

Parameters:

Name	Type	Description	Default
`model`	`str`	Model name (e.g. "claude-sonnet-4-20250514").	`'claude-sonnet-4-20250514'`
`api_key`	`str \| None`	API key. Defaults to `ANTHROPIC_API_KEY` env var.	`None`
`max_tokens`	`int \| None`	Max output tokens per request (default: model limit).	`None`
`timeout`	`float`	HTTP timeout in seconds (default: 120).	`120.0`

`max_output_tokens` `property` ¶

Return the max output tokens.

`model_name` `property` ¶

Return the model name.

`generate_chat(messages, **kwargs)` ¶

Call Anthropic Messages API with retry on transient failures.

Anthropic uses a separate system parameter instead of a system message in the array, so we extract it automatically.

Returns (output_text, finish_reason).

`count_tokens(text)` ¶

Count tokens using Anthropic's tokenizer if available.

Falls back to ~3.5 chars/token heuristic (Claude uses a modified BPE tokenizer similar to GPT-4's).

`context_window_size()` ¶

Return the current context window count.

Returns:

Type	Description
`int`	`int`.

`cost_per_1k_tokens()` ¶

Anthropic pricing per 1K tokens (USD) - updated 2025-Q2.

`generate_chat_stream(messages, **kwargs)` ¶

Stream token chunks from Anthropic.

Yields individual text deltas. Return value is finish_reason.

`providers.base`¶

`crp.providers.base` ¶

LLMProvider abstract base class - §6.1; CRP-SPEC-008.

Every provider adapter MUST implement generate_chat, count_tokens, and context_window_size. The protocol uses these three to compute window budgets and dispatch tasks.

Relevant specifications

CRP specification §6.1: Provider interface
CRP-SPEC-008: Dispatch & Provider Adaptation

`LLMProvider` ¶

Bases: ABC

Abstract interface for LLM backends.

Implementations: OpenAIAdapter, AnthropicAdapter, LlamaCppAdapter, CustomProvider.

Subclasses must implement the three core abstract methods: generate_chat, count_tokens, and context_window_size. The remaining methods provide optional metadata, cost estimates, and tool or streaming support.

`max_output_tokens` `property` ¶

Optional provider-reported max output tokens.

Returns:

Type	Description
`int \| None`	Maximum output tokens, or None if not known.

`model_name` `property` ¶

Human-readable model identifier for diagnostics.

Returns:

Type	Description
`str`	Class name by default; subclasses may override with a specific
`str`	model identifier.

`is_thinking_model` `property` ¶

Return True if the model produces reasoning_content alongside output.

Thinking models (qwen3, deepseek-r1, o1, etc.) spend a significant portion of tokens on internal reasoning, requiring a larger generation reserve to avoid starving the final content output.

Returns:

Type	Description
`bool`	True when the provider represents a thinking model.

`generate_chat(messages, **kwargs)` `abstractmethod` ¶

Generate text completion from a message array.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, str]]`	List of chat messages, e.g. `[{"role": "system", "content": "..."}, ...]`.	required
`**kwargs`	`object`	Provider-specific overrides (max_tokens, temperature, etc.)	`{}`

Returns:

Type	Description
`tuple[str, str]`	`(output_text, finish_reason)` where `finish_reason` is one of: - `"length"` - model hit generation reserve limit (physical wall) - `"stop"` - model produced natural EOS - `"error"` - provider returned an error

`count_tokens(text)` `abstractmethod` ¶

Count tokens in text using this provider's exact tokenizer.

MUST return accurate per-model counts - no estimates.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text to tokenise.	required

Returns:

Type	Description
`int`	Number of tokens in `text`.

`context_window_size()` `abstractmethod` ¶

Return the model's physical context window size in tokens.

E.g. 128_000 for Claude 3 Opus, 8_192 for GPT-3.5.

Returns:

Type	Description
`int`	Context window size in tokens.

`cost_per_1k_tokens()` ¶

Return (input_cost_per_1k, output_cost_per_1k) in USD.

Returns:

Type	Description
`float`	`(input_cost, output_cost)` per 1,000 tokens. Defaults to
`float`	`(0.0, 0.0)` for local models or unknown pricing.

`supports_tools()` ¶

Return True if this provider supports function/tool calling.

Override in subclasses that implement generate_chat_with_tools(). The orchestrator uses this to decide between push (envelope) and pull (tool-mediated) context relay.

Returns:

Type	Description
`bool`	True when tool-mediated dispatch is supported.

`generate_chat_with_tools(messages, tools, **kwargs)` ¶

Generate with tool/function calling support.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, object]]`	Chat messages (may include tool role messages).	required
`tools`	`list[dict[str, object]]`	Tool definitions in OpenAI-compatible format.	required
`**kwargs`	`object`	Provider-specific overrides.	`{}`

Returns:

Name	Type	Description
	`str`	`(output_text, finish_reason, tool_calls, raw_assistant_message)`
`where`	`str`
	`list[dict[str, object]] \| None`	`output_text`: Generated text content (may be empty if tool_calls)
	`dict[str, object] \| None`	`finish_reason`: "stop", "length", "tool_calls", or "error"
	`tuple[str, str, list[dict[str, object]] \| None, dict[str, object] \| None]`	`tool_calls`: List of tool call dicts if finish_reason=="tool_calls"
	`tuple[str, str, list[dict[str, object]] \| None, dict[str, object] \| None]`	`raw_assistant_message`: The full assistant message dict (for appending to conversation history in the tool loop)

Raises:

Type	Description
`NotImplementedError`	By default; subclasses must override this method when `supports_tools()` returns True.

`generate_chat_stream(messages, **kwargs)` ¶

Stream token chunks from the LLM.

Yields individual token chunks as strings. The return value (accessible via StopIteration.value) is the finish_reason ("stop" or "length").

Default implementation falls back to generate_chat() and yields the full output as a single chunk. Override in subclasses for real streaming.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, str]]`	Chat messages.	required
`**kwargs`	`object`	Provider-specific overrides.	`{}`

Yields:

Type	Description
`str`	Token chunks as strings.

Returns:

Type	Description
`str`	The finish reason string.

`providers.custom`¶

`crp.providers.custom` ¶

CustomProvider - user-supplied LLM adapter (Axiom 6: Portability; CRP-SPEC-008).

Allows any LLM to be used with CRP if the user provides three callables

generate_fn(messages) → (output, finish_reason) count_tokens_fn(text) → int context_size → int

Relevant specifications

CRP-SPEC-008: Dispatch & Provider Adaptation
Axiom 6: Portability

`CustomProvider` ¶

Bases: LLMProvider

User-supplied LLM backend.

Example::

provider = CustomProvider(
    generate_fn=my_generate,
    count_tokens_fn=my_tokenizer,
    context_size=8192,
)

Parameters:

Name	Type	Description	Default
`generate_fn`	`Callable[[list[dict[str, str]]], tuple[str, str]]`	Callable accepting a message list and returning `(output_text, finish_reason)`.	required
`count_tokens_fn`	`Callable[[str], int]`	Callable returning the token count for a string.	required
`context_size`	`int`	Model context window size in tokens.	required
`name`	`str`	Human-readable provider name.	`'custom'`
`max_output`	`int \| None`	Optional maximum output token count.	`None`

`max_output_tokens` `property` ¶

Return the configured maximum output tokens, if any.

`model_name` `property` ¶

Return the configured provider name.

`generate_chat(messages, **kwargs)` ¶

Generate a completion using the user-supplied function.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, str]]`	Chat messages.	required
`**kwargs`	`Any`	Ignored by this adapter; accepted for API compatibility.	`{}`

Returns:

Type	Description
`tuple[str, str]`	`(output_text, finish_reason)` from `generate_fn`.

`count_tokens(text)` ¶

Count tokens using the user-supplied tokenizer.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text to tokenise.	required

Returns:

Type	Description
`int`	Token count.

`context_window_size()` ¶

Return the configured context window size.

Returns:

Type	Description
`int`	Context window size in tokens.

`providers.diagnostic`¶

`crp.providers.diagnostic` ¶

ProviderDiagnostic - 6-step health check (§05, §06 §26.3).

Executed at init() and on demand via diagnose(). Each step produces a diagnostic code string indicating pass/fail/warning.

`DiagnosticCode` ¶

Bases: str, Enum

All provider diagnostic result codes.

`DiagnosticResult` `dataclass` ¶

Result of a single diagnostic step.

`passed` `property` ¶

Return whether the passed condition holds.

`DiagnosticReport` `dataclass` ¶

Full 6-step diagnostic report.

`all_passed` `property` ¶

Return whether the all passed condition holds.

`summary` `property` ¶

Return the summary.

`ProviderDiagnostic` ¶

Execute 6-step diagnostic sequence on an LLM provider.

`diagnose(provider)` ¶

Run all 6 diagnostic steps and return a report.

`providers.discovery`¶

`crp.providers.discovery` ¶

Local LLM discovery - detect running models and their capabilities (§6.2).

CRP can introspect the inference layer it is about to govern. Before a single token is dispatched, :func:discover_local_llms probes the common local-LLM runtimes (LM Studio, Ollama, llama.cpp / llama-server, and any other OpenAI-compatible server) and reports, for every model it finds:

the runtime serving it and the endpoint,
whether the model is currently loaded into memory,
its architecture, publisher and quantization,
its maximum context length vs. the currently loaded context length,
whether it advertises tool / function calling (MCP-compatible), and
whether it is a thinking / reasoning model (extended chain-of-thought).

This matters because CRP's window budgeting, generation-reserve sizing and tool-vs-envelope dispatch decisions all depend on the real capabilities of the model actually answering - not on what a config file claims.

Everything here uses only the Python standard library (urllib); no runtime needs to be installed for discovery to work.

Usage::

from crp.providers.discovery import discover_local_llms

report = discover_local_llms()
for model in report.loaded_models:
    print(model.id, model.runtime, model.max_context_length,
          "tools" if model.supports_tools else "",
          "reasoning" if model.is_reasoning_model else "")

`RuntimeKind` ¶

Bases: str, Enum

The local inference runtime serving a model.

`ModelState` ¶

Bases: str, Enum

Whether the model is resident in memory and ready to serve.

`DetectedModel` `dataclass` ¶

A model discovered on a local runtime, with its capabilities.

`is_loaded` `property` ¶

Return whether this object is loaded.

`context_utilisation` `property` ¶

Loaded window as a fraction of the model's maximum (0.0–1.0).

0.03 here means the runtime allocated only 3 % of what the model can actually handle - a strong signal that aggressive context management (CKF, continuation, windowing) is required.

`to_provider()` ¶

Return a CRP provider adapter configured for this detected model.

Returns None if the runtime kind is not recognised or the model is not an LLM.

`to_dict()` ¶

Serialize the detected model to a dict.

`DetectedRuntime` `dataclass` ¶

A local runtime endpoint and the models it serves.

`to_dict()` ¶

Serialize the detected runtime to a dict.

`DiscoveryReport` `dataclass` ¶

The result of probing all local runtimes.

`reachable_runtimes` `property` ¶

Return the reachable runtimes.

`models` `property` ¶

Return the models.

`loaded_models` `property` ¶

Return the loaded models.

`any_reachable` `property` ¶

Return whether the any reachable condition holds.

`primary_model()` ¶

Best candidate to dispatch to: a loaded LLM, else any LLM.

`to_dict()` ¶

Serialize the full discovery report to a dict.

`discover_local_llms(endpoints=None, *, timeout=2.0)` ¶

Probe local LLM runtimes and report every model and its capabilities.

Parameters:

Name	Type	Description	Default
`endpoints`	`tuple[_Endpoint, ...] \| None`	Override the set of endpoints to probe. Defaults to the standard LM Studio / Ollama / llama.cpp / vLLM ports.	`None`
`timeout`	`float`	Per-request timeout in seconds. Discovery is best-effort - unreachable runtimes are reported as `reachable=False` and never raise.	`2.0`

Returns:

Name	Type	Description
`A`	`DiscoveryReport`	class:`DiscoveryReport`.

`providers.llamacpp`¶

`crp.providers.llamacpp` ¶

llama.cpp adapter - local model inference via llama-cpp-python or HTTP (§6.1).

Supports two modes

Python binding: pip install llama-cpp-python (in-process inference).
HTTP server: llama-server --model model.gguf --port 8080 (talks to llama.cpp's OpenAI-compatible endpoint).

Usage (Python binding)::

from crp.providers.llamacpp import LlamaCppAdapter

provider = LlamaCppAdapter(model_path="/models/llama3-8b.gguf")
output, reason = provider.generate_chat([
    {"role": "user", "content": "Hello!"},
])

Usage (HTTP server)::

provider = LlamaCppAdapter(server_url="http://localhost:8080")

`LlamaCppAdapter` ¶

Bases: LLMProvider

llama.cpp adapter - local inference or HTTP server.

Parameters:

Name	Type	Description	Default
`model_path`	`str \| None`	Path to a GGUF model file (Python binding mode).	`None`
`server_url`	`str \| None`	Base URL for llama.cpp's HTTP server (e.g. "http://localhost:8080"). If provided, model_path is ignored.	`None`
`context_size`	`int`	Context window size in tokens (default: 4096).	`4096`
`max_tokens`	`int`	Max output tokens per generation (default: 2048).	`2048`
`n_gpu_layers`	`int`	GPU layers for Python binding (default: -1 = all).	`-1`
`n_threads`	`int \| None`	CPU threads for Python binding (default: os.cpu_count()).	`None`

`max_output_tokens` `property` ¶

Return the max output tokens.

`model_name` `property` ¶

Return the model name.

`generate_chat(messages, **kwargs)` ¶

Generate via Python binding or HTTP server.

`count_tokens(text)` ¶

Count tokens via llama.cpp's tokenizer or heuristic fallback.

`context_window_size()` ¶

Return the current context window count.

Returns:

Type	Description
`int`	`int`.

`supports_tools()` ¶

llama.cpp supports OpenAI-compatible tool calling.

In HTTP-server mode this depends on the server exposing /v1/chat/completions with tool support. In Python-binding mode it depends on the underlying llama-cpp-python version and model. CRP advertises support and lets the runtime fail if the model or server does not implement it.

`generate_chat_with_tools(messages, tools, **kwargs)` ¶

Generate with llama.cpp tool/function calling.

Returns (text, finish_reason, tool_calls, raw_assistant_message).

`providers.manager`¶

`crp.providers.manager` ¶

LLMProviderManager - multi-provider routing with fallback chain (§5; CRP-SPEC-008).

Supports

Primary provider selection
Fallback chain: if primary fails, try registered providers in order
Provider registration and retrieval by name

Relevant specifications

CRP specification §5: Provider routing
CRP-SPEC-008: Dispatch & Provider Adaptation

`LLMProviderManager` ¶

Routes requests to one or more LLM providers with fallback.

Usage::

mgr = LLMProviderManager(primary_provider)
mgr.register(fallback_provider)

# Generate with automatic fallback
output, reason = mgr.generate_with_fallback(messages)

Parameters:

Name	Type	Description	Default
`provider`	`LLMProvider`	Primary LLM provider. It is also registered under its `model_name` for internal lookup.	required

`primary` `property` ¶

Return the primary provider.

`provider_count` `property` ¶

Return the number of registered providers.

`provider_names` `property` ¶

Return the names of all registered providers.

`register(provider)` ¶

Register an additional provider for fallback routing.

Parameters:

Name	Type	Description	Default
`provider`	`LLMProvider`	Provider to add to the routing table.	required

`get(name=None)` ¶

Get a provider by name, or the primary if name is None.

Parameters:

Name	Type	Description	Default
`name`	`str \| None`	Provider name, or None to retrieve the primary provider.	`None`

Returns:

Type	Description
`LLMProvider`	The requested `LLMProvider`.

Raises:

Type	Description
`ProviderError`	If `name` is supplied but no matching provider is registered.

`generate_with_fallback(messages, **kwargs)` ¶

Generate using primary, fall back to registered providers on failure.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, str]]`	Chat messages to send.	required
`**kwargs`	`object`	Provider-specific overrides passed to `generate_chat`.	`{}`

Returns:

Type	Description
`str`	`(output_text, finish_reason, provider_name)` - the provider that
`str`	successfully generated is identified in `provider_name`.

Raises:

Type	Description
`ProviderError`	If ALL providers fail.

`generate_with_tools_fallback(messages, tools, **kwargs)` ¶

Generate with tools using primary, fall back on failure.

Only providers that report supports_tools() == True are considered.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, object]]`	Chat messages (may include tool messages).	required
`tools`	`list[dict[str, object]]`	OpenAI-compatible tool definitions.	required
`**kwargs`	`object`	Provider-specific overrides passed to `generate_chat_with_tools`.	`{}`

Returns:

Type	Description
`tuple[str, str, list[dict[str, object]] \| None, dict[str, object] \| None, str]`	`(output_text, finish_reason, tool_calls, raw_message, provider_name)`.

Raises:

Type	Description
`ProviderError`	If no tool-capable provider succeeds.

`providers.ollama`¶

`crp.providers.ollama` ¶

Ollama adapter - local model inference via Ollama REST API (§6.1).

Requires a running Ollama instance (ollama serve).

Usage::

from crp.providers.ollama import OllamaAdapter

provider = OllamaAdapter(model="llama3.1")
output, reason = provider.generate_chat([
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "Hello!"},
])

`OllamaAdapter` ¶

Bases: LLMProvider

Ollama REST API adapter for local model inference.

Parameters:

Name	Type	Description	Default
`model`	`str`	Model name (e.g. "llama3.1", "mistral", "gemma2").	`'llama3.1'`
`base_url`	`str \| None`	Ollama API base URL. Defaults to `OLLAMA_HOST` env var or `http://localhost:11434`.	`None`
`context_size`	`int \| None`	Override context window size (tokens).	`None`
`max_tokens`	`int`	Max output tokens per request (default: 2048).	`2048`
`timeout`	`float`	HTTP timeout in seconds (default: 300 - local models can be slow on CPU).	`300.0`

`max_output_tokens` `property` ¶

Return the max output tokens.

`model_name` `property` ¶

Return the model name.

`generate_chat(messages, **kwargs)` ¶

Call Ollama /api/chat endpoint with retry on transient failures.

Returns (output_text, finish_reason).

`count_tokens(text)` ¶

Estimate token count (Ollama doesn't expose tokenizer directly).

Uses a conservative ~3.5 chars/token estimate for most models. For exact counts, use the LlamaCppAdapter with model_path instead.

`context_window_size()` ¶

Return the current context window count.

Returns:

Type	Description
`int`	`int`.

`supports_tools()` ¶

Ollama supports OpenAI-compatible tool calling from 0.3.0+.

Actual availability depends on the model (e.g. llama3.1, qwen2.5, mistral). CRP advertises support here and lets the server fail gracefully if the loaded model is not tool-capable.

`generate_chat_with_tools(messages, tools, **kwargs)` ¶

Generate with Ollama tool/function calling.

Returns (text, finish_reason, tool_calls, raw_assistant_message). Finish reason is "tool_calls" when the model emits tool calls.

`providers.openai`¶

`crp.providers.openai` ¶

OpenAI adapter - GPT-4o, GPT-4, GPT-3.5-turbo, o1/o3 families (§6.1).

Requires openai>=1.0 (pip install crprotocol[full]).

Usage::

from crp.providers.openai import OpenAIAdapter

provider = OpenAIAdapter(model="gpt-4o")
output, reason = provider.generate_chat([
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "Hello!"},
])

`OpenAIAdapter` ¶

Bases: LLMProvider

OpenAI chat completions adapter.

Works with OpenAI API and any OpenAI-compatible server (LM Studio, vLLM, llama.cpp server, Ollama OpenAI compat, TGI, etc.).

Model capabilities are auto-discovered via 3-layer resolution: 1. Exact match against known OpenAI models 2. Prefix match against 50+ open-source model families 3. Server-side probing (for vLLM, Ollama-compat endpoints) 4. Conservative fallback (8K context) - safe for unknown models

Parameters:

Name	Type	Description	Default
`model`	`str`	Model name (e.g. "gpt-4o", "qwen3-4b", "llama3.1").	`'gpt-4o'`
`api_key`	`str \| None`	API key. Defaults to `OPENAI_API_KEY` env var.	`None`
`base_url`	`str \| None`	Override API base URL (for LM Studio, vLLM, etc.).	`None`
`context_size`	`int \| None`	Override auto-discovered context window (tokens).	`None`
`max_tokens`	`int \| None`	Override auto-discovered max output tokens per request.	`None`
`timeout`	`float`	HTTP timeout in seconds (default: 120).	`120.0`

`max_output_tokens` `property` ¶

Return the max output tokens.

`model_name` `property` ¶

Return the model name.

`is_thinking_model` `property` ¶

Detect if the current model is a thinking/reasoning model.

`generate_chat(messages, **kwargs)` ¶

Call OpenAI chat completions API with retry on transient failures.

Handles "thinking" models (Qwen3, DeepSeek-R1, o1, etc.) that split output into reasoning_content + content fields. CRP extracts the final content and preserves the full reasoning for downstream extraction.

Returns (output_text, finish_reason).

`count_tokens(text)` ¶

Count tokens using tiktoken (exact) or fallback heuristic.

`context_window_size()` ¶

Return the current context window count.

Returns:

Type	Description
`int`	`int`.

`cost_per_1k_tokens()` ¶

OpenAI pricing per 1K tokens (USD) - updated 2025-Q2.

`supports_tools()` ¶

OpenAI and compatible servers support function/tool calling.

`generate_chat_with_tools(messages, tools, **kwargs)` ¶

Generate with OpenAI tool/function calling.

Returns (text, finish_reason, tool_calls, raw_assistant_message). When the model wants to call tools, finish_reason="tool_calls" and the tool_calls list contains structured call requests. The raw_assistant_message is the full message dict for appending to conversation history (required by the OpenAI tool protocol).

`generate_chat_stream(messages, **kwargs)` ¶

Stream token chunks from OpenAI.

Yields individual token deltas. Return value is finish_reason.

`providers.tokenizers`¶

`crp.providers.tokenizers` ¶

Per-provider tokenizer reconciliation (§06 §6.4).

Three-layer hierarchy

Layer 1: Model-specific tokenizer (best - 100% accuracy) Layer 2: Provider API token counting (good - 99%) Layer 3: Character-to-token fallback (acceptable - 70-80%)

`TokenizerRegistry` ¶

Cache and resolve tokenizers per provider.

Phase 1 implementation delegates to LLMProvider.count_tokens() which each adapter must implement with its own tokenizer. The registry adds the Layer 3 fallback heuristic for providers that raise.

`count_tokens(text, provider)` ¶

Count tokens using the best available method for provider.

Layer 1/2: provider.count_tokens() - exact model tokenizer or API. Layer 3: chars/4 fallback if the provider raises.

`validate_roundtrip(text, provider)` ¶

Validate encode→decode→encode is lossless (best-effort).

crp.providers¶

providers¶

crp.providers ¶

LLMProvider ¶

max_output_tokens property ¶

model_name property ¶

is_thinking_model property ¶

generate_chat(messages, **kwargs) abstractmethod ¶

count_tokens(text) abstractmethod ¶

context_window_size() abstractmethod ¶

cost_per_1k_tokens() ¶

supports_tools() ¶

generate_chat_with_tools(messages, tools, **kwargs) ¶

generate_chat_stream(messages, **kwargs) ¶

CustomProvider ¶

max_output_tokens property ¶

model_name property ¶

generate_chat(messages, **kwargs) ¶

count_tokens(text) ¶

context_window_size() ¶

DetectedModel dataclass ¶

is_loaded property ¶

context_utilisation property ¶

to_provider() ¶

to_dict() ¶

DetectedRuntime dataclass ¶

to_dict() ¶

DiscoveryReport dataclass ¶

reachable_runtimes property ¶

models property ¶

loaded_models property ¶

any_reachable property ¶

primary_model() ¶

to_dict() ¶

ModelState ¶

RuntimeKind ¶

LlamaCppAdapter ¶

max_output_tokens property ¶

model_name property ¶

generate_chat(messages, **kwargs) ¶

count_tokens(text) ¶

context_window_size() ¶

supports_tools() ¶

generate_chat_with_tools(messages, tools, **kwargs) ¶

OllamaAdapter ¶

max_output_tokens property ¶

model_name property ¶

generate_chat(messages, **kwargs) ¶

count_tokens(text) ¶

context_window_size() ¶

supports_tools() ¶

generate_chat_with_tools(messages, tools, **kwargs) ¶

OpenAIAdapter ¶

max_output_tokens property ¶

model_name property ¶

is_thinking_model property ¶

generate_chat(messages, **kwargs) ¶

count_tokens(text) ¶

context_window_size() ¶

cost_per_1k_tokens() ¶

supports_tools() ¶

generate_chat_with_tools(messages, tools, **kwargs) ¶

generate_chat_stream(messages, **kwargs) ¶

discover_local_llms(endpoints=None, *, timeout=2.0) ¶

providers.anthropic¶

crp.providers.anthropic ¶

AnthropicAdapter ¶

max_output_tokens property ¶

model_name property ¶

generate_chat(messages, **kwargs) ¶

count_tokens(text) ¶

context_window_size() ¶

cost_per_1k_tokens() ¶

generate_chat_stream(messages, **kwargs) ¶

providers.base¶

crp.providers.base ¶

LLMProvider ¶

max_output_tokens property ¶

model_name property ¶

is_thinking_model property ¶

`providers`¶

`crp.providers` ¶

`LLMProvider` ¶

`max_output_tokens` `property` ¶

`model_name` `property` ¶

`is_thinking_model` `property` ¶

`generate_chat(messages, **kwargs)` `abstractmethod` ¶

`count_tokens(text)` `abstractmethod` ¶

`context_window_size()` `abstractmethod` ¶

`cost_per_1k_tokens()` ¶

`supports_tools()` ¶

`generate_chat_with_tools(messages, tools, **kwargs)` ¶

`generate_chat_stream(messages, **kwargs)` ¶

`CustomProvider` ¶

`max_output_tokens` `property` ¶

`model_name` `property` ¶

`generate_chat(messages, **kwargs)` ¶

`count_tokens(text)` ¶

`context_window_size()` ¶

`DetectedModel` `dataclass` ¶

`is_loaded` `property` ¶

`context_utilisation` `property` ¶

`to_provider()` ¶

`to_dict()` ¶

`DetectedRuntime` `dataclass` ¶

`to_dict()` ¶

`DiscoveryReport` `dataclass` ¶

`reachable_runtimes` `property` ¶

`models` `property` ¶

`loaded_models` `property` ¶

`any_reachable` `property` ¶

`primary_model()` ¶

`to_dict()` ¶

`ModelState` ¶

`RuntimeKind` ¶

`LlamaCppAdapter` ¶

`max_output_tokens` `property` ¶

`model_name` `property` ¶

`generate_chat(messages, **kwargs)` ¶

`count_tokens(text)` ¶

`context_window_size()` ¶

`supports_tools()` ¶

`generate_chat_with_tools(messages, tools, **kwargs)` ¶

`OllamaAdapter` ¶

`max_output_tokens` `property` ¶

`model_name` `property` ¶

`generate_chat(messages, **kwargs)` ¶

`count_tokens(text)` ¶

`context_window_size()` ¶

`supports_tools()` ¶

`generate_chat_with_tools(messages, tools, **kwargs)` ¶

`OpenAIAdapter` ¶

`max_output_tokens` `property` ¶

`model_name` `property` ¶

`is_thinking_model` `property` ¶

`generate_chat(messages, **kwargs)` ¶

`count_tokens(text)` ¶

`context_window_size()` ¶

`cost_per_1k_tokens()` ¶

`supports_tools()` ¶

`generate_chat_with_tools(messages, tools, **kwargs)` ¶

`generate_chat_stream(messages, **kwargs)` ¶

`discover_local_llms(endpoints=None, *, timeout=2.0)` ¶

`providers.anthropic`¶

`crp.providers.anthropic` ¶

`AnthropicAdapter` ¶

`max_output_tokens` `property` ¶

`model_name` `property` ¶

`generate_chat(messages, **kwargs)` ¶

`count_tokens(text)` ¶

`context_window_size()` ¶

`cost_per_1k_tokens()` ¶

`generate_chat_stream(messages, **kwargs)` ¶

`providers.base`¶

`crp.providers.base` ¶

`LLMProvider` ¶

`max_output_tokens` `property` ¶

`model_name` `property` ¶

`is_thinking_model` `property` ¶

`generate_chat(messages, **kwargs)` `abstractmethod` ¶