Skip to content

crp.adapters

Auto-generated reference for the crp.adapters subpackage.

adapters

crp.adapters

Compatibility alias - from crp.adapters import ...

This module re-exports everything from :mod:crp.providers so that documentation examples using crp.adapters work unchanged.

Canonical location is crp.providers.

CustomProvider

Bases: LLMProvider

User-supplied LLM backend.

Example::

provider = CustomProvider(
    generate_fn=my_generate,
    count_tokens_fn=my_tokenizer,
    context_size=8192,
)

Parameters:

Name Type Description Default
generate_fn Callable[[list[dict[str, str]]], tuple[str, str]]

Callable accepting a message list and returning (output_text, finish_reason).

 required
count_tokens_fn Callable[[str], int]

Callable returning the token count for a string.

 required
context_size int

Model context window size in tokens.

 required
name str

Human-readable provider name.

 'custom'
max_output int | None

Optional maximum output token count.

 None

max_output_tokens property

Return the configured maximum output tokens, if any.

model_name property

Return the configured provider name.

generate_chat(messages, **kwargs)

Generate a completion using the user-supplied function.

Parameters:

Name Type Description Default
messages list[dict[str, str]]

Chat messages.

 required
**kwargs Any

Ignored by this adapter; accepted for API compatibility.

 {}

Returns:

Type Description
tuple[str, str]

(output_text, finish_reason) from generate_fn.

count_tokens(text)

Count tokens using the user-supplied tokenizer.

Parameters:

Name Type Description Default
text str

Text to tokenise.

 required

Returns:

Type Description
int

Token count.

context_window_size()

Return the configured context window size.

Returns:

Type Description
int

Context window size in tokens.

LlamaCppAdapter

Bases: LLMProvider

llama.cpp adapter - local inference or HTTP server.

Parameters:

Name Type Description Default
model_path str | None

Path to a GGUF model file (Python binding mode).

 None
server_url str | None

Base URL for llama.cpp's HTTP server (e.g. "http://localhost:8080"). If provided, model_path is ignored.

 None
context_size int

Context window size in tokens (default: 4096).

 4096
max_tokens int

Max output tokens per generation (default: 2048).

 2048
n_gpu_layers int

GPU layers for Python binding (default: -1 = all).

 -1
n_threads int | None

CPU threads for Python binding (default: os.cpu_count()).

 None

max_output_tokens property

Return the max output tokens.

model_name property

Return the model name.

generate_chat(messages, **kwargs)

Generate via Python binding or HTTP server.

count_tokens(text)

Count tokens via llama.cpp's tokenizer or heuristic fallback.

context_window_size()

Return the current context window count.

Returns:

Type Description
int

int.

supports_tools()

llama.cpp supports OpenAI-compatible tool calling.

In HTTP-server mode this depends on the server exposing /v1/chat/completions with tool support. In Python-binding mode it depends on the underlying llama-cpp-python version and model. CRP advertises support and lets the runtime fail if the model or server does not implement it.

generate_chat_with_tools(messages, tools, **kwargs)

Generate with llama.cpp tool/function calling.

Returns (text, finish_reason, tool_calls, raw_assistant_message).

LLMProvider

Bases: ABC

Abstract interface for LLM backends.

Implementations: OpenAIAdapter, AnthropicAdapter, LlamaCppAdapter, CustomProvider.

Subclasses must implement the three core abstract methods: generate_chat, count_tokens, and context_window_size. The remaining methods provide optional metadata, cost estimates, and tool or streaming support.

max_output_tokens property

Optional provider-reported max output tokens.

Returns:

Type Description
int | None

Maximum output tokens, or None if not known.

model_name property

Human-readable model identifier for diagnostics.

Returns:

Type Description
str

Class name by default; subclasses may override with a specific
str

model identifier.

is_thinking_model property

Return True if the model produces reasoning_content alongside output.

Thinking models (qwen3, deepseek-r1, o1, etc.) spend a significant portion of tokens on internal reasoning, requiring a larger generation reserve to avoid starving the final content output.

Returns:

Type Description
bool

True when the provider represents a thinking model.

generate_chat(messages, **kwargs) abstractmethod

Generate text completion from a message array.

Parameters:

Name Type Description Default
messages list[dict[str, str]]

List of chat messages, e.g. [{"role": "system", "content": "..."}, ...].

 required
**kwargs object

Provider-specific overrides (max_tokens, temperature, etc.)

 {}

Returns:

Type Description
tuple[str, str]

(output_text, finish_reason) where finish_reason is one of: - "length" - model hit generation reserve limit (physical wall) - "stop" - model produced natural EOS - "error" - provider returned an error

count_tokens(text) abstractmethod

Count tokens in text using this provider's exact tokenizer.

MUST return accurate per-model counts - no estimates.

Parameters:

Name Type Description Default
text str

Text to tokenise.

 required

Returns:

Type Description
int

Number of tokens in text.

context_window_size() abstractmethod

Return the model's physical context window size in tokens.

E.g. 128_000 for Claude 3 Opus, 8_192 for GPT-3.5.

Returns:

Type Description
int

Context window size in tokens.

cost_per_1k_tokens()

Return (input_cost_per_1k, output_cost_per_1k) in USD.

Returns:

Type Description
float

(input_cost, output_cost) per 1,000 tokens. Defaults to
float

(0.0, 0.0) for local models or unknown pricing.

supports_tools()

Return True if this provider supports function/tool calling.

Override in subclasses that implement generate_chat_with_tools(). The orchestrator uses this to decide between push (envelope) and pull (tool-mediated) context relay.

Returns:

Type Description
bool

True when tool-mediated dispatch is supported.

generate_chat_with_tools(messages, tools, **kwargs)

Generate with tool/function calling support.

Parameters:

Name Type Description Default
messages list[dict[str, object]]

Chat messages (may include tool role messages).

 required
tools list[dict[str, object]]

Tool definitions in OpenAI-compatible format.

 required
**kwargs object

Provider-specific overrides.

 {}

Returns:

Name Type Description
str

(output_text, finish_reason, tool_calls, raw_assistant_message)
where str
list[dict[str, object]] | None
  • output_text: Generated text content (may be empty if tool_calls)
dict[str, object] | None
  • finish_reason: "stop", "length", "tool_calls", or "error"
tuple[str, str, list[dict[str, object]] | None, dict[str, object] | None]
  • tool_calls: List of tool call dicts if finish_reason=="tool_calls"
tuple[str, str, list[dict[str, object]] | None, dict[str, object] | None]
  • raw_assistant_message: The full assistant message dict (for appending to conversation history in the tool loop)

Raises:

Type Description
NotImplementedError

By default; subclasses must override this method when supports_tools() returns True.

generate_chat_stream(messages, **kwargs)

Stream token chunks from the LLM.

Yields individual token chunks as strings. The return value (accessible via StopIteration.value) is the finish_reason ("stop" or "length").

Default implementation falls back to generate_chat() and yields the full output as a single chunk. Override in subclasses for real streaming.

Parameters:

Name Type Description Default
messages list[dict[str, str]]

Chat messages.

 required
**kwargs object

Provider-specific overrides.

 {}

Yields:

Type Description
str

Token chunks as strings.

Returns:

Type Description
str

The finish reason string.

OllamaAdapter

Bases: LLMProvider

Ollama REST API adapter for local model inference.

Parameters:

Name Type Description Default
model str

Model name (e.g. "llama3.1", "mistral", "gemma2").

 'llama3.1'
base_url str | None

Ollama API base URL. Defaults to OLLAMA_HOST env var or http://localhost:11434.

 None
context_size int | None

Override context window size (tokens).

 None
max_tokens int

Max output tokens per request (default: 2048).

 2048
timeout float

HTTP timeout in seconds (default: 300 - local models can be slow on CPU).

 300.0

max_output_tokens property

Return the max output tokens.

model_name property

Return the model name.

generate_chat(messages, **kwargs)

Call Ollama /api/chat endpoint with retry on transient failures.

Returns (output_text, finish_reason).

count_tokens(text)

Estimate token count (Ollama doesn't expose tokenizer directly).

Uses a conservative ~3.5 chars/token estimate for most models. For exact counts, use the LlamaCppAdapter with model_path instead.

context_window_size()

Return the current context window count.

Returns:

Type Description
int

int.

supports_tools()

Ollama supports OpenAI-compatible tool calling from 0.3.0+.

Actual availability depends on the model (e.g. llama3.1, qwen2.5, mistral). CRP advertises support here and lets the server fail gracefully if the loaded model is not tool-capable.

generate_chat_with_tools(messages, tools, **kwargs)

Generate with Ollama tool/function calling.

Returns (text, finish_reason, tool_calls, raw_assistant_message). Finish reason is "tool_calls" when the model emits tool calls.

OpenAIAdapter

Bases: LLMProvider

OpenAI chat completions adapter.

Works with OpenAI API and any OpenAI-compatible server (LM Studio, vLLM, llama.cpp server, Ollama OpenAI compat, TGI, etc.).

Model capabilities are auto-discovered via 3-layer resolution: 1. Exact match against known OpenAI models 2. Prefix match against 50+ open-source model families 3. Server-side probing (for vLLM, Ollama-compat endpoints) 4. Conservative fallback (8K context) - safe for unknown models

Parameters:

Name Type Description Default
model str

Model name (e.g. "gpt-4o", "qwen3-4b", "llama3.1").

 'gpt-4o'
api_key str | None

API key. Defaults to OPENAI_API_KEY env var.

 None
base_url str | None

Override API base URL (for LM Studio, vLLM, etc.).

 None
context_size int | None

Override auto-discovered context window (tokens).

 None
max_tokens int | None

Override auto-discovered max output tokens per request.

 None
timeout float

HTTP timeout in seconds (default: 120).

 120.0

max_output_tokens property

Return the max output tokens.

model_name property

Return the model name.

is_thinking_model property

Detect if the current model is a thinking/reasoning model.

generate_chat(messages, **kwargs)

Call OpenAI chat completions API with retry on transient failures.

Handles "thinking" models (Qwen3, DeepSeek-R1, o1, etc.) that split output into reasoning_content + content fields. CRP extracts the final content and preserves the full reasoning for downstream extraction.

Returns (output_text, finish_reason).

count_tokens(text)

Count tokens using tiktoken (exact) or fallback heuristic.

context_window_size()

Return the current context window count.

Returns:

Type Description
int

int.

cost_per_1k_tokens()

OpenAI pricing per 1K tokens (USD) - updated 2025-Q2.

supports_tools()

OpenAI and compatible servers support function/tool calling.

generate_chat_with_tools(messages, tools, **kwargs)

Generate with OpenAI tool/function calling.

Returns (text, finish_reason, tool_calls, raw_assistant_message). When the model wants to call tools, finish_reason="tool_calls" and the tool_calls list contains structured call requests. The raw_assistant_message is the full message dict for appending to conversation history (required by the OpenAI tool protocol).

generate_chat_stream(messages, **kwargs)

Stream token chunks from OpenAI.

Yields individual token deltas. Return value is finish_reason.