Rust Package

The Rust package target is sipp-rs. It publishes the sipp library crate for Rust applications and re-exports the high-level client API plus selected runtime, backend, lifecycle, shard, provider, and gateway types.

sipp-rs depends on sipp-sys, the native llama.cpp FFI crate. Installing sipp-rs from crates.io builds the native backend from source on the target machine; it is not a binary wheel-style package.

See the Library API Overview for the shared add, query, chat, and embed contracts.

Install

cargo add sipp-rs

The release workflow publishes sipp-sys first, then publishes sipp-rs. Applications depend on the sipp-rs package and import the sipp crate.

Build Requirements

Rust applications that depend on sipp-rs need the normal Rust toolchain plus the native build tools used by sipp-sys:

A C/C++ compiler for the target platform.
CMake.
Ninja or a compatible CMake generator.
Platform SDKs required by the selected backend.

The CPU native backend is the baseline and does not require a Cargo feature. Backend features add their own requirements:

cuda: CUDA Toolkit plus a compatible NVIDIA driver.
metal: macOS with Xcode command line tools.
vulkan: Vulkan SDK or system Vulkan development libraries.
openmp: OpenMP compiler/runtime support for the target platform.

Use It For

Rust applications that need local GGUF inference.
Gateway-backed query, chat, and embedding calls.
Direct provider descriptors behind the providers feature.
Shared Sipp value types across application boundaries.

Local GGUF Query

#![allow(unused)]
fn main() {
use sipp::{
    SippClient, SippQueryRequest, SippTextOptions, EndpointDescriptor,
    LocalTextOptions,
};
use sipp::engine::{
    CacheRuntimeConfig, ContextRuntimeConfig, KvReuseMode, NativeRuntimeConfig,
    ObservabilityRuntimeConfig, SchedulerRuntimeConfig,
};

async fn run(
    model_path: std::path::PathBuf,
) -> Result<(), Box<dyn std::error::Error>> {
    let mut client = SippClient::new();
    let endpoint = client
        .add(
            "default",
            EndpointDescriptor::local(model_path, runtime_config()),
        )
        .await?;

    let response = client
        .query(SippQueryRequest {
            endpoint: Some(endpoint),
            prompt: "Explain Sipp in one sentence.".to_string(),
            options: SippTextOptions {
                max_tokens: Some(64),
                ..Default::default()
            },
            local: LocalTextOptions {
                context_key: Some("rust-local".to_string()),
                ..Default::default()
            },
            ..Default::default()
        })
        .await?;
    println!("{}", response.text);
    Ok(())
}

fn runtime_config() -> NativeRuntimeConfig {
    NativeRuntimeConfig {
        context: ContextRuntimeConfig {
            n_ctx: Some(2048),
            ..Default::default()
        },
        scheduler: SchedulerRuntimeConfig {
            continuous_batching: true,
            prefill_chunk_size: 0,
            ..Default::default()
        },
        cache: CacheRuntimeConfig {
            mode: KvReuseMode::LiveSlotPrefix,
            ..Default::default()
        },
        observability: ObservabilityRuntimeConfig {
            runtime_metrics: true,
            backend_profiling: false,
        },
        ..Default::default()
    }
}
}

See Runtime Options for the shared runtime config groups and request option boundaries.

Gateway Query

#![allow(unused)]
fn main() {
use sipp::{
    SippClient, SippQueryRequest, SippTextOptions, EndpointDescriptor,
    GatewayAuthentication, GatewayEndpointConfig, GatewayRoutes, GatewaySecret,
    GatewayTimeoutPolicy,
};

let mut client = SippClient::new();
let endpoint = client
    .add(
        "gateway",
        EndpointDescriptor::gateway(GatewayEndpointConfig {
            target: std::env::var("SIPP_GATEWAY_TARGET")?,
            base_url: std::env::var("SIPP_GATEWAY_URL")?,
            routes: GatewayRoutes::default(),
            authentication: GatewayAuthentication::Bearer(GatewaySecret::new(
                std::env::var("SIPP_GATEWAY_TOKEN")?,
            )),
            static_headers: Default::default(),
            timeouts: GatewayTimeoutPolicy::default(),
            protocol_options: Default::default(),
        }),
    )
    .await?;

let response = client
    .query(SippQueryRequest {
        endpoint: Some(endpoint),
        prompt: "Explain gateway inference.".to_string(),
        options: SippTextOptions {
            max_tokens: Some(64),
            ..Default::default()
        },
        ..Default::default()
    })
    .await?;
println!("{}", response.text);
}

Keyboard shortcuts