Ollaya runs local 'decision models' that answer typed questions in milliseconds

Ollaya is a local runtime for decision models — small networks that answer typed questions (choice, score, yes/no) about a piece of text or JSON in a single forward pass, skipping the token-by-token generation of a chat LLM. That design is the whole pitch: a five-question request to the bundled ‘laya’ model runs in roughly 8–10 ms end-to-end on an RTX 4090, versus 236–276 ms for the comparable hosted API it mimics. Every model also returns calibrated probabilities, so you can set real confidence thresholds rather than trusting a raw label.

The project follows the Ollama playbook: one binary, one command (ollaya run laya), with open weights pulled from their authors’ Hugging Face repos, pinned to a commit and verified by sha256. It ships several models with different tradeoffs — laya for speed, ‘decider’ for accuracy, ‘von’ for an 8k-token context, and qwen3guard for safety screening — and exposes drop-in-compatible /v1/systemone and /v1/models endpoints, so an existing hosted-API SDK works unchanged against a local server. The runtime is Apache-2.0 and runs on CPU everywhere, with NVIDIA GPU acceleration on Linux, WSL 2, or Docker.

The angle that matters most is privacy. Decision models are typically pointed at support tickets, emails, and user messages — often the most sensitive text an organization holds — and Ollaya scores that data in place, binding to 127.0.0.1 by default with no per-token metering or external API calls. For teams already sending classification traffic to a hosted service, it’s a path to keep the data on their own hardware while cutting both latency and cost.