WARP: a runtime-free inference engine for MoE models in C

</> AI-friendly version

WARP is the rare case where the C-engine for inferring MoE models does without any external runtime, and memory is saved so that even a MacBook Pro with 64 GB of RAM can run the full Kimi K3 model with 2.78 trillion parameters directly from an NVMe drive. Speed? 0.45-0.62 tokens per second is not a record, but for such a machine without a GPU it is at least interesting. And if you need something lighter, there is Kimi-Linear with 48 billion parameters: it fits into a 19 GB container, takes only 1.32 GB of RAM and already issues 10.6 tokens/s. All this with no CUDA, no BLAS, minimal stack on C11 and Python only for conversion and an OpenAI compatible server.

What is WARP and what makes it unique

WARP is an inference engine written in pure C with no dependencies on third-party runtimes. Its main feature is that it is optimized specifically for Mixture of Experts (MoE) models, which usually require a ton of memory. Instead of keeping all the experts in RAM, WARP stores only the general part of the model and loads the rest directly from the NVMe drive – and does it so efficiently that even a laptop with 64 GB of RAM can run a behemoth like Kimi K3 (2.78 trillion parameters) at 0.45-0.62 tokens per second.

Why is it important? Because most modern MoE models simply do not fit into the RAM of ordinary machines. WARP solves this problem by minimizing RAM requirements: for example, Kimi-Linear 48B (19 GB on disk) requires only 1.32 GB of RAM, while the inference speed is 10.6 tokens/s. And all this without GPU, CUDA or BLAS – just C11 and Python to convert the models and run an OpenAI-compatible server.

The project is completely open source, built via make, tested via make check, and requires nothing more than a compiler and a willingness to experiment. If you need overhead-free inference, WARP is worth considering.

WARP Performance: Tests and Figures

WARP proves that MoE model inference can be done quickly — even without a GPU. On a MacBook Pro with 64GB RAM, the full Kimi K3 model (2.78T parameters in a 982GB container) generates tokens at 0.45-0.62 per second. Yes, these are not hundreds of tokens, but considering the size of the model and the lack of a video card, the result is impressive. Experts are loaded directly from NVMe, and the general part of the model sits in RAM, minimizing delays.

With Kimi-Linear 48B, WARP shows a completely different picture: 10.6 tokens/s on the same machine. The model takes up only 19 GB on disk, and only 1.32 GB is needed in RAM. The difference in performance is simply explained: smaller model, fewer experts, faster switching. Both tests confirm the main feature of WARP — efficient use of resources. No BLAS, no CUDA, just pure C11 and minimal dependencies. Even Python is here only to convert models and run an OpenAI-compatible server.

Assembly via make, tests via make check – everything is transparent and predictable. WARP does not claim record speeds, but it allows you to run large models where it seemed impossible: on a laptop, without cloud GPUs, without complex settings. And most importantly, without compromises with security or stability.

Technical stack and ease of integration

WARP is a minimalist MoE model inference engine written in pure C11 with no third-party runtimes. The technology stack is deliberately concise: C for the core, Python only for converting models and running an OpenAI-compatible server. No GPU, CUDA or BLAS: everything runs on the CPU, and the experts boot directly from the NVMe drive, saving RAM.

Assembly is simple: make and you’re done. Testing? make check will check the correctness of the operation on the example of Kimi-Linear 48B (19 GB container, 1.32 GB RAM, 10.6 token/s on MacBook Pro). For full-sized models like the Kimi K3 2.78T (982GB), WARP keeps the common part in RAM, and the experts upload on the fly – 0.45-0.62 tokens/s on the same machine with 64GB of memory.

The code is open source and available on GitHub. Integration is a matter of several commands: you convert the model, start the server, and that’s it – the OpenAI-compatible API works without additional dependencies. Simple, fast, without magic.

How WARP affects AI developers and SEO specialists

WARP is a breakthrough for AI developers tired of relying on expensive GPUs. Now, large MoE models like Kimi K3’s 2.78 trillion parameters can be run on a regular MacBook Pro with 64GB of RAM — without cloud services or dedicated graphics cards. Speed? Up to 0.62 tokens/s for giant models and 10.6 tokens/s for lightweight versions (like the Kimi-Linear 48B, which only takes up 1.32GB of memory). This means: local development, quick idea testing and minimal infrastructure overhead.

WARP opens new horizons for SEO specialists. Local inference allows you to analyze content in real time without delays in requests to external APIs. Imagine: you generate optimized texts directly on your laptop, test their relevance with the help of MoE models and immediately see the result – without the limitations of cloud tariffs. And this is also a step towards personalization of content on the fly: models will be able to adapt to a specific audience without the risk of data leakage.

The future of MoE models with WARP looks even more attractive. Currently, the engine works with containers from 19 GB (like Kimi-Linear) to almost a terabyte (full Kimi K3), but optimization is ongoing. Expect that soon even smartphones will be able to run lightweight versions of such models — this will change the rules of the edge AI game. For developers, this is a chance to create products that previously seemed like a fantasy: autonomous chatbots, offline analytics of big data, built-in AI assistants in mobile applications.