A homelab builder has fitted two Nvidia RTX 3090 graphics cards into an Optane-based system running the Kimi K3 Q2_K AI model, reaching 3.11 tokens per second for generation and 16.8 tokens per second for prompt evaluation with 45 of the model's 94 shards living on a RAM drive.
That RAM drive holds 414 GB of the 865 GB model file, with the remainder in Optane. The arrangement follows a pattern familiar to enthusiasts who reserve system memory as a fast, disk-like tier, even though it remains volatile and demands a large amount of RAM. With both GPUs but no RAM drive, the same build ran at 2.20 tokens per second for generation and 13.3 for prompt evaluation.
The GPUs plainly cannot hold the model. Two RTX 3090s at 24 GB apiece offer 48 GB of VRAM, roughly 5.5% of the 865 GB file, which is likely measured in gibibytes but reported in GB as written by the builder. Kimi K3 is a mixture-of-experts model with 896 experts, 16 selected per token plus two shared, giving 104B active parameters out of 2.8T. At 16-bit precision the full 2.8T parameters would take about 5.6 TB, while Moonshot's 4-bit MXFP4 release comes to about 1.56 TB and the Q2_K file used here shrinks that to 865 GB, averaging under 3 bits per parameter.
In this configuration the routed-expert tensors stay on the CPU side, and the RTX 3090s handle attention, the dense feed-forward layer that is not split into experts, the shared experts, other active-dense components, the KV cache, the stored context of the current conversation and GPU compute buffers. Those GPU-resident parts are used on every token, so gains there come from GPU compute and memory bandwidth. Moving shards into Linux's tmpfs, which keeps files in virtual memory, means that portion of the model no longer has to be read from slower memory, even though the original build was designed to save money with Optane. The builder has said DRAM is faster than Optane.
The result landed inside the builder's earlier guess of somewhere between 2 and 4 tokens per second. Installing the GPUs required cutting metal out of the case and relocating the PSU to fit the cards and their cooling, and the builder noted the improved throughput did not change the original thermal warning. Two 12W blower fans for the Optane DIMMs remain in place alongside the dual 350W-class GPUs.
Further optimizations are possible, including customized quantization along the lines of techniques used by lawyer Nick Mermiges. Selecting quantization per tensor or layer, or by where data lives, might ease bandwidth and capacity pressure on the slower tiers and make the setup faster yet. The builder also noted that tokens per second alone is not enough and that output quality must be weighed in any future plan.
More AI news from TechManNews.







