Niko Veit, developer of the open-source AI inference engine "Strata," announced on October 24th that the AI model "DeepSeek-V4.1-Flash" can now be run on a PC with 64GB of main memory. This utilizes a quantized version of the model, employing the "GSQ-RCO" compression method developed by DASLab at the Institute of Science and Technology Austria.
Strata is software designed to run AI models on your personal computer. Instead of solely fitting the model into GPU memory (VRAM), it also places data in main memory and on SSDs, and comprehensively processes it by utilizing the CPU and other resources.
The supported "DeepSeek-V4.1-Flash-1017" is a massive model with approximately 763 billion parameters.
The publicly available models come in three versions with average weight precisions of 1.75-bit, 2-bit, and 2.25-bit. The download sizes, including related files, are approximately 239GB, 257GB, and 274GB, respectively, with the 2-bit version requiring about 265GB of free space.
In a measurement example using "Strata 0.2.5," a GeForce RTX 5070 with 12GB of dedicated memory and 64GB of main memory was used, generating 4 to 6 tokens per second with the 2-bit version.
The provider of the quantized version states that the 2-bit version achieved the same average score as the original model in three evaluations: mathematics, science, and code generation. While this expands the possibility of trying out massive models on a home PC, hundreds of gigabytes of free space and considerable waiting times are still required.
Made with AI
