Run a 321-Billion-Parameter Model on Your Own GPU: How Project Maya Brings Datacenter-Scale AI Home

What it does

Project Maya does one very specific and very appealing thing: it lets you run Z.ai's open-source GLM-5.3-Flash, a 321-billion-parameter model, on the graphics cards you already own. This model has 321B total parameters but only activates about 18B per token, and it supports a context window of up to one million tokens. Models at this scale normally require a datacenter. Maya's goal is to squeeze that datacenter into your desktop — from a single GPU all the way up to sixteen, on NVIDIA or AMD hardware, under Linux or Windows. It ships with its own inference engine, a local server, a browser-based chat panel, and a live monitoring dashboard, and it exposes an OpenAI- and Anthropic-compatible API so you can plug it straight into your coding agent. In plain terms, if you want a private assistant with near-flagship capability that never leaves your machine, Maya lowers the barrier from "server room" to "desk drawer."

Why it blew up

It gained traction fast because it hits two nerves at once. First, more and more people care about keeping their data and conversations off someone else's cloud. Second, GLM-5.3-Flash is open-source and genuinely capable, yet practically impossible for a normal person to run. Maya is not a thin wrapper. It is a purpose-built "tiered memory scheduler" for this one model: it keeps the experts your conversation uses most on the GPU, the next tier in RAM, and the rest on your NVMe SSD, shuffling them as you chat. On top of that, the author published a flood of real-world benchmarks from actual users — 118 tokens/s on four RTX 4090s, 81 on a single RTX PRO 6000, and 30+ on a lone RTX 5090 — and those visible, reproducible numbers spread through the local-LLM community far faster than vague marketing ever could.

Technical highlights

The headline feature is expert placement across storage tiers. Because a MoE model only wakes a small subset of experts per step, Maya tracks which experts you use most, keeps the hot ones in VRAM, the warm ones in system memory, and the cold ones on disk, wasting almost none of your precious GPU memory. It also probes your machine's GPU, CPU, RAM, and SSD performance on first run and tunes itself automatically, so you never have to hand-edit obscure config files.

On speed, the published real numbers are impressive: up to 118 tok/s on four RTX 4090s, 81 tok/s on one RTX PRO 6000, and a steady 30+ tok/s even on a single RTX 5090. On quality, Maya's own quantization preserves between 97.7% and 99.2% of the FP8 model's zero-shot accuracy, so you basically cannot see the drop.

At the interface level it is fully OpenAI- and Anthropic-compatible with tool-call support, meaning coding agents like Claude Code or Codex can use it as a backend immediately. It also handles up to eight concurrent requests — handy when several people share one machine or one agent spawns many sessions. The browser UI bundles a chat window and a live monitor, supports images and adjustable reasoning depth, and the whole stack comes up with a single setup command.

Who it is for

If you are a local-LLM enthusiast who wants a private assistant that works without any network connection, Maya is almost tailor-made for you. Privacy-conscious users who refuse to hand their chat history to a cloud provider will love that nothing leaves the chassis. It also suits developers who want to give a coding agent a strong local model without fighting API rate limits and monthly bills. The only real prerequisite is a recent NVIDIA or AMD card plus enough disk space for the hefty weight files.

Quick start

Clone the repo and bring the whole environment up with one command:

git clone https://github.com/mw00/project-maya
cd project-maya
# run the one-command setup described in the README

Model weights are hosted on HuggingFace under the GLM-5.3-Flash-Maya-GGUF repo; download the quantization that fits your hardware.

Compared with alternatives

The closest cousins are general-purpose local inference engines like llama.cpp and Ollama. The difference is that Maya is not a "runs anything" tool; it is deeply specialized for exactly one model, GLM-5.3-Flash, which is precisely why it can push a 321B MoE onto consumer hardware at usable speeds and accuracy. The general engines win on breadth of supported models and ecosystem size, but they rarely go as aggressive on tiered expert scheduling for ultra-large MoEs. If your only goal is to run GLM-5.3-Flash fast and faithful, Maya is currently the most painless path; if you constantly swap between many smaller models, a general engine stays more flexible.

Where it fits and what to watch

It is worth being clear about the hardware reality. Running a 321B model, even with aggressive tiering, still demands serious disk and RAM: the weights alone are hundreds of gigabytes, and you will want fast NVMe because the cold experts live there. On a single consumer card you will see lower speeds and tighter context limits, so treat the headline 118 tok/s as a four-card number. The project is also young and led by a small team, so expect rapid changes and the occasional rough edge in setup. None of that undermines the core achievement: it proves a datacenter-class MoE can live on a desk. If you already own recent NVIDIA or AMD hardware and want a private, coding-agent-ready model, this is one of the most exciting self-hosting experiments of the moment.