SparkX 2.5 pairs a 4-billion-parameter model with a configured context window of just over one million tokens, using a hybrid attention layout to keep most of that history cheap. The architecture cuts the KV cache to roughly a quarter of what full attention across every layer would need at the same window, but the full window is still measured in tens of gigabytes. The published benchmarks show task-specific wins rather than across-the-board superiority, and the memory math changes completely at smaller context budgets.

https://www.youtube.com/watch?v=UzYlemYewFU

4B parameters vs. 1M tokens

Two numbers define the model, and they measure different things. The 4B figure is the count of learned parameters stored in the weights — the model itself. The million-token figure is the conversation window, the space available for prompts, source files, logs, and tool results. Tokens are pieces of text, not words or files, so a million tokens is not a fixed amount of code.

The official repository configures --context-length at 1,048,576 and warns that this setting requires sufficient device memory, advising a lower value when memory runs short (https://github.com/XHToken/Spark-X2.5). A PR verifying the 4B checkpoint reports a parameter count of 4,112,079,360, with 36 layers for the 4B variant (https://github.com/pytorch/executorch/pull/22865). The 4B describes the whole model, unlike a mixture of experts that activates only a subset of parameters per token.

Hybrid attention: 512-token local windows plus global layers

Most of Spark's attention layers see only a short local stretch of the conversation. The configuration repeats three sliding-window layers followed by one full-attention layer across the 36 layers, which works out to 27 local layers and nine global ones if the 3:1 pattern holds throughout. The local direct view is limited to 512 tokens; the global layers keep a path to earlier positions across the whole window.

That distinction is about which positions a given layer can access directly, not about the model forgetting everything past 512 tokens. Information can still move through the layer stack, and the global layers preserve long-range reach. The practical consequence: an instruction placed early can remain reachable through the global layers even after it falls outside every local window. It makes the connection possible, but it does not guarantee correct recall.

Shared KV storage and long-context training

Spark is described as using grouped query attention, with 16 query heads sharing four sets of keys and values. Several query heads read the same stored notes instead of keeping a separate copy of the key and value history for every head. The specific 16-to-4 split is not confirmed by sources outside the project's own description, so treat it as the project's stated configuration rather than an independently verified figure.

The window also involved training, not just a setting. The project describes a dedicated long-context stage with sequences reaching one million tokens, which is what supports the native context claim. That training detail, too, is not independently verifiable from the sources checked. Even if it holds, length in training does not establish reliable recall at every position inside the window.

KV cache budget: 39 GB at full window, much less at smaller budgets

The KV cache is the working memory a conversation builds as it runs — stored keys and values that later tokens can consult. Holding keys and values at 16-bit precision, the estimate is about 4.1 KB per token per attention layer. At the full configured window, the nine global caches account for almost all of that storage; including the bounded local caches, the total comes to roughly 39 GB of cache alone, before weights, temporary buffers, or the runtime itself. These are calculated storage requirements from the project, not measurements from a laptop.

For comparison, the same dimensions with all 36 layers keeping full history would need roughly 155 GB — also an estimate rather than a measurement. Spark brings that down to about a quarter. A large reduction can still leave a large object. At 32,768 tokens the same 16-bit estimate is about 1.3 GB, and at 131,000 tokens it is about 4.9 GB. The local portion barely changes once its window fills; the global portion grows with the conversation. Runtime allocation strategies vary, and simultaneous conversations need more memory, so these figures are a starting budget rather than a promise that a particular GPU will fit the whole workload.

Weight files and cache compression are separate costs

Compressing the weights does not compress the conversation. The BF16 safetensors for the 4B checkpoint total 8,224,158,720 bytes, roughly 8.2 GB (https://github.com/pytorch/executorch/pull/22865). A community GGUF listing puts the Q4_K_M file at 2.60 GB (https://huggingface.co/stornic56/Spark-X2.5-4B-GGUF). An 8-bit option has been described at around 4.4 GB, though that figure was not confirmed by the sources checked.

Choosing the smaller download does not automatically shrink a 16-bit KV cache. Runtimes may also support cache compression, which attacks the second bill, but its format support and quality tradeoffs need their own checks. There is also the time required to read a large prompt before answering; reducing the number of global layers does not eliminate their long-range attention work, so fitting a million tokens does not by itself make processing them feel interactive.

Benchmarks show task-specific wins, not universal superiority

The published evaluation table reports strong results in coding, tool use, browsing, and mathematics, in thinking mode, with some competitor scores taken from other cards or papers. These are the project's own numbers and have not been independently reproduced for every compressed download. Spark scores 44.4 on SWE-bench Pro against 29.4 for Qwen 3.5 4B, but on SWE-bench Verified it scores 41.6 against 53.1 for Qwen 9B — even the coding category does not produce one universal winner.

Other headline rows include 40.9 on BrowseComp, 54.6 on MCP Atlas, and 90.7 on AIME 2026, covering browsing, tool-based tasks, and competition mathematics respectively. The less flattering rows matter too: on GPQA, Spark scores 67.4 against Qwen 9B's 77.2. Instruction following is mixed — Spark leads the listed Qwen models on some rows, while Gemma 4 12B beats it on another. The sensible reading depends on both the task and the comparison model.

Runtime support and packaging have caught up—verify your setup

Local support has moved quickly, and older advice about custom builds is no longer the whole picture. The model card lists native support versions and states that llama.cpp has added Spark support. The llama.cpp release pages available for checking discuss Nvidia DGX Spark hardware rather than confirming this model's addition, so treat an advertised command as untested until your installed runtime reports the architecture.

Download options are broader than a single file: community GGUF quantizations and a 4-bit MLX conversion are documented, and the card also points to an INT8 checkpoint, 8-bit MLX, and an ONNX export, though those were not confirmed in the sources checked. A filename tells you how the weights are stored, not whether your chosen app understands the architecture. The MLX cards point to Spark's dedicated runtime rather than the standard library, and the original checkpoint route has its own instructions; the two are not interchangeable.

Runtime documentation is described as recommending 8-bit or full precision for tool calling and warning that 4-bit can reduce structured-argument accuracy. That is guidance rather than a measured quality result, and a community GGUF page separately warns that mixed quantizations may fail strict tool-class checks three to four times more often than uniform quants. If the model fits only by breaking the tool arguments your workflow depends on, the saved memory has not bought a working agent.

A practical trial: context budget, precision, and tests

Pick a repository whose expected behavior you already know. Place a relevant instruction early in the material, then request a change that depends on it. Inspect the patch and run the tests. Repeat at a few context lengths while recording memory use and time to the first answer, and for agents, validate the tool arguments against their schemas before execution. This is a proposed evaluation, not a result reported by the project.

Spark advertises more than 200 languages and ships under Apache 2.0. Broad language coverage does not mean equal quality everywhere, so if an English benchmark does not represent your repository's comments or support documents, include those in the trial. The architecture reduces the cost of keeping history, and the published strengths fit coding and tool workflows. Choose a supported runtime, a context budget that matches the task, and enough weight precision to preserve the behavior you need. A larger window removes some packing work; it does not remove the need to check that the model used the right evidence.

Takeaways

• SparkX 2.5 is a 4B model whose official example configures a 1,048,576-token context window, with device memory as the limiting factor.
• Its attention layout repeats three sliding-window layers per full-attention layer across 36 layers, leaving nine global layers to carry long-range reach.
• At 16-bit precision, the project estimates roughly 39 GB of KV cache at the full window and about 1.3 GB at 32,768 tokens; an all-layer full-history design would need roughly 155 GB.
• Weight file size and conversation cache size are separate budgets: a 4-bit GGUF around 2.6 GB does not shrink a 16-bit cache.
• Published benchmark results are the project's own and are task-specific, with wins on SWE-bench Pro, BrowseComp, MCP Atlas, and AIME 2026 alongside losses on SWE-bench Verified and GPQA.