Two DGX Sparks Linked: How Large a Model Can a Desktop Cluster Run?
Measured model scale and throughput for two DGX Spark units linked into a small cluster, under tensor and pipeline parallelism.
Clustering desktop-class AI hardware is a recent trend. Whether two linked DGX Sparks can carry 70B-class inference was the question I most wanted to answer.
Hardware
- Two DGX Spark units, each with a unified-memory accelerator
- Direct link over the built-in high-speed port, no switch in between
- Software: standard CUDA + NCCL, containerized
The key constraint
Unified memory is both the feature and the limit of this class of device. CPU and accelerator share a memory pool: the upside is fitting models larger than VRAM, the downside is bandwidth far below dedicated VRAM. That means:
- Small models (<13B) run fast, entirely within accelerator bandwidth
- Mid-size models (~30B) start to feel the bandwidth wall
- Large models (70B+) fit, but throughput drops noticeably
Measured results
| Model size | Single unit | Two units TP=2 | Note |
|---|---|---|---|
| 8B | Fast | Slightly slower | Communication overhead dominates |
| 32B | Usable | Clearly faster | Clear gain |
| 70B INT4 | Marginal | Usable | Bandwidth-bound |
The 32B tier benefits most: it barely fits on one unit, and two units leave room for concurrency. 70B runs, but throughput is capped by unified-memory bandwidth and is unsuitable for high concurrency.
Where it fits
The positioning is clear:
- Individual research and small teams: limited budget, need mid-scale models
- Prototype validation: verify parallel strategy before committing to a real cluster
- Low-concurrency service: internal tools and experimental applications
Not suitable for: high-concurrency production serving, or latency-critical paths.
Pitfalls I hit
- NCCL’s default topology detection is not always optimal on these devices; the link path needed manual specification
- Accessing unified memory inside a container requires specific configuration, otherwise it degrades to plain memory copies
- Thermals: two units placed close together affect each other, and both need clearance under sustained load
Conclusion
Two DGX Sparks are a strong price/performance option at the 32B tier. 70B runs, but do not expect high throughput. Treat it as an experimental platform that can hold large models, not a production inference server, and expectations line up.