AI Hardware & Software Notes
Notes on local LLM deployment, inference acceleration, GPU clusters, and edge AI.
Writing
Writing →- The VRAM Budget for a 70B Model: Quantization, KV Cache, and Concurrency
A practical accounting of where VRAM actually goes when serving a 70B model, and how quantization level and concurrency constrain each other.
- Interconnect Choices for Multi-Node Inference: Ethernet vs RoCE vs InfiniBand
Measured performance of three interconnect options for tensor-parallel inference on a two-node 8-GPU cluster, and when InfiniBand is worth the premium.
- Running a 7B Model at the Edge: The Power, Latency, and Thermal Trade-off
Practical trade-offs when deploying a 7B model on Jetson-class and domestic edge accelerators, focused on power ceilings and thermal limits under sustained load.
- Two DGX Sparks Linked: How Large a Model Can a Desktop Cluster Run?
Measured model scale and throughput for two DGX Spark units linked into a small cluster, under tensor and pipeline parallelism.
- Choosing an Inference Framework: Where vLLM, SGLang, and TensorRT-LLM Each Fit
How the three mainstream inference frameworks differ in latency, throughput, and deployment complexity, and how to pick by workload.