AI Hardware & Software Notes https://ai.147444.xyz/en/ Notes on local LLM deployment, inference acceleration, GPU clusters, and edge AI. en The VRAM Budget for a 70B Model: Quantization, KV Cache, and Concurrency https://ai.147444.xyz/en/posts/vram-70b-quantization/ https://ai.147444.xyz/en/posts/vram-70b-quantization/ Tue, 20 Oct 2026 00:00:00 GMT A practical accounting of where VRAM actually goes when serving a 70B model, and how quantization level and concurrency constrain each other. Interconnect Choices for Multi-Node Inference: Ethernet vs RoCE vs InfiniBand https://ai.147444.xyz/en/posts/interconnect-tensor-parallel/ https://ai.147444.xyz/en/posts/interconnect-tensor-parallel/ Mon, 12 Oct 2026 00:00:00 GMT Measured performance of three interconnect options for tensor-parallel inference on a two-node 8-GPU cluster, and when InfiniBand is worth the premium. Running a 7B Model at the Edge: The Power, Latency, and Thermal Trade-off https://ai.147444.xyz/en/posts/edge-7b-power-thermal/ https://ai.147444.xyz/en/posts/edge-7b-power-thermal/ Mon, 28 Sep 2026 00:00:00 GMT Practical trade-offs when deploying a 7B model on Jetson-class and domestic edge accelerators, focused on power ceilings and thermal limits under sustained load. Two DGX Sparks Linked: How Large a Model Can a Desktop Cluster Run? https://ai.147444.xyz/en/posts/dual-dgx-spark-cluster/ https://ai.147444.xyz/en/posts/dual-dgx-spark-cluster/ Tue, 15 Sep 2026 00:00:00 GMT Measured model scale and throughput for two DGX Spark units linked into a small cluster, under tensor and pipeline parallelism. Choosing an Inference Framework: Where vLLM, SGLang, and TensorRT-LLM Each Fit https://ai.147444.xyz/en/posts/inference-framework-selection/ https://ai.147444.xyz/en/posts/inference-framework-selection/ Wed, 02 Sep 2026 00:00:00 GMT How the three mainstream inference frameworks differ in latency, throughput, and deployment complexity, and how to pick by workload.