← Brown Cyber Services
Theme Text Layout

Geryon Split Planner

This planner shows why GPU memory doesn't add up across machines, and compares what each way of splitting a large model costs per token on the hardware you describe. It's for anyone deciding whether to spread a model across several computers, and you'll see which approaches fit and which would be painfully slow, as estimates, not benchmarks.

Why you can't pool VRAM across boxes, and what you get instead. Pick a model, set up the GPU hosts and the links between them, and compare per-token cost under each strategy. Numbers are first-order estimates for single-user decode; they explain the shape of the problem, not a benchmark.

Model

Hosts

Pipeline order is top to bottom. Each hop uses the slower of the two adjacent links. VRAM overhead of 0.6 GB per card is reserved for the runtime.

Memory

Per-token cost by strategy

Compute (weights read from VRAM)Link latency × sync countBytes over the wire

Bars share one scale, capped at the slowest strategy that fits. A strategy that doesn't fit is shown in red with the reason.

Pipeline layer split

Layers are assigned proportional to free VRAM after each card's share of the KV cache. The hop count is hosts − 1, plus one return trip to the host that owns the sampler.

Prefill (prompt processing)

How the numbers are built

Compute

Decode at batch 1 is bandwidth-bound: every token reads every weight once. Time per card = bytes on that card ÷ VRAM bandwidth. The card's FLOPS barely matter here, which is why an old card with wide memory can keep up.

Pooled VRAM (why it's absurd)

A transparent address space means the card doing the work pulls weights it doesn't hold over the slowest link, every token. Cost = non-resident bytes ÷ link bandwidth. Over Ethernet this is seconds to minutes per token. No software fixes this; it's the link.

Pipeline parallel

Contiguous layer blocks live on each card and never move. Per token: sum of every card's compute, plus one latency per hop, plus the activation vector (hidden × 2 bytes) per hop. Hops ≈ host count, so latency is paid a handful of times. Cards work one after another, never together, so throughput is roughly the serial sum. Adding a card adds capacity, not speed.

Tensor parallel

Every layer is sliced across all cards, which all work on the same layer at once. Each transformer block needs two all-reduces (after attention, after the MLP), so sync count = layers × 2. Each sync pays a full link round trip and moves roughly 2 × (n−1)/n × hidden × 2 bytes. With equal slicing, the slowest card gates every step. Over NVLink the syncs are microseconds and this is the fastest strategy; over Ethernet the 80-odd syncs per token dominate.

MoE and hybrid models

Total params set what has to sit in VRAM; active params set what is read per token, so a 180B-A6B model fits like a 180B and decodes like a 6B. KV layers counts only full-attention layers: Mamba, Gated DeltaNet and linear-attention layers keep a small fixed state instead of a per-token cache, so hybrids like Nemotron Nano and Qwen3.5/3.8 carry far less KV than their layer count suggests. For MLA models (GLM-5.3) the GQA ratio is set so the formula lands near the real compressed cache size.

Assumptions you should distrust

Link latency is a one-way application-level figure for a small message (TCP RPC on Ethernet, not raw PHY latency). KV cache per token = 2 × layers × hidden × 2 bytes ÷ GQA ratio. No overlap, no batching, no speculative decode, no MoE. Real llama.cpp RPC runs tend to land at 60–80% of the pipeline estimate here.

© 2026 Brown Cyber Services Inc · All rights reserved · Colorado · Access logs retained 90 days