GPU Memory Calculator for LLM Training

Training Configuration

Model Settings

Preset Model:

Model Name:

Parameters:

Layers:

Hidden Size:

Attention Heads:

Vocab Size:

Max Seq Length:

Mixture of Experts (MoE)

Enable MoE

Training Settings

Batch Size:

Gradient Accumulation:

Optimizer:

Precision:

Activation Checkpointing:

Parallelism

Tensor PP:

Pipeline PP:

Data PP:

Sequence Parallel

Effective GPUs: 8

Training Engine

Engine Type:

ZeRO Stage:

Offload Optimizer:

Offload Parameters:

ZeRO Init (ZeRO-3)

Sharding Strategy:

Megatron-LM Options:

Use Distributed Optimizer

Num Micro-Batches (PP):

Advanced Training Options:

Gradient Clipping:

Weight Decay:

Learning Rate:

Warmup Steps:

Hardware

Number of GPUs:

GPU Model:

Results

Memory Breakdown

Per GPU: -- GB

Total All GPUs: -- GB

CPU Memory: -- GB

Component Breakdown

Model Parameters: -- GB

Gradients: -- GB

Optimizer States: -- GB

Activations: -- GB

Overhead: -- GB

Params Grads Opt Act

Feasibility

Status: --

Utilization: --%

Formula Explanation

Run a calculation to see the formula breakdown.

🚀 GPU Memory Calculator

Training Configuration

Model Settings

Mixture of Experts (MoE)

Training Settings

Parallelism

Training Engine

Hardware

Results

Memory Breakdown

Component Breakdown

Feasibility

Formula Explanation

References:

Inference Configuration

Model Settings

Inference Settings

TGI-Specific Settings

vLLM-Specific Settings

TensorRT-LLM-Specific Settings

SGLang-Specific Settings

Inference Results

Memory Breakdown

Performance Estimates

Multi-Node Training Configuration

Model Settings

Node Configuration

Training Configuration

Parallelism Strategy

Multi-Node Results

Network Overhead

Time Impact

Optimization Suggestions