nvprobe Documentation

nvprobe runs its own optimized CUDA workloads via CuPy and cuBLAS — no external model files, no PyTorch/TensorFlow wrappers. Every benchmark is self-contained. Install, configure, and get GPU performance numbers in minutes.

Quickstart

pip install nvprobe
nvprobe setup -f
nvprobe run --local
nvprobe report

The run --local command reads nvprobe/configs/local.yaml by default. Generated reports land in nvprobe/reports/.

Config Reference

Every benchmark in a YAML config supports its own params block. Below are all recognized parameters per benchmark.

bandwidth

Parameter	Type	Default	Description
`sizes_mb`	list[int]	`[1, 4, 16, 64, 256, 1024]`	Buffer sizes in MB for H2D / D2H / D2D transfers
`iterations`	int	`100`	Measurement iterations per size (higher = more stable)

- name: bandwidth
  enabled: true
  params:
    sizes_mb: [1, 4, 16, 64, 256, 1024, 4096]
    iterations: 100

stream

Parameter	Type	Default	Description
`sizes`	list[int]	`[10000000, 100000000]`	Array element counts for STREAM operations
`iterations`	int	`20`	Measurement iterations per size (higher = more stable)

- name: stream
  enabled: true
  params:
    sizes: [10000000, 100000000, 1000000000]
    iterations: 20

STREAM benchmark (John D. McCalpin) measuring sustainable memory bandwidth via four operations: COPY (a[i] = b[i]), SCALE (a[i] = q * b[i]), ADD (a[i] = b[i] + c[i]), and TRIAD (a[i] = b[i] + q * c[i]). Reports bandwidth in GB/s with mean/min/max/std statistics.

memtest

Parameter	Type	Default	Description
`sizes_mb`	list[int]	`[1024]`	VRAM segment to test (MB per run). Actual size capped at 85 % free memory.

- name: memtest
  enabled: true
  params:
    sizes_mb: [1024, 4096]

Runs 7 patterns: solid 0x00 / 0xFF, checkerboard 0xAA / 0x55, pseudo-random, walking‑1 high/low. Reports total errors per pattern.

burn

Parameter	Type	Default	Description
`duration_sec`	int	`30`	Sustained load duration in seconds
`matrix_size`	int	`8192`	Square matrix dimension (N) for cuBLAS SGEMM

- name: burn
  enabled: true
  params:
    duration_sec: 60
    matrix_size: 16384

Samples SM/MEM clocks, temperature, and power every second. Throttling is flagged if the SM clock drops >15 % from its initial value.

custom

Parameter	Type	Default	Description
`kernels`	list[str]	`[matmul, attention]`	Kernels to run: `matmul`, `tiled_matmul`, `attention`, `conv2d`
`matrix_sizes`	list[int]	`[512, 1024, 2048]`	Problem sizes (N for N×N, or seq_len for attention)
`iterations`	int	`50`	Measurement iterations per kernel per size

- name: custom
  enabled: true
  params:
    kernels: [matmul, tiled_matmul, attention, conv2d]
    matrix_sizes: [512, 1024, 2048, 4096]
    iterations: 50

Requires CuPy [ctk] extras. Without CUDA toolkit headers, custom kernels fall back with an allocation error. Install with pip install cupy-cuda12x[ctk].

hpl

Parameter	Type	Default	Description
`problem_sizes`	list[int]	`[4096, 8192]`	HPL matrix sizes N. Auto-calculated from GPU memory if empty.
`binary`	str	`~/.nvprobe/tools/xhpl`	Path to xhpl binary

- name: hpl
  enabled: true
  params:
    problem_sizes: [4096, 8192, 16384]
    binary: "~/.nvprobe/tools/xhpl"

Datacenter GPUs only (A100, H100, B200, L40S). RTX cards may crash (SIGSEGV). On Blackwell GPUs ensure OpenMPI ≥ 5.x — older MPI versions may segfault.

hpcg

Parameter	Type	Default	Description
`grid_sizes`	list[int]	`[128, 256]`	3D grid sizes (uniform n_x = n_y = n_z)
`binary`	str	`~/.nvprobe/tools/xhpcg`	Path to xhpcg binary

- name: hpcg
  enabled: true
  params:
    grid_sizes: [128, 256, 512]
    binary: "~/.nvprobe/tools/xhpcg"

Same GPU / MPI constraints as HPL.

mlperf

Parameter	Type	Default	Description
`model`	str	`resnet50`	MLPerf model name
`framework`	str	`onnxruntime`	Inference framework
`scenario`	str	`Offline`	MLPerf scenario
`category`	str	`edge`	`edge` or `datacenter`
`implementation`	str	`reference`	MLPerf implementation
`test_query_count`	int	`100`	Number of queries for test mode
`mode`	str	`test`	`test`, `find_performance`, or `full`
`batch_size`	int	—	Optional custom batch size override

- name: mlperf
  enabled: false  # requires: pip install cmx4mlperf
  params:
    model: resnet50
    framework: onnxruntime
    scenario: Offline
    category: edge
    implementation: reference
    test_query_count: 100

score

The AI Accelerator Score is a CLI command (not a benchmark) that calculates normalized performance scores for AI accelerators.

Parameter	Type	Required	Description
`--name`	str	No	Accelerator name for display (default: "accelerator")
`--bandwidth`	float	Yes	Sustained memory bandwidth in GB/s
`--compute`	str	Yes	Compute throughput per precision (e.g., fp32=67,fp16=134,fp8=268)

nvprobe score --name "H100 SXM" --bandwidth 3350 --compute fp32=67,bf16=134,fp16=134,fp8=268,int8=268

Output includes per-precision breakdown, Global Performance (GP), and Normalized Score (H100 = 1.00). Uses Roofline-inspired model: P[p] = min(C[p], BW × I_stream[p] / f).

Example Configs

Local workstation

name: workstation
precisions: [fp32, fp16]
batch_sizes: [1, 32, 64]
benchmarks:
  - name: bandwidth
    params:
      sizes_mb: [1, 4, 16, 64, 256]
      iterations: 50
  - name: memtest
  - name: burn
    params:
      duration_sec: 30

Infrastructure audit only

name: node-audit
gpu:
  min_count: 1
benchmarks:
  - name: memtest
    params:
      sizes_mb: [4096, 16384]
  - name: burn
    params:
      duration_sec: 60
      matrix_size: 16384
  - name: bandwidth
    params:
      sizes_mb: [256, 1024, 4096]
      iterations: 100

HPC cluster (Slurm)

name: cluster-suite
slurm:
  enabled: true
  partition: gpu
  nodes: 4
  gpus_per_node: 8
  time_limit: "02:00:00"
precisions: [fp32, fp16]
batch_sizes: [1, 32, 64, 128, 256]
benchmarks:
  - name: bandwidth
    params:
      sizes_mb: [1, 4, 16, 64, 256, 1024, 4096]
      iterations: 100
  - name: memtest
    params:
      sizes_mb: [4096, 16384]
  - name: burn
    params:
      duration_sec: 60
      matrix_size: 16384
  - name: hpl
    params:
      problem_sizes: [4096, 8192, 16384, 32768]
  - name: hpcg
    params:
      grid_sizes: [128, 256, 512]

How Benchmarks Work

Each benchmark runs as an independent subprocess launched via sys.executable -m nvprobe.benchmarks._cuda.<name>. Output is captured as JSON, parsed, and stored in the local SQLite database.

bandwidth, memtest, burn — use CuPy + NumPy. No CUDA toolkit required.
custom — requires CuPy with [ctk] extras (CUDA toolkit headers for JIT kernel compilation).
hpl, hpcg — call NVIDIA HPC Benchmarks binaries (xhpl / xhpch) via MPI.
mlperf — drives cmx4mlperf (the MLCommons cm framework).

nvprobe is not a PyTorch/TensorFlow wrapper. It does not run user-provided model scripts or datasets. Benchmarks are self-contained CUDA workloads optimized for hardware validation.

Slurm Quickstart

1. Prepare a config

Use the HPC cluster example above or generate one:

nvprobe init -f
nvprobe run --config nvprobe/configs/default.yaml --dry-run

2. Submit to Slurm

nvprobe slurm --config nvprobe/configs/default.yaml --action full

This single command:

Generates one sbatch script per node.
Submits all scripts.
Monitors job progress (polls every 30 s).
Collects & merges per-node databases into a single benchmarks.db.

3. Generate the report

nvprobe report --results nvprobe/results

Each node writes its own database (node_0/benchmarks.db, node_1/benchmarks.db, …). GPUs are re-indexed globally and hostnames are appended to model names in the merged environment table.

Per-node, not per-benchmark. One Slurm job runs all benchmarks on all GPUs of a single node. This keeps scheduler overhead low and captures a clean per-node fingerprint.

Environment Diagnostics

Every run captures a full environment fingerprint stored as JSON alongside the benchmark results. The HTML report shows these fields in the Environment table:

Field	Source	Notes
GPU model	nvidia-smi	Hostname appended in Slurm merged reports
Memory total	nvidia-smi	MiB
SM Clock (max)	`clocks.max.sm`	Hardware ceiling — detects degraded silicon
Mem Clock (max)	`clocks.max.mem`	Hardware ceiling
Power Limit	`power.limit`	Watts — N/A on WDDM (Windows)
ECC mode	`ecc.mode.current`	Enabled / Disabled / N/A
ECC errors	`ecc.errors.corrected / uncorrected`	Cumulative volatile counts
PCIe link (max)	`pcie.link.gen.max` × `pcie.link.width.max`	Maximum negotiated link speed
NVLink active links	`nvlink.active.links` + fallback `nvlink -s`	B200 / NVSwitch compatible
Driver / CUDA	nvidia-smi banner	Also shown in overview cards

Baseline today, re-run in 6 months. If the max SM clock dropped or an NVLink link went inactive, the report makes the regression instantly attributable. No guesswork.