📊
Pre-built Benchmarks
Memory bandwidth, matmul, attention, convolution — run standard CUDA workloads with zero setup.
Why nvProbe
🏎️
HPL & HPCG
Self-contained Linpack and CG benchmarks via NVIDIA HPC Benchmarks — auto-downloaded on setup.
🧠
MLPerf Inference
End-to-end MLPerf via cmx4mlperf with configurable model, framework, and precision.
📈
Infrastructure Audits
Capture max clocks, power caps, ECC state, PCIe link speed & NVLink — baseline today, re-run in 6 months, spot degradation instantly.
📊
Interactive Charts
Chart.js canvas-based charts with tooltip hover, zoom, and oscilloscope-style glow.
📦
Bundled CUDA Runtime
No system CUDA toolkit required — CuPy with [ctk] bundles everything via pip.
🖥️
Slurm Integration
Generate, submit, monitor, and collect results from HPC clusters — all from a single CLI.
Quick Start
Install
pip install nvprobe
Setup
Install CuPy, download HPL/HPCG binaries, generate configs
nvprobe setup
Check environment
Verify GPU detection, driver, and CUDA version
nvprobe env
Run benchmarks
nvprobe run --local
Generate report
nvprobe report --open
More commands
nvprobe compare --a results/run1 --b results/run2Compare runs
nvprobe run --config configs/cluster.yamlCustom config
nvprobe slurm submit --config configs/cluster.yamlSlurm submit
nvprobe setup --cuda 13Specific CUDA
Benchmarks
Bandwidth
H2D, D2H, D2D memory bandwidth across buffer sizes
CuPyBurn
Sustained MatMul burn with clock/temp tracking and throttling detection
CuPyMemtest
VRAM integrity: solid, checkerboard, random, walking-1 patterns
CuPyCustom Kernels
MatMul, tiled MatMul, fused attention, convolution
CUDA CHPL
High Performance Linpack (FP64)
MPI Datacenter GPUsHPCG
Conjugate Gradients
MPI Datacenter GPUsMLPerf
MLPerf Inference — test / find_performance / full
ONNX RuntimeFAQ
Can I run nvprobe on a workstation RTX GPU for HPL and HPCG benchmarks?
The NVIDIA HPC Benchmarks binaries (xhpl and xhpch) are validated for datacenter GPUs such as A100, H100, B200, and L40S. On RTX-series workstation cards the GPU initialization step may crash with a SIGSEGV. However, all other benchmarks — memory bandwidth, custom CUDA kernels (matmul, attention, convolution), and MLPerf inference — work on any CUDA-capable GPU. You can selectively disable HPL and HPCG in your YAML config and still run the full suite of bandwidth and custom kernel tests.
How does nvprobe handle MLPerf Inference setup and cuDNN detection?
nvprobe drives MLPerf Inference through cmx4mlperf (the MLCommons cm framework), supporting model, framework, scenario, and precision configuration. The mlcr tool discovers cuDNN through system CUDA library paths by default. If you installed cuDNN via pip install nvidia-cudnn-cuXX, pre-register the path manually: mlcr get,cudnn,nvidia --input=$(python3 -c 'import nvidia.cudnn; print(nvidia.cudnn.__path__[0])'). NVPROBE's setup command also installs CuPy with bundled CUDA libraries, so no system CUDA toolkit is required for the bandwidth and custom kernel benchmarks.
Does nvprobe support Slurm-based HPC clusters for multi-GPU benchmarking?
Yes, nvprobe has native Slurm integration. Use nvprobe slurm submit --config configs/cluster.yaml to generate job scripts, submit them to the specified Slurm partition, monitor their status with nvprobe slurm status, and automatically collect results once they complete. The YAML config lets you define gpus_per_node, partition name, and per-node benchmark parameters, making it suitable for large-scale HPC clusters with multiple GPU nodes running HPL, HPCG, or MLPerf in parallel.