Overview

Graphics Processing Units (GPUs) are critical components for tasks like machine learning, data processing, and gaming. Monitoring GPU performance metrics helps you understand resource utilization, identify bottlenecks, and ensure optimal performance. This guide explains how to check and monitor key GPU metrics using NVIDIA's System Management Interface (nvidia-smi) tool.

Prerequisites

• A system with one or more NVIDIA GPUs installed
• NVIDIA GPU drivers properly installed
• Terminal or command-line access to your system

Step by Step Guide

1. Check basic GPU information using the simple nvidia-smi command:
   bash

    

   nvidia-smi

   This displays a summary of all GPUs including model, driver version, and basic utilization metrics.
2. View comprehensive GPU metrics with a custom query:
   bash

    

   nvidia-smi --query-gpu=timestamp,index,name,temperature.gpu,power.draw,clocks.gr,clocks.mem,utilization.gpu,utilization.memory,memory.used,memory.total,pstate,fan.speed --format=csv,noheader,nounits

   
   This command displays the following metrics:
   • timestamp: Current system time
   • index: GPU device index (useful for multi-GPU systems)
   • name: GPU model name
   • temperature.gpu: GPU temperature in Celsius
   • power.draw: Current power consumption in Watts
   • clocks.gr: Graphics clock speed in MHz
   • clocks.mem: Memory clock speed in MHz
   • utilization.gpu: GPU utilization percentage
   • utilization.memory: GPU memory utilization percentage
   • memory.used: Used GPU memory in MiB
   • memory.total: Total GPU memory in MiB
   • pstate: Current performance state (P0 to P12, with P0 being highest performance)
   • fan.speed: Fan speed percentage

1. 3. Set up continuous monitoring by combining with the watch command:
      bash

       

      watch -n 1 "nvidia-smi --query-gpu=timestamp,index,name,temperature.gpu,power.draw,utilization.gpu,memory.used,memory.total --format=csv,noheader,nounits

      This updates the display every second with a simplified set of metrics for easier viewing.

4. Format output as needed using nvidia-smi options:
   • For CSV format (good for logging):

     nvidia-smi --query-gpu=name,temperature.gpu,utilization.gpu --format=csv > gpu_log.csv

   • For XML format:

     nvidia-smi --query-gpu=name,temperature.gpu,utilization.gpu --format=xml

1. 5. Target a specific GPU in a multi-GPU system:

      nvidia-smi -i 0 --query-gpu=temperature.gpu,power.draw --format=csv

      Replace 0 with the index of the GPU you want to monitor.

1. 6. Monitor processes using GPU resources:

      nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

      This shows which processes are using GPU resources and how much memory they're consuming.

Additional Notes & Tips

• Set up automated monitoring by creating a script that runs nvidia-smi commands and logs the output to a file
• Consider using graphical monitoring tools like NVIDIA-SMI GUI, GPU-Z, or NVIDIA's DCGM for more advanced visualizations
• Critical thresholds to watch for:
  • Temperature: Generally keep below 85°C for most GPUs
  • Memory utilization: Near 100% can indicate memory bottlenecks
  • Power consumption: Consistently high values may indicate inefficient usage or potential thermal issues