How-To Article: Debugging Common Memory Errors on Large-Scale Systems

1. Document Scope

This guide is designed for researchers, developers, and HPC (High Performance Computing) administrators who encounter memory-related issues on large-scale computing systems. It provides a systematic approach to diagnosing and fixing memory errors such as leaks, out-of-bounds access, or out-of-memory (OOM) events. The steps below focus on tools and best practices applicable to distributed environments with potentially hundreds or thousands of compute nodes.

Target Audience

• HPC software developers
• Researchers performing large-scale simulations or data processing
• Cluster and supercomputer administrators

Prerequisites

• Familiarity with job schedulers (e.g., Slurm, PBS, LSF)
• Basic Linux command-line skills
• Some experience with debugging or profiling tools (e.g., Valgrind, Intel Inspector)

2. Steps to Debug Common Memory Errors

Step 1: Verify Job Resource Requests

1. Check Allocated vs. Used Memory

   • Slurm Example:

     sacct -j <job_id> --format=JobID,MaxRSS,AveRSS,Elapsed,State
     

     • MaxRSS (Maximum Resident Set Size) shows peak memory usage.
   • PBS Example:

     qstat -fx <job_id> | grep resources_used.mem
     

   • Interpretation: If your actual usage (MaxRSS) is at or near the requested memory limit, you might encounter OOM errors or slow performance due to paging.

2. Live Monitoring

   • Commands:

     free -h         # Displays free/used memory in a human-readable format
     top / htop      # Shows process-level memory usage
     

   • Interpretation: Useful for diagnosing if any single process is consuming excessive memory on a specific node.

Step 2: Check System and Scheduler Logs

1. Kernel Logs for OOM-Killer

   • Commands:

     dmesg | grep -i oom
     journalctl -k | grep -i oom
     

   • Interpretation: When the system triggers the OOM killer, these logs typically identify which process was killed and why.

2. Scheduler Logs

   • Slurm: Check slurmd.log on compute nodes for memory-related error messages.
   • PBS: Look in mom_logs on the node where the job ran.

Step 3: Use Debugging Tools to Find Memory Leaks and Corruption

1. Valgrind (Single-Node Debugging)

   • Command:

     valgrind --leak-check=full --track-origins=yes ./my_app
     

   • Interpretation: Valgrind identifies invalid memory reads/writes, uninitialized variables, and memory leaks. Often used on a small subset of nodes before scaling.

2. Intel Inspector

   • Command:

     inspxe-cl -collect mi1 -- ./my_app
     

   • Interpretation: Locates memory errors and thread race conditions in Intel-compiled HPC applications.

3. ARM DDT (Scalable Parallel Debugger)

   • Command:

     ddt mpirun -n <num_procs> ./my_app
     

   • Interpretation: Allows debugging across numerous processes in an MPI job, highlighting memory issues across distributed ranks.

4. CUDA memcheck (GPU Memory Errors)

   • Command:

     cuda-memcheck ./my_gpu_app
     

   • Interpretation: Useful for GPU-accelerated workloads, detecting out-of-bounds or misaligned accesses in CUDA kernels.

Step 4: Profile and Isolate Memory Hotspots

1. HPCToolkit

   • Command:

     hpcrun -e MEM_UOPS_RETIRED -o hpctoolkit-out ./my_app
     hpcviewer hpctoolkit-out
     

   • Interpretation: Generates call-path profiles highlighting the most memory-intensive portions of your application.

2. Intel VTune Profiler

   • Command:

     amplxe-cl -collect memory-access -- ./my_app
     

   • Interpretation: Helps analyze memory bandwidth usage, cache misses, and NUMA behaviors.

3. ARM MAP

   • Command:

     map --profile ./my_app
     

   • Interpretation: Shows memory usage timelines alongside CPU load, facilitating correlation between code segments and memory spikes.

Step 5: Apply Fixes and Optimize

1. Fix Detected Leaks or Corruption

   • Action: Resolve issues flagged by Valgrind, Intel Inspector, etc. This may involve deallocating buffers, re-checking array bounds, or initializing variables properly.

2. Data Layout and Blocking

   • Why: Inefficient data structures or poor memory alignment can lead to high cache miss rates.
   • Action: Consider blocking or tiling techniques, and ensure arrays are aligned to cache boundaries.

3. NUMA Binding

   • Command:

     numactl --cpunodebind=0 --membind=0 ./my_app
     

   • Interpretation: Allocates memory on the same NUMA node(s) as the CPU cores. Minimizes remote memory access overhead.

4. Huge Pages

   • Why: Larger page sizes reduce TLB misses, improving performance for memory-intensive workloads.
   • Example: Some clusters provide module load craype-hugepages2M or similar modules.

5. Revise Resource Requests

   • Slurm:

     #SBATCH --mem=64G
     

   • PBS:

     #PBS -l mem=64GB
     

   • Interpretation: Make sure you request enough memory, but avoid vast over-provisioning that can lead to inefficient cluster usage.

Step 6: Re-Test and Scale Up

1. Check for Regressions

   • Run your application again under Valgrind or other tools on a single node or small subset of nodes to confirm fixes.

2. Incremental Scaling

   • Scale the job to more nodes, monitoring memory usage (sstat in Slurm or logs in PBS) to ensure stable behavior.

3. Continuous Integration (CI)

   • Integrate memory checks into automated pipelines if possible, catching future issues before large production runs.

3. Conclusion

Debugging memory errors on large-scale systems demands a methodical approach. By verifying your job’s resource requests, examining system logs, and leveraging specialized debugging and profiling tools, you can locate and resolve common memory pitfalls—ranging from leaks to NUMA imbalances. Implementing data layout optimizations, ensuring proper resource requests, and gradually validating changes at scale will help maintain stable, efficient HPC environments.

Key Takeaways

• Match Memory Requests to Usage: Prevent OOM by aligning requested resources to actual consumption.
• Employ Advanced Tools: Use Valgrind, Intel Inspector, ARM DDT, or CUDA memcheck to detect leaks or mismanagement, especially in parallel or GPU contexts.
• Focus on Data Layout: Optimize memory access patterns to reduce overhead and improve performance across hundreds of nodes.
• Test, Fix, and Scale: Always verify on a small scale first, then ramp up to full production size, monitoring for new or recurring issues.

Armed with these strategies, HPC practitioners can confidently tackle memory errors on the largest supercomputers and cluster architectures, ensuring high throughput and reliable scientific or data-driven results.