NVIDIA SDK

Description

In addition to Cray's Craypat and VIHPS' Scorep, Nvidia provides the following performance analysis tools on Piz Daint:

  • nvprof (command line profiler)

  • nvvp (visual profiler)

 

Remote profiling

First collect performance data on the remote system using nvprof:

unset COMPUTE_PROFILE 
export PMI_NO_FORK=1 
srun -C gpu -n1 nvprof -o nvprof.output.%h.%p <myexe>

 

%h (and %p) will report the node name (and process ID) in the output file name ( nvprof -h for more details ).

  • cudatoolkit/7.5 requires to export LD_LIBRARY_PATH=$CRAY_CUDATOOLKIT_DIR/lib64

Then view and analyze the data using nvvp:

nvvp: File -> Import -> Nvprof
or
nvprof -i <nvprof output file>

 

 

 

 

Debugging

For cpu and gpu debugging, please use DDT.

For reference, Nvidia debugging tools are:

 

  • cuda-gdb (debugger)
  • cuda-memcheck (memory checker):

    • srun -n1 cuda-memcheck --force-blocking-launches yes <executable path>

  • cuobjdump (object dump for binaries)
  • nvdiasm (cubin extraction)

Further Information

Please refer to Nvidia's Documentation for more details.

For questions on using Nvidia tools, please contact us at help(at)cscs.ch.