Of special mention is the fact that article deals only with single precision computational bottlenecks. Even on Nvidia’s latest generation of GPUs (the “Fermi” series), double precision lags single precision performance by a factor of two or so. That said, much of what Volkov has to say is equally applicable to double precision.

Volkov’s motivation for the article — if any is ever needed for seeking improved computational performance — are comparative benchmarks from 2007, in which an Intel Core2 Quad 2.4GHz and a GeForce 8800 GTX are compared for matrix multiplication and LU factorization. The comparison is between Intel’s MKL (Math Kernel Library) and Nvidia’s CUBLAS (CUDA Basic Linear Algebra Subroutines). On a raw computational basis, the GPU has a roughly 5:1 advantage over the CPU. In practice, Volkov shows the advantage to be between 2:1 and zero! A rather disappointing result, but one that motivated Volkov to dig deeper into the problem.

GPUs obtain their raw computational power advantage over CPUs by having vastly more ALUs (Arithmetic Logic Units). However, this advantage in raw compute power can only be realized if the ALUs are all kept busy. Failure to fully occupy the ALUs will throw away this raw compute advantage; but to fully occupy the ALUs with a given computational task requires a good understanding of the GPU architecture.

Using the coding techniques suggested by Volkov yields impressive performance improvements over code written in a way less “sympathetic” to the low-level architecture of GPU. Moreover, the techniques appear to yield a higher fraction of theoretical peak performance over a range of GPU architectures from old to new (GeForce 8600 GTS, 8800 GTX, 9800 GTX, and GTX 280). The net result is about a doubling of performance over Nvidia’s CULAS code. And the same techniques suggested by Volkov have also yielded performance gains for other numerical kernels, such as for the FFT (Fast Fourier Transform).

Some of the bottlenecks pinpointed by Volkov will no doubt be improved in future GPU architectures. That will go some way to relieving the burden of optimizing performance from the application programmer. But anyone wanting to make their computations truly fly on GPUs would do well to be mindful of the how instructions and memory interact on GPUs at a very low level as discussed in Volkov’s article.

Volkov’s BLAS algorithms are now incorporated into newer versions of CULBAS.

Key takeaways:

• Raw computational power of a GPU scales with the number of cores.
• Memory bandwidth on a GPU scales with the number of memory controllers.
• GPUs have no scalar capability in that scalar and pointer operations use full vector length.
• GPU supports variable vector length, with 512 being the maximum. Often, maximum performance can be achieved with vectors of shorter length; 64 elements, for example.
• There is a hard ceiling of 2/3 of peak raw performance when using an operand with shared memory. Volkov contends that this is an inherent bottleneck in the GPU architecture.
• Row pivoting in column-major layout on GPU is slow.
• Don’t run small tasks (vectors and matrices with “N” and “M” dimensions less than several thousand) on GPU because the overhead isn’t worth it.
• Large matrix operations can benefit significantly if spread across multiple GPUs.
• Advice: Keep as much data as possible in registers, but bear in mind that doing so only helps with vector operations.
• Advice: Use shared memory, which is smaller and slower than register memory, as a communication device only. It is not only an issue of memory access (fetch and store) because some instructions run slower (require more clock cycles) when using shared memory.
• Advice: For scalar and pointer operations use shorter vectors, not longer. And, more generally, use shorter vectors not longer; indeed, divide longer vectors and their operations into shorter ones in application code, if necessary.
• Advice: It is sometimes the case that it is better to reduce memory traffic by recomputing results rather than storing and retrieving them from memory.
• Advice: Try fast algorithms that might fail occasionally; check for failure, and recompute if needed using a slower algorithm.
• Advice: Use heterogeneous algorithms (of the type that can run on either the CPU or GPU) and make the switch to GPU only for problems that exceed a certain size, or based on other problem characteristics.
• My advice: Question all assumptions, and all coding advice! If the cost of doing experiments with your code is low, do them.

Notes:

• [1] “Understanding performance bottlenecks in numerical kernels on GPUs” by Vasily Volkov, May 21, 2010, can be found here: http://www.math.ntu.edu.tw/~wwang/mtxcomp2010/download/volkov2010-NTU1.pdf

BLAS is divided into three levels: 1) BLAS1 for vector operations, 2) BLAS2 for vector-matrix operations, and 3) BLAS3 for matrix-matrix operations. For each level, functions are available in single, double and complex (single and double precision).

Like many GPU libraries, using CUBLAS requires and understanding of what’s going on at the hardware level. Because CPU memory is separate from GPU memory, as a programmer you are in the position of having to manage objects in two memory spaces. For this reason, CUBLAS provides some “helper functions” to simplify use of the library. These helper functions, such as cublasAlloc() to allocate memory on the GPU device, and cublasSetVector() to move a vector from CPU memory to GPU memory, all return a cublasStatus code. cublasStatus is defined as unsigned int in the header file cublas.h together with symbolic names for the success and failure return codes, such as CUBLAS_STATUS_SUCCESS and CUBLAS_STATUS_ARCH_MISMATCH. The library must be initialized before use by calling cublasInit(), and shutdown in an orderly fashion after use by calling cublasShutdown(). As the programmer is largely responsible for managing memory on both the CPU host and GPU device, care with allocating and freeing memory is essential.

A careful perusal of the cublas.h header file is informative in itself, but is essential if you want to really understand how to best call the library from your own code. For example, if you want to call CUBLAS from C# using P/Invoke you will need to pay careful attention to function signatures in cublas.h, as these two examples illustrate.

// cublas.h: cublasInit() function signature.
cublasStatus CUBLASAPI cublasInit (void);

// C# wrapper for cublasInit()
[DllImport("cublas.dll",CallingConvention=CallingConvention.StdCall)]
public static extern cublasStatus cublasInit();

// cublas.h: cublasAlloc() function signature.
cublasStatus CUBLASAPI cublasAlloc (int n, int elemSize, void **devicePtr);

// C# wrapper for cublasAlloc()
[DllImport("cublas.dll", CallingConvention = CallingConvention.StdCall)]
public static extern cublasStatus cublasAlloc(int n, int elemSize, out IntPtr devicePtr);

Given the history of BLAS and the fact that it was originally written in FORTRAN it uses vector and matrix indexing that starts at one, in contrast to typical C/C++ libraries that use indexing that starts at zero. CUBLAS is a C/C++ based library API, but the documentation does discuss how to call CUBLAS functions from FORTRAN. Another wrinkle associated with CUBLAS’ FORTRAN past is that matrices are stored in memory using “column-major” indexing, which is opposite to the “row-major” indexing of C/C++. When passing matrices to CUBLAS functions be sure to pass them in column-major order. These choices — to use indexing that starts at 1 and column-major order — were made so that the CUBLAS has maximum compatibility with existing FORTRAN environments.

The library is distributed under the “Modified Berkeley Software Distribution License”.

Key takeaways:

• Care must be taken when using the CUBLAS library because vector and matrix indexing starts at 1 and matrices are stored in column-major order..
• CUBLAS is a numerical library optimized for GPU using CUDA that is distributed under a liberal (in the sense of being largely unfettered) redistribution license.

Notes:

• [1] The “CUBLAS library” is distributed as part of Nvidia’s “GPU Computing Toolkit” and includes the binary, header file (cublas.h) and documentation (CUBLAS_Library.pdf).
• [2] You can learn more about BLAS on Wikipedia: Basic Linear Algebra Subprograms. And over at Netlib: BLAS (Basic Linear Algebra Subprograms).
• [3] Optimized versions for BLAS are also available for Intel and AMD CPUs.
• [4] Sparse matrix versions of BLAS also exist.
• [5] You can learn about the history of BLAS from the “oral histories” section of the Society for Industrial and Applied Mathematics (SIAM) website: The History of Numerical Analysis and Scientific Computing. Jack Dongarra and Charles Lawson were key contributors in the development of BLAS.

Here you will find my thoughts and experiments with programming with GPUs. There is a bias towards application of GPUs in finance, but that’s simply because that’s where I do my work.