1.1   Debug and Verification

make BUILD=debug BENCHMARK=off

Running make with BUILD=debug builds the debug version of the benchmark kernels, where no optimizations are performed, line numbers and debug symbols are included as well as DEBUG will be defined. The resulting binary will be found in the bin subdirectory and named lbmbenchk-linux-<compiler>-debug.

Specifying BENCHMARK=off turns on verification (VERIFICATION=on), statistics (STATISTICS=on), and VTK output (VTK_OUTPUT=on) enabled.

Please note that the generated binary will therefore exhibit a poor performance.

1.2   Benchmarking

To generate a binary for benchmarking run make with

make

As default BENCHMARK=on and BUILD=release is set, where BUILD=release turns optimizations on and BENCHMARK=on disables verfification, statistics, and VTK output.

1.3   Release and Verification

Verification with the debug builds can be extremely slow. Hence verification capabilities can be build with release builds:

make BENCHMARK=off

1.4   Compilers

Currently only the GCC and Intel compiler under Linux are supported. Between both configuration can be chosen via CONFIG=linux-gcc or CONFIG=linux-intel.

1.5   Cleaning

For each configuration and build (debug/release) a subdirectory under the src/obj directory is created where the dependency and object files are stored. With

make CONFIG=... BUILD=... clean

a specific combination is select and cleaned, whereas with

make clean-all

all object and dependency files are deleted.

1.6   Options Summary

Options that can be specified when building the framework with make:

2.1   Command Line Parameters

Running the binary with -h list all available parameters:

Usage:
./lbmbenchk -list
./lbmbenchk
    [-dims XxYyZ] [-geometry box|channel|pipe|blocks[-<block size>]] [-iterations <iterations>] [-lattice-dump-ascii]
    [-rho-in <density>] [-rho-out <density] [-omega <omega>] [-kernel <kernel>]
    [-periodic-x]
    [-t <number of threads>]
    [-pin core{,core}*]
    [-verify]
    -- <kernel specific parameters>

-list           List available kernels.

-dims XxYxZ     Specify geometry dimensions.

-geometry blocks-<block size>
                Geometetry with blocks of size <block size> regularily layout out.

If an option is specified multiple times the last one overrides previous ones. This holds also true for -verify which sets geometry dimensions, iterations, etc, which can afterward be override, e.g.:

$ bin/lbmbenchk-linux-intel-release -verfiy -dims 32x32x32

Kernel specific parameters can be opatained via selecting the specific kernel and passing -h as parameter:

$ bin/lbmbenchk-linux-intel-release -kernel kernel-name -- -h
...
Kernel parameters:
[-blk <n>] [-blk-[xyz] <n>]

A list of all available kernels can be obtained via -list:

$ ../bin/lbmbenchk-linux-gcc-debug -list
Lattice Boltzmann Benchmark Kernels (LbmBenchKernels) Copyright (C) 2016, 2017 LSS, RRZE
This program comes with ABSOLUTELY NO WARRANTY; for details see LICENSE.
This is free software, and you are welcome to redistribute it under certain conditions.

LBM Benchmark Kernels 0.1, compiled Jul  5 2017 21:59:22, type: verification
Available kernels to benchmark:
   list-aa-pv-soa
   list-aa-ria-soa
   list-aa-soa
   list-aa-aos
   list-pull-split-nt-1s-soa
   list-pull-split-nt-2s-soa
   list-push-soa
   list-push-aos
   list-pull-soa
   list-pull-aos
   push-soa
   push-aos
   pull-soa
   pull-aos
   blk-push-soa
   blk-push-aos
   blk-pull-soa
   blk-pull-aos

2.2   Kernels

The following list shortly describes available kernels:

• push-soa/push-aos/pull-soa/pull-aos: Unoptimized kernels (but stream/collide are already fused) using two grids as source and destination. Implement push/pull semantics as well structure of arrays (soa) or array of structures (aos) layout.
• blk-push-soa/blk-push-aos/blk-pull-soa/blk-pull-aos: The same as the unoptimized kernels without the blk prefix, except that they support spatial blocking, i.e. loop blocking of the three loops used to iterate over the lattice. Here manual work sharing for OpenMP is used.
• list-push-soa/list-push-aos/list-pull-soa/list-pull-aos: The same as the unoptimized kernels without the list prefix, but for indirect addressing. Here only a 1D vector of is used to store the fluid nodes, omitting the obstacles. An adjacency list is used to recover the neighborhood associations.
• list-pull-split-nt-1s-soa/list-pull-split-nt-2s-soa: Optimized variant of list-pull-soa. Chunks of the lattice are processed as once. Postcollision values are written back via nontemporal stores in 18 (1s) or 9 (2s) loops.
• list-aa-aos/list-aa-soa: Unoptimized implementation of the AA pattern for the 1D vector with adjacency list. Supported are array of structures (aos) and structure of arrays (soa) data layout is supported.
• list-aa-ria-soa: Implementation of AA pattern with intrinsics for the 1D vector with adjacency list. Furthermore it contains a vectorized even time step and run length coding to reduce the loop balance of the odd time step.
• list-aa-pv-soa: All optimizations of list-aa-ria-soa. Additional with partial vectorization of the odd time step.

Note that all array of structures (aos) kernels might require blocking (depending on the domain size) to reach the performance of their structure of arrays (soa) counter parts.

The following table summarizes the properties of the kernels. Here D means direct addressing, i.e. full array, I means indirect addressing, i.e. 1D vector with adjacency list, x means supported, whereas -- means unsupported. The loop balance B_l is computed for D3Q19 model with double precision floating point for PDFs (8 byte) and 4 byte integers for the index (adjacency list). As list-aa-ria-soa and list-aa-pv-soa support run length coding their effective loop balance depends on the geometry. The effective loop balance is printed during each run.

3.1   Padding

With correct padding cache and TLB thrashing can be avoided. Therefore the number of (fluid) nodes used in the data layout is artificially increased.

Currently automatic padding is active for kernels which support it. It can be controlled via the kernel parameter (i.e. parameter after the --) -pad. Supported values are auto (default), no (to disable padding), or a manual padding.

Automatic padding tries to avoid cache and TLB thrashing and pads for a 32 entry (huge pages) TLB with 8 sets and a 512 set (L2) cache. This reflects the parameters of current Intel based processors.

Manual padding is done via a padding string and has the format mod_1+offset_1(,mod_n+offset_n), which specifies numbers of bytes. SoA data layouts can exhibit TLB thrashing. Therefore we want to distribute the 19 pages with one lattice (36 with two lattices) we are concurrently accessing over as much sets in the TLB as possible. This is controlled by the distance between the accessed pages, which is the number of (fluid) nodes in between them and can be adjusted by adding further (fluid) nodes. We want the distance d (in bytes) between two accessed pages to be e.g. d % (PAGE_SIZE * TLB_SETS) = PAGE_SIZE. This would distribute the pages evenly over the sets. Hereby PAGE_SIZE * TLB_SETS would be our mod_1 and PAGE_SIZE (after the =) our offset_1. Measurements show that with only a quarter of half of a page size as offset higher performance is achieved, which is done by automatic padding. On top of this padding more paddings can be added. They are just added to the padding string and are separated by commas.

A zero modulus in the padding string has a special meaning. Here the corresponding offset is just added to the number of nodes. A padding string like -pad 0+16 would at a static padding of two nodes (one node = 8 b).