1 <?xml version="1.0" encoding="utf-8" ?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
5 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
6 <meta name="generator" content="Docutils 0.12: http://docutils.sourceforge.net/" />
7 <title>LBM Benchmark Kernels Documentation</title>
8 <style type="text/css">
11 :Author: David Goodger (goodger@python.org)
12 :Id: $Id: html4css1.css 7614 2013-02-21 15:55:51Z milde $
13 :Copyright: This stylesheet has been placed in the public domain.
15 Default cascading style sheet for the HTML output of Docutils.
17 See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
18 customize this style sheet.
21 /* used to remove borders from tables and images */
22 .borderless, table.borderless td, table.borderless th {
25 table.borderless td, table.borderless th {
26 /* Override padding for "table.docutils td" with "! important".
27 The right padding separates the table cells. */
28 padding: 0 0.5em 0 0 ! important }
31 /* Override more specific margin styles with "! important". */
32 margin-top: 0 ! important }
34 .last, .with-subtitle {
35 margin-bottom: 0 ! important }
41 text-decoration: none ;
48 margin-bottom: 0.5em }
50 object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] {
54 /* Uncomment (and remove this text!) to get bold-faced definition list terms
62 div.abstract p.topic-title {
66 div.admonition, div.attention, div.caution, div.danger, div.error,
67 div.hint, div.important, div.note, div.tip, div.warning {
69 border: medium outset ;
72 div.admonition p.admonition-title, div.hint p.admonition-title,
73 div.important p.admonition-title, div.note p.admonition-title,
74 div.tip p.admonition-title {
76 font-family: sans-serif }
78 div.attention p.admonition-title, div.caution p.admonition-title,
79 div.danger p.admonition-title, div.error p.admonition-title,
80 div.warning p.admonition-title, .code .error {
83 font-family: sans-serif }
85 /* Uncomment (and remove this text!) to get reduced vertical space in
87 div.compound .compound-first, div.compound .compound-middle {
88 margin-bottom: 0.5em }
90 div.compound .compound-last, div.compound .compound-middle {
99 div.dedication p.topic-title {
107 div.footer, div.header {
116 div.line-block div.line-block {
122 margin: 0 0 0.5em 1em ;
123 border: medium outset ;
125 background-color: #ffffee ;
130 div.sidebar p.rubric {
131 font-family: sans-serif ;
134 div.system-messages {
137 div.system-messages h1 {
141 border: medium outset ;
144 div.system-message p.system-message-title {
151 h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
152 h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
164 img.align-left, .figure.align-left, object.align-left {
169 img.align-right, .figure.align-right, object.align-right {
174 img.align-center, .figure.align-center, object.align-center {
190 /* reset inner alignment in figures */
192 text-align: inherit }
194 /* div.align-center * { */
195 /* text-align: left } */
197 ol.simple, ul.simple {
201 list-style: decimal }
204 list-style: lower-alpha }
207 list-style: upper-alpha }
210 list-style: lower-roman }
213 list-style: upper-roman }
227 white-space: nowrap }
236 font-family: sans-serif ;
241 font-family: sans-serif ;
252 pre.literal-block, pre.doctest-block, pre.math, pre.code {
256 pre.code .ln { color: grey; } /* line numbers */
257 pre.code, code { background-color: #eeeeee }
258 pre.code .comment, code .comment { color: #5C6576 }
259 pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold }
260 pre.code .literal.string, code .literal.string { color: #0C5404 }
261 pre.code .name.builtin, code .name.builtin { color: #352B84 }
262 pre.code .deleted, code .deleted { background-color: #DEB0A1}
263 pre.code .inserted, code .inserted { background-color: #A3D289}
266 font-family: sans-serif ;
267 font-style: oblique }
269 span.classifier-delimiter {
270 font-family: sans-serif ;
274 font-family: sans-serif }
277 white-space: nowrap }
285 span.section-subtitle {
286 /* font-size relative to parent (h1..h6 element) */
290 border-left: solid 1px gray;
298 margin-bottom: 0.5em }
301 border-left: solid 1px black;
304 table.docutils td, table.docutils th,
305 table.docinfo td, table.docinfo th {
306 padding-left: 0.5em ;
307 padding-right: 0.5em ;
308 vertical-align: top }
310 table.docutils th.field-name, table.docinfo th.docinfo-name {
313 white-space: nowrap ;
316 /* "booktabs" style (no vertical lines) */
317 table.docutils.booktabs {
319 border-top: 2px solid;
320 border-bottom: 2px solid;
321 border-collapse: collapse;
323 table.docutils.booktabs * {
326 table.docutils.booktabs th {
327 border-bottom: thin solid;
331 h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
332 h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
336 list-style-type: none }
339 <style type="text/css">
342 h1, h2, h3, h4, h5, h6 {
343 font-family: sans-serif;
345 background-color: #dcdcdc;
349 background-color: gray;
365 table, table.docutils, td, th {
369 table.citation, table.footnote {
374 background-color: lavender ;
378 xxbackground-color: aliceblue;
379 background-color: white;
382 xxbackground-color: lavender;
383 background-color: whitesmoke;
391 <div class="document">
394 <div class="line-block">
395 <div class="line">Copyright</div>
396 <div class="line-block">
397 <div class="line">Markus Wittmann, 2016-2018</div>
398 <div class="line">RRZE, University of Erlangen-Nuremberg, Germany</div>
399 <div class="line">markus.wittmann -at- fau.de or hpc -at- rrze.fau.de</div>
400 <div class="line"><br /></div>
401 <div class="line">Viktor Haag, 2016</div>
402 <div class="line">LSS, University of Erlangen-Nuremberg, Germany</div>
403 <div class="line"><br /></div>
404 <div class="line">Michael Hussnaetter, 2017-2018</div>
405 <div class="line">University of Erlangen-Nuremberg, Germany</div>
406 <div class="line">michael.hussnaetter -at- fau.de</div>
407 <div class="line"><br /></div>
409 <div class="line">This file is part of the Lattice Boltzmann Benchmark Kernels (LbmBenchKernels).</div>
410 <div class="line"><br /></div>
411 <div class="line">LbmBenchKernels is free software: you can redistribute it and/or modify</div>
412 <div class="line">it under the terms of the GNU General Public License as published by</div>
413 <div class="line">the Free Software Foundation, either version 3 of the License, or</div>
414 <div class="line">(at your option) any later version.</div>
415 <div class="line"><br /></div>
416 <div class="line">LbmBenchKernels is distributed in the hope that it will be useful,</div>
417 <div class="line">but WITHOUT ANY WARRANTY; without even the implied warranty of</div>
418 <div class="line">MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the</div>
419 <div class="line">GNU General Public License for more details.</div>
420 <div class="line"><br /></div>
421 <div class="line">You should have received a copy of the GNU General Public License</div>
422 <div class="line">along with LbmBenchKernels. If not, see <<a class="reference external" href="http://www.gnu.org/licenses/">http://www.gnu.org/licenses/</a>>.</div>
424 <p><strong>LBM Benchmark Kernels Documentation</strong></p>
425 <div class="contents topic" id="contents">
426 <p class="topic-title first">Contents</p>
427 <ul class="auto-toc simple">
428 <li><a class="reference internal" href="#introduction" id="id5">1 Introduction</a></li>
429 <li><a class="reference internal" href="#compilation" id="id6">2 Compilation</a><ul class="auto-toc">
430 <li><a class="reference internal" href="#debug-and-verification" id="id7">2.1 Debug and Verification</a></li>
431 <li><a class="reference internal" href="#release-and-verification" id="id8">2.2 Release and Verification</a></li>
432 <li><a class="reference internal" href="#benchmarking" id="id9">2.3 Benchmarking</a></li>
433 <li><a class="reference internal" href="#compilers" id="id10">2.4 Compilers</a></li>
434 <li><a class="reference internal" href="#floating-point-precision" id="id11">2.5 Floating Point Precision</a></li>
435 <li><a class="reference internal" href="#cleaning" id="id12">2.6 Cleaning</a></li>
436 <li><a class="reference internal" href="#options-summary" id="id13">2.7 Options Summary</a></li>
439 <li><a class="reference internal" href="#invocation" id="id14">3 Invocation</a><ul class="auto-toc">
440 <li><a class="reference internal" href="#command-line-parameters" id="id15">3.1 Command Line Parameters</a></li>
441 <li><a class="reference internal" href="#kernels" id="id16">3.2 Kernels</a></li>
444 <li><a class="reference internal" href="#id2" id="id17">4 Benchmarking</a><ul class="auto-toc">
445 <li><a class="reference internal" href="#intel-compiler" id="id18">4.1 Intel Compiler</a></li>
446 <li><a class="reference internal" href="#pinning" id="id19">4.2 Pinning</a></li>
447 <li><a class="reference internal" href="#general-remarks" id="id20">4.3 General Remarks</a></li>
448 <li><a class="reference internal" href="#padding" id="id21">4.4 Padding</a></li>
451 <li><a class="reference internal" href="#geometries" id="id22">5 Geometries</a></li>
452 <li><a class="reference internal" href="#performance-results" id="id23">6 Performance Results</a><ul class="auto-toc">
453 <li><a class="reference internal" href="#machine-specifications" id="id24">6.1 Machine Specifications</a></li>
454 <li><a class="reference internal" href="#single-socket-results" id="id25">6.2 Single Socket Results</a></li>
457 <li><a class="reference internal" href="#licence" id="id26">7 Licence</a></li>
458 <li><a class="reference internal" href="#acknowledgements" id="id27">8 Acknowledgements</a></li>
459 <li><a class="reference internal" href="#bibliography" id="id28">9 Bibliography</a></li>
462 <div class="section" id="introduction">
463 <h1><a class="toc-backref" href="#id5">1 Introduction</a></h1>
464 <p>The lattice Boltzmann (LBM) benchmark kernels are a collection of LBM kernel
466 <p><strong>AS SUCH THE LBM BENCHMARK KERNELS ARE NO FULLY EQUIPPED CFD SOLVER AND SOLELY
467 SERVES THE PURPOSE OF STUDYING POSSIBLE PERFORMANCE OPTIMIZATIONS AND/OR
468 EXPERIMENTS.</strong></p>
469 <p>Currently all kernels utilize a D3Q19 discretization and the
470 two-relaxation-time (TRT) collision operator <a class="citation-reference" href="#ginzburg-2008" id="id1">[ginzburg-2008]</a>.
471 All operations are carried out in double or single precision arithmetic.</p>
473 <div class="section" id="compilation">
474 <h1><a class="toc-backref" href="#id6">2 Compilation</a></h1>
475 <p>The benchmark framework currently supports only Linux systems and the GCC and
476 Intel compilers. Every other configuration probably requires adjustment inside
477 the code and the makefiles. Furthermore some code might be platform or at least
479 <p>The benchmark can be build via <tt class="docutils literal">make</tt> from the <tt class="docutils literal">src</tt> subdirectory. This will
480 generate one binary which hosts all implemented benchmark kernels.</p>
481 <p>Binaries are located under the <tt class="docutils literal">bin</tt> subdirectory and will have different names
482 depending on compiler and build configuration.</p>
483 <p>Compilation can target debug or release builds. Combined with both build types
484 verification can be enabled, which increases the runtime and hence is not
485 suited for benchmarking.</p>
486 <div class="section" id="debug-and-verification">
487 <h2><a class="toc-backref" href="#id7">2.1 Debug and Verification</a></h2>
488 <pre class="literal-block">
489 make BUILD=debug BENCHMARK=off
491 <p>Running <tt class="docutils literal">make</tt> with <tt class="docutils literal">BUILD=debug</tt> builds the debug version of
492 the benchmark kernels, where no optimizations are performed, line numbers and
493 debug symbols are included as well as <tt class="docutils literal">DEBUG</tt> will be defined. The resulting
494 binary will be found in the <tt class="docutils literal">bin</tt> subdirectory and named
495 <tt class="docutils literal"><span class="pre">lbmbenchk-linux-<compiler>-debug</span></tt>.</p>
496 <p>Specifying <tt class="docutils literal">BENCHMARK=off</tt> turns on verification
497 (<tt class="docutils literal">VERIFICATION=on</tt>), statistics (<tt class="docutils literal">STATISTICS=on</tt>), and VTK output
498 (<tt class="docutils literal">VTK_OUTPUT=on</tt>) enabled.</p>
499 <p>Please note that the generated binary will therefore
500 exhibit a poor performance.</p>
502 <div class="section" id="release-and-verification">
503 <h2><a class="toc-backref" href="#id8">2.2 Release and Verification</a></h2>
504 <p>Verification with the debug builds can be extremely slow. Hence verification
505 capabilities can be build with release builds:</p>
506 <pre class="literal-block">
510 <div class="section" id="benchmarking">
511 <h2><a class="toc-backref" href="#id9">2.3 Benchmarking</a></h2>
512 <p>To generate a binary for benchmarking run make with</p>
513 <pre class="literal-block">
516 <p>As default <tt class="docutils literal">BENCHMARK=on</tt> and <tt class="docutils literal">BUILD=release</tt> is set, where
517 <tt class="docutils literal">BUILD=release</tt> turns optimizations on and <tt class="docutils literal">BENCHMARK=on</tt> disables
518 verfification, statistics, and VTK output.</p>
519 <p>See Options Summary below for further description of options which can be
520 applied, e.g. TARCH as well as the Benchmarking section.</p>
522 <div class="section" id="compilers">
523 <h2><a class="toc-backref" href="#id10">2.4 Compilers</a></h2>
524 <p>Currently only the GCC and Intel compiler under Linux are supported. Between
525 both configuration can be chosen via <tt class="docutils literal"><span class="pre">CONFIG=linux-gcc</span></tt> or
526 <tt class="docutils literal"><span class="pre">CONFIG=linux-intel</span></tt>.</p>
528 <div class="section" id="floating-point-precision">
529 <h2><a class="toc-backref" href="#id11">2.5 Floating Point Precision</a></h2>
530 <p>As default double precision data types are used for storing PDFs and floating
531 point constants. Furthermore, this is the default for the intrincis kernels.
532 With the <tt class="docutils literal">PRECISION=sp</tt> variable this can be changed to single precision.</p>
533 <pre class="literal-block">
534 make PRECISION=sp # build for single precision kernels
536 make PRECISION=dp # build for double precision kernels (defalt)
539 <div class="section" id="cleaning">
540 <h2><a class="toc-backref" href="#id12">2.6 Cleaning</a></h2>
541 <p>For each configuration and build (debug/release) a subdirectory under the
542 <tt class="docutils literal">src/obj</tt> directory is created where the dependency and object files are
545 <pre class="literal-block">
546 make CONFIG=... BUILD=... clean
548 <p>a specific combination is select and cleaned, whereas with</p>
549 <pre class="literal-block">
552 <p>all object and dependency files are deleted.</p>
554 <div class="section" id="options-summary">
555 <h2><a class="toc-backref" href="#id13">2.7 Options Summary</a></h2>
556 <p>Options that can be specified when building the suite with make:</p>
557 <table border="1" class="docutils">
564 <thead valign="bottom">
565 <tr><th class="head">name</th>
566 <th class="head">values</th>
567 <th class="head">default</th>
568 <th class="head">description</th>
572 <tr><td>BENCHMARK</td>
575 <td>If enabled, disables VERIFICATION, STATISTICS, VTK_OUTPUT. If disabled enables the three former options.</td>
578 <td>debug, release</td>
580 <td>debug: no optimization, debug symbols, DEBUG defined. release: optimizations enabled.</td>
583 <td>linux-gcc, linux-intel</td>
585 <td>Select GCC or Intel compiler.</td>
588 <td>avx512, avx, sse</td>
590 <td>Determines which ISA extension is used for macro definitions of the intrinsics. This is <em>not</em> the architecture the compiler generates code for.</td>
595 <td>OpenMP, i.e. threading support.</td>
597 <tr><td>PRECISION</td>
600 <td>Floating point precision used for data type, arithmetic, and intrincics.</td>
602 <tr><td>STATISTICS</td>
605 <td>View statistics, like density etc, during simulation.</td>
610 <td>Via TARCH the architecture the compiler generates code for can be overridden. The value depends on the chosen compiler.</td>
612 <tr><td>VERIFICATION</td>
615 <td>Turn verification on/off.</td>
617 <tr><td>VTK_OUTPUT</td>
620 <td>Enable/Disable VTK file output.</td>
624 <p><strong>Suboptions for ``ISA=avx512``</strong></p>
625 <table border="1" class="docutils">
632 <thead valign="bottom">
633 <tr><th class="head">name</th>
634 <th class="head">values</th>
635 <th class="head">default</th>
636 <th class="head">description</th>
640 <tr><td>ADJ_LIST_MEM_TYPE</td>
642 <td><ul class="first last simple">
646 <td>Determines memory location of adjacency list array, DRAM or HBM.</td>
648 <tr><td>PDF_MEM_TYPE</td>
650 <td><ul class="first last simple">
654 <td>Determines memory location of PDF array, DRAM or HBM.</td>
656 <tr><td>SOFTWARE_PREFETCH_LOOKAHEAD_L1</td>
659 <td>Software prefetch lookahead of elements into L1 cache, value is multiplied by vector size (<tt class="docutils literal">VSIZE</tt>).</td>
661 <tr><td>SOFTWARE_PREFETCH_LOOKAHEAD_L2</td>
664 <td>Software prefetch lookahead of elements into L2 cache, value is multiplied by vector size (<tt class="docutils literal">VSIZE</tt>).</td>
668 <p>Please note this options require AVX-512 PF support of the target processor.</p>
671 <div class="section" id="invocation">
672 <h1><a class="toc-backref" href="#id14">3 Invocation</a></h1>
673 <p>Running the binary will print among the GPL licence header a line like the following:</p>
674 <pre class="literal-block">
675 LBM Benchmark Kernels 0.1, compiled Jul 5 2017 21:59:22, type: verification
677 <p>if verfication was enabled during compilation or</p>
678 <pre class="literal-block">
679 LBM Benchmark Kernels 0.1, compiled Jul 5 2017 21:59:22, type: benchmark
681 <p>if verfication was disabled during compilation.</p>
682 <div class="section" id="command-line-parameters">
683 <h2><a class="toc-backref" href="#id15">3.1 Command Line Parameters</a></h2>
684 <p>Running the binary with <tt class="docutils literal"><span class="pre">-h</span></tt> list all available parameters:</p>
685 <pre class="literal-block">
689 [-dims XxYxZ] [-geometry box|channel|pipe|blocks[-<block size>]] [-iterations <iterations>] [-lattice-dump-ascii]
690 [-rho-in <density>] [-rho-out <density] [-omega <omega>] [-kernel <kernel>]
692 [-t <number of threads>]
695 -- <kernel specific parameters>
697 -list List available kernels.
699 -dims XxYxZ Specify geometry dimensions.
701 -geometry blocks-<block size>
702 Geometetry with blocks of size <block size> regularily layout out.
704 <p>If an option is specified multiple times the last one overrides previous ones.
705 This holds also true for <tt class="docutils literal"><span class="pre">-verify</span></tt> which sets geometry dimensions,
706 iterations, etc, which can afterward be override, e.g.:</p>
707 <pre class="literal-block">
708 $ bin/lbmbenchk-linux-intel-release-dp -verfiy -dims 32x32x32
710 <p>Kernel specific parameters can be obtained via selecting the specific kernel
711 and passing <tt class="docutils literal"><span class="pre">-h</span></tt> as parameter:</p>
712 <pre class="literal-block">
713 $ bin/lbmbenchk-linux-intel-release-dp -kernel kernel-name -- -h
716 [-blk <n>] [-blk-[xyz] <n>]
718 <p>A list of all available kernels can be obtained via <tt class="docutils literal"><span class="pre">-list</span></tt>:</p>
719 <pre class="literal-block">
720 $ ../bin/lbmbenchk-linux-gcc-debug-dp -list
721 Lattice Boltzmann Benchmark Kernels (LbmBenchKernels) Copyright (C) 2016, 2017 LSS, RRZE
722 This program comes with ABSOLUTELY NO WARRANTY; for details see LICENSE.
723 This is free software, and you are welcome to redistribute it under certain conditions.
725 LBM Benchmark Kernels 0.1, compiled Jul 5 2017 21:59:22, type: verification
726 Available kernels to benchmark:
731 list-pull-split-nt-1s-soa
732 list-pull-split-nt-2s-soa
747 <div class="section" id="kernels">
748 <h2><a class="toc-backref" href="#id16">3.2 Kernels</a></h2>
749 <p>The following list shortly describes available kernels:</p>
751 <li><strong>push-soa/push-aos/pull-soa/pull-aos</strong>:
752 Unoptimized kernels (but stream/collide are already fused) using two grids as
753 source and destination. Implement push/pull semantics as well structure of
754 arrays (soa) or array of structures (aos) layout.</li>
755 <li><strong>blk-push-soa/blk-push-aos/blk-pull-soa/blk-pull-aos</strong>:
756 The same as the unoptimized kernels without the blk prefix, except that they support
757 spatial blocking, i.e. loop blocking of the three loops used to iterate over
758 the lattice. Here manual work sharing for OpenMP is used.</li>
759 <li><strong>aa-aos/aa-soa</strong>:
760 Straight forward implementation of AA pattern on full array with blocking support.
761 Manual work sharing for OpenMP is used. Domain is partitioned only along the x dimension.</li>
762 <li><strong>aa-vec-soa/aa-vec-sl-soa</strong>:
763 Optimized AA kernel with intrinsics on full array. aa-vec-sl-soa uses only
764 one loop for iterating over the lattice instead of three nested ones.</li>
765 <li><strong>list-push-soa/list-push-aos/list-pull-soa/list-pull-aos</strong>:
766 The same as the unoptimized kernels without the list prefix, but for indirect addressing.
767 Here only a 1D vector of is used to store the fluid nodes, omitting the
768 obstacles. An adjacency list is used to recover the neighborhood associations.</li>
769 <li><strong>list-pull-split-nt-1s-soa/list-pull-split-nt-2s-soa</strong>:
770 Optimized variant of list-pull-soa. Chunks of the lattice are processed as
771 once. Postcollision values are written back via nontemporal stores in 18 (1s)
772 or 9 (2s) loops.</li>
773 <li><strong>list-aa-aos/list-aa-soa</strong>:
774 Unoptimized implementation of the AA pattern for the 1D vector with adjacency
775 list. Supported are array of structures (aos) and structure of arrays (soa)
776 data layout is supported.</li>
777 <li><strong>list-aa-ria-soa</strong>:
778 Implementation of AA pattern with intrinsics for the 1D vector with adjacency
779 list. Furthermore it contains a vectorized even time step and run length
780 coding to reduce the loop balance of the odd time step.</li>
781 <li><strong>list-aa-pv-soa</strong>:
782 All optimizations of list-aa-ria-soa. Additional with partial vectorization
783 of the odd time step.</li>
785 <p>Note that all array of structures (aos) kernels might require blocking
786 (depending on the domain size) to reach the performance of their structure of
787 arrays (soa) counter parts.</p>
788 <p>The following table summarizes the properties of the kernels. Here <strong>D</strong> means
789 direct addressing, i.e. full array, <strong>I</strong> means indirect addressing, i.e. 1D
790 vector with adjacency list, <strong>x</strong> means supported, whereas <strong>--</strong> means unsupported.
791 The loop balance B_l is computed for D3Q19 model with <strong>double precision</strong> floating
792 point for PDFs (8 byte) and 4 byte integers for the index (adjacency list).
793 As list-aa-ria-soa and list-aa-pv-soa support run length coding their effective
794 loop balance depends on the geometry. The effective loop balance is printed
796 <table border="1" class="docutils">
806 <thead valign="bottom">
807 <tr><th class="head">kernel name</th>
808 <th class="head">prop. step</th>
809 <th class="head">data layout</th>
810 <th class="head">addr.</th>
811 <th class="head">parallel</th>
812 <th class="head">blocking</th>
813 <th class="head">B_l [B/FLUP]</th>
817 <tr><td>push-soa</td>
825 <tr><td>push-aos</td>
833 <tr><td>pull-soa</td>
841 <tr><td>pull-aos</td>
849 <tr><td>blk-push-soa</td>
857 <tr><td>blk-push-aos</td>
865 <tr><td>blk-pull-soa</td>
873 <tr><td>blk-pull-aos</td>
897 <tr><td>aa-vec-soa</td>
905 <tr><td>aa-vec-sl-soa</td>
913 <tr><td>list-push-soa</td>
921 <tr><td>list-push-aos</td>
929 <tr><td>list-pull-soa</td>
937 <tr><td>list-pull-aos</td>
945 <tr><td>list-pull-split-nt-1s</td>
953 <tr><td>list-pull-split-nt-2s</td>
961 <tr><td>list-aa-soa</td>
969 <tr><td>list-aa-aos</td>
977 <tr><td>list-aa-ria-soa</td>
985 <tr><td>list-aa-pv-soa</td>
997 <div class="section" id="id2">
998 <h1><a class="toc-backref" href="#id17">4 Benchmarking</a></h1>
999 <p>Correct benchmarking is a nontrivial task. Whenever benchmark results should be
1000 created make sure the binary was compiled with:</p>
1002 <li><tt class="docutils literal">BENCHMARK=on</tt> (default if not overriden) and</li>
1003 <li><tt class="docutils literal">BUILD=release</tt> (default if not overriden) and</li>
1004 <li>the correct ISA for macros (i.e. intrinsics) is used, selected via <tt class="docutils literal">ISA</tt> and</li>
1005 <li>use <tt class="docutils literal">TARCH</tt> to specify the architecture the compiler generates code for.</li>
1007 <div class="section" id="intel-compiler">
1008 <h2><a class="toc-backref" href="#id18">4.1 Intel Compiler</a></h2>
1009 <p>For the Intel compiler one can specify depending on the target ISA extension:</p>
1011 <li>SSE: <tt class="docutils literal"><span class="pre">TARCH=-xSSE4.2</span></tt></li>
1012 <li>AVX: <tt class="docutils literal"><span class="pre">TARCH=-xAVX</span></tt></li>
1013 <li>AVX2 and FMA: <tt class="docutils literal"><span class="pre">TARCH=-xCORE-AVX2,-fma</span></tt></li>
1014 <li>AVX512: <tt class="docutils literal"><span class="pre">TARCH=-xCORE-AVX512</span></tt></li>
1015 <li>KNL: <tt class="docutils literal"><span class="pre">TARCH=-xMIC-AVX512</span></tt></li>
1017 <p>Compiling for an architecture supporting AVX (Sandy Bridge, Ivy Bridge):</p>
1018 <pre class="literal-block">
1019 make ISA=avx TARCH=-xAVX
1021 <p>Compiling for an architecture supporting AVX2 (Haswell, Broadwell):</p>
1022 <pre class="literal-block">
1023 make ISA=avx TARCH=-xCORE-AVX2,-fma
1025 <p>WARNING: ISA is here still set to <tt class="docutils literal">avx</tt> as currently we have the FMA intrinsics not
1026 implemented. This might change in the future.</p>
1027 <!-- TODO: add isa=avx512 and add docu for knl -->
1028 <!-- TODO: kein prefetching wenn AVX-512 PF nicht unterstuetz wird -->
1029 <p>Compiling for an architecture supporting AVX-512 (Skylake):</p>
1030 <pre class="literal-block">
1031 make ISA=avx512 TARCH=-xCORE-AVX512
1033 <p>Please note that for the AVX512 gather kernels software prefetching for the
1034 gather instructions is disabled per default.
1035 To enable it set <tt class="docutils literal">SOFTWARE_PREFETCH_LOOKAHEAD_L1</tt> and/or
1036 <tt class="docutils literal">SOFTWARE_PREFETCH_LOOKAHEAD_L2</tt> to a value greater than <tt class="docutils literal">0</tt> during
1037 compilation. Note that this requires AVX-512 PF support from the target
1039 <p>Compiling for MIC architecture KNL supporting AVX-512 and AVX-512 PF:</p>
1040 <pre class="literal-block">
1041 make ISA=avx512 TARCH=-xMIC-AVX512
1043 <p>or optionally with software prefetch enabled:</p>
1044 <pre class="literal-block">
1045 make ISA=avx512 TARCH=-xMIC-AVX512 SOFTWARE_PREFETCH_LOOKAHEAD_L1=<value> SOFTWARE_PREFETCH_LOOKAHEAD_L2=<value>
1048 <div class="section" id="pinning">
1049 <h2><a class="toc-backref" href="#id19">4.2 Pinning</a></h2>
1050 <p>During benchmarking pinning should be used via the <tt class="docutils literal"><span class="pre">-pin</span></tt> parameter. Running
1051 a benchmark with 10 threads and pin them to the first 10 cores works like</p>
1052 <pre class="literal-block">
1053 $ bin/lbmbenchk-linux-intel-release-dp ... -t 10 -pin $(seq -s , 0 9)
1056 <div class="section" id="general-remarks">
1057 <h2><a class="toc-backref" href="#id20">4.3 General Remarks</a></h2>
1058 <p>Things the binary does nor check or control:</p>
1060 <li>transparent huge pages: when allocating memory small 4 KiB pages might be
1061 replaced with larger ones. This is in general a good thing, but if this is
1062 really the case, depends on the system settings (check e.g. the status of
1063 <tt class="docutils literal">/sys/kernel/mm/transparent_hugepage/enabled</tt>).
1064 Currently <tt class="docutils literal">madvise(MADV_HUGEPAGE)</tt> is used for allocations which are aligned to
1065 a 4 KiB page, which should be the case for the lattices.
1066 This should result in huge pages except THP is disabled on the machine.
1067 (NOTE: madvise() is used if <tt class="docutils literal">HAVE_HUGE_PAGES</tt> is defined, which is currently
1068 hard coded defined in <tt class="docutils literal">Memory.c</tt>).</li>
1069 <li>CPU/core frequency: For reproducible results the frequency of all cores
1070 should be fixed.</li>
1071 <li>NUMA placement policy: The benchmark assumes a first touch policy, which
1072 means the memory will be placed at the NUMA domain the touching core is
1073 associated with. If a different policy is in place or the NUMA domain to be
1074 used is already full memory might be allocated in a remote domain. Accesses
1075 to remote domains typically have a higher latency and lower bandwidth.</li>
1076 <li>System load: interference with other application, especially on desktop
1077 systems should be avoided.</li>
1078 <li>Padding: For SoA based kernels the number of (fluid) nodes is automatically
1079 adjusted so that no cache or TLB thrashing should occur. The parameters are
1080 optimized for current Intel based systems. For more details look into the
1081 padding section.</li>
1082 <li>CPU dispatcher function: the compiler might add different versions of a
1083 function for different ISA extensions. Make sure the code you might think is
1084 executed is actually the code which is executed.</li>
1087 <div class="section" id="padding">
1088 <h2><a class="toc-backref" href="#id21">4.4 Padding</a></h2>
1089 <p>With correct padding cache and TLB thrashing can be avoided. Therefore the
1090 number of (fluid) nodes used in the data layout is artificially increased.</p>
1091 <p>Currently automatic padding is active for kernels which support it. It can be
1092 controlled via the kernel parameter (i.e. parameter after the <tt class="docutils literal"><span class="pre">--</span></tt>)
1093 <tt class="docutils literal"><span class="pre">-pad</span></tt>. Supported values are <tt class="docutils literal">auto</tt> (default), <tt class="docutils literal">no</tt> (to disable padding),
1094 or a manual padding.</p>
1095 <p>Automatic padding tries to avoid cache and TLB thrashing and pads for a 32
1096 entry (huge pages) TLB with 8 sets and a 512 set (L2) cache. This reflects the
1097 parameters of current Intel based processors.</p>
1098 <p>Manual padding is done via a padding string and has the format
1099 <tt class="docutils literal"><span class="pre">mod_1+offset_1(,mod_n+offset_n)</span></tt>, which specifies numbers of bytes.
1100 SoA data layouts can exhibit TLB thrashing. Therefore we want to distribute the
1101 19 pages with one lattice (36 with two lattices) we are concurrently accessing
1102 over as much sets in the TLB as possible.
1103 This is controlled by the distance between the accessed pages, which is the
1104 number of (fluid) nodes in between them and can be adjusted by adding further
1106 We want the distance d (in bytes) between two accessed pages to be e.g.
1107 <strong>d % (PAGE_SIZE * TLB_SETS) = PAGE_SIZE</strong>.
1108 This would distribute the pages evenly over the sets. Hereby <strong>PAGE_SIZE * TLB_SETS</strong>
1109 would be our <tt class="docutils literal">mod_1</tt> and <strong>PAGE_SIZE</strong> (after the =) our <tt class="docutils literal">offset_1</tt>.
1110 Measurements show that with only a quarter of half of a page size as offset
1111 higher performance is achieved, which is done by automatic padding.
1112 On top of this padding more paddings can be added. They are just added to the
1113 padding string and are separated by commas.</p>
1114 <p>A zero modulus in the padding string has a special meaning. Here the
1115 corresponding offset is just added to the number of nodes. A padding string
1116 like <tt class="docutils literal"><span class="pre">-pad</span> 0+16</tt> would at a static padding of two nodes (one node = 8 b).</p>
1119 <div class="section" id="geometries">
1120 <h1><a class="toc-backref" href="#id22">5 Geometries</a></h1>
1121 <p>TODO: supported geometries: channel, pipe, blocks, fluid</p>
1123 <div class="section" id="performance-results">
1124 <h1><a class="toc-backref" href="#id23">6 Performance Results</a></h1>
1125 <p>The sections lists performance values measured on several machines for
1126 different kernels and geometries and <strong>double precision</strong> floating point data/arithmetic.
1127 The <strong>RFM</strong> column denotes the expected performance as predicted by the
1128 Roofline performance model <a class="citation-reference" href="#williams-2008" id="id3">[williams-2008]</a>.
1129 For performance prediction of each kernel a memory bandwidth benchmark is used
1130 which mimics the kernels memory access pattern and the kernel's loop balance
1131 (see <a class="citation-reference" href="#kernels" id="id4">[kernels]</a> for details).</p>
1132 <div class="section" id="machine-specifications">
1133 <h2><a class="toc-backref" href="#id24">6.1 Machine Specifications</a></h2>
1134 <p><strong>Ivy Bridge, Intel Xeon E5-2660 v2</strong></p>
1136 <li>Ivy Bridge architecture, AVX</li>
1137 <li>10 cores, 2.2 GHz</li>
1138 <li>SMT enabled</li>
1139 <li>memoy bandwidth:<ul>
1140 <li>copy-19 32.7 GB/s</li>
1141 <li>copy-19-nt-sl 35.6 GB/s</li>
1142 <li>update-19 37.4 GB/s</li>
1146 <p><strong>Haswell, Intel Xeon E5-2695 v3</strong></p>
1148 <li>Haswell architecture, AVX2, FMA</li>
1149 <li>14 cores, 2.3 GHz</li>
1150 <li>2 x 7 cores in cluster-on-die (CoD) mode enabled</li>
1151 <li>SMT enabled</li>
1152 <li>memory bandwidth:<ul>
1153 <li>copy-19 47.3 GB/s</li>
1154 <li>copy-19-nt-sl 47.1 GB/s</li>
1155 <li>update-19 44.0 GB/s</li>
1159 <p><strong>Broadwell, Intel Xeon E5-2630 v4</strong></p>
1161 <li>Broadwell architecture, AVX2, FMA</li>
1162 <li>10 cores, 2.2 GHz</li>
1163 <li>SMT disabled</li>
1164 <li>memory bandwidth:<ul>
1165 <li>copy-19 48.0 GB/s</li>
1166 <li>copy-nt-sl-19 48.2 GB/s</li>
1167 <li>update-19 51.1 GB/s</li>
1171 <p><strong>Skylake, Intel Xeon Gold 6148</strong></p>
1173 <li>Skylake server architecture, AVX2, AVX512, 2 FMA units</li>
1174 <li>20 cores, 2.4 GHz</li>
1175 <li>SMT enabled</li>
1176 <li>memory bandwidth:<ul>
1177 <li>copy-19 89.7 GB/s</li>
1178 <li>copy-19-nt-sl 92.4 GB/s</li>
1179 <li>update-19 93.6 GB/s</li>
1183 <p><strong>Zen, AMD EPYC 7451</strong></p>
1185 <li>Zen architecture, AVX2, FMA</li>
1186 <li>24 cores, 2.3 GHz</li>
1187 <li>SMT enabled</li>
1188 <li>memory bandwidth:<ul>
1189 <li>copy-19 111.9 GB/s</li>
1190 <li>copy-19-nt-sl 111.7 GB/s</li>
1191 <li>update-19 109.2 GB/s</li>
1195 <p><strong>Zen, AMD Ryzen 7 1700X</strong></p>
1197 <li>Zen architecture, AVX2, FMA</li>
1198 <li>8 cores, 3.4 GHz</li>
1199 <li>SMT enabled</li>
1200 <li>memory bandwidth:<ul>
1201 <li>copy-19 27.2 GB/s</li>
1202 <li>copy-19-nt-sl 27.1 GB/s</li>
1203 <li>update-19 26.1 GB/s</li>
1208 <div class="section" id="single-socket-results">
1209 <h2><a class="toc-backref" href="#id25">6.2 Single Socket Results</a></h2>
1211 <li>Geometry dimensions are for all measurements 500x100x100 nodes.</li>
1212 <li>Note the <strong>different scaling on the y axis</strong> of the plots!</li>
1214 <table border="1" class="docutils">
1216 <col width="100%" />
1218 <tbody valign="top">
1219 <tr><td>Ivy Bridge, Intel Xeon E5-2660 v2, Double Precision</td>
1221 <tr><td><img alt="perf_emmy_dp" src="images/benchmark-emmy-dp.png" style="width: 1000.0px; height: 250.0px;" /></td>
1223 <tr><td>Ivy Bridge, Intel Xeon E5-2660 v2, Single Precision</td>
1225 <tr><td><img alt="perf_emmy_sp" src="images/benchmark-emmy-sp.png" style="width: 1000.0px; height: 250.0px;" /></td>
1227 <tr><td>Haswell, Intel Xeon E5-2695 v3, Double Precision</td>
1229 <tr><td><img alt="perf_hasep1_dp" src="images/benchmark-hasep1-dp.png" style="width: 1000.0px; height: 250.0px;" /></td>
1231 <tr><td>Haswell, Intel Xeon E5-2695 v3, Single Precision</td>
1233 <tr><td><img alt="perf_hasep1_sp" src="images/benchmark-hasep1-sp.png" style="width: 1000.0px; height: 250.0px;" /></td>
1235 <tr><td>Broadwell, Intel Xeon E5-2630 v4, Double Precision</td>
1237 <tr><td><img alt="perf_meggie_dp" src="images/benchmark-meggie-dp.png" style="width: 1000.0px; height: 250.0px;" /></td>
1239 <tr><td>Broadwell, Intel Xeon E5-2630 v4, Single Precision</td>
1241 <tr><td><img alt="perf_meggie_sp" src="images/benchmark-meggie-sp.png" style="width: 1000.0px; height: 250.0px;" /></td>
1243 <tr><td>Skylake, Intel Xeon Gold 6148, Double Precision</td>
1245 <tr><td><img alt="perf_skylakesp2_dp" src="images/benchmark-skylakesp2-dp.png" style="width: 1000.0px; height: 250.0px;" /></td>
1247 <tr><td>Skylake, Intel Xeon Gold 6148, Single Precision</td>
1249 <tr><td><img alt="perf_skylakesp2_sp" src="images/benchmark-skylakesp2-sp.png" style="width: 1000.0px; height: 250.0px;" /></td>
1251 <tr><td>Zen, AMD Ryzen 7 1700X, Double Precision</td>
1253 <tr><td><img alt="perf_summitridge1_dp" src="images/benchmark-summitridge1-dp.png" style="width: 1000.0px; height: 250.0px;" /></td>
1255 <tr><td>Zen, AMD Ryzen 7 1700X, Single Precision</td>
1257 <tr><td><img alt="perf_summitridge1_sp" src="images/benchmark-summitridge1-sp.png" style="width: 1000.0px; height: 250.0px;" /></td>
1259 <tr><td>Zen, AMD EPYC 7451, Double Precision</td>
1261 <tr><td><img alt="perf_naples1_dp" src="images/benchmark-naples1-dp.png" style="width: 1000.0px; height: 250.0px;" /></td>
1263 <tr><td>Zen, AMD EPYC 7451, Single Precision</td>
1265 <tr><td><img alt="perf_naples1_sp" src="images/benchmark-naples1-sp.png" style="width: 1000.0px; height: 250.0px;" /></td>
1271 <div class="section" id="licence">
1272 <h1><a class="toc-backref" href="#id26">7 Licence</a></h1>
1273 <p>The Lattice Boltzmann Benchmark Kernels are licensed under GPLv3.</p>
1275 <div class="section" id="acknowledgements">
1276 <h1><a class="toc-backref" href="#id27">8 Acknowledgements</a></h1>
1277 <p>If you use the benchmark kernels you can cite us:</p>
1278 <p>M. Wittmann, V. Haag, T. Zeiser, H. Köstler, and G. Wellein: Lattice Boltzmann
1279 Benchmark Kernels as a Testbed for Performance Analysis, (2018), Computer &
1280 Fluids, Special Issue DSFD2017. doi:10.1016/j.compfluid.2018.03.030.</p>
1281 <p>Bibtex entry:</p>
1282 <pre class="literal-block">
1283 @article{wittmann-2018,
1284 author = {M. Wittmann and V. Haag and T. Zeiser and H. K\"ostler and G. Wellein},
1285 title = {Lattice {B}oltzmann benchmark kernels as a testbed for performance analysis},
1286 journal = {Computers \& Fluids},
1289 doi = {10.1016/j.compfluid.2018.03.030},
1292 <p>This work was funded by BMBF, grant no. 01IH15003A (project SKAMPY).</p>
1293 <p>This work was funded by KONWHIR project OMI4PAPS.</p>
1295 <div class="section" id="bibliography">
1296 <h1><a class="toc-backref" href="#id28">9 Bibliography</a></h1>
1297 <table class="docutils citation" frame="void" id="ginzburg-2008" rules="none">
1298 <colgroup><col class="label" /><col /></colgroup>
1299 <tbody valign="top">
1300 <tr><td class="label"><a class="fn-backref" href="#id1">[ginzburg-2008]</a></td><td>I. Ginzburg, F. Verhaeghe, and D. d'Humières.
1301 Two-relaxation-time lattice Boltzmann scheme: About parametrization, velocity, pressure and mixed boundary conditions.
1302 Commun. Comput. Phys., 3(2):427-478, 2008.</td></tr>
1305 <table class="docutils citation" frame="void" id="williams-2008" rules="none">
1306 <colgroup><col class="label" /><col /></colgroup>
1307 <tbody valign="top">
1308 <tr><td class="label"><a class="fn-backref" href="#id3">[williams-2008]</a></td><td>S. Williams, A. Waterman, and D. Patterson.
1309 Roofline: an insightful visual performance model for multicore architectures.
1310 Commun. ACM, 52(4):65-76, Apr 2009. doi:10.1145/1498765.1498785</td></tr>
1313 <p>Document was generated at 2018-06-06 10:38.</p>