nicula.xyz blog

Bypassing the branch predictor

Mon, 10 Mar 2025 22:00:00 +0200

A couple of days ago I was thinking about what you can do when the branch predictor is effectively working against you, and thus pessimizing your program instead of optimizing it.

Let’s work with something relatively simple & concrete: consider that we want to write some kind of financial system (maybe a trading system) and all of our transaction requests arrive at a certain function before being either (a) sent out to some authoritative server, or (b) abandoned. Let’s also assume that:

The vast majority of our transaction requests end up being abandoned at the last step.
We care A LOT about the speed of the ‘send’ path and we want to make it as fast as possible.
We don’t care at all about the speed of the ‘abandon’ path.

The code would look roughly like this:

/* Defined somewhere else */
struct Transaction;
bool should_send(Transaction *t);
void send(Transaction *t);
void abandon(Transaction *t);

void resolve(Transaction *t)
{
    if (should_send(t)) {
        send(t);
    } else {
        abandon(t);
    }
}

The implication of assumption #1 is that the branch predictor will be heavily primed to predict that should_send(t) returns false. What this means is that when we finally get a transaction that needs to be sent out, we’ll likely have to pay a branch misprediction penalty (~20 cycles); plus, our send() function may not be in the instruction cache just yet, because it’s so rarely executed. Also, we won’t get the advantage of pipelining whatever instructions are needed for send().

Since we only care about the speed of the send() path, I was wondering if there’s a way of telling the CPU that we don’t actually want to rely on the branch predictor at all when executing send() or abandon(): we always want to assume that send() will be executed.

A low-level solution?

I asked Claude if there is such a way to basically hard-code branch prediction rules into the machine code, and the answer was that there’s no way to do this on x86, but there is a way on ARM: the BEQP (predict branch taken) and BEQNP (predict branch not taken) instructions.

Those ARM instructions are just hallucinated, and the reality is actually the other way around: ARM doesn’t have a way of hard-coding ‘predictions’, but x86 does. More accurately: some old x86 processors do. On the Pentium 4 series, those rules/hints are encoded as instruction prefixes:

0x2E (branch not taken);
0x3E (branch taken).

So if a jump instruction came in with the prefix 0x3E, then the processor would assume that the branch is taken.

On modern x86 processors, those instruction prefixes are simply ignored, so compilers obviously won’t even bother generating them when you’re targeting such a CPU.

Another ’low-level’ approach that doesn’t work here are the [[likely]] and [[unlikely]] attributes introduced in C++20. Those attributes will typically just reorder some labels/paths around to make sure that the path which we mark as [[likely]] will require less jumps. Whether we mark the send() branch as likely or we leave the code as-is, Clang and GCC generate the same assembly¹:

resolve(Transaction*):
    push    rbx
    mov     rbx, rdi
    call    should_send(Transaction*)
    mov     rdi, rbx
    test    al, al
    je      .L2
    pop     rbx
    jmp     send(Transaction*)
.L2:
    pop     rbx
    jmp     abandon(Transaction*)

It won’t really matter that the send() branch doesn’t need to do the jump at line 7, because the branch predictor will be primed to assume the abandon() branch nonetheless. If you’re writing code for a modern x86 processor, you can consider that [[likely]] and [[unlikely]] are completely unrelated to the CPU branch predictor, because the microarchitecture itself won’t have any way of overriding its predictions. So those attributes are just a reordering mechanism. As noted in the proposal doc, however, the compiler could use the likely/unlikely hints to override the branch predictor:

Some of the possible code generation improvements from using branch probability hints include:

[…]

Some microarchitectures, such as Power, have branch hints which can override dynamic branch prediction.

A higher-level solution?

So if we can’t bypass the branch predictor on modern x86 processors with some fine-grained and guaranteed mechanism, and we also don’t want to go buy a bunch of Pentium 4 CPUs and run our code on them, we need to think about a higher-level solution that works well enough; it doesn’t need to be guaranteed to work 100% of the time.

One such solution that I know of is one that Carl Cook talked about during his CppCon 17 talk²: we can fill our system with mocked transaction data for which should_send(t) returns true. We have to do this enough times such that the mocked transaction data becomes the vast majority over the real data, so the branch predictor will be primed to assume that the send() path will be executed practically on every resolve() call. Overall, this may actually be a better approach than hard-coding prediction rules, because those rules wouldn’t really guarantee us that the whole send() path would get executed (the assumption may just lead to partial execution, until the should_send(t) condition is actually evaluated and the pipeline is flushed; so at the end we may still have important stuff not placed in the instruction/data cache).

For getting rid of those mocked transactions after they get ‘sent out’ from our program, Carl Cook mentions that network cards are capable of identifying and discarding such messages without adding significant overhead to the real transactions, so we can just leave it at that.

As mentioned in his talk, this whole ‘mocked/dummy transaction’ system gives him a 5 microsecond speed-up which, as the talk’s title suggests, matters a lot for his use case.

Thanks for reading!

You can check out the r/cpp discussion regarding this blog post here.

References:

Carl Cook’s CppCon talk (see footnote);
CppCon 2022: C++20’s [[likely]] Attribute - Optimizations, Pessimizations, and [[unlikely]] Consequences, page 33;
Answer to “Is there a compiler hint for GCC to force branch prediction to always go a certain way?” on Stackoverflow.

This is not to say that [[likely]] and [[unlikely]] don’t have an effect generally speaking. It’s just that the default codegen for the code snippet that I gave is already one that makes the first branch ’likely’. If you negated the condition and swapped the 2 branches around, marking the send() branch as [[likely]] would change the generated assembly. ↩︎
CppCon 2017: Carl Cook “When a Microsecond Is an Eternity: High Performance Trading Systems in C++” ↩︎

Improving on std::count_if()'s auto-vectorization

Sat, 08 Mar 2025 18:00:00 +0200

The problem

Let’s consider the problem with the following description:

We have an array of arbitrary length filled with uint8_t values;
We want to count the number of even values in the array;
Before doing the calculation we know that the number of even values in the array is between 0 and 255¹.

A typical solution

To start off, we can leverage the STL for this calculation. std::count_if() more precisely:

auto count_even_values_v1(const std::vector<uint8_t> &vec)
{
    return std::count_if(
        vec.begin(),
        vec.end(),
        [](uint8_t x) { return x % 2 == 0; }
    );
}

Let’s take a look at the assembly (generated by Clang 19.1.0 with flags -O3 -march=rocketlake -fno-unroll-loops). This is the main loop which counts the even numbers:

.LCPI0_1:
    .byte   1
count_even_values_v1():
;   ............
    vpbroadcastb    xmm1, byte ptr [rip + .LCPI0_1]
.LBB0_6:
    vmovd   xmm2, dword ptr [rsi + rax]
    vpandn  xmm2, xmm2, xmm1
    vpmovzxbq       ymm2, xmm2
    vpaddq  ymm0, ymm0, ymm2
    add     rax, 4
    cmp     r8, rax
    jne     .LBB0_6

We can see that the assembly generated by Clang is vectorized, and iterates the array DWORD-by-DWORD (i.e. four 8-bit values at a time).

Let’s inspect the algorithm:

First off, at line 5 in the snippet (vpbroadcastb xmm1, byte ptr [rip + .LCPI0_1]), we fill xmm1 with 16 x 8-bit masks, all equal to 0b1. So xmm1 will look like this:
1

xmm1[127:0] := [0b1, 0b1, 0b1, 0b1, 0b1, 0b1, 0b1, 0b1, 0b1, 0b1, 0b1, 0b1, 0b1, 0b1, 0b1, 0b1, 0b1]
Because we’re iterating DWORD-by-DWORD, only the first 32 bits of the xmm2 register will be filled with relevant data on each iteration, so we can disregard the rest which will be set to zero (for both xmm2 and xmm1). So we’ll consider xmm1 as only having 4 packed 8-bit masks, actually:
1

xmm1[31:0] := [0b1, 0b1, 0b1, 0b1]
At line 7 (vmovd xmm2, dword ptr [rsi + rax]), we load a DWORD into the first 32 bits of the xmm2 register, i.e. 4 packed 8-bit integers. So xmm2 will look like this (where W, X, Y, Z represent the bit pattern of the first, second, third, and fourth of the loaded integers, respectively):
1

xmm2[31:0] := [0bZZZZZZZZ, 0bYYYYYYYY, 0bXXXXXXXX, 0bWWWWWWWW]
At line 8 (vpandn xmm2, xmm2, xmm1), we do an AND NOT operation. vpandn DEST, SRC1, SRC2 is defined as DEST := NOT(SRC1) AND SRC2, which means:
1

xmm2[31:0] := [~0bZZZZZZZZ & 0b1, ~0bYYYYYYYY & 0b1, ~0bXXXXXXXX & 0b1, ~0bWWWWWWWW & 0b1]
In simpler terms, this is effectively selecting the first bit from each of the packed values, and then flipping that bit. When a number is even, the first bit in its binary representation will be zero, and the result of the AND NOT operation will be 0b1. Similarly, for odd numbers, the result will be 0b0.
At line 9 (vpmovzxbq ymm2, xmm2), we zero-extend the four 8-bit results of the AND NOT operation to four equivalent 64-bit results (i.e. for each packed value, we add 56 bits of zero-padding). R1, R2, R3, R4 stand for the first, second, third, and fourth result of the earlier AND NOT operation, respectively:
1

ymm2[255:0] := [zero-extended R4, zero-extended R3, zero-extended R2, zero-extended R1]

At line 10 (vpaddq ymm0, ymm0, ymm2) we simply add the 4 results from the current iteration (ymm2) to the accumulator (ymm0):

ymm0[63:0]    := ymm0[63:0]    + ymm2[63:0];
ymm0[127:64]  := ymm0[127:64]  + ymm2[127:64];
ymm0[191:128] := ymm0[191:128] + ymm2[191:128];
ymm0[255:192] := ymm0[255:192] + ymm2[255:192];

After the previous step, we either go back to step 1 if we still have DWORD-sized chunks to process, or we end the current loop and process chunks of lower size (not relevant for this investigation so we won’t go into further detail about those other branches).

Concrete example of running the algorithm

Consider we have the array [1, 2, 1, 2, 1, 2, 1, 2] as input, or [0b01, 0b10, 0b01, 0b10, 0b01, 0b10, 0b01, 0b10] in binary.

The individual steps are:

-> vpbroadcastb xmm1, byte ptr [rip + .LCPI0_1]
xmm1[31:0] := [0b1, 0b1, 0b1, 0b1]

Iteration 1:
   -> vmovd xmm2, dword ptr [rsi + rax]
   xmm2[31:0] := [0b10, 0b01, 0b10, 0b01]

   -> vpandn xmm2, xmm2, xmm1
   xmm2[31:0] := [~0b10 & 0b1, ~0b01 & 0b1, ~0b10 & 0b1, ~0b01 & 0b1]
              := [0b1, 0b0, 0b1, 0b0]

   -> vpmovzxbq ymm2, xmm2
   ymm2[255:0] := [0b00...1, 0b00..0, 0b00...1, 0b00..0]

   -> vpaddq  ymm0, ymm0, ymm2
   ymm0[63:0]    := ymm0[63:0]    + ymm2[63:0]    = 0;
   ymm0[127:64]  := ymm0[127:64]  + ymm2[127:64]  = 1;
   ymm0[191:128] := ymm0[191:128] + ymm2[191:128] = 0;
   ymm0[255:192] := ymm0[255:192] + ymm2[255:192] = 1;

Iteration 2:
   -> vmovd xmm2, dword ptr [rsi + rax]
   xmm2[31:0] := [0b10, 0b01, 0b10, 0b01]

   -> vpandn xmm2, xmm2, xmm1
   xmm2[31:0] := [~0b10 & 0b1, ~0b01 & 0b1, ~0b10 & 0b1, ~0b01 & 0b1]
              := [0b1, 0b0, 0b1, 0b0]

   -> vpmovzxbq ymm2, xmm2
   ymm2[255:0] := [0b00...1, 0b00..0, 0b00...1, 0b00..0]

   -> vpaddq  ymm0, ymm0, ymm2
   ymm0[63:0]    := ymm0[63:0]    + ymm2[63:0]    = 0;
   ymm0[127:64]  := ymm0[127:64]  + ymm2[127:64]  = 2;
   ymm0[191:128] := ymm0[191:128] + ymm2[191:128] = 0;
   ymm0[255:192] := ymm0[255:192] + ymm2[255:192] = 2;

So at the end we have two partial results equal to 0, and two partial results equal to 2. Adding them up, we get the final answer 4.

Improving the typical solution

Now, with all of this in mind, you might be wondering: why are we zero-extending those four 8-bit values to four 64-bit values?

The answer is: because the type of the accumulator that std::count_if() uses is the difference_type of the iterator that we’re passing². In this case, the type of the accumulator will be long, a 64-bit integer type (on the platform that we’re targeting). This effectively means that whenever we want to increment the accumulator, we’re telling the compiler that we require 64-bit precision. Otherwise, if the compiler were to generate the increments on 32-bit integers (or smaller), the result might overflow or wrap around, which would lead to a wrong answer at the end of the calculation (so the compiler is not allowed to do that).

However, considering the constraint that was specified in the problem description (the total number of even values in the array is between 0 and 255), we know that we do NOT need 64-bit precision when doing the increments. In our case, 8 bits will suffice. Therefore, the type of the accumulator can be uint8_t, same as the array’s element type.

Let’s then write our own std::count_if() implementation which lets us control the type of the accumulator (regardless of the difference_type of the iterator):

template<typename Acc, typename It, typename Pred>
Acc custom_count_if(It begin, It end, Pred pred)
{
    Acc result = 0;
    for (auto it = begin; it != end; it++)
    {
        if (pred(*it))
            result++;
    }
    return result;
}

We can then use this custom implementation as follows:

auto count_even_values_v2(const std::vector<uint8_t> &vec)
{
    return custom_count_if<uint8_t>(
        vec.begin(),
        vec.end(),
        [](uint8_t x) { return x % 2 == 0; }
    );
}

Let’s check the assembly of the main loop again:

.LCPI0_2:
    .byte   1
count_even_values_v2():
;   ............
    vpbroadcastb    ymm1, byte ptr [rip + .LCPI0_2]
.LBB0_11:
    vmovdqu ymm2, ymmword ptr [rdx + rax]
    vpandn  ymm2, ymm2, ymm1
    vpaddb  ymm0, ymm2, ymm0
    add     rax, 32
    cmp     rdi, rax
    jne     .LBB0_11

Great. We went from iterating with DWORD-sized chunks (4 x 8-bit values) in the previous version, to iterating with YMMWORD-sized chunks (32 x 8-bit values) in the current version³. That’s because those partial results packed in the YMM accumulator aren’t filled up with useless zero-padding anymore, so we can use the now-available bit positions to pack 32 partial results, not just 4.

Let’s do some benchmarks⁴ with libbenchmark to see if we’ve actually improved anything significantly.

I am taking 2^10, 2^12, ..., 2^30 random uint8_t values, calling the first version and the second version of the function to get the count of even numbers, and then comparing the results. Here’s the (lightly) edited benchmark output⁵. Note that the times are in nanoseconds (lower is better), and v1 represents count_even_values_v1() and v2 represents count_even_values_v2():

$ clang++ benchmark.cpp -std=c++23 -O3 -fno-unroll-loops -march=native -lbenchmark 
$ ./a.out
Benchmark                   Time_V1      Time_V2     Relative speedup
---------------------------------------------------------------------
[v1 vs. v2]/1024            145          25          5.8x
[v1 vs. v2]/4096            556          77          7.2x
[v1 vs. v2]/16384           2220         287         7.7x
[v1 vs. v2]/65536           8572         1286        6.6x
[v1 vs. v2]/262144          34280        6034        5.6x
[v1 vs. v2]/1048576         137068       25245       5.4x
[v1 vs. v2]/4194304         548051       102163      5.3x
[v1 vs. v2]/16777216        2379866      918532      2.5x
[v1 vs. v2]/67108864        9729229      3804606     2.5x
[v1 vs. v2]/268435456       39623747     15267566    2.5x
[v1 vs. v2]/1073741824      157596298    61126020    2.5x

If we allow loop unrolling, these are the results⁶:

$ clang++ benchmark.cpp -std=c++23 -O3 -march=native -lbenchmark 
$ ./a.out
Benchmark                   Time_V1      Time_V2     Relative speedup
---------------------------------------------------------------------
[v1 vs. v2]/1024            105          12          8.7x
[v1 vs. v2]/4096            425          48          8.8x
[v1 vs. v2]/16384           1703         178         9.5x
[v1 vs. v2]/65536           6804         950         7.1x
[v1 vs. v2]/262144          27233        5421        5.0x
[v1 vs. v2]/1048576         109215       25116       4.3x
[v1 vs. v2]/4194304         436428       101143      4.3x
[v1 vs. v2]/16777216        1900205      926924      2.0x
[v1 vs. v2]/67108864        7836094      3841985     2.0x
[v1 vs. v2]/268435456       31811774     15366099    2.0x
[v1 vs. v2]/1073741824      126397482    61204442    2.0x

So for our workload, the second version wins by a pretty large margin. At best it’s ~9.5 times faster and at worst it’s ~2.0 times faster. This is yet another example of exploiting your input’s constraints in order to make significant performance improvements.

Note: This optimization works out for other element types, too. For example, if we had an array filled with uint32_t values and we had to count the even ones, the fastest solution would be the one that uses uint32_t as the type of the accumulator (not long, nor uint8_t).

Bonus: GCC regression

If you’re using a GCC version between 9.1 and 14.2 (most recent release as of March 8, 2025), there’s an interesting regression that has only been fixed after 5 years. The regression is that GCC 8.5 used to be able to vectorize count_even_values_v1(), the version that uses std::count_if(), but later versions until GCC 15.0 aren’t able to, and generate assembly that traverses the array one byte (uint8_t) at a time.

I say ‘interesting regression’ because GCC fails to vectorize the loop for even/odd checks. If you change the condition to x % 3 == 0 or x % 4201337 == 0, the loop is vectorized (although the vectorization is poorer than Clang’s). I reported this issue (#117776) some time ago and it was fixed in the trunk. It won’t be backported though, so you’ll only have the fix starting with GCC 15.0.

Thanks for reading!

Source code:

count_even_values_v1(): raw, Godbolt
count_even_values_v2(): raw, Godbolt
benchmark.cpp⁷: raw, Godbolt

See discussions of this blog post on:

Another constraint that would allow the same optimizations would be to say that we don’t know the range of the answer beforehand, but that we want the answer modulo 256. Similar constraints for larger types, e.g. uint16_t or uint32_t, also enable similar optimizations (we could say that we have uint16_t / uint32_t data and we know that the answer is between 0 and 2¹⁶ - 1 or 2³² - 1, respectively). ↩︎
std::count_if() page on cppreference. ↩︎
If you change the signature of the first version to uint8_t count_even_values_v1(const std::vector<uint8_t>&) (i.e. you return uint8_t instead of auto), Clang is smart enough to basically interpret that as using a uint8_t accumulator in the first place, and thus generates identical assembly to count_even_values_v2(). However, GCC is NOT smart enough to do this, and the signature change has no effect (so it does vectorize the code, but poorly). Generally, I’d rather be explicit and not rely on those implicit/explicit conversions to be recognized and used appropriately by the optimizer. Thanks to @total_order_ for commenting with a Rust solution on Reddit that basically does what I described in this footnote (I’m guessing it comes down to the same LLVM optimization pass). It’s also debatable whether or not Clang’s optimization pass results in better codegen in most cases that you care about, so I wouldn’t jump to bash GCC just yet (though it may be worth opening an issue to see what the maintainers think). ↩︎
CPU specs. ↩︎
Raw output. ↩︎
Raw output with inlining allowed. ↩︎
I’m using std::span for those functions instead of std::vector because I’m only generating one global vector, but this doesn’t affect the measurements. ↩︎

Getting rid of unwanted branches with __builtin_unreachable()

Sun, 23 Feb 2025 08:00:00 +0200

Consider that we have an array of 8-bit unsigned integers and we want to calculate their sum modulo 256.

In C++ we can do it like this:

uint8_t sum(const uint8_t *data, size_t len)
{
    return std::accumulate(data, data + len, uint8_t(0));
}

When compiling with Clang 19.1.0 and flags -O3 -march=rocketlake -fno-unroll-loops, we get the following assembly:

sum():
        test    rsi, rsi
        je      .LBB0_1
        cmp     rsi, 16
        jae     .LBB0_4
        xor     eax, eax
        mov     rdx, rdi
        jmp     .LBB0_15
.LBB0_1:
        xor     eax, eax
        ret
.LBB0_4:
        cmp     rsi, 32
        jae     .LBB0_10
        xor     eax, eax
        xor     ecx, ecx
.LBB0_6:
        mov     r8, rsi
        and     r8, -16
        lea     rdx, [rdi + r8]
        movzx   eax, al
        vmovd   xmm0, eax
.LBB0_7:
        vpaddb  xmm0, xmm0, xmmword ptr [rdi + rcx]
        add     rcx, 16
        cmp     r8, rcx
        jne     .LBB0_7
        vpshufd xmm1, xmm0, 238
        vpaddb  xmm0, xmm0, xmm1
        vpxor   xmm1, xmm1, xmm1
        vpsadbw xmm0, xmm0, xmm1
        vmovd   eax, xmm0
        cmp     r8, rsi
        jne     .LBB0_15
        jmp     .LBB0_9
.LBB0_10:
        mov     rcx, rsi
        and     rcx, -32
        vpxor   xmm0, xmm0, xmm0
        xor     eax, eax
.LBB0_11:
        vpaddb  ymm0, ymm0, ymmword ptr [rdi + rax]
        add     rax, 32
        cmp     rcx, rax
        jne     .LBB0_11
        vextracti128    xmm1, ymm0, 1
        vpaddb  xmm0, xmm0, xmm1
        vpshufd xmm1, xmm0, 238
        vpaddb  xmm0, xmm0, xmm1
        vpxor   xmm1, xmm1, xmm1
        vpsadbw xmm0, xmm0, xmm1
        vmovd   eax, xmm0
        cmp     rcx, rsi
        je      .LBB0_9
        test    sil, 16
        jne     .LBB0_6
        add     rcx, rdi
        mov     rdx, rcx
.LBB0_15:
        add     rdi, rsi
.LBB0_16:
        add     al, byte ptr [rdx]
        inc     rdx
        cmp     rdx, rdi
        jne     .LBB0_16
.LBB0_9:
        vzeroupper
        ret

The code is nicely vectorized, but that’s quite a lot of assembly (68 lines).

Looking closer at it, we can see that the ‘main’ loop is this one, which works on YMMWORD-sized chunks (256-bits worth of data or, in this case, 32 x 8-bit integers):

.LBB0_11:
        vpaddb  ymm0, ymm0, ymmword ptr [rdi + rax]
        add     rax, 32
        cmp     rcx, rax
        jne     .LBB0_11

There’s also this part that adds up a XMMWORD-sized chunk if len % 32 >= 16¹.

.LBB0_7:
        vpaddb  xmm0, xmm0, xmmword ptr [rdi + rcx]
        add     rcx, 16
        cmp     r8, rcx
        jne     .LBB0_7

This part deals with the elements that remain after processing all YMMWORD and XMMWORD chunks (i.e. it adds up the remaining len % 16 integers):

.LBB0_16:
        add     al, byte ptr [rdx]
        inc     rdx
        cmp     rdx, rdi
        jne     .LBB0_16

And last but not least, there’s also this branch that’s used for returning zero immediately whenever we pass len == 0 to the function:

sum():
        test    rsi, rsi
        je      .LBB0_1
;       ................
.LBB0_1:
        xor     eax, eax
        ret

If we care about vectorization and minimizing code sizes (for example because we might want to do aggressive inlining, but at the same time we want to keep important things relatively close together in the instruction cache), then we can significantly reduce the assembly by imposing or exploiting a couple of constraints on our input. For example, let’s assume that:

The length of the array that we want to sum up is always a multiple of 32 (the number of 8-bit values that fit in a YMM register). This constraint can be a natural property of our input, or we can artificially add zero values as padding until we reach a length that is a multiple of 32.
The array is non-empty. By convention, we can say that the sum function must NOT be called on empty arrays.

We can express these constraints with __builtin_unreachable()², a compiler built-in which tells the compiler that a specific code location/code path is unreachable:

uint8_t sum_with_constraints(const uint8_t *data, size_t len)
{
    constexpr size_t U8_VALUES_PER_YMMWORD = 32;
    if (len % U8_VALUES_PER_YMMWORD != 0)
        __builtin_unreachable(); // `len` is always a multiple of 32.
    if (len == 0)
        __builtin_unreachable(); // `len` is never zero.

    return std::accumulate(data, data + len, uint8_t(0));
}

Let’s see where this gets us, with the same compiler/flags as before:

sum_with_constraints():
        vpxor   xmm0, xmm0, xmm0
        xor     eax, eax
.LBB0_1:
        vpaddb  ymm0, ymm0, ymmword ptr [rdi + rax]
        add     rax, 32
        cmp     rsi, rax
        jne     .LBB0_1
        vextracti128    xmm1, ymm0, 1
        vpaddb  xmm0, xmm0, xmm1
        vpshufd xmm1, xmm0, 238
        vpaddb  xmm0, xmm0, xmm1
        vpxor   xmm1, xmm1, xmm1
        vpsadbw xmm0, xmm0, xmm1
        vmovd   eax, xmm0
        vzeroupper
        ret

Pretty good. That’s the entire output. We’re down to 17 lines, with very little effort. In terms of machine code size, we’re down from 222 bytes (in the initial unoptimized version) to 65 bytes (the version above). This is because the compiler now knows it’s redundant to generate those branches that deal with empty arrays, or those for processing an XMMWORD chunk, or single elements. Since we indicated that the entire array can be traversed in YMMWORD-sized chunks, the compiler disregards all other branches from the first version.

This may be a small and simple example, but generally speaking, you may be surprised how much performance/code size improvements you can squeeze out by simply exploiting your input’s constraints. And these things can eventually add up.

If you’ve identified a bottleneck, try reasoning about whether or not your input has any exploitable properties. For example, if you’re doing a lot of work parsing integers with std::stoi()³, but your integers are actually between 0 and 100, try writing a custom parsing function that uses this information and see if it improves the speed. Or, if you’re decoding some JSON data that necessarily adheres to a certain schema, try NOT decoding that JSON data as an arbitrary JSON, because you might end up paying a lot for various validations or storing metadata⁴ for abstract JSON types, even though you already know your actual types before decoding.

What about GCC?

Often times this kind of optimizations won’t translate well (or not at all) when switching from one compiler to another.

In this case GCC does optimize the sum_with_constraints() version as expected (i.e. it eliminates all branches except for the YMMWORD loop).

However, when we try using a std::vector instead of a raw pointer and a length, things get messy:

uint8_t sum_vec_with_constraints(const std::vector<uint8_t> &vec)
{
    constexpr size_t U8_VALUES_PER_YMMWORD = 32;
    if (vec.size() % U8_VALUES_PER_YMMWORD != 0)
        __builtin_unreachable(); // `len` is always a multiple of 32.
    if (vec.size() == 0)
        __builtin_unreachable(); // `len` is never zero.

    return std::accumulate(vec.begin(), vec.end(), uint8_t(0));
}

The good news is that Clang is still capable of optimizing this, just like sum_with_constraints(). The bad news is that GCC isn’t capable. In this case, __builtin_unreachable() is ineffective (the same assembly gets generated when we omit those __builtin_unreachable() calls altogether).

We can try making the sum_vec_with_constraints() function use sum_with_constraints(), like this:

__always_inline
uint8_t sum_with_constraints(const uint8_t *data, size_t len)
{
    constexpr size_t U8_VALUES_PER_YMMWORD = 32;
    if (len % U8_VALUES_PER_YMMWORD != 0)
        __builtin_unreachable(); // `len` is always a multiple of 32.
    if (len == 0)
        __builtin_unreachable(); // `len` is never zero.

    return std::accumulate(data, data + len, uint8_t(0));
}

uint8_t sum_vec_with_constraints_v2(const std::vector<uint8_t> &vec)
{
    const uint8_t *data = vec.begin().base();
    const size_t   len  = vec.size();
    return sum_with_constraints(data, len);
}

But even this does not work. I tried a few other ways of helping the compiler figure this out, but none of them were successful. If you give it a try and find something that works, let me know (see update⁵)!

I opened an issue on GCC’s tracker with this problem, and it seems to be related to how expressions get optimized out in certain contexts. We call __builtin_unreachable() on a constraint for a certain expression, but that expression ends up not being used in the actual code, from the perspective of the compiler after a few optimization passes. Andrew Pinski described the problem more concretely:

This is because len is vec.end() - vec.begin() where end() is a pointer load.
And then data + len gets re-optimized to just vec.end().
And so len is no longer taken into account in the std::accumulate loop.

This goes to show how:

A seemingly trivial optimization pass can completely throw off another optimization pass, especially when you’re doing this kind of very fine-grained manual optimizations without dropping down to assembly.
A ‘zero-cost’ abstraction (which is what some people may call std::vector, for example) can end up costing a lot more than zero compared to a more C-like counterpart.

I appreciate the GCC team’s quick responses to these reports. Hopefully they find a solution for this issue.

Source code:

sum(): raw, Godbolt
sum_with_constraints(): raw, Godbolt
sum_vec_with_constraints(): raw, Godbolt
sum_vec_with_constraints_v2(): raw, Godbolt

It only adds up 1 XMMWORD-sized chunk, because 2 or more would constitute at least a YMMWORD chunk (and those have already been processed at this point). ↩︎
If you prefer not using the compiler built-in, C23 has unreachable(), and C++23 has std::unreachable(). ↩︎
Or the function in your language of choice that parses arbitrary 32-bit integers. This advice is not language-specific. ↩︎
E.g. union tags, array pointers/lengths. If according to some pre-defined schema you have a million JSON arrays filled with a single integer element each, you can end up paying more for the internal array representations in your language of choice, than for the actual array elements; instead, you could write a custom JSON decoder that only allocates a single array with a million elements worth of capacity, and append each integer to it while decoding the JSON; this avoids wasting a lot of space for the internal representations of one million arrays, assuming that’s how your non-schema-based JSON decoder would do it. ↩︎
@namniav commented on Twitter with a few solutions. You can use asm volatile, std::launder()⁶, or a volatile qualifier to basically make the compiler disconnect len from vec.end() - vec.begin(). After that, the __builtin_unreachable() requirements will still hold in the std::accumulate() loop. The asm volatile approach produces the shortest assembly; the other ones have a bit of overhead. See the solutions here. ↩︎
Who even chooses these names? ↩︎

My approach to writing

Thu, 20 Feb 2025 03:59:57 +0200

Short (meta) blog post about what kind of educational materials I prefer, and why:

I generally prefer learning material that is impersonal, and doesn’t contain author-specific idiosyncrasies. When I’m reading or watching something, I’m not interested in hearing the author give a ‘performance’, I’m typically interested strictly in the key information being conveyed. As such, I don’t like seeing jokes, quirky formulations or other similar things through which the author seemingly seeks to show off their personality, because that just serves as a distraction from the (supposedly) important information.

That’s the approach I like to see, and that’s the approach I try to employ as much as possible. When I write something, I want to respect the reader’s time. Therefore, I try to iterate on paragraphs/chapters/etc. until the information is very, very clear. After that, I try to reduce the size of the material as much as possible while keeping the clarity constant.

So, with this approach, jokes or other such things have no place in the material. I’m not trying to rhetorically impress you, I’m not trying to show off my personality or my quirks. I’m only trying to lay out some information that hopefully is a bit useful to you (at the very least), and do so with as much clarity and conciseness as possible.

And just to be extra clear: I’m not saying that this approach is better than others in an objective sense. I’m only saying what subjectively works better for me.

A good example of this style is TCP/IP Illustrated by Kevin R. Fall & W. Richard Stevens. I’ve been reading Volume I for a while and from early on in the book I was really impressed by the style of writing and how easy it was to follow.

A Clang regression related to switch statements and inlining

Sun, 16 Feb 2025 01:25:19 +0200

After my previous post, Eliminating redundant bound checks (read it for context if you haven’t already), I wanted to do a benchmark using the ‘optimized’ version of the increment() function, which didn’t contain any bound checks when compiled with Clang, even though we used .at() for indexing into the array. Here’s that last version of the function from the previous post (I made it static because I’m using a single compilation unit):

static void increment(std::array<uint8_t, 1024> &data, uint8_t first_idx)
{
    size_t second_idx = first_idx_to_second_idx(first_idx);
    data.at(second_idx)++;
}

My benchmark goes like this:

int main()
{
    std::vector<uint8_t> indexes = generate(); // Generate 300 million indexes.

    auto t0 = std::chrono::high_resolution_clock::now();

    std::array<uint8_t, 1024> arr = {};
    for (auto idx : indexes)
        increment(arr, idx);

    auto t1 = std::chrono::high_resolution_clock::now();

    // Use the results from increment() so the function call is not optimized away.
    auto sum = std::accumulate(arr.begin(), arr.end(), 0);

    // Print the elapsed time and the sum.
    auto dur = std::chrono::duration_cast<std::chrono::milliseconds>(t1 - t0);
    std::println("elapsed: {} sum: {}", dur, sum);
}

I generate 300 million instances for first_idx, and then I call increment() with them. In order to avoid inlining messing with the results, I also benchmarked a version of increment() that is not inlined:

__attribute__((noinline))
static void increment_non_inlined(std::array<uint8_t, 1024> &data, uint8_t first_idx)
{
    size_t second_idx = first_idx_to_second_idx(first_idx);
    data.at(second_idx)++;
}

The results were as follows:

$ clang++ inlined.cpp -stdlib=libc++ -std=c++23 -O3 -march=native
$ ./a.out
elapsed: 2810ms sum: 27392
$ clang++ non-inlined.cpp -stdlib=libc++ -std=c++23 -O3 -march=native
$ ./a.out
elapsed: 547ms sum: 29696

That’s rather… unexpected. Allowing the function to be inlined, and thus giving it more context regarding the way it’s called, is making the performance worse, not better. And not just by a little, it’s roughly 5 times slower.

What’s causing the issue? Well, the behavior of the increment() function vastly changes when it’s inlined compared to non-inlined. Let’s look at the assembly.

The non-inlined version does what we saw in the previous post. It gets the second_idx by indexing into a lookup table¹ with first_idx, and then it increments the integer at that position.

increment:
        add     sil, -128
        movzx   eax, sil
      # size_t second_idx = first_idx_to_second_idx(first_idx);
        lea     rcx, [rip + .Lswitch.table.increment]
        mov     rax, qword ptr [rcx + 8*rax]
      # data.at(second_idx)++;
        inc     byte ptr [rdi + rax]
        ret

On the other hand, the inlined version does something very bad for performance. Every case in the switch has a different branch (or label), and each one will feed into a main branch that does the actual increment (.LBB0_266):

.LBB0_11:
        lea     rcx, [rsp + 561] # case 1: return 553;
        jmp     .LBB0_266
.LBB0_12:
        lea     rcx, [rsp + 549] # case 2: return 541;
        jmp     .LBB0_266
#       ...
.LBB0_266:
        inc     byte ptr [rcx] # data.at(second_idx)++;
        inc     r12
        cmp     r12, 268435456
        je      .LBB0_8
        movzx   ecx, byte ptr [rbx + r12] # Get the next first_idx.
        movsxd  rdx, dword ptr [rax + 4*rcx] # Get the next switch case label.
        add     rdx, rax
        mov     rcx, rbp
        jmp     rdx # Jump to the switch case label of the current first_idx value.

Since the indexes are randomly generated, the branch predictor won’t be able to accurately predict to which of those switch case labels to jump on each iteration. If the RNG that we use is good, we’d expect the branch predictor to miss on almost every iteration. We can verify this with perf stat:

$ sudo perf stat ./a.out # increment()
elapsed: 2815ms sum: 27136

 Performance counter stats for './a.out':
        ...
        4809459795      branches      # 542.294 M/sec
         299087225      branch-misses # 6.22% of all branches

$ sudo perf stat ./a.out # increment_non_inlined()
elapsed: 546ms sum: 27648

 Performance counter stats for './a.out':
        ...
        4804843199      branches      # 735.653 M/sec
             79808      branch-misses # 0.00% of all branches

So the inlined version generates 299087225 branch misses, while the non-inlined version generates 79808. The number 299087225 is pretty close to the total number of first_idx‘es that we generate (300 million), so the expectation seems to be correct². No wonder the performance tanks so hard.

Next, I wanted to check if this botched optimization could be fixed by some flags given to Clang. The zig toolchain has some defaults which have given me better performance in the past when simply replacing clang++ with zig c++ in the compilation command. Testing the inlined version which performed badly with clang++:

1
2

$ zig c++ inlined.cpp -std=c++23 -O3 -march=native && ./a.out
elapsed: 243ms sum: 28928

That did it, it’s now 11 times faster than what we compiled with Clang.

I have Zig 13 locally, which uses Clang 18.1.6. My previous clang++ commands were using version 19.1.6. After trying to add some flags (that the zig c++ toolchain sets) to the previous clang++ command without managing to improve the performance, I was thinking that this may just be a regression in Clang 19. Testing with Clang 18.1.8:

1
2
3

$ /tmp/clang-18/bin/clang++ inlined.cpp -stdlib=libc++ -std=c++23 -O3 -march=native
$ ./a.out
elapsed: 239ms sum: 27648

So indeed, this seems to be a regression³ (at the very least, for the purposes of this use case). I opened an issue on Clang/LLVM’s repo; you can find it here if you’re interested in this problem’s conclusion. Someone already responded saying that it’s a trade-off between two optimization passes called SROA (scalar replacement of aggregates) and SimplifyCFG (simplify control flow graph). Remains to be seen if this was intentional and/or if they want to do something about it going forward.

You can see the benchmark code here, or play around with a smaller reproduced version on Godbolt.

Note: this lookup table is not related to the TABLE variable that we used in the first version of the increment() function in the previous post. This lookup table is the assembly representation that Clang generates for the big switch statement. ↩︎
I also ran the same code but with 100 million first_idx‘es instead of 300 million, just to verify that those ~300 million branch misses we saw are directly related to the number of increment() invocations, and not to something else. In this case the number of branch misses was roughly 100 million, which further confirms the original expectation. ↩︎
Note (27 March 2025): The issue is still present in the newest release – Clang/LLVM 20.1.1. ↩︎

Eliminating redundant bound checks

Wed, 12 Feb 2025 23:33:07 +0200

Problem

Consider that you have some mapping from indexes in the range [0, 255] to indexes in the range [0, 1023]. The requirements are:

Write a function that takes an array of 1024 8-bit unsigned integers and increments the value at an arbitrary index.
Only use safe methods of indexing that do bounds checking (e.g. using .at() in C++ instead of the subscript operator).
Avoid constructs such as __builtin_unreachable() or __builtin_assume().
Arrive at an implementation where, even though you’re using checked indexing methods, those checks are omitted from the generated assembly.

Finding a solution

Based on this description, my first instinct would be to write a constexpr table defining the index mapping:

// Pre-defined index mapping:
static constexpr std::array<size_t, 256> TABLE = {
    798, 553, 541, 345, 276, 698, 861, 448, 898, 588, 678,  593, 265,  611, 915,
    835, 893, 3,   411, 769, 792, 115, 526, 836, 356, 454,  279, 876,  924, 644,
//  ............................................................................
};

And then use the table like this to increment the value at a given position:

void increment(std::array<uint8_t, 1024> &data, uint8_t first_idx)
{
    size_t second_idx = TABLE.at(first_idx);
    data.at(second_idx)++;
}

When you look at the generated assembly you’ll see that GCC and Clang are smart enough to realize that doing bound checks on TABLE.at(first_idx) is redundant, because TABLE is of length 256 and the range of values for a uint8_t index is [0, 255]; therefore, the access is always going to be within bounds. Great!

However, GCC and Clang are not smart enough (yet) to realize that doing bound checks on data.at(second_idx) is also redundant because, as per the problem statement, TABLE contains values strictly in the [0, 1023] range and data has length equal to 1024. So the access at second_idx is always going to be valid in this case too.

Clang’s output goes like this, and GCC’s is similar:

increment(std::array<unsigned char, 1024ul>&, unsigned char):
        mov     eax, esi
        lea     rcx, [rip + TABLE]
        mov     rsi, qword ptr [rcx + 8*rax]
        cmp     rsi, 1024 # REDUNDANT CHECK!
        jae     .LBB0_2
        inc     byte ptr [rdi + rsi]
        ret
.LBB0_2: # REDUNDANT BRANCH!
        push    rax
        lea     rdi, [rip + .L.str]
        mov     edx, 1024
        xor     eax, eax
        call    std::__throw_out_of_range_fmt(char const*, ...)@PLT

TABLE:
        .quad   798
        .quad   553
#       ...........

That extra branch is probably not the end of the world, but we can try to do better.

So… what’s happening here? Well, it seems that Clang and GCC don’t have enough insight regarding the range of values in TABLE in order to make the inference that checking bounds on data.at(second_idx) is redundant. I submitted this missed optimization opportunity to both GCC¹ and Clang/LLVM², so hopefully they get this implemented.

In the meantime, we can get around this issue by exploring other methods of representing the index mapping (i.e. not as a constexpr table).

From Andrew Pinski’s comment in the report on bugzilla:

Maybe the best way to solve this is have an IPA pass over analyzes the static variables for their ranges and store that range off and then when VRP comes a long and reads the range and records it. In a similar sense as we handle return value ranges. Note there might be other bug reports for a similar thing …

I understood that GCC does have a mechanism which applies this type of optimization, called Value Range Propagation (VRP), but it’s either not implemented at all for table lookups, or its capabilities are more primitive compared to the ‘return value range propagation’ that Andrew Pinski mentioned.

So, we can then try representing the index mapping as a function which uses a big switch statement with a case for every index in the range [0, 255], which returns its appropriate mapped value:

static size_t first_idx_to_second_idx(uint8_t first_idx)
{
    // Pre-defined index mapping:
    switch (first_idx) {
    case 0: return 798; case 1: return 553; case 2: return 541; case 3: return 345;
    case 4: return 276; case 5: return 698; case 6: return 861; case 7: return 448;
//  ...............................................................................
}

Now, if we adapt increment() to use this function instead, like so:

void increment(std::array<uint8_t, 1024> &data, uint8_t first_idx)
{
    size_t second_idx = first_idx_to_second_idx(first_idx); // Use this instead of TABLE
    data.at(second_idx)++;
}

Unfortunately, GCC still doesn’t apply the optimization, but Clang does:

increment(std::array<unsigned char, 1024ul>&, unsigned char):
        add     sil, -128
        movzx   eax, sil
        lea     rcx, [rip + .Lswitch.table.increment(std::array<unsigned char, 1024ul>&, unsigned char)]
        mov     rax, qword ptr [rcx + 8*rax]
        inc     byte ptr [rdi + rax]
        ret

.Lswitch.table.increment(std::array<unsigned char, 1024ul>&, unsigned char):
        .quad   782
        .quad   582
#       ...........

Finally, it sees that the bounds check is redundant, and generates branchless code. So representing the mapping as a function was enough to help VRP see through the range of possible index values, and then optimize the code better.

Job done! You can see the full code here, or play around with it on Godbolt.