I've implemented counters exactly like this. I actually migrated away from the tls_counter to something like the cas_multi_counter.

The problem with the tls_counter is memory usage that scales with the number of threads. In this example you need 16 bytes per thread because only one counter is supported, I believe we ended up with 48 bytes per thread. So with 10,000 threads you can end up with >=160KB of data for every counter. And 10,000 threads isn't that far-fetched if most are sleeping (though not ideal). Then you end up with 10,000 counters and you're spending >= 1.6 GB of RAM on counters. Where the cas_multi_counter would only be using 40.96 MB. And read() scales with the number of threads, so can get pretty slow, but that is secondary.

One optimization you can use to make the cas_multi_counter more memory efficient is to share the backing memory. You have 4096 bytes, but are only using the first 8 bytes of each 64-byte stripe. You can multiplex 8 counters into the same 4096 byte storage. The first counter gets offset 0, the second offset 8, and so on. This only adds 1 indirection in the hot operator++() path, and doesn't add any extra false sharing as long as the 4096 byte buffer is 64-byte aligned. Construction and destruction is more expensive, but not terribly so, and that is probably cold anyway. Now you're at 512 bytes per counter, and the size is independent of the cache-line size. If you can count the number of active CPUs you can dynamically size your buffer to only have # CPUs cachelines, further saving memory.

The author correctly identifies the problems with sched_yield() [1], but then used it anyway. In my experience, using nanosleep() with an exponential backoff and a threshold performs well and results in the kind of behavior people want when they use sched_yield(). I think the difference is enough to affect the results, so I’m surprised the author didn’t experiment with it.

[1] for an additional reference, https://smackernews.com/item/10926654 HN

Great writeup to use as introduction and reference.

The striping to have a memory location per core is great for performance but can have a noticeable memory overhead.

It would be interesting to have Intel TSX in the mix. It can be a benefit to some workloads and does not require many adjustments to your code.

Really enjoyed this article, thank you for sharing

Great article! Will definately keep this hierarchy in mind when writing concurrent code.

Wow. Hellava good write up. Def appreciated. I learned a lot from last two examples.

Great article. Loved it!