SELECT SearchPhrase
FROM hits
WHERE SearchPhrase IS NOT NULL
ORDER BY SearchPhrase
LIMIT 10;

It returns the top 10 rows sorted by non-NULL SearchPhrase column values in ascending order. The query is taken from the ClickBench benchmark.

In the case of large datasets, such queries can be efficiently executed on multiple threads. This requires several stages:

1. The "query owner" thread divides the dataset into small enough frames. The maximum size for each frame varies in the range of 10K-1M rows in popular analytical databases. Say, DuckDB uses 10K frames while in QuestDB they're up to 1M rows in size. Then it publishes tasks containing the frames in an SPMC queue. The tasks should also contain data required to execute the filter (WHERE SearchPhrase IS NOT NULL in our case).

2. Worker threads pick up the tasks from the queue and process them. First, they apply the filter. This step is optional as there may be no filter in the query. Then they try adding filtered rows to a sorted data structure, like min/max heap or R/B tree. The data structure may contain row IDs, in case the storage format supports random access, or materialized columns returned by the query, in case random access isn't possible due to data compression. As a result of a task execution, the data structure contains up to the top 10 rows. Each worker thread should have its instance of the data structure and use it when handling all tasks belonging to the given query.

3. Once all tasks are processed, the "query owner" thread needs to merge the worker "top 10 rows" data structures into a single one, so that it can return the result set back to the client. This can be done eagerly, e.g. the "query owner" thread may iterate through all worker data structures and put their rows into its data structure, or lazily via the k-way merge algorithm.

The 1st and 3rd stages above are serial, but they assume very little work while the main body of work is done in the 2nd stage. Thanks to this, this execution model scales very nicely with the number of worker threads and CPU cores.

I'm interested in learning more about analytical query execution models and not only, so if you have anything to share, don't hesitate to write a comment. Have fun coding and see you next time.

As you may already know, hash tables play a crucial role not only in hash joins, but in parallel GROUP BY execution. There is no "silver bullet" design for a hash table, so having data structures specialized for a given scenario is the key to efficient query execution. QuestDB's SQL engine currently uses 6 hash tables written in Java and one written in C++. The numbers are very likely to grow in the future.

The hash table of our interest is aimed for GROUP BY queries over a single varchar key, like the following one:

SELECT URL, COUNT(*) AS c
FROM hits
ORDER BY c DESC
LIMIT 10;

It's a hash table with linear probing. The data is stored in a contiguous chunk of native memory and has the following layout for key/value pairs:

| Hash code 32 LSBs |    Size    | Varchar pointer | Value columns 0..V |
+-------------------+------------+-----------------+--------------------+
|       4 bytes     |  4 bytes   |     8 bytes     |         -          |
+-------------------+------------+-----------------+--------------------+

The trick is that the 3rd field here contains pointers to mmapped memory. So, the hash table itself doesn't hold copies of varchar byte arrays but instead points at external (stable) memory. This way, we don't need to allocate additional memory and do a varchar copy. Another nice side effect is a lower memory footprint, all thanks to page cache memory being elastic, i.e. the OS evicts pages on memory pressure and reloads them from the disk on future access.

Based on our benchmarks, this hash table is 10-30% faster than our "default" hash table. While not much, in the case of slow queries, the execution time is reduced by several seconds, so it's worth it.

Have fun coding and see you next time.

To keep things simple, we'll write an atomic tuple class holding 3 long values. The structure of the class looks like the following:

public class AtomicLongTuple {

    // Version is used for the seqlock.
    long version;
    // 3 long fields:
    long x;
    long y;
    long z;

    // Var handles for the fields:
    private static final VarHandle VH_VERSION;
    private static final VarHandle VH_X;
    private static final VarHandle VH_Y;
    private static final VarHandle VH_Z;

    // Holder class used by readers and writers.
    public static class TupleHolder {
        public long x;
        public long y;
        public long z;
    }

    // Readers provide a holder to avoid an allocation.
    public void read(TupleHolder holder) {
        // Implementation goes here...
    }

    // Writers provide a function to modify the latest state.
    public void write(Consumer<TupleHolder> writer) {
        // Implementation goes here...
    }
}

Here we have the tuple, i.e. 3 long fields, plus one more long field called version . As we'll see now, the version is needed to implement the seqlock. Of course, the above code doesn't include boring details such as padding or VarHandle initialization, but you may find the full source code here.

To learn about the seqlock, let's start with the write() method:

public void write(Consumer<TupleHolder> writer) {
    for (;;) {
        final long version = (long) VH_VERSION.getAcquire(this);
        if ((version & 1) == 1) {
            // Another write is in progress. Back off and keep spinning.
            LockSupport.parkNanos(1);
            continue;
        }

        // Try to update the version to an odd value (write intent).
        // We don't use compareAndExchangeRelease here to avoid
        // an additional full fence following this operation.
        final long currentVersion = (long) VH_VERSION.compareAndExchange(this, version, version + 1);
        if (currentVersion != version) {
            // Someone else started writing. Back off and try again.
            LockSupport.parkNanos(10);
            continue;
        }

        // Apply the write.
        writerHolder.x = (long) VH_X.getOpaque(this);
        writerHolder.y = (long) VH_Y.getOpaque(this);
        writerHolder.z = (long) VH_Z.getOpaque(this);
        writer.accept(writerHolder);
        VH_X.setOpaque(this, writerHolder.x);
        VH_Y.setOpaque(this, writerHolder.y);
        VH_Z.setOpaque(this, writerHolder.z);

        // Update the version to an even value (write finished).
        VH_VERSION.setRelease(this, version + 2);
        return;
    }
}

The above code uses acquire/release memory semantics here, so if you're not familiar with it, refer to this post. Other than that, the code is quite simple: each writer spins trying to increment the version field to an odd value to prevent concurrent writes (think, a spinlock) and, when it succeeds, applies the given function to the current tuple state. Finally, it increments the version, so that it has an even value telling readers and other writers that there is no ongoing write.

Notice that we're using plain loads and stores to fetch and modify the tuple in the "critical section". These operations don't need to be atomic, all thanks to the atomicity (and ordering guarantees) of the surrounding operations and memory fences.

Now, let's check what readers do:

public void read(TupleHolder holder) {
    for (;;) {
        final long version = (long) VH_VERSION.getAcquire(this);
        if ((version & 1) == 1) {
            // A write is in progress. Back off and keep spinning.
            LockSupport.parkNanos(1);
            continue;
        }

        // Read the tuple.
        holder.x = (long) VH_X.getOpaque(this);
        holder.y = (long) VH_Y.getOpaque(this);
        holder.z = (long) VH_Z.getOpaque(this);

        // We don't want the below load to bubble up, hence the fence.
        VarHandle.loadLoadFence();

        final long currentVersion = (long) VH_VERSION.getAcquire(this);
        if (currentVersion == version) {
            // The version didn't change, so the atomic snapshot succeeded.
            return;
        }
    }
}

The reader's code is even simpler than the writer's one. Each reader spins trying to read an even version, then reads the tuple state. Finally, it reads the version once again and compares it with the previously read value. If the value hasn't changed, the reader was able to read the tuple atomically, so the operation finishes. Again, the tuple state operations are non-atomic, but thanks to the surrounding operations and fences that's not needed.

A "classical" seqlock assumes a separate sequence number and a mutex. Using a mutex over a hand-made spinlock is a wise choice since mutexes in modern runtimes and OSes came a long path and they're not as expensive as they used to be. In our case, the version field combines the sequence number and the mutex (to be more precise, a spinlock). For educational purposes, that's not really important since the algorithm is the same. Thanks to seqlock, we were able to implement atomic memory snapshots with a few lines of code.

As you may have noticed, the algorithm is not lock-free since if a writer thread gets blocked after it has incremented the version to an odd value, no readers and writers would be able to make progress. Considering this, you may be asking yourself how practical this approach is. It's certainly not versatile and only makes sense in use cases when the writes are infrequent and the total amount of memory to be read stays within a few cache lines (ideally, within a single cache line, i.e. 64 bytes on most modern machines). In other scenarios, it's better to use a good old exclusive or shared mutex.

If you're curious about how many wasteful spins a reader has to make before making a successful snapshot, there is a test for that. It starts a few reader threads that spin over the read() method and a single writer thread that calls write() and then emulates a bit of work by calling Blackhole.consumeCPU(50) on the Blackhole class from JMH. When the test finishes, each reader thread reports the wasteful_spins / total_spins ratio which, on my Linux machine, is around 0.7-0.9. So, in the presence of relatively frequent writes, no reader has to do more than a couple of spins on average.

That's it for today. Good luck and see you next time.

The core idea of the algorithm is that writers do a loop of CAS operations on each stored word. They try to swap the value (or a pointer) with a pointer to the so-called operation descriptor structure. The structure includes old and new values, as well as the operation status. Once a writer successfully completes all CASes, it does one more CAS on the status field marking the operation as complete. So, the algorithm requires k+1 single-word CAS operations per k-word CAS.

Indeed, this k-word CAS algorithm is lock-free and linearizable, but if you also want to be able to read all k words atomically, the algorithm won't be of any help. Of course, you can do a no-op k-word CAS to do an atomic read, but such read may be costly. One more downside is that being able to swap a primitive value with a pointer means that the algorithm is not meant to be used in any language with a GC. In theory, it can be still used in languages with non-moving GC, such as Golang, but with unsafe things like pointer tagging. Also, if you're fine with less plain memory layout, then, say, in Java the algorithm may be modified to use an array of AtomicReference<Object> instead of an array of primitive type (usually, long[]).

If lock-freedom is not a must, a seqlock-like approach might do just fine. The sequence field could be used to preserve exclusive writer access, as well as enough help for readers to determine whether they got an atomic snapshot of all k-words. If lock-freedom is important, then in languages with GC there is an even simpler option which is to use an immutable data structure and let the writers do a single-word CAS swapping the pointer to the old values with the new one. Finally, a good old lock could be used, optionally a much more scalable reader-writer one.

First, let's check what POSIX.1-2017 specification says about fsync:

The fsync() function shall request that all data for the open file descriptor named by fildes is to be transferred to the storage device associated with the file described by fildes. The nature of the transfer is implementation-defined.

The above description is rather vague. If the OS issues operations to write the data to the disk's volatile cache, that's a "transfer", so formally such OS would be POSIX-compliant. The informative section of the spec sheds more light on what a proper fsync implementation should do:

The fsync() function is intended to force a physical write of data from the buffer cache, and to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk. Since the concepts of "buffer cache", "system crash", "physical write", and "non-volatile storage" are not defined here, the wording has to be more abstract.

OK, that's much more specific. Once an fsync call is made, the data should become durable in the face of a system crash, e.g. due to a power loss. But is it really something Linux does on an fsync?

As with many other FS-related system calls, most (if not all) file systems have their own implementation of fsync. To keep things simpler, we're going to check the ext4 implementation in the recent 6.x kernel code base. We should be looking at the `ext4_sync_file()` function which is invoked on an fsync() call. It involves the following steps:

1. First, it writes all dirty pages belonging to the file that corresponds to the input file descriptor to the disk. That's done by the `file_write_and_wait_range()` function. As a result, the data may be sitting in the disk volatile cache, so that's not what we're looking for.

2. Next, it writes the inode's metadata to the disk. Depending on whether journaling is enabled on the FS or not, it's done via a specific function, e.g. `ext4_fsync_journal()`. Again, not something we're in search of.

3. Finally, if the needs_barrier variable is true, it calls the blkdev_issue_flush() function. That's probably what we need, isn't it?

Let's leave the needs_barrier variable out of the equation for now and check what blkdev_issue_flush() does. This function queues a flush operation to the block device and waits until it's finished. The operation has the REQ_PREFLUSH bit set among the flags. If we open kernel docs, we'll find some information on this flag (and not only):

In addition the REQ_PREFLUSH flag can be set on an otherwise empty bio structure, which causes only an explicit cache flush without any dependent I/O. It is recommend to use the blkdev_issue_flush() helper for a pure cache flush.

As we expected, the REQ_PREFLUSH flag (as well as the REQ_FUA flag) tells the block device that it should flush its volatile cache to the persistent storage. Drivers for any well-behaved disk with a volatile write cache should handle this flag properly. Obviously, disks without such cache don't need to bother with these operations and flags.

Now, what's the buzz with needs_barrier? In both journaled and non-journaled ext4 code paths, it appears to be nothing more, but an optimization to avoid sending flush requests to the disk multiple times. Note that you can also configure ext4 not to issue flush operations. For example, in non-journaled mode it's done via EXT4_DEFM_NOBARRIER mount option.

Other file systems have their own specifics, but the overall logic should be close enough to the ext4's one. So unlike macOS, Linux does its best to transfer the data to persistent storage on fsync. Of course, this doesn't protect you from a flawed driver implementation written for a cheap no-name drive, but it also means that if you have a decent SSD from a well-known brand, you may be fine without an enterprise-grade disk.

To be slightly more concrete, here is a simple example of a query that can benefit from scatter-gather execution:

SELECT sensor_id, max(temp), min(temp), avg(temp)
FROM temperature
WHERE sensor_id IN (402, 1202, 3983)
GROUP BY sensor_id;

To be able to use the scatter-gather model, the storage format used in the database needs to support parallelism, i.e. it should be possible to divide the on-disk data into individual parts that can be independently scanned. In practice, this usually means log-structured append-only storage format (which is in many cases also columnar) but not necessarily.

Now, let's consider the execution of the above query.

Scatter

For the sake of simplicity, let's assume that there is no index on the `sensor_id` column, and table columns are stored in append-only log files. In this case, it's trivial to split the log files to be scanned into N chunks based on offsets. That's basically what the original thread does when scattering the query execution. Each of the chunks is serialized into a task object/struct along with the query execution plan details, such as selected columns, aggregate functions and filter, and written into an in-memory queue to be picked up by worker threads.

Let's refer to the original thread as the orchestrator thread. If the original thread belongs to the same thread pool, it must participate as a worker, i.e. it should poll tasks from the queue and execute them. That's to avoid starvation and deadlocks in the situation when all threads try to orchestrate their own queries.

When a worker thread picks up a task, it starts executing it. It scans through the data, applies the filter (WHERE clause) and calculates the intermediate result for each aggregate function (min, max and avg in our example). A convenient way to store the result is a hash table holding <int, <int, int, float>> key-value pairs. Here, we use int type for the key assuming that sensor_id is an integer column and each <int, int, float> tuple stands for the intermediate results of our three aggregate functions.

Once a task is executed, the worker must write the intermediate result (hash table) into an in-memory queue to be consumed and gathered by the orchestrator thread.

Gather (and merge)

The orchestrator thread needs a hash table to store query results. Initially, it's empty, but as soon as the thread consumes (gathers) a result from one of the workers, it needs to merge two tables. The merge is simple thanks to the natural properties of the aggregate functions:

min(A ∪ B) = min(min(A), min(B))
max(A ∪ B) = max(max(A), max(B))
count(A ∪ B) = count(A) + count(B)
sum(A ∪ B) = sum(A) + sum(B)
avg(A ∪ B) = sum(A ∪ B) / count(A ∪ B)

It's not a complete list, but most scalar aggregate functions assume little data to be stored as their state, so they fit into scatter-gather nicely.

As soon as the orchestrator gathers and merges the last task result, it has the query result to be returned to the client.

Conclusion

Scatter-gather(-merge) is a simple single-stage execution model. It works nicely for relatively trivial GROUP BY queries with an optional WHERE clause while more complex queries involving JOINs require a more complex multi-stage parallel execution.

Of course, Amdahl's law applies to the scatter-gather model, so if the serial part of the total work is significant, the speed up from parallelism will be humble. This may be fixed by an approach similar to radix-partitioning. The idea is to split each worker's hash table into a fixed number of hash tables based on a few of the highest bytes of the hash code. Then at the later stage, the merge can be done in parallel for each set of these hash tables. As a nice side effect, it enables parallelism for later stages like ORDER BY + LIMIT.

Another advantage of this model is that it naturally applies to distributed databases. We can easily swap "thread" with "node" in the above text with no other changes except for the storage requirement. The data has to be sharded across cluster nodes and the orchestrator node has to be aware of the sharding scheme so that it's aware of the data location when it distributes the work.

I'm interested in learning more about analytical query execution models and not only, so if you have anything to share, don't hesitate to write a comment. Have fun coding and see you next time.

Wait-freedom means that any thread can make progress in a finite number of steps regardless of external factors, such as other threads blocking. A trivial example of a wait-free data structure is an atomic counter (in x86 it would use a LOCK XADD instruction), e.g. Java's j.u.c.AtomicInteger.

public class Counter {

    private final AtomicInteger cnt = new AtomicInteger();

    public int add(int delta) {
        return cnt.addAndGet(delta);
    }    

    public int get() {
        return cnt.get();
    }
}

Lock-freedom means that the application as a whole can make progress regardless of anything. So, while individual threads may be blocked, at least one of them would be making progress. A trivial example would be the same atomic counter based on a loop with a CAS operation.

public class Counter {

    private static final AtomicIntegerFieldUpdater<Counter> updater =
        AtomicIntegerFieldUpdater.newUpdater(Counter.class, "cnt");
    private volatile int cnt;

    public int add(int delta) {
        int cur;
        do {
            cur = cnt;
        } while (!updater.compareAndSet(this, cur, cur + delta));
        return cur + delta;
    }

    public int get() {
        return cnt;
    }
}

Obstruction-freedom means that a thread can make progress only if there is no contention from other threads. This guarantee is the weakest one on the list. It's hard to illustrate this definition with a simple enough example I'm not aware of a trivial example of this one, but you may refer to this paper.

Is it fair enough to say that wait-free data structures and algorithms are faster than lock-free ones and that lock-freedom means something better than obstruction-freedom and blocking code? Not really. You may limit yourself with wait-freedom if you want to limit the maximum latency of an individual operation, e.g. if you're building a real-time OS. But in most cases, you should consider all possible algorithms and their combinations. For instance, your data structure may be quite fast while it implements wait-free or lock-free reads and blocking writes based on a striped lock (wink-wink Java's ConcurrentHashMap and xsync's Map/MapOf).

If you want to learn more about multithreaded programming and scalable concurrent data structures, I highly recommend Dmitry Vyukov's old blog. Just go through all posts starting with the intro one.

Have fun coding and see you next time.

On the one hand we have ConcurrentHashMap, also known as CHM. It's a brilliant concurrent hash table: no writes to shared memory in read-only operations, hybrid linked list / tree used for buckets depending on the number of stored nodes, volatile value fields in nodes allowing to update the value without having to allocate a new node, a striped counter for the current size, and so on. CHM has been improved across many versions of Java, for many years, so reaching its level of performance isn't an easy task.

On the other hand, MapOf borrows ideas from multiple sources: buckets are organized in unrolled linked lists of cache line size thanks to Cache-Line Hash Table (CLHT), hash codes stored in the buckets to avoid extra hash function calls and pointer chasing, immutable entries to avoid extra synchronization on reads and also reduce GC pressure, also a striped counter for the current size.

Both maps have a bunch of handy atomic operations available to the developer. But enough of this boring stuff, let's do the benchmarking.

Our test stand is nothing more, but a laptop machine with an i7-1185G7 CPU (8 HT cores) and Ubuntu 22.04 running 64-bit builds of Go 1.19.3 and OpenJDK 17.0.4.

The CHM benchmark can be found here, while the Go benchmark is in the xsync repo.

The benchmarks start with a pre-warmed map holding 1,000 64-bit integer key-value pairs. The "99% reads" scenario assumes that each thread spends 99% of its calls on get (Load) operations and 0.5% on both put (Store) and remove (Delete) operations. Each operation is called for a randomly selected key, so there are no hot spots. The "75% reads" scenario is more write-heavy as write operations get 12.5% of total number of operations each.

The benchmarks were run with different number of threads/cores to be used, staring with 1 core and ending with all 8 available cores. That's to see how well both maps scale on multi-core machines. Finally, the below results are based on average values collected after 10 runs of each benchmark.

Let's start with the write-heavy scenario. CHM has an advantage here since it allows updating values in-place, so it might show better results.

As expected, CHM is slightly better on all core counts except for 4 cores. Both maps scale well enough.

Let's see results for the read-heavy scenario which might be more common in many real world applications.

Again, both maps scale well with a slight advantage of MapOf. I'm pretty happy with the results. There are certainly rough edges, but MapOf has proven its worthiness.

Have fun coding and see you next time.

This brief post is based on the talk I gave a while ago. I assume that you're familiar with the concept of state striping in concurrent counters since it's important for understanding the end application.

First of all, while Golang assigns identifiers to goroutines, it doesn't expose them. That's by the design:

Goroutines do not have names; they are just anonymous workers. They expose no unique identifier, name, or data structure to the programmer. Some people are surprised by this, expecting the go statement to return some item that can be used to access and control the goroutine later.

The fundamental reason goroutines are anonymous is so that the full Go language is available when programming concurrent code. By contrast, the usage patterns that develop when threads and goroutines are named can restrict what a library using them can do.

The same applies to the worker threads used by the Golang scheduler to run your goroutines. But the whole idea of a striped counter depends on being able to identify the current thread, so that subsequent calls are (most of the time) run on the same CPU core and, hence, avoid contention.

There are two straightforward approaches to the problem:

1. CPUID x86 instruction - not portable to other architectures and also requires some assembly or FFI calls to be used.
2. gettid(2) Linux-only system call - not portable to other OSes.

Both of these options are non-versatile and, ideally, we want to have a Go-native and cross-platform solution. Luckily there is one.

I'm talking of sync.Pool. If you're familiar with its source code, you already know that it uses thread-local pools under the hood. If we allocate a struct and place it in the pool, the next time we request it one the same thread (but not necessarily same goroutine) we should get the same struct.

Let's take a look at a fragment of the xsync.Counter's code:

// pool for P tokens
var ptokenPool sync.Pool

// ptoken is used to point at the current OS thread (P)
// on which the goroutine is run; exact identity of the thread,
// as well as P migration tolerance, is not important since
// it's used to as a best effort mechanism for assigning
// concurrent operations (goroutines) to different stripes of
// the counter.
type ptoken struct {
    idx uint32
}

// Counter is a striped int64 counter.
type Counter struct {
    stripes []cstripe
    mask    uint32
}

type cstripe struct {
    c int64
    // The padding prevent false sharing.
    pad [cacheLineSize - 8]byte
}

func NewCounter() *Counter {
    // Consider the number of CPU cores
    // when deciding on the number of stripes.
    nstripes := nextPowOf2(parallelism())
    c := Counter{
        stripes: make([]cstripe, nstripes),
        mask:    nstripes - 1,
    }
    return &c
}

// Value returns the current counter value.
func (c *Counter) Value() int64 {
    v := int64(0)
    for i := 0; i < len(c.stripes); i++ {
        stripe := &c.stripes[i]
        v += atomic.LoadInt64(&stripe.c)
    }
    return v
}

// Add adds the delta to the counter.
func (c *Counter) Add(delta int64) {
    // Pick up a token from the pool. If Add was called recently
    // on the same thread, we'll probably get the same ptoken.
    t, ok := ptokenPool.Get().(*ptoken)
    if !ok {
        // Allocate a new token and pick up a random stripe index.
        t = new(ptoken)
        t.idx = fastrand() & c.mask
    }
    for {
        stripe := &c.stripes[t.idx]
        cnt := atomic.LoadInt64(&stripe.c)
        if atomic.CompareAndSwapInt64(&stripe.c, cnt, cnt+delta) {
            // We were able to update the stripe, so all done.
            break
        }
        // CAS failed, so there is some contention over the stripe.
        // Give a try with another randomly selected stripe.
        t.idx = fastrand() & c.mask
    }
    // Return ptoken back to the pool, so that another goroutine
    // running on the same thread can use it.
    ptokenPool.Put(t)
}

Here, in the Add method, we using the ptoken structs to hold (not-so) thread-local state. Once we obtain a ptoken, we try to change the corresponding stripe in a CAS-based loop. This allows the goroutines to self-organize: they detect contention via a failed CAS operation and then change the stripe. The goal is to avoid contention due to unlucky thread-to-stripe distribution.

You may ask if piggybacking on a sync.Pool's implementation detail is worth hassle. My answer would be "no, unless you really know what you're doing". Say, single-threaded performance of a primitive atomic int64 would be better. There is also an overhead in the Value() method since it needs to read values from all stripes. So, this trick is certainly from the "don't try that at home" category. But if you aim for scalability of your write operations, it's certainly worth it:

$ go test -benchmem -run=^$ -bench "Counter|Atomic"
goos: linux
goarch: amd64
pkg: github.com/puzpuzpuz/xsync/v2
cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
BenchmarkCounter-8           409773502             2.909 ns/op           0 B/op           0 allocs/op
BenchmarkAtomicInt64-8       92472007            14.09 ns/op           0 B/op           0 allocs/op
PASS
ok      github.com/puzpuzpuz/xsync/v2    3.024s

If you run the same benchmark on a machine with more cores, the int64's result would only get worse due to contention.

Both Map and MapOf, concurrent hash maps from xsync library, use a variation of a striped counter internally to track the current map size. Naturally, they do a counter increment or decrement on each write operation, but read the counter value rarely when a resize happens.

One more example of application of this trick is RBMutex, a reader biased reader/writer mutual exclusion lock that implements BRAVO algorithm. I'm leaving learning the internals of this one to the curious reader.

As promised, the post is a short one, so that's it for today. Have fun coding and see you next time.

type point struct {
    x int
    y int
}
// create a MapOf for with keys and values of point type
m := NewTypedMapOf[point, point](func(p point) uint64 {
    // hash function to be used by the map
    return uint64(31*p.x + p.y)
})
// load the existing value or compute it lazily, if it's absent
v, loaded := m.LoadOrCompute(point{42, 42}, func() int {
    return point{0, 0}
})

The above wasn't possible until xsync v1.5.0. That's because the generic version of the concurrent map available in the library supported only string keys. But now you can use any comparable type as a key, so xsync.MapOf became a real, more scalable alternative to the good old sync.Map. Today we're going to discuss the recent changes in the data structure that allowed using arbitrary key types and also do some (as always, non-scientific) microbenchmarking.

The challenges

The original intent behind xsync library was to provide faster alternatives for the built-in concurrent data structures available in the Golang standard library. Most of the alternatives, like RBMutex or MPMCQueue, aren't suitable for general purpose, i.e. they're tailored for niche use cases.

But xsync.Map is different as it was aimed to replace sync.Map in the same scenarios and beyond. It's based on a (noticeably) modified version of Cache-Line Hash Table (CLHT). Read-only operations, such as Load or Range, are obstruction-free while write operations and rehashing use fine-grained locking (lock sharding). Refer to this blog post to learn more on the algorithm. The only significant limitation was in keys limited to the string type.

The original version of the map was non-generic, so you had to deal with unpleasant interface{} rituals in your code. When Go got a stable version of generics, Viacheslav Poturaev contributed a generics-friendly version of the map, xsync.MapOf. This was a nice step forward, but it still supported string keys only.

A few month ago, the library repo received a pull request from Rob Mason. The goal was to allow arbitrary comparable types for keys while maintaining backwards compatibility. Since Golang doesn't expose the built-in hash functions except for what's in the hash/maphash package, the hash function has to be provided by the user which isn't a big deal. The only problem was the layout of the data structure.

Each bucket in the underlying hash table had the following structure:

Here we have 128 bytes (two cache-lines on most modern CPUs) holding a sync.Mutex, 7 key/value pairs, and a uint64 with cached hash codes for the keys present in the bucket. The mutex is being used while writing to the bucket. The key/values pairs hold pointers to the map entries. Finally, the hash code cache holds one most significant byte per each key’s hash code.

So, each bucket was able to hold up to 7 key/value pairs and, in case if the bucket was fully on an insert, a rehashing would have to be made. Such design is sufficient if you have a high quality hash function, such as the built-in one for strings. But if the function is user-provided chances of having 7+ perfect hash code collisions in the same map increase dramatically.

To fix that, mutex and hashes fields were merged into a single uint64-based data structure with the following layout:

| key 0's top hash | ... | key 7's top hash | bitmap for keys |
|      1 byte      | ... |      1 byte      |     1 byte      |

Here, we have 8 most significant bits (MSBs) of the hash code for each key stored in the bucket. They're used to avoid many expensive hash code computations on each look up. Before calculating the hash code for a key/value pair, we compare the MSBs with the MSBs of the hash code of the user-provided key and, if they don't match, we move on to the next pair in the bucket. Next, the least significant bit in the "bitmap for keys" byte is used to implement a mutex (a TTAS spinlock, to be more precise).

Merging the mutex and the hash code MSBs together saved gorgeous 8 bytes of memory in each bucket. The saved bytes hold pointers to the next bucket in the chain, so as in the very first xsync version, hash table buckets are now organized in unrolled linked lists. Hence, in face of perfect hash code collisions, the linked list trivially growths until the map load factor is met.

Needless to say that once all of the above was done, supporting any comparable key type finally became possible.

At this point you may be asking yourself if it's using a non-standard map is worth it. Let's see.

The promised benchmarks

The following results were obtained on i7-1185G7, Ubuntu 22.04 x86-64 and Go 1.19.2. The benchmark source code itself can be found in the xsync repo.

We start with a benchmark that uses a pre-warmed map with 1,000 key/value pairs. The keys are strings while the values are ints. The benchmark uses all 8 goroutines to load all 8 HT cores available on the machine. Each goroutine selects a key randomly and executes either Load, Store or Delete operation on it.

Here, the x axis stands for the percentage of Load operation while the two remaining operations have equal chances to be called, e.g. for the 90% value we have 90% of Load calls, 5% of Stores and 5% of Deletes. The y axis stands for the average time in nanoseconds spend on an operation.

Things get even more interesting with 1M key/value pairs.

As usual, take the above results with a grain of salt. There might be some scenarios when the standard map is faster, so make sure to test things close to what your application does. Nevertheless, due to the design the standard map is quite limited in terms of scalability in the presence of even a small fraction of concurrent writes. After all, a data structure guarded with a single lock will scale worse than the one with sharded locks. When it comes to reads which are the strongest point of sync.Map, both data structures are on par.

Lessons learned

Hopefully, this post has convinced you to try xsync.MapOf in action and maybe to contribute to xsync. This story is another evidence of the power of open source communities. Without so many contributors, xsync.Map would be still limited in its capabilities. I'm pretty sure that it's not the end of the story and the library would continue evolving in future. Have fun coding and see you next time.

To narrow down the topic, we're going to use Java and try to cover the SPSC queue we built recently with a minimal set of tests. While the observed tests are minimal, you should be able to write tests for your own concurrent code after reading this blog post. As for the programming language, everything we talk of should be applicable to any language with threads or green threads, so C/C++, Rust, Golang, Zig and many others apply.

The interface of our queue is very simple and consists of two methods:

public class SpscBoundedQueue<E> {

    // Some boring stuff, like fields and constructor.

    /**
     * Publishes an item to the tail of the queue, if it's not full.
     */
    public boolean offer(E e) { /* some code goes here */ }

    /**
     * Removes and returns the head of the queue, if it's not empty.
     */
    public E poll() { /* some code goes here */ }
}

So, where to start when testing it?

Where to start?

The best thing to do as the first step is to write good old single-threaded tests. Many aspects of your code can (and should) be covered with such tests. Those are input validation, boundary checks, basic invariants of your data structure, methods that aren't thread-safe and, hence, will be always called from a single thread - all of these should be covered with "cheap" (in terms of the execution time) tests. Keep in mind that the aforementioned list is not complete. You should always try to cover as much as possible with single-threaded tests.

In our case, we should cover the following things:

1. Input validation - our original code was minimal, so it lacked things like positive size validation in the constructor. Such validation is a perfect candidate for a single-threaded test.
2. Boundary checks - our queue has a limited size, so we expect offer() to return false when the queue is full, as well as poll() to return a null when the queue is empty.
3. Basic invariants - we should test the "First in, first out" (FIFO) property of our data structure.
4. Non thread-safe methods - again, we omitted many other methods that are handy, such as clear() method. Due to the queue design, those methods have to be called from a single thread in absence of any other queue mutation calls. Single-threaded tests are to the rescue.

Here is a test that illustrates items 2 and 3 from the above list:

@Test
public void testSerial() {
    SpscBoundedQueue<Integer> queue = new SpscBoundedQueue<>(10);

    Assert.assertNull(queue.poll());

    for (int i = 0; i < 10; i++) {
        Assert.assertTrue(queue.offer(i));
    }
    Assert.assertFalse(queue.offer(42));

    for (int i = 0; i < 10; i++) {
        Assert.assertEquals((Integer) i, queue.poll());
    }
    Assert.assertNull(queue.poll());
}

It's a bit dense and could be split into multiple, more focused tests, but it's not a big deal considering that it's an illustration of the concept. The test verifies both boundary checks, as well as the FIFO property.

Enough silly serial tests, let's run things in parallel!

How to break things?

Our main goal is to find any thread-safety violations. But what does it mean in practice? Such violations may be very infrequent and hard to reproduce. Sometimes race conditions, data races and other unpleasant things may even remain unnoticed until you hit a certain edge case. The sad truth is that testing concurrent code is hard and you can never be sure that your test suite is good enough. But that means that you should do your best at writing concurrent tests to eliminate most, if not all, thread-safety bugs.

Each concurrent test scenario has to be thought separately. It should reproduce a use case and involve a set of related methods of your data structure(s). In our example, things are simple: we need to test the offer() and poll() methods running on two different threads (remember, we deal with a Single Producer Single Consumer queue). But is it enough to call these methods like crazy from separate threads? Not really. Just like with single-threaded tests, we have to think of the invariants we have in the thread-safe part of the code.

To keep things practical, let's start with the skeleton of the test:

@Test
public void testHammer() throws InterruptedException {
    final int iterations = 1_000_000;

    // Prepare the data structure.
    SpscBoundedQueue<Integer> queue = new SpscBoundedQueue<>(10);

    // Prepare helper data structures (test infra).
    CyclicBarrier barrier = new CyclicBarrier(2);
    CountDownLatch latch = new CountDownLatch(2);
    AtomicInteger anomalies = new AtomicInteger();

    // Prepare and start the threads.
    ConsumerThread consumer = new ConsumerThread(queue, barrier, latch, anomalies, iterations);
    consumer.start();
    ProducerThread producer = new ProducerThread(queue, barrier, latch, anomalies, iterations);
    producer.start();

    // Wait for the threads to finish.
    latch.await();

    // Verify that there were no thread-safety violations.
    Assert.assertEquals(0, anomalies.get());
}

The above code is quite straightforward. The test runs two threads and involves a queue, as well as a number of helper synchronization primitives. Once the threads are done, it checks the anomalies counter to verify that there were no thread-safety violations. As the test name suggests, it's a hammer style test, i.e. it aims to "bash" the queue from multiple threads until it breaks (or not). Such tests sometimes called stress tests for concurrent code.

Now, let's see what our threads actually do. We start with the producer thread:

private static class ProducerThread extends Thread {

    // Boring stuff such as fields and constructor goes here...

    @Override
    public void run() {
        try {
            // Await for the consumer thread, so we start simultaneously.
            barrier.await();
            // Start publishing incrementing numbers to the queue.
            for (int i = 0; i < iterations; i++) {
                while (!queue.offer(i)) {
                    // Yes, we want to busy spin.
                }
            }
        } catch (Exception e) {
            // Any exception we get when producing is an anomaly.
            e.printStackTrace();
            anomalies.incrementAndGet();
        } finally {
            // Notify the main thread that we're done.
            latch.countDown();
        }
    }
}

Producer's code is simple and illustrative. Notice that we're publishing incrementing numbers to the queue. As we're going to see in the consumer's code, that's to be able to verify our main invariant - the FIFO property. One more important thing here is that we don't have any kind of back-off calls in the while loop. Instead, we prefer to busy spin. That's because calls like Thread#sleep() or LockSupport.parkNanos() or anything similar involve synchronization that might fix your otherwise broken code. Also, if you need to emulate some local work as the back-off or on successful operation, prefer using Blackhole#consumeCpu() from JMH or similar methods of your choice. Finally, due to the same consideration, it's definitely a bad idea to call System.out.println() or log anything is the main loop.

The consumer thread's code is also pretty simple:

private static class ConsumerThread extends Thread {

    // Boring stuff such as fields and constructor goes here...

    @Override
    public void run() {
        try {
            barrier.await();
            // Consume all items from the queue.
            int prev = -1;
            while (prev != iterations - 1) {
                Integer element = queue.poll();
                if (element == null) {
                    // Again, we busy spin.
                    continue;
                }
                // Check that we received the incremented number.
                if (element != prev + 1) {
                    anomalies.incrementAndGet();
                }
                prev = element;
            }
            // We expect the queue to be empty now.
            if (queue.poll() != null) {
                anomalies.incrementAndGet();
            }
        } catch (Exception e) {
            e.printStackTrace();
            anomalies.incrementAndGet();
        } finally {
            latch.countDown();
        }
    }
}

The above code completes the picture: our test aims to verify the FIFO property of the queue and nothing more than that.

Variations of the concurrent tests are important. If we would be testing a MPMC queue, it would be a good idea to have multiple tests with different number of producer and consumer threads: single producer - single consumer, single producer - multiple consumers, multiple producers - single consumer, multiple producers - multiple consumers. If we have some kind of local work emulation in the tests, it would be nice to test it with different CPU time too. Same applies to data structure capacity and any other things that may affect the flow of your code. Thread-safety violations are a question of unlucky (or lucky, if you want to find bugs) ordering and visibility, so the more variations of the scenario you run, the higher chances to find a violation.

The complete test source code may be found here. If you're proficient in Golang and fancy to see a more complex application of the above principles, see tests of xsync library. The library consists of a number of concurrent data structures that are certainly more complex than our SPSC queue.

How to run the tests?

Before we wrap up, let's discuss a few tips to squeeze everything from your multi-threaded tests.

First of all, it is a good habit to run the newly written concurrent tests on your dev machine for a few minutes. This might show failures early, without involving many CI runs.

Next, if your code runs on different CPU architectures, make sure to run tests on those. For instance, ARM CPUs have a weaker hardware memory model when compared with x86 ones.

Some language ecosystems have race detector tools, like ThreadSanitizer or Golang's Data Race Detector. If applicable, make sure to configure your CI to run the tests with enabled race detector. It's also worth mentioning jcstress and Lincheck frameworks available in JVM ecosystem. Unlike the aforementioned race detectors, these frameworks require writing dedicated tests, so, in case of concurrent data structure testing, they can be seen as an alternative to the hand-written tests we're discussing today.

Finally, if some of your concurrent tests appear to be flaky, i.e. infrequently fail due to an unknown reason, that may be an indication of an actual bug. Make sure to do your best to reproduce the failure, analyze it and fix the cause.

Let's recap?

Writing thread-safe concurrent code is hard. Writing sufficient tests for such code may be even harder. Here is the summary of what we discussed today:

• Always try to cover as much as possible with single-threaded tests.
• Write your concurrent tests to stress your code and verify a set of invariants.
• Avoid calls that involve additional synchronization in the main loops of your tests.
• Variations of the concurrent tests are important.
• Test on various CPU architectures (wink-wink ARM).
• If applicable, configure your CI server to run the concurrent tests with a race detector.
• Flaky tests are your friends. Always do your best to reproduce and analyze them.

I hope you've learned something new today. Good luck with your concurrent tests and see you next time.

We start with the skeleton of the data structure, i.e. its fields and interface:

public class SpscBoundedQueue<E> {

    private final Object[] data;
    private final PaddedAtomicInteger producerIdx = new PaddedAtomicInteger();
    private final PaddedAtomicInteger producerCachedIdx = new PaddedAtomicInteger();
    private final PaddedAtomicInteger consumerIdx = new PaddedAtomicInteger();
    private final PaddedAtomicInteger consumerCachedIdx = new PaddedAtomicInteger();

    public SpscBoundedQueue(int size) {
        this.data = new Object[size + 1];
    }

    public boolean offer(E e) {
        // The code will follow...
    }

    public E poll() {
        // The code will follow...
    }

    static class PaddedAtomicInteger extends AtomicInteger {
        @SuppressWarnings("unused")
        private int i1, i2, i3, i4, i5, i6, i7, i8,
                i9, i10, i11, i12, i13, i14, i15;
    }
}

Here, we have an array of queue elements plus a number of index fields where consumer and producer each get a pair of PaddedAtomicInteger. The PaddedAtomicInteger is basically the standard j.u.c.a.AtomicInteger class with some padding added to prevent false sharing. Alternatively, we could keep the memory layout flat with all indexes declared as primitive fields right in the SpscBoundedQueue class, but this would make the code much less readable.

You may also notice that only offer() and poll() methods are implemented. Again, that's to keep the code compact and readable. Adding other useful methods, like the batch flavor ones, is simple enough and left as an exercise for curious readers.

The array of queue items is used as a ring buffer of arbitrary size, i.e. there is no power of two restriction for the size like in some ring buffer implementations. The producerIdx and consumerIdx fields are used to synchronize producer's and consumer's accesses to the array. Both producer and consumer check each other's index to understand if they can insert or read the next item and, if the check succeeds, perform the action and update their own index. Two other fields are used to cache the index seen during the latest check. We'll discuss why such caching improves the end performance in a moment.

Let's see how it all works for the producer:

public boolean offer(E e) {
    // Read producer's own index.
    final int idx = producerIdx.getOpaque();
    int nextIdx = idx + 1;
    if (nextIdx == data.length) {
        nextIdx = 0;
    }
    // Read the last seen consumer's index.
    int cachedIdx = consumerCachedIdx.getPlain();
    if (nextIdx == cachedIdx) {
        // If we have reached the known index, we need to read the current value.
        cachedIdx = consumerIdx.getAcquire();
        // Make sure to update the cached value.
        consumerCachedIdx.setPlain(cachedIdx);
        if (nextIdx == cachedIdx) {
            // The queue is full.
            return false;
        }
    }
    // There is an empty slot, so we can insert the item.
    data[idx] = e;
    // Make sure to update our own index.
    // We use release semantics while the consumer has an acquire edge.
    producerIdx.setRelease(nextIdx);
    return true;
}

The above code uses acquire/release semantics to keep the emitted instructions as lightweight as possible from the memory barriers perspective. Other than that, the code does pretty much as what we discussed before.

As it was already mentioned, the manipulations with the consumerCachedIdx field are important for the end performance. All reads and writes on this field are thread-local, i.e. only the producer thread accesses this field, so we don't need to use costly atomic operations. This reduces cache coherency traffic dramatically and lets the CPU core on its own non-shared data in those cases when there multiple empty slots are available in the queue.

Consumer's part of the picture may be seen in the full source code available here.

Finally, we're going compare our queue with the good old j.u.c.ArrayBlockingQueue and a couple of SPSC queue implementations from JCTools library. If you're not familiar with JCTools and never used it, I advice you to put it on your radar.

The benchmark we'll be running is available here. When run, it starts a couple of threads to play a ping-pong game. Each operation, a.k.a. a ping-pong round, assumes sending/receiving a single item over the SPSC queue combined with a bit of work done for each successful attempt.

Here is a reduced JMH benchmark output on my laptop running Ubuntu 20.04 and OpenJDK 17.0.4 64-bit:

Benchmark                                            (type)   Mode  Cnt          Score          Error  Units
SpscQueueBenchmark.group                         SPSC_QUEUE  thrpt    3  107503612.612 ± 16230253.288  ops/s
SpscQueueBenchmark.group               ARRAY_BLOCKING_QUEUE  thrpt    3    7158948.722 ±  8635350.468  ops/s
SpscQueueBenchmark.group                      JCTOOLS_QUEUE  thrpt    3  120533694.168 ±  4686758.722  ops/s
SpscQueueBenchmark.group               JCTOOLS_ATOMIC_QUEUE  thrpt    3  101704017.278 ± 18252611.281  ops/s

As expected, JCTools' queues and our own one are significantly faster than the ArrayBlockingQueue queue. Also, surprisingly, our SPSC queue keeps on par with the JCTools' queues which is not something I was expecting, to be honest. Does it mean that you should go for an in-house implementation instead of JCTools? Not really. If you can afford yourself 3rd-party dependencies, go for JCTools. JCTools' data structures are certainly more efficient, as well as much better tested and benchmarked than our toy queue. So, you'd have to spend quite some time reaching the same level of stability for a DIY queue.

Needless to say that this algorithm is not something new. You may see it in this great blog post by Erik Rigtorp, as well as recognize it in the SPSequence and SCSequence classes in QuestDB's source code. Yet, I hope that this data structure would be a nice addition to your engineering toolkit. See you next time.