- Generated by
1.8.11
|
proxygen
|
F14 is a 14-way probing hash table that resolves collisions by double hashing. Up to 14 keys are stored in a chunk at a single hash table position. Vector instructions (SSE2 on x86_64, NEON on aarch64) are used to filter within a chunk; intra-chunk search takes only a handful of instructions. F14 refers to the fact that the algorithm F**ilters up to **14 keys at a time. This strategy allows the hash table to be operated at a high maximum load factor (12/14) while still keeping probe chains very short.
F14 provides compelling replacements for most of the hash tables we use in production at Facebook. Switching to it can improve memory efficiency and performance at the same time. The hash table implementations widely deployed in C++ at Facebook exist along a spectrum of space/time tradeoffs. The fastest is the least memory efficient, and the most memory efficient (google::sparse_hash_map) is much slower than the rest. F14 moves the curve, simultaneously improving memory efficiency and performance when compared to most of the existing algorithms.
The core hash table implementation has a pluggable storage strategy, with three policies provided:
F14NodeMap stores values indirectly, calling malloc on each insert like std::unordered_map. This implementation is the most memory efficient for medium and large keys. It provides the same iterator and reference stability guarantees as the standard map while being faster and more memory efficient, so you can substitute F14NodeMap for std::unordered_map safely in production code. F14's filtering substantially reduces indirection (and cache misses) when compared to std::unordered_map.
F14ValueMap stores values inline, like google::dense_hash_map. Inline storage is the most memory efficient for small values, but for medium and large values it wastes space. Because it can tolerate a much higher load factor, F14ValueMap is almost twice as memory efficient as dense_hash_map while also faster for most workloads.
F14VectorMap keeps values packed in a contiguous array. The main hash array stores 32-bit indexes into the value vector. Compared to the existing internal implementations that use a similar strategy, F14 is slower for simple keys and small or medium-sized tables (because of the cost of bit mixing), faster for complex keys and large tables, and saves about 16 bytes per entry on average.
We also provide:
F14FastMap inherits from either F14ValueMap or F14VectorMap depending on entry size. When the key and mapped_type are less than 24 bytes, it inherits from F14ValueMap. For medium and large entries, it inherits from F14VectorMap. This strategy provides the best performance, while also providing better memory efficiency than dense_hash_map or the other hash tables in use at Facebook that don't individually allocate nodes.
F14FastMap is a good default choice. If you care more about memory efficiency than performance, F14NodeMap is better for medium and large entries. F14NodeMap is the only F14 variant that doesn't move its elements, so in the rare case that you need reference stability you should use it.
In some cases it makes sense to define hash and key equality across types. For example, StringPiece's hash and equality are capable of accepting std::string (because std::string is implicitly convertible to StringPiece). If you mark the hash functor and key equality functor as transparent, then F14 will allow you to search the table directly using any of the accepted key types without converting the key.
For example, using H = folly::transparent<folly::hasher<folly::StringPiece>> and E = folly::transparent<std::equal_to<folly::StringPiece>>, an F14FastSet<std::string, H, E> will allow you to use a StringPiece key without the need to construct a std::string.
Heterogeneous lookup and erase works for any key types that can be passed to operator() on the hasher and key_equal functors. For operations such as operator[] that might insert there is an additional constraint, which is that the passed-in key must be explicitly convertible to the table's key_type. F14 maps understand all possible forms that can be used to construct the underlying std::pair<key_type const, value_type), so heterogeneous keys can be used even with insert and emplace.
Assuming that you have a magic wand that lets you search all of the keys in a chunk in a single step (our wand is called _mm_cmpeq_epi8), then using chunks fundamentally improves the load factor/collision tradeoff. The cost is proportional only to the number of chunks visited to find the key.
It's kind of like the birthday paradox in reverse. In a room with 23 people there is a 50/50 chance that two of them have the same birthday (overflowing a chunk with capacity 1), but the chance that 8 of them were born in the same week (overflowing a chunk with capacity 7) is very small. Even though the chance of any two people being born in the same week is higher (1/52 instead of 1/365), the larger number of coincidences required means that the final probability is much lower (less than 1 in a million). It would require 160 people to reach a 50/50 chance that 8 of them were born in the same week.
Chaining to a new chunk on collision is not very memory efficient, because the new chunk is almost certain to be under-filled. We tried chaining to individual entries, but that bloated the lookup code and can't match the performance of a probing strategy.
At our max load factor of 12/14, the expected probe length when searching for an existing key (find hit) is 1.04, and fewer than 1% of keys are not found in one of the first 3 chunks. When searching for a key that is not in the map (find miss) the expected probe length at max load factor is 1.275 and the P99 probe length is 4.
Hash tables with a complex probing strategy (quadratic or double-hashing) typically use a tombstone on erase, because it is very difficult to find the keys that might have been displaced by a full bucket (i.e., chunk in F14). If the probing strategy allows only a small number of potential destinations for a displaced key (linear probing, Robin Hood hashing, or Cuckoo hashing), it is also an option to find a displaced key, relocate it, and then recursively repair the new hole.
Tombstones must be eventually reclaimed to deal with workloads that continuously insert and erase. google::dense_hash_map eventually triggers a rehash in this case, for example. Unfortunately, to avoid quadratic behavior this rehash may have to halve the max load factor of the table, resulting in a huge decrease in memory efficiency.
Although most probing algorithms just keep probing until they find an empty slot, probe lengths can be substantially reduced if you track whether a bucket has actually rejected a key. This "overflow bit" is set when an attempt is made to place a key into the bucket but the bucket was full. (An especially unlucky key might have to try several buckets, setting the overflow bit in each.) Amble and Knuth describe an overflow bit in the "Further development" section of "Ordered hash tables" (https://academic.oup.com/comjnl/article/17/2/135/525363).
The overflow bit subsumes the role of a tombstone, since a tombstone's only effect is to cause a probe search to continue. Unlike a tombstone, however, the overflow bit is a property of the keys that were displaced rather than the key that was erased. It's only a small step to turn this into a counter that records the number of displaced keys, and that can be decremented on erase. Overflow counts give us both an earlier exit from probing and the effect of a reference-counted tombstone. They automatically clean themselves up in a steady-state insert and erase workload, giving us the upsides of double-hashing without the normal downsides of tombstones.
F14 computes a secondary hash value for each key, which we call the key's tag. Tags are 1 byte: 7 bits of entropy with the top bit set. The 14 tags are joined with 2 additional bytes of metadata to form a 16-byte aligned __m128i at the beginning of the chunk. When we're looking for a key we can compare the needle's tag to all 14 tags in a chunk in parallel. The result of the comparison is a bitmask that identifies only slots in a chunk that might have a non-empty matching key. Failing searches are unlikely to perform any key comparisons, successful searches are likely to perform exactly 1 comparison, and all of the resulting branches are pretty predictable.
The vector search is coded using SIMD intrinsics, SSE2 on x86_64 and NEON on aarch64. These instructions are a non-optional part of those platforms (unlike later SIMD instruction sets like AVX2 or SVE), so no special compilation flags are required. The exact vector operations performed differs between x86_64 and aarch64 because aarch64 lacks a movemask instruction, but the F14 algorithm is the same.
The F14 algorithm works well for large tables, because the tags can fit in cache even when the keys and values can't. Tiny hash tables are by far the most numerous, however, so it's important that we minimize the footprint when the table is empty or has only 1 or 2 elements. Conveniently, tags cause keys to be densely packed into the bottom of a chunk and filter all memory accesses to the portions of a chunk that are not used. That means that we can also support capacities that are a fraction of 1 chunk with no change to any of the search and insertion algorithms. The only change required is in the check to see if a rehash is required. F14's first three capacities all use one chunk and one 16-byte metadata vector, but allocate space for 2, 6, and then 12 keys.
No. F14 does provide full support for stateful allocators, fancy pointers, and as many parts of the C++ standard for unordered associative containers as it can, but it is not fully standards-compliant.
We don't know of a way to efficiently implement the full bucket API in a table that uses double-hashed probing, in particular size_type bucket(key_type const&). This function must compute the bucket index for any key, even before it is inserted into the table. That means that a local_iterator range can't partition the key space by the chunk that terminated probing during insert; the only partition choice with reasonable locality would be the first-choice chunk. The probe sequence for a key in double-hashing depends on the key, not the first-choice chunk, however, so it is infeasible to search for all of the displaced keys given only their first-choice location. We're unwilling to use an inferior probing strategy or dedicate space to the required metadata just to support the full bucket API. Implementing the rest of the bucket API, such as local_iterator begin(size_type), would not be difficult.
F14 does not allow max_load_factor to be adjusted. Probing tables can't support load factors greater than 1, so the standards-required ability to temporarily disable rehashing by temporarily setting a very high max load factor just isn't possible. We have also measured that there is no performance advantage to forcing a low load factor, so it's better just to omit the field and save space in every F14 instance. This is part of the way we get empty maps down to 32 bytes. The void max_load_factor(float) method is still present, but does nothing. We use the default max_load_factor of 1.0f all of the time, adjusting the value returned from size_type bucket_count() so that the externally-visible load factor reaches 1 just as the actual internal load factor reaches our threshold of 12/14.
The standard requires that a hash table be iterable in O(size()) time regardless of its load factor (rather than O(bucket_count()). That means if you insert 1 million keys then erase all but 10, iteration should be O(10). For std::unordered_map the cost of supporting this scenario is an extra level of indirection in every read and every write, which is part of why we can improve substantially on its performance. Low load factor iteration occurs in practice when erasing keys during iteration (for example by repeatedly calling map.erase(map.begin())), so we provide the weaker guarantee that iteration is O(size()) after erasing any prefix of the iteration order. F14VectorMap doesn't have this problem.
The standard requires that clear() be O(size()), which has the practical effect of prohibiting a change to bucket_count. F14 deallocates all memory during clear() if it has space for more than 100 keys, to avoid leaving a large table that will be expensive to iterate (see the previous paragraph). google::dense_hash_map works around this tradeoff by providing both clear() and clear_no_resize(); we could do something similar.
As stated above, F14NodeMap and F14NodeSet are the only F14 variants that provides reference stability. When running under ASAN the other storage policies will probabilistically perform extra rehashes, which makes it likely that reference stability problems will be found by the address sanitizer.
An additional subtlety for hash tables that don't provide reference stability is whether they rehash before evaluating the arguments passed to insert(). F14 tables may rehash before evaluating the arguments to a method that causes an insertion, so it's not safe to write something like map.insert(k2, map[k1]) with F14FastMap, F14ValueMap, or F14VectorMap. This behavior matches google::dense_hash_map and the excellent absl::flat_hash_map.
F14NodeMap does not currently support the C++17 node API, but it could be trivially added.
1.8.11