bloom-filter.cpp/DEV.md

Dev Docs

I've scaffolded but it I feel it makes more sense to just have a one file impl first in some .cpp file and take it from there. So off we go to top level main.cpp

There is an original paper but I'm reading off from wiki for now https://en.wikipedia.org/wiki/Bloom_filter

Making k hashes

This was the first problem. Just like open addressing hash tables do it.

Idea is you take 2 hash functions and then use them as

h_i(x) = ( h1(x) + i.h2(x) ) mod m

Then if I say that gcd(h2(x),m) = 1 then thi will loop over all residues and I have a uniform spread over the range m.

I can't do h(x) + i as that's a bad idea since it'll be consecutive same with i . h(x) as there will be just equally spaced strides.

This double hashed func almost behaves like a random number generator.

std::hash

you make a hasher object and them immediately call it

Designing the api

Now that I have a basic version ready I need to design some sort of an api going forward - for the header. For now I'm sticking to the traditional impl.

There was a google's impl https://github.com/google/guava/blob/master/guava/src/com/google/common/hash/BloomFilter.java

and also another cpp lib that's around 10y old https://www.partow.net/programming/bloomfilter/index.html

Ok going step by step, first the hash function

Hash func

The guava one uses MurmurHash3 ( first I've heard of it ) There are two sorts of hashes - cryptographic and non crypto

The diff is explained here: https://security.stackexchange.com/questions/11839/what-is-the-difference-between-a-hash-function-and-a-cryptographic-hash-function

The point being - cryptographic ones ensure security against some adversary and guarantee some stuff like

1. preimage resistance - given hash find message
2. second preimage resitance, given just x and h(x), find another x' with same hash
3. collision resistance - free to choose any pair but need to match hashes

When you loosen these constraints - faster hashes are possible.

In any case - I googled and seems like xxhash is the fastest now so I'll use that.

From there I downloaded the simplest impl https://create.stephan-brumme.com/xxhash/ the v2 version for 64. It's a single header so that should be good enough.

Back to actual api?

I liked Google's api for it - especially the naming and all. I won't b supporting any set operations at all.

mightContain(x) expectedFalsePositiveProbab put approxElemCount() - estimates from fraction of set bits

Constructors - there should be these atleast (expectedPuts, numHashes, falsePositiveityRate) (bitSize, numHashes)

The headers

using vector as it's bit packed internally and allows for runtime size declaration

I ended up writing the entire impl in the header file and then split it. Earlier I was going for a templated thing but that needed a lot of additional handling due to how something with internal pointers would manage data ( say std::string ). Now it's similar to xxhash where you pass in a void pointer and a length to it.

Fixing clangd

Post the split, I needed to fix clangd as it did not register variables across files as I did not have a valid cmake. So added that and ran cmake -S . -B build and all worked fine.

---

The Math

Better to quickly go over the math as well n - no of keys added ( expected ), m -> filter size in bits, k -> no of hash functions

Now there are m bits, and consider one insertion, and just one hash function Proba that it's still 0 is (1-1/m) as 1/m is the chance that bit is occupied and hence 1. Now there are n keys and k such hash functions. This takes you to

(1-1/m) ^ nk

and then P(some particular bit is set) is 1-(1-1/m)^nk

For a test k, it's false positive if all of the k bits are set which gives you

(1 - (1-1/m)^nk ) ^ k

For large m, (1-1/x)^x ~= 1/e so the inner part becomes 1 - e^(-kn/m)

This last step is incorrect as the assuming some bitstates to be independent P(bit A is 1 and bit B is 1) = P(bit A is 1) * P(bit B is 1)

Now this looks correct except that it is not https://www.math.umd.edu/~immortal/CMSC420/notes/bloomfilters.pdf

But that's besides the point for now.

My error probab function?

Based off of Guava, it uses the actual free bits at runtime to give a "live" estimate bits filled fraction is P(any arbitrary bit is 1) False positive = all of those bits end up being 1 for some hash = P^k

This has the same independence issue in assumption though.

Plotting stuff

Found out about gnuplot which I guess is a bit of an overkill having a binary for the whole thing but fine the plotting itself is easier. In the end the plots came out in line with theoretical which was great.

Links

There are many variations for bloom filters that I'll get to implementing and testing eventually which I hope is soon enough.