130 lines
4.3 KiB
Markdown
130 lines
4.3 KiB
Markdown
# Dev Docs
|
|
|
|
I've scaffolded but it I feel it makes more sense to just have a one file impl first in some .cpp file
|
|
and take it from there. So off we go to top level main.cpp
|
|
|
|
There is an original paper but I'm reading off from wiki for now https://en.wikipedia.org/wiki/Bloom_filter
|
|
|
|
## Making k hashes
|
|
|
|
This was the first problem. Just like open addressing hash tables do it.
|
|
|
|
Idea is you take 2 hash functions and then use them as
|
|
```
|
|
h_i(x) = ( h1(x) + i.h2(x) ) mod m
|
|
```
|
|
Then if I say that `gcd(h2(x),m) = 1` then thi will loop over all residues and I have a uniform spread over the range m.
|
|
|
|
I can't do h(x) + i as that's a bad idea since it'll be consecutive same with i . h(x) as there will be just equally spaced strides.
|
|
|
|
This double hashed func almost behaves like a random number generator.
|
|
|
|
## std::hash
|
|
|
|
you make a hasher object and them immediately call it
|
|
|
|
## Designing the api
|
|
|
|
Now that I have a basic version ready I need to design some sort
|
|
of an api going forward - for the header. For now I'm sticking
|
|
to the traditional impl.
|
|
|
|
There was a google's impl
|
|
https://github.com/google/guava/blob/master/guava/src/com/google/common/hash/BloomFilter.java
|
|
|
|
and also another cpp lib that's around 10y old
|
|
https://www.partow.net/programming/bloomfilter/index.html
|
|
|
|
Ok going step by step, first the hash function
|
|
|
|
### Hash func
|
|
|
|
The guava one uses MurmurHash3 ( first I've heard of it )
|
|
There are two sorts of hashes - cryptographic and non crypto
|
|
|
|
The diff is explained here: https://security.stackexchange.com/questions/11839/what-is-the-difference-between-a-hash-function-and-a-cryptographic-hash-function
|
|
|
|
The point being - cryptographic ones ensure security against some
|
|
adversary and guarantee some stuff like
|
|
1. preimage resistance - given hash find message
|
|
2. second preimage resitance, given just x and h(x), find another x'
|
|
with same hash
|
|
3. collision resistance - free to choose any pair but need to match
|
|
hashes
|
|
|
|
When you loosen these constraints - faster hashes are possible.
|
|
|
|
In any case - I googled and seems like xxhash is the fastest now
|
|
so I'll use that.
|
|
|
|
From there I downloaded the simplest impl
|
|
https://create.stephan-brumme.com/xxhash/ the v2 version for 64.
|
|
It's a single header so that should be good enough.
|
|
|
|
### Back to actual api?
|
|
|
|
I liked Google's api for it - especially the naming and all.
|
|
I won't b supporting any set operations at all.
|
|
|
|
mightContain(x)
|
|
expectedFalsePositiveProbab
|
|
put
|
|
approxElemCount() - estimates from fraction of set bits
|
|
|
|
Constructors - there should be these atleast
|
|
(expectedPuts, numHashes, falsePositiveityRate)
|
|
(bitSize, numHashes)
|
|
|
|
### The headers
|
|
|
|
using vector<bool> as it's bit packed internally and allows for
|
|
runtime size declaration
|
|
|
|
I ended up writing the entire impl in the header file and then split it.
|
|
Earlier I was going for a templated thing but that needed a lot of additional handling
|
|
due to how something with internal pointers would manage data ( say std::string ).
|
|
Now it's similar to xxhash where you pass in a void pointer and a length to it.
|
|
|
|
## Fixing clangd
|
|
|
|
Post the split, I needed to fix clangd as it did not register variables across files as I did not
|
|
have a valid cmake. So added that and ran `cmake -S . -B build` and all worked fine.
|
|
|
|
---
|
|
|
|
# The Math
|
|
|
|
Better to quickly go over the math as well
|
|
n - no of keys added ( expected ), m -> filter size in bits, k -> no of hash functions
|
|
|
|
Now there are m bits, and consider one insertion, and just one hash function
|
|
Proba that it's still 0 is (1-1/m) as 1/m is the chance that bit is occupied and hence 1.
|
|
Now there are n keys and k such hash functions. This takes you to
|
|
|
|
(1-1/m) ^ nk
|
|
|
|
and then P(some particular bit is set) is 1-(1-1/m)^nk
|
|
|
|
For a test k, it's false positive if all of the k bits are set which gives you
|
|
|
|
(1 - (1-1/m)^nk ) ^ k
|
|
|
|
For large m, (1-1/x)^x ~= 1/e
|
|
so the inner part becomes 1 - e^(-kn/m)
|
|
|
|
This last step is incorrect as the assuming some bitstates to be independent
|
|
P(bit A is 1 and bit B is 1) = P(bit A is 1) * P(bit B is 1)
|
|
|
|
Now this looks correct except that it is not
|
|
https://www.math.umd.edu/~immortal/CMSC420/notes/bloomfilters.pdf
|
|
|
|
But that's besides the point for now.
|
|
|
|
## My error probab function?
|
|
|
|
Based off of Guava, it uses the actual free bits at runtime to give a "live" estimate
|
|
bits filled fraction is P(any arbitrary bit is 1)
|
|
False positive = all of those bits end up being 1 for some hash = P^k
|
|
|
|
This has the same independence issue in assumption though.
|