91 lines
3.0 KiB
Markdown
91 lines
3.0 KiB
Markdown
# Dev Docs
|
|
|
|
I've scaffolded but it I feel it makes more sense to just have a one file impl first in some .cpp file
|
|
and take it from there. So off we go to top level main.cpp
|
|
|
|
There is an original paper but I'm reading off from wiki for now https://en.wikipedia.org/wiki/Bloom_filter
|
|
|
|
## Making k hashes
|
|
|
|
This was the first problem. Just like open addressing hash tables do it.
|
|
|
|
Idea is you take 2 hash functions and then use them as
|
|
```
|
|
h_i(x) = ( h1(x) + i.h2(x) ) mod m
|
|
```
|
|
Then if I say that `gcd(h2(x),m) = 1` then thi will loop over all residues and I have a uniform spread over the range m.
|
|
|
|
I can't do h(x) + i as that's a bad idea since it'll be consecutive same with i . h(x) as there will be just equally spaced strides.
|
|
|
|
This double hashed func almost behaves like a random number generator.
|
|
|
|
## std::hash
|
|
|
|
you make a hasher object and them immediately call it
|
|
|
|
## Designing the api
|
|
|
|
Now that I have a basic version ready I need to design some sort
|
|
of an api going forward - for the header. For now I'm sticking
|
|
to the traditional impl.
|
|
|
|
There was a google's impl
|
|
https://github.com/google/guava/blob/master/guava/src/com/google/common/hash/BloomFilter.java
|
|
|
|
and also another cpp lib that's around 10y old
|
|
https://www.partow.net/programming/bloomfilter/index.html
|
|
|
|
Ok going step by step, first the hash function
|
|
|
|
### Hash func
|
|
|
|
The guava one uses MurmurHash3 ( first I've heard of it )
|
|
There are two sorts of hashes - cryptographic and non crypto
|
|
|
|
The diff is explained here: https://security.stackexchange.com/questions/11839/what-is-the-difference-between-a-hash-function-and-a-cryptographic-hash-function
|
|
|
|
The point being - cryptographic ones ensure security against some
|
|
adversary and guarantee some stuff like
|
|
1. preimage resistance - given hash find message
|
|
2. second preimage resitance, given just x and h(x), find another x'
|
|
with same hash
|
|
3. collision resistance - free to choose any pair but need to match
|
|
hashes
|
|
|
|
When you loosen these constraints - faster hashes are possible.
|
|
|
|
In any case - I googled and seems like xxhash is the fastest now
|
|
so I'll use that.
|
|
|
|
From there I downloaded the simplest impl
|
|
https://create.stephan-brumme.com/xxhash/ the v2 version for 64.
|
|
It's a single header so that should be good enough.
|
|
|
|
### Back to actual api?
|
|
|
|
I liked Google's api for it - especially the naming and all.
|
|
I won't b supporting any set operations at all.
|
|
|
|
mightContain(x)
|
|
expectedFalsePositiveProbab
|
|
put
|
|
approxElemCount() - estimates from fraction of set bits
|
|
|
|
Constructors - there should be these atleast
|
|
(expectedPuts, numHashes, falsePositiveityRate)
|
|
(bitSize, numHashes)
|
|
|
|
### The headers
|
|
|
|
using vector<bool> as it's bit packed internally and allows for
|
|
runtime size declaration
|
|
|
|
I ended up writing the entire impl in the header file and then split it.
|
|
Earlier I was going for a templated thing but that needed a lot of additional handling
|
|
due to how something with internal pointers would manage data ( say std::string ).
|
|
Now it's similar to xxhash where you pass in a void pointer and a length to it.
|
|
|
|
## Fixing clangd
|
|
|
|
Post the split, I needed to fix clangd as it did not
|