Deduplication

CS336 Notes: Lecture 14 - Data 2

Data filtering and deduplication at scale: n-gram language models, fastText classifiers, importance sampling, MinHash, LSH, and Bloom filters for efficient web-scale processing.