When processing a lot of documents that often have similar sections into a single Postgres cluster, deduplication becomes much harder because not all users have the same view to the processed chunks. This means that you have to store all chunks in order to make sure no chunks are removed simply because there exists an inaccessible one elsewhere for the user.
By using hybrid search and a MinHash + LSH clustering scheme, we can produce on-demand deduplicated search results quickly and efficiently for a user by clustering chunks when ingesting documents.
