We’re super excited to announce the release of semhash, our semantic deduplication and dataset multitool (other features coming soon).

This blogpost describes the Tokenlearn method, which is a method to pre-train Model2Vec models.

This blog was first posted on the Hugging Face blog. We’re also posting it here for archival purposes.