Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese data processing and cleaning methods in MassiveText.
-
Updated
Jun 7, 2023 - Python
#
massivetext
Here is 1 public repository matching this topic...
Add this topic to your repo
To associate your repository with the massivetext topic, visit your repo's landing page and select "manage topics."