Better filtering & dedup based on RedPajava v2

[syoyo]

https://www.together.ai/blog/redpajama-data-v2

• The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only https://arxiv.org/abs/2306.01116

  • RefinedWeb approach is already considered in CharShu v1(except for line-wise filtering)

• Scaling Language Models: Methods, Analysis & Insights from Training Gopher https://arxiv.org/abs/2112.11446

• A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity https://arxiv.org/abs/2305.13169

• Data Selection for Language Models via Importance Resampling https://arxiv.org/abs/2302.03169

  • code https://github.com/p-lambda/dsir