diff --git a/README.md b/README.md index 459c70a..301245e 100644 --- a/README.md +++ b/README.md @@ -125,6 +125,7 @@ Tools listed in this section can help you evaluate, generate, and explore datase ### Data scraping * [**Trafilatura**](https://github.com/adbar/trafilatura): Python and command-line tool to gather text and metadata on the web. Used for the creation of [RefinedWeb](https://arxiv.org/abs/2306.01116). * [**Marker**](https://github.com/VikParuchuri/marker): Quickly convert PDFs to markdown text. +* [**SearchApi**](https://www.searchapi.io/): Retrieve real-time SERP API results from engines like Google, Bing, Baidu, YouTube, Yahoo, and more. ### Data filtering * **Rule-based filtering**: Remove samples based on a list of unwanted words, like refusals and "As an AI assistant" ([example](https://huggingface.co/datasets/cognitivecomputations/WizardLM_alpaca_evol_instruct_70k_unfiltered/blob/main/wizardlm_clean.py)).