hugging-downloader

---

Installation

pip install -r requirements.txt

or

pip install git+https://github.com/huggingface/transformers

VPN configuration

cd ./hf_downloader/profiles/

修改__init__.py中的vpn配置，配置格式如下，其中34789为vpn的本地代理端口，

import os

os.environ["http_proxy"] = "http://127.0.0.1:34789"
os.environ["https_proxy"] = "http://127.0.0.1:34789"

另一种方式是，在命令行中配置环境变量

export https_proxy=http://127.0.0.1:34789;export http_proxy=http://127.0.0.1:34789;export all_proxy=socks5://127.0.0.1:34789

Usage:

spider and request model from huggingface

python hf_downloader/spider_hf_downloader.py 

Download models from huggingface

python hf_downloader/huggingface_download.py --repo-id prajjwal1/bert-tiny --cache-dir ../cache --local-dir ../bert-tiny

example

python hf_downloader/huggingface_download.py --repo-id prajjwal1/bert-tiny --cache-dir ../cache --local-dir ../bert-tiny

Download models use git ifs

python hf_downloader/git_lfs_clone.py --repo_url https://huggingface.co/prajjwal1/bert-tiny --repo_path bert-tiny