AI-Toolbox

Command

ait-download

Download hf model,dataset and will retry after a failure, default 10 times.

ait-download \
--repo_info_path=/your_repo_info.json \
--retry_count=2 \
--thread_nums=5

Args description

Demo repo_info.json

{
    "model":[
        "OpenAssistant/falcon-40b-megacode2-oasst",
        "stabilityai/stable-diffusion-xl-base-1.0"
    ],
    "dataset":[
        "OpenAssistant/oasst1",
        "PygmalionAI/PIPPA"
    ]
}

ait-fork

fork hf model, dataset and will retry after a failure, default 10 times.

# fork with fork_repo_info.json
ait-fork \
--repo_info_path=/your_repo_info.json \
--retry_count=2 \
--hf_username=your_hf_username \
--hf_token=your_hf_token \
--separator=__ \
--thread_nums=5

# fork without fork_repo_info.json
ait-fork \
--retry_count=2 \
--hf_username=your_hf_username \
--hf_token=your_hf_token \
--separator=__ \
--source_repo=source_repo \
--target_repo=your_target_repo \
--repo_type=dataset|model \
--disable-multi-task

Args description

you must pass --repo_info_path or --source_repo at least one!

Demo fork_repo_info.json

{
    "model":[
        {
            "source":"OpenAssistant/falcon-40b-megacode2-oasst",
            "target":"your_name/your_repo"
        }        
    ],
    "dataset":[
        {
            "source":"PygmalionAI/PIPPA",
            "target":"your_name/your_repo"
        }
    ]
}