Skip to main content

Swiftly get tons of images from indexed tars on Huggingface.

Project description

cheesechaser

PyPI PyPI - Python Version Loc Comments

Code Test Package Release codecov

Discord GitHub Org's stars GitHub stars GitHub forks GitHub commit activity GitHub issues GitHub pulls Contributors GitHub license

Swiftly get tons of images from indexed tars on Huggingface

Installation

pip install cheesechaser

How this library works

This library is based on the mirror datasets on huggingface.

For the Gelbooru mirror dataset repository, such as deepghs/gelbooru_full, each data packet includes a tar archive file and a corresponding JSON index file. The JSON index file contains detailed information about the files within the tar archive, including file size, offset, and file fingerprint.

The files in this dataset repository are organized according to a fixed pattern based on their IDs. For example, a file with the ID 114514 will have a modulus result of 4514 when divided by 10000. Consequently, it is stored in images/4/0514.tar.

Utilizing the quick download feature from hfutils.index, users can instantly access individual files. Since the download service is provided through Huggingface's LFS service and not the original website or an image CDN, there is no risk of IP or account blocking. The only limitations to your download speed are your network bandwidth and disk read/write speeds.

This efficient system ensures a seamless and reliable access to the dataset without any restrictions.

Batch Download Images

  • Danbooru
from cheesechaser.datapool import DanbooruNewestDataPool

pool = DanbooruNewestDataPool()

# download danbooru #2010000-2010300, to directory /data/exp2
pool.batch_download_to_directory(
    resource_ids=range(2010000, 2010300),
    dst_dir='/data/exp2',
    max_workers=12,
)
  • Danbooru With Tags Query
from cheesechaser.datapool import DanbooruNewestDataPool
from cheesechaser.query import DanbooruIdQuery

pool = DanbooruNewestDataPool()
my_waifu_ids = DanbooruIdQuery(['surtr_(arknights)', 'solo'])

# download danbooru images with surtr+solo, to directory /data/exp2_surtr
pool.batch_download_to_directory(
    resource_ids=my_waifu_ids,
    dst_dir='/data/exp2_surtr',
    max_workers=12,
)
  • Konachan (Gated dataset, you should be granted first and set HF_TOKEN environment variable)
from cheesechaser.datapool import KonachanDataPool

pool = KonachanDataPool()

# download konachan #210000-210300, to directory /data/exp2
pool.batch_download_to_directory(
    resource_ids=range(210000, 210300),
    dst_dir='/data/exp2',
    max_workers=12,
)
  • Civitai (this mirror repository on hf is private for now, you have to use hf token of an authorized account)
from cheesechaser.datapool import CivitaiDataPool

pool = CivitaiDataPool()

# download civitai #7810000-7810300, to directory /data/exp2
# should contain one image and one json metadata file
pool.batch_download_to_directory(
    resource_ids=range(7810000, 7810300),
    dst_dir='/data/exp2',
    max_workers=12,
)

More supported:

  • RealbooruDataPool (Gated Dataset)
  • ThreedbooruDataPool (Gated Dataset)
  • FancapsDataPool (Gated Dataset)
  • BangumiBaseDataPool (Gated Dataset)
  • AnimePicturesDataPool (Gated Dataset)
  • KonachanDataPool (Gated Dataset)
  • YandeDataPool (Gated Dataset)
  • ZerochanDataPool (Gated Dataset)
  • GelbooruDataPool and GelbooruWebpDataPool (Gated Dataset)
  • DanbooruNewestDataPool and DanbooruNewestWebpDataPool

Batch Retrieving Images

from itertools import islice

from cheesechaser.datapool import DanbooruNewestDataPool
from cheesechaser.pipe import SimpleImagePipe, PipeItem

pool = DanbooruNewestDataPool()
pipe = SimpleImagePipe(pool)

# select from danbooru 7349990-7359990
ids = range(7349990, 7359990)
with pipe.batch_retrieve(ids) as session:
    # only need 20 images
    for i, item in enumerate(islice(session, 20)):
        item: PipeItem
        print(i, item)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cheesechaser-0.2.0.tar.gz (42.6 kB view details)

Uploaded Source

Built Distribution

cheesechaser-0.2.0-py3-none-any.whl (57.2 kB view details)

Uploaded Python 3

File details

Details for the file cheesechaser-0.2.0.tar.gz.

File metadata

  • Download URL: cheesechaser-0.2.0.tar.gz
  • Upload date:
  • Size: 42.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.10

File hashes

Hashes for cheesechaser-0.2.0.tar.gz
Algorithm Hash digest
SHA256 6556e9409808a0e142bc119ec982dc9286e9d1b5d4d407867530ac533ad18784
MD5 cffd9ef7381d799dc669f5de600d0973
BLAKE2b-256 b8858545a81eb465020d7de3d73da23e003bf1413d290b354d7a4bd3d67b8945

See more details on using hashes here.

File details

Details for the file cheesechaser-0.2.0-py3-none-any.whl.

File metadata

  • Download URL: cheesechaser-0.2.0-py3-none-any.whl
  • Upload date:
  • Size: 57.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.10

File hashes

Hashes for cheesechaser-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 8e7beb5e9e1be1c6c66278eedb37c1338754b8c16e4382b0e3095e5d7b8fa800
MD5 8fe42402030312d299560966eaf6312c
BLAKE2b-256 a199efaa199742bde882a12e0858f7ed0914e06733f6f739d8a8eb77e55f5f99

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page