hfdol 0.1.18

Simple Mapping interface to HuggingFace

hfdol

Simple Mapping interface to HuggingFace.

(Note -- was hf but realeased the name to Huggingface itself for their tool.)

To install: pip install hfdol

You'll also need a Hugginface token. See more about this here.

Motivation

The Python packages datasets and huggingface_hub provide a remarkably clean, well-documented, and comprehensive API for accessing datasets, models, spaces, and papers hosted on Hugging Face.
Yet, as elegant as these APIs are, they remain their own language. Every library—no matter how intuitive—inevitably carries its own conventions, abstractions, and domain-specific semantics. When working with one or two APIs, this diversity is harmless, even stimulating. But when juggling dozens or hundreds of them, the cognitive overhead accumulates.

Despite their differences, most APIs share a small set of universal primitives — retrieve something by key, list what's available, check existence, store, update, delete.
In Python, these operations are embodied by the Mapping interface, the conceptual model behind dictionaries. It's a minimal, ubiquitous, and instantly recognizable abstraction.

This package offers such a Mapping-based façade to Hugging Face datasets and models, allowing you to browse, query, and access them as if they were simple Python dictionaries. The goal isn't to replace the original API, but to provide a thin, ergonomic layer for the most common operations — so you can spend less time remembering syntax, and more time working with data.

Examples

This package provides four ready-to-use singleton instances, each offering a dictionary-like interface to different types of HuggingFace resources:

import hfdol

Working with Datasets

The hfdol.datasets singleton provides a Mapping (i.e. read-only-dictionary-like) interface to HuggingFace datasets:

List Local Datasets

As with dictionaries, hfdol.datasets is an iterable. An iterable of keys. The keys are repository ids for those datasets you've downloaded. See what datasets you already have cached locally like this:

list(hfdol.datasets)  # Lists locally cached datasets
# ['stingning/ultrachat', 'allenai/WildChat-1M', 'google-research-datasets/go_emotions']

Access Local Datasets

The values of hfdol.datasets are the DatasetDict (from Huggingface's datasets package) instances that give you access to the dataset. If you already have the dataset downloaded locally, it will load it from there, if not it will download it, then give it to you (and it will be cached locally for the next time you access it).

data = hfdol.datasets['stingning/ultrachat']  # Loads the dataset
print(data)  # Shows dataset information and structure

Search for Remote Datasets

hfdol.datasets also offers a search functionality, so you can search "remote" repositories:

# Search for music-related datasets
search_results = hfdol.datasets.search('music', gated=False)
print(f"search_results is a {type(search_results).__name__}")  # It's a generator

# Get the first result (it will be a `DatasetInfo` instance contain information on the dataset)
result = next(search_results)
print(f"Dataset ID: {result.id}")
print(f"Description: {result.description[:80]}...")

# Download and use it directly
data = hfdol.datasets[result]  # You can pass the DatasetInfo object directly

Note that the gated=False was to make sure you get models that you have access to. For more search options, see the HuggingFace Hub documentation.

A useful recipe: Get a table of result infos

You can use this to get a dataframe of the first/next n results of the results iterable:

def table_of_results(results, n=10):
    import itertools, operator, pandas as pd

    results_table = pd.DataFrame(  # make a table with
        map(
            operator.attrgetter('__dict__'),  # the attributes dicts
            itertools.islice(results, n),  # ... of the first 10 search results
        )
    )
    return results_table

Example:

results_table = table_of_results(search_results)
results_table

                          id            author                                       sha ...
0   Genius-Society/hoyoMusic    Genius-Society  4f7e5120c0e8e26213d4bb3b52bcce76e69dfce4 ...
1      Genius-Society/emo163    Genius-Society  6b8c3526b66940ddaedf15602d01083d24eb370c ...
2  ccmusic-database/acapella  ccmusic-database  4cb8a4d4cb58cc55f30cb8c7a180fee1b5576dc5 ...
3    ccmusic-database/pianos  ccmusic-database  db2b3f74c4c989b4fbda4b309e6bc925bfd8f5d1 ...
...

Working with Models

The hfdol.models singleton provides the same dictionary-like interface for models:

Search for Models

Find models by keywords:

model_search_results = hfdol.models.search('embeddings', gated=False)
model_result = next(model_search_results)
print(f"Model: {model_result.id}")

Download Models

Get the local path to a model (downloads if not cached):

model_path = hfdol.models[model_result]
print(f"Model downloaded to: {model_path}")

List Local Models

See what models you have cached:

list(hfdol.models)  # Lists all locally cached models

Working with Spaces

The hfdol.spaces singleton provides access to HuggingFace Spaces (interactive ML demos and applications):

Search for Spaces

Find interesting Spaces by keywords:

space_search_results = hfdol.spaces.search('gradio', limit=5)
space_result = next(space_search_results)
print(f"Space: {space_result.id}")

Access Space Information

Get detailed information about a Space:

space_info = hfdol.spaces[space_result]
print(f"Space info: {space_info}")

List Local Spaces

See what spaces you have cached locally:

list(hfdol.spaces)  # Lists all locally cached spaces

Working with Papers

The hfdol.papers singleton provides access to research papers hosted on HuggingFace:

Search for Papers

Find research papers by topic:

paper_search_results = hfdol.papers.search('transformer', limit=5)
paper_result = next(paper_search_results)
print(f"Paper: {paper_result.id}")

Access Paper Information

Get detailed information about a paper:

paper_info = hfdol.papers[paper_result]
print(f"Paper title: {paper_info.title}")
print(f"Abstract: {paper_info.summary[:100]}...")

Note: Papers are metadata objects only—they contain information about research papers but don't have downloadable files like datasets or models.

Getting Repository Sizes

You can check the size of any repository before downloading using the get_size function. The repo_type parameter is required to avoid ambiguity when repositories exist as multiple types:

from hfdol import get_size

# Get size of a dataset (specify repo_type explicitly)
dataset_size = get_size('ccmusic-database/music_genre', repo_type='dataset')
print(f"Dataset size: {dataset_size:.2f} GiB")

# Get size of a model 
model_size = get_size('ccmusic-database/music_genre', repo_type='model')
print(f"Model size: {model_size:.2f} GiB")

# Using RepoType enum for type safety
from hfdol.base import RepoType
size_with_enum = get_size('some-repo', repo_type=RepoType.DATASET)

# Get size in different units (e.g., bytes)
size_in_bytes = get_size('some-repo', repo_type='dataset', unit_bytes=1)

Pro tip: Use the singleton instances for automatic repo_type handling:

# These automatically know their repo_type
dataset_size = hfdol.datasets.get_size('ccmusic-database/music_genre')
model_size = hfdol.models.get_size('ccmusic-database/music_genre')

Unified Interface

The beauty of this approach is that whether you're working with datasets, models, spaces, or papers, the interface remains familiar and consistent—just like working with Python dictionaries. All four singleton instances support the same core operations:

• Dictionary-style access: resource = hfdol.datasets[key], model_path = hfdol.models[key]
• Local listing: list(hfdol.datasets), list(hfdol.models)
• Remote searching: hfdol.datasets.search(query), hfdol.models.search(query)
• Existence checking: key in hfdol.datasets, key in hfdol.models

This unified interface means you can switch between different types of HuggingFace resources without learning new APIs—it's all just dictionaries! And since they're singleton instances, they're always ready to use without any setup.

Design & Architecture

Design Philosophy

This package is designed as a thin façade over the excellent huggingface_hub and datasets libraries. Rather than reinventing functionality, it provides a unified Mapping interface that wraps the most common operations, making them feel like native Python dictionary operations.

The design balances two sometimes-competing goals:

1. Simplicity: Keep the codebase small, readable, and maintainable
2. Single Source of Truth (SSOT): Minimize hardcoded knowledge about the underlying APIs

Ideally, this interface would be entirely auto-generated through static analysis of the wrapped packages. While we achieve this partially, practical constraints require some manual intervention—but we've minimized it as much as possible.

Key Architectural Patterns

1. Configuration-Driven Design (SSOT)

The repo_type_helpers dictionary serves as the single source of truth for all repo-type-specific behavior:

repo_type_helpers = dict(
    dataset=dict(
        loader_func=load_dataset,
        search_func=list_datasets,
    ),
    model=dict(
        loader_func=snapshot_download,
        search_func=list_models,
    ),
    # ... etc
)

This declarative approach means:

• Adding a new repo type requires only updating this configuration
• No duplication of logic across different repo types
• Clear visibility of how each type differs

2. Dynamic Signature Injection

Rather than manually replicating the signatures of wrapped functions (which would violate SSOT), we use signature extraction and injection via the sign_kwargs_with decorator:

@sign_kwargs_with(search_func)
def search(self, filter, **kwargs):
    return self.search_func(filter=filter, **kwargs)

This means:

• Each .search() method automatically inherits the correct signature from its underlying function
• IDEs and type checkers see the actual parameters available
• When HuggingFace updates their APIs, our signatures update automatically
• Documentation stays accurate without manual synchronization

Note: The list_papers function required special handling (_list_papers wrapper) because it uses query instead of filter as its parameter name. This is the type of pragmatic compromise we make—we normalize the interface rather than exposing the inconsistency.

3. Separation of Concerns

The architecture cleanly separates:

• Configuration (repo_type_helpers): What differs between types
• Base functionality (HfMapping): Shared behavior for all types
• Type-specific classes (HfDatasets, HfModels, etc.): Minimal subclasses that mainly provide:
  • Clear, discoverable class names
  • Type-specific documentation
  • Future extensibility points
• Convenience layer (module-level singletons): Zero-setup access for users

4. Module-Level Singletons

The pre-instantiated datasets, models, spaces, and papers instances follow Python's convenience instance pattern (seen in sys.stdout, np.random, etc.):

# Ready to use immediately
datasets = HfDatasets()
models = HfModels()

This works because these instances:

• Have no mutable state
• Require no configuration for basic use
• Represent logical singletons ("the datasets mapping")

5. Progressive Disclosure

The API supports multiple levels of sophistication:

# Simplest: Use pre-configured singletons
data = hfdol.datasets['some/dataset']

# Advanced: Create custom instances with configuration
my_datasets = HfDatasets()

# Power user: Parameterized mapping for dynamic repo types
custom = HfMapping(RepoType.DATASET)

Design Compromises

Several compromises were made for pragmatism:

1. Manual wrappers: _list_papers normalizes the papers API to match others
2. Enum + string hybrid: RepoType(str, Enum) allows both type safety and string convenience
3. Explicit repo_type in get_size: Required parameter to avoid ambiguity when repos exist as multiple types
4. Signature injection limitations: Works well for keyword arguments but can't handle complex overloads

Contributing Guidelines

When contributing to this package, please maintain these principles:

✅ DO:

• Add configuration to repo_type_helpers rather than creating new methods
• Use signature extraction (sign_kwargs_with) when wrapping functions with many parameters
• Keep HfMapping generic and push specialization to configuration
• Document why special cases exist (like _list_papers)
• Test against actual HuggingFace APIs to catch signature drift

❌ AVOID:

• Duplicating knowledge about wrapped APIs
• Hardcoding parameter lists or types that could be extracted
• Adding stateful behavior to mapping instances
• Creating wrapper methods that simply pass through to underlying functions

When in doubt:

• Ask "Could this be driven by configuration?"
• Prefer declarative patterns over imperative logic
• Keep the codebase small and the configuration visible

The goal is a package where 80% of the code is just wiring and configuration, and the HuggingFace packages do the actual work. This maximizes maintainability and minimizes drift as those packages evolve.