wandbfsspec 0.1.1

fsspec interface for Weights & Biases (wandb)

🍱 fsspec interface for Weights & Biases (wandb)

Quoting Weights and Biases (wandb), "Weights & Biases is the machine learning platform for developers to build better models faster. Use W&B's lightweight, interoperable tools to quickly track experiments, version and iterate on datasets, evaluate model performance, reproduce models, visualize results and spot regressions, and share findings with colleagues.". Reference at https://docs.wandb.ai/

So you may be thinking, what does wandb have to do with anything close to a File System? Well, it's not but it actually provides a way to upload/download files and store them in a remote, which makes it somehow a File System. Also, wandb provides an API that lets you interact with that "File System", so this is why wandbfsspec makes sense, in order to ease that interface between wandb's File System and anyone willing to use it.

The wandbfsspec implementation is based on https://github.com/fsspec/filesystem_spec.

🔮 Future TODOs

Obviously, since wandb's main purpose is to track and monitor ML experiments, it contains an artifact store, so as to dump there the experiment artifacts for data versioning and model tracking. More information in https://wandb.ai/site/artifacts.

So on, a new interface will be implemented in wandbfsspec not just to handle the files that can be uploaded/downloaded to/from wandb, but also the artifacts. So the next release will implement a new AbstractFileSystem class named WandbArtifactStore with the protocol wandbas in order to access the artifact store as if it was a default File System.

Some more notes on how to actually use wandb's artifact store at https://docs.wandb.ai/guides/artifacts.

Once that's done, we'll fill a PR in https://github.com/fsspec/filesystem_spec, so as to register both protocols supported by wandbfsspec: wandbfs and wandbas.

🚸 Usage

Here's an example on how to locate and open a file:

>>> from wandbfsspec.core import WandbFileSystem
>>> fs = WandbFileSystem(api_key="YOUR_API_KEY")
>>> fs.ls("alvarobartt/wandbfsspec-tests/3s6km7mp")
['alvarobartt/wandbfsspec-tests/3s6km7mp/config.yaml', 'alvarobartt/wandbfsspec-tests/3s6km7mp/file.yaml', 'alvarobartt/wandbfsspec-tests/3s6km7mp/files', 'alvarobartt/wandbfsspec-tests/3s6km7mp/output.log', 'alvarobartt/wandbfsspec-tests/3s6km7mp/requirements.txt', 'alvarobartt/wandbfsspec-tests/3s6km7mp/wandb-metadata.json', 'alvarobartt/wandbfsspec-tests/3s6km7mp/wandb-summary.json']
>>> with fs.open("alvarobartt/wandbfsspec-tests/3s6km7mp/file.yaml", "rb") as f:
...     print(f.read())
b'some: data\nfor: testing'

📌 Note that it can also be done through fsspec as long as wandbfsspec is installed:

>>> import fsspec
>>> fs = fsspec.filesystem("wandbfs")
>>> fs.ls("alvarobartt/wandbfsspec-tests/3s6km7mp")
['alvarobartt/wandbfsspec-tests/3s6km7mp/config.yaml', 'alvarobartt/wandbfsspec-tests/3s6km7mp/file.yaml', 'alvarobartt/wandbfsspec-tests/3s6km7mp/files', 'alvarobartt/wandbfsspec-tests/3s6km7mp/output.log', 'alvarobartt/wandbfsspec-tests/3s6km7mp/requirements.txt', 'alvarobartt/wandbfsspec-tests/3s6km7mp/wandb-metadata.json', 'alvarobartt/wandbfsspec-tests/3s6km7mp/wandb-summary.json']
>>> with fs.open("alvarobartt/wandbfsspec-tests/3s6km7mp/file.yaml", "rb") as f:
...     print(f.read())
b'some: data\nfor: testing'

📝 Documentation

Coming soon... (https://github.com/mkdocs/mkdocs)

🧪 How to test it

In order to test it, you should first set the following environment variables so as to use wandb as a file system for the tests.

WANDB_ENTITY = ""
WANDB_PROJECT = ""
WANDB_API_KEY = ""

Both entity and project values can be found in your https://wandb.ai/ account, as the entity name is your account name, and the project name can either be already created or you can just specify it and it'll be created during pytest init. Then, regarding the API Key, you just need to go to https://wandb.ai/settings, scroll down to Danger Zone -> API Keys, and copy your personal API Key from there.

⚠️ Make sure that you don't publish your API Key anywhere, that's why we're defining it as an environment value, so as to avoid potential issues on commiting code with the actual API Key value.

Then, in order to actually run the tests you can either run:

• poetry run pytest
• poetry run make tests

Or, if you're not using poetry, you can just run both those commands without it.