fs-synapse
A Synapse implementation of the fsspec interface.
fs-synapse allows us to leverage the fsspec API to interface with Synapse files, folders, and projects. By learning this API, you can write code that is agnostic to where your files are physically located. This is achieved by referring to Synapse entities using URLs. Commented examples are included below.
syn://syn50545516 # Synapse project
syn://syn50557597 # Folder in the above Synapse project
syn://syn50545516/syn50557597 # Same folder, but using a full path
syn://syn50545516/TestSubDir # Same folder, but referenced by name
syn://syn50555279 # File in the above Synapse project
syn://syn50545516/syn50555279 # Same file, but using a full path
syn://syn50545516/test.txt # Same file, but referenced by name
syn://syn50545516/ExploratoryTests/report.json # Nested file
Benefits
There are several benefits to using the fs-synapse API over synapseclient.
from synapsefs import SynapseFS
fs = SynapseFS()
# Or using fsspec directly
import fsspec
fs = fsspec.filesystem("syn")
Interact with Synapse using a Pythonic interface
file_url = "syn://syn50555279"
with fs.open(file_url, "a") as fp:
fp.write("Appending some text to a Synapse file")
Access to several convenience functions
folder_url = "syn://syn50696438"
fs.makedirs(f"{folder_url}/creating/nested/folders/with/one/operation")
Refer to Synapse files and folders by name
You don't have to track as many Synapse IDs. You only need to care about the top-level projects or folders and refer to subfolders and files by name.
project_url = "syn://syn50545516"
data_url = f"{project_url}/data/raw.csv"
output_url = f"{project_url}/outputs/processed.csv"
with fs.open(data_url, "r") as data_fp, fs.open(output_url, "a") as output_fp:
results = number_cruncher(data)
output.write(results)
Write Synapse-agnostic code
Unfortunately, every time you use synapseclient for file and folder operations, you are hard-coding a dependency on Synapse into your project. Leveraging fs-synapse helps avoid this hard dependency and makes your code more portable to other file backends (e.g. S3). You can swap for any other file system by using their URL scheme (e.g. s3://). Here's an index of available file systems that you can swap for.
Rely on code covered by integration tests
So you don't have to write the Synapse integration tests yourself! These tests tend to be slow, so delegating that responsibilty to an externally managed package like fs-synapse keeps your test suite fast and focused on what you care about.
In your test code, you can use the memory filesystem for faster I/O instead of storing and retrieving files on Synapse.
def test_some_feature_of_your_code():
fs = fsspec.filesystem("memory")
cruncher = NumberCruncher(fs=fs)
cruncher.save("report.json")
assert fs.exists("report.json")
Migration from PyFilesystem2 to fsspec
This package previously used PyFilesystem2 (fs) as its base. It now uses fsspec. The table below maps the old API to the new one.
Initialization
| Old (PyFilesystem2) | New (fsspec) |
|---|---|
from fs import open_fs |
import fsspec |
fs = open_fs("syn://") |
fs = fsspec.filesystem("syn") |
fs = open_fs("syn://syn50545516") |
fs = SynapseFS(root="syn50545516") |
File operations
| Old (PyFilesystem2) | New (fsspec) |
|---|---|
fs.open(path, "r") |
fs.open(path, "r") |
fs.readtext(path) |
fs.cat_file(path).decode() |
fs.readbytes(path) |
fs.cat_file(path) |
fs.writetext(path, text) |
fs.pipe_file(path, text.encode()) |
fs.writebytes(path, data) |
fs.pipe_file(path, data) |
fs.create(path) / fs.touch(path) |
fs.touch(path) |
fs.download(name, file_obj) |
fs.get(path, local_path) |
Directory operations
| Old (PyFilesystem2) | New (fsspec) |
|---|---|
fs.listdir(path) |
fs.ls(path, detail=False) (returns full paths) |
fs.makedir(path) |
fs.mkdir(path) |
fs.makedirs(path) |
fs.makedirs(path) |
fs.opendir(path) |
(no equivalent; use full paths) |
fs.tree(path=path) |
fs.ls(path, detail=True) |
Removal
| Old (PyFilesystem2) | New (fsspec) |
|---|---|
fs.remove(path) |
fs.rm(path) |
fs.removedir(path) |
fs.rmdir(path) |
fs.removetree(path) |
fs.rm(path, recursive=True) |
Info and metadata
| Old (PyFilesystem2) | New (fsspec) |
|---|---|
info = fs.getinfo(path, namespaces=["details", "synapse"]) |
info = fs.info(path) |
info.name |
info["name"] |
info.is_dir |
info["type"] == "directory" |
info.get("details", "size") |
info["size"] |
info.get("synapse", "id") |
info["synapse_id"] |
info.get("synapse", "content_type") |
info["synapse_content_type"] |
info.get("synapse", "etag") |
info["synapse_etag"] |
fs.getsize(path) |
fs.info(path)["size"] |
fs.gettype(path) |
fs.info(path)["type"] |
fs.exists(path) |
fs.exists(path) |
Errors
| Old (PyFilesystem2) | New (fsspec) |
|---|---|
fs.errors.ResourceNotFound |
FileNotFoundError |
fs.errors.FileExists / DirectoryExists |
FileExistsError |
fs.errors.FileExpected |
IsADirectoryError |
fs.errors.DirectoryExpected |
NotADirectoryError |
fs.errors.CreateFailed |
ValueError |
fs.errors.ResourceInvalid |
ValueError |
fs.errors.RemoveRootError |
PermissionError |
fs.errors.DirectoryNotEmpty |
OSError |
PyScaffold
This project has been set up using PyScaffold 4.3. For details and usage information on PyScaffold see PyScaffold.
putup --name fs-synapse --markdown --github-actions --pre-commit --license Apache-2.0 fs-synapse
1 maintainer
Metadata
1 maintainer
Release files for fs-synapse 3.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| fs_synapse-3.0.2.tar.gz | 119.1 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| fs_synapse-3.0.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 138.1 kB
Release files / fs_synapse-3.0.2.tar.gz
| Download URL | fs_synapse-3.0.2.tar.gz |
|---|---|
| Size | 119.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9833655c55d914f11b727931cfe583ac24254196a41d44b2a51e16496fbcccd6
|
|
BLAKE2b-256 checksum How to use checksums |
d27b653c3c064fe181466370b0b09696ff2b9c88805e0cb4953a913a8468c82e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
Release files / fs_synapse-3.0.2-py3-none-any.whl
| Download URL | fs_synapse-3.0.2-py3-none-any.whl |
|---|---|
| Size | 19.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
089418a48953a4000d22ed83ec3fac8c2b73f1eedf95ee387182b5923a941ad9
|
|
BLAKE2b-256 checksum How to use checksums |
5f2ef7a35303b1c466e2b8609f12567a201aa46c459a6f29a1f786a697d99cf0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
Security for fs-synapse 3.0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
