Skip to main content

Convenient filesystem interface over Oracle Cloud's Object Storage

Project description

Oracle Cloud Infrastructure Object Storage fsspec Implementation

PyPI Python

​ The Oracle Cloud Infrastructure Object Storage service is an internet-scale, high-performance storage platform that offers reliable and cost-efficient data durability. With Object Storage, you can safely and securely store or retrieve data directly from the internet or from within the cloud platform. ​ ocifs is part of the fsspec intake/filesystem_spec ecosystem

a template or specification for a file-system interface, that specific implementations should follow, so that applications making use of them can rely on a common interface and not have to worry about the specific internal implementation decisions with any given backend. ​ ocifs joins the list of file systems supported with this package. ​ The intake/filesystem_spec project is used by Pandas, Dask and other data libraries in python, this package adds Oracle OCI Object Storage capabilties to these libraries. ​

OCIFS file system style operations Example:

from ocifs import OCIFileSystem

fs = OCIFilesystem("~/.oci/config")
# 1.Create empty file or truncate in OCI objectstorage bucket
 fs.touch("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt", truncate=True, data=b"Writing to Object Storage!")
 # 2.Fetch(potentially multiple paths' contents"oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt")
 # 3.Get metadata about a file from a head or list call"oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt")
 # 4.Get directory listing page"oci://<my_bucket>@<my_namespace>/<my_prefix>/", detail=True)
 # 5.Is this entry directory-like?
 # 6.Is this entry file-like?
 # 7.If there is a file at the given path (including broken links)
 # 8.List of files for the given path
 fs.listdir("oci://<my_bucket>@<my_namespace>/<my_prefix>", detail=True)
 # 9.Get the first ``size`` bytes from file
 fs.head("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt", size=1024)
 # 10.Get the last ``size`` bytes from file
 fs.tail("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt", size=1024)
 # 11.Hash of file properties, to tell if it has changed
 # 12.Size in bytes of file
 # 13.Size in bytes of each file in a list of paths
 paths = ["oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt"]
 # 14.Normalise OCI path string into bucket and key.
 # 15.Delete  a file from the  bucket
 # 16.Get the contents of the file as a byte
 fs.read_bytes("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt", start=0, end=13)
 # 17.Get the contents of the file as a string
 fs.read_text("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt", encoding=None, errors=None, newline=None)
 # 18.Get the contents of the file as a byte
 fs.read_block("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt", 0, 13)
 # 19.Open a file for writing/flushing into file in OCI objectstorage bucket
 # Ocifs sets the best-guessed content-type for hello.txt i.e "text/plain"
 with"oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt", 'w', autocommit=True) as f:
        f.write("Writing data to buffer, before manually flushing and closing.") # data is flushed and file closed
# Ocifs uses the specified content-type passed in the open while writing to OCI objectstorage bucket
 with"oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt", 'w',content_type='text/plain') as f:
        f.write("Writing data to buffer, before manually flushing and closing.") # data is flushed and file closed
 # 20.Open a file for reading a file from OCI objectstorage bucket
 with"oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt") as f:
 # 21.Space used by files and optionally directories within a path
 # 22.Find files by glob-matching.
 # 23.Renames an object in a particular bucket in tenancy namespace on OCI
 fs.rename("oci://<my_bucket>@<my_namespace>/<my_prefix>/hello.txt", "oci://<my_bucket>@<my_namespace>/<my_prefix>/hello2.txt")
 # 24.Delete multiple files from the same bucket
 pathlist = ["oci://<my_bucket>@<my_namespace>/<my_prefix>/hello2.txt"]

Or Use With Pandas

import pandas as pd
import ocifs

df = pd.read_csv(
    storage_options={"config": "~/.oci/config"},

Or Use With PyArrow

import pandas as pd
import ocifs

df = pd.read_csv(
    "oci://my_bucket@my_namespace/my_object.csv",storage_options={"config": "~/.oci/config"})

Or Use With ADSDataset

import ads
import pandas as pd
from ads.common.auth import default_signer
from ads.dataset.dataset import ADSDataset

    ads.set_auth(auth="api_key", oci_config_location="~/.oci/config", profile="<profile_name>")
    ds = ADSDataset(
        df=pd.read_csv(f"oci://my_bucket@my_namespace/my_object.csv", storage_options=default_signer()),

Getting Started

python3 -m pip install ocifs

Software Prerequisites

Python >= 3.6

Environment Variables for Authentication:

export OCIFS_IAM_TYPE=api_key
export OCIFS_CONFIG_LOCATION=~/.oci/config

Note, if you are operating on OCI with an alternative valid signer, such as resource principal, instead set the following:

export OCIFS_IAM_TYPE=resource_principal

Environment Variables for enabling Logging:

To quickly see all messages, you can set the environment variable OCIFS_LOGGING_LEVEL=DEBUG.




The built-in filesystems in fsspec are maintained by the intake project team, where as ocifs is an external implementation (similar to s3fs, gcsfs, adl/abfs, and so on), which is maintained by Oracle.


This project welcomes contributions from the community. Before submitting a pull request, please review our contribution guide


Please consult the security guide for our responsible security vulnerability disclosure process


Copyright (c) 2021, 2023 Oracle and/or its affiliates.

Released under the Universal Permissive License v1.0 as shown at

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ocifs-1.3.1.tar.gz (57.3 kB view hashes)

Uploaded Source

Built Distribution

ocifs-1.3.1-py3-none-any.whl (67.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page