Skip to main content

Filestore

The filestore is an easy way to to process files with multiple processes without needing locks.

Initialize a FileStore

Create a filestore in a temporary area:

>>> import tempfile, os
>>> temp_dir = tempfile.mkdtemp()
>>> store_dir = os.path.join(temp_dir, 'store1')
>>> os.mkdir(store_dir)
>>> from gocept.filestore import FileStore
>>> filestore = FileStore(store_dir)
>>> filestore
<gocept.filestore.filestore.FileStore object at 0x...>

So far nothing has happend:

>>> import os
>>> os.listdir(store_dir)
[]

Before using the store we need to prepare it:

>>> filestore.prepare()

Prepare has created the tmp/new/cur directory structure:

>>> sorted(os.listdir(store_dir))
['cur', 'new', 'tmp']

Calling prepare again does nothing:

>>> filestore.prepare()
>>> sorted(os.listdir(store_dir))
['cur', 'new', 'tmp']

If the store_dir is removed, it is created again by calling prepare.

>>> import shutil
>>> shutil.rmtree(store_dir)
>>> os.listdir(temp_dir)
[]
>>> filestore.prepare()
>>> os.listdir(temp_dir)
['store1']
>>> sorted(os.listdir(store_dir))
['cur', 'new', 'tmp']

Use a FileStore

Adding files to the store works with the create method:

>>> f = filestore.create('a-file')

Files are created in the ‘tmp’ area with the ‘w’ mode (if not specified):

>>> f.name
'.../tmp/a-file'
>>> f.mode
'w'

We find the file in the tmp area. Note that filestore.list lists files with their full path names, so we could feed the name directly to file/open:

>>> filestore.list('tmp')
['.../tmp/a-file']

We got a plain file back, so write to it:

>>> _ = f.write('Die Ente bleibt draussen!')
>>> f.close()

We have finished writing our file, so we can move it to the new space for other applications to pick it up:

>>> filestore.move('a-file', 'tmp', 'new')
>>> filestore.list('tmp')
[]
>>> filestore.list('new')
['.../new/a-file']

The move operation uses os.move which is supposed to be atomic. When another processes “sees” the file it can directly work with it and move it to ‘cur’:

>>> filestore.move('a-file', 'new', 'cur')
>>> filestore.list('new')
[]
>>> filestore.list('cur')
['.../cur/a-file']

Files can be copied, too:

>>> filestore.copy('a-file', 'cur', 'tmp')
>>> filestore.list('cur')
['.../cur/a-file']
>>> filestore.list('tmp')
['.../tmp/a-file']

Finally, files can be removed:

>>> filestore.remove('a-file', 'cur')
>>> filestore.list('cur')
[]

Cleanup

Remove the temporary directory after testing:

>>> import shutil
>>> shutil.rmtree(store_dir)

Changes

1.0 (2023-07-14)

  • Drop support for Python 2.7, 3.5, 3.6.

0.5 (2023-03-16)

  • Add support for Python 3,9, 3.10, 3.11.

  • Use GitHub actions as CI.

0.4 (2019-11-29)

  • Migrate repository to Bitbucket.

  • Migrate repository to GitHub.

  • Made Python 3 compatible (tested with Python 2.7, 3.7 and 3.8).

  • Replace bootstrap/buildout with tox.

  • Increase test coverage to 100%.

0.3 (2009-10-08)

  • Added copy() method.

0.2 (2007-08-30)

  • Initial public release.

Release files for gocept.filestore 1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for gocept.filestore 1.0
File Size Uploaded
gocept.filestore-1.0.tar.gz 6.3 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for gocept.filestore 1.0
File Interpreter ABI Platform
gocept.filestore-1.0-py2.py3-none-any.whl Python 2, Python 3 none any Details

Total release size: 13.7 kB

Release files / gocept.filestore-1.0.tar.gz

Download URL gocept.filestore-1.0.tar.gz
Size 6.3 kB
Tags Source
SHA-256 checksum
How to use checksums
3fbec68b20cf2b2df54d046aafb6a382a83a772e1b5c59b459c44ec4dd843786
BLAKE2b-256 checksum
How to use checksums
e5a798c1abc3c87a7a7a3aade652b7fea069c6a49429f14e087451bdcf0e7f52
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.4

Release files / gocept.filestore-1.0-py2.py3-none-any.whl

Download URL gocept.filestore-1.0-py2.py3-none-any.whl
Size 7.4 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
f2488366f26e8c32e7c53fcb09e8d22c1918c4bcbbeba6a9afa1324c471bbcc6
BLAKE2b-256 checksum
How to use checksums
8f25069ec6906b96827a4ccb9ed6af458964d7320c8e331a5d5de2da57a4451b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.4

Release history Release notifications | RSS feed

This release

1.0 This release

2 release files

0.5

2 release files

0.4

1 release file

0.3

1 release file

0.2

1 release file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page