Skip to main content

Find the next file in a sequence of files in a thread-safe way.

Project description

BuildStatus BuildStatusWin Coverage

Generate sequential file names in a thread-safe way.

Usage

If you want to create sequential files to write output to (say from different threads or from consecutive runs of the same program), then you can use this module from within your program or from the command line.

From the commandline

$ seqfile
usage: seqfile [-h] [-m MAX_ATTEMPTS] [-b BASE] prefix [suffix] [folder]
seqfile: error: too few arguments

# Assume files "./a.0.txt" and "./a.3.txt" exist.
$ seqfile a. .txt .
./a.4.txt

# Assume no other files exist.
# This will create files 100 files with names test.*.txt in the current
# folder.
$ seq 100 | xargs -n 1 -P 100 -I{} \
    bash -c "fname=\$(seqfile test. .txt); \
             echo 'Do something awesome with {}' > \$fname"

From your program

# Assume files "./a.0.txt" and "./a.3.txt" exist.
>> import seqfile

# Providing prefix and suffix finds the next file in the sequence
# ignoring any gaps
>> seqfile.findNextFile('.', prefix='a.', suffix='.txt')
'./a.4.txt'

# Providing a file name generator as fnameGen produces the first file
# which it was successful in creating.
>> seqfile.findNextFile('.', fnameGen=lambda x: 'a.' + str(x) + '.txt')
'./a.1.txt'

The returned file will exist, be empty, and can be opened for writing.

Installing

This package is available from PyPi <https://pypi.python.org/pypi/seqfile>.

pip install seqfile

You can install the bleeding edge directly from the source:

pip install git+https://github.com/musically-ut/seqfile.git@master#egg=seqfile

Tests

The tests can be run using nosetests (if all dependencies are installed in the environment) or using python setup.py test.

Caveats

It is best not to provide unicode characters in the prefix, suffix, or folder. Though the support has been tested on all major OSes, it has not been verfied that all OS/filesystem combinations will work.

Also, the O_CREAT | O_EXCEL trick used to create files atomically may not work on old linux kernels while writing to an NFS.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

seqfile-0.2.0.tar.gz (56.0 kB view details)

Uploaded Source

File details

Details for the file seqfile-0.2.0.tar.gz.

File metadata

  • Download URL: seqfile-0.2.0.tar.gz
  • Upload date:
  • Size: 56.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for seqfile-0.2.0.tar.gz
Algorithm Hash digest
SHA256 fc2831537ee3547c9fc1e8d21e28c777e6cf52d2d07d5f8d51b241d57e955154
MD5 6820e3d8a341b2954931a98bbe85baba
BLAKE2b-256 1c9d2521a5126f9d7d92d5e5005490e80526bbb9b71228ff9621b2c3a122ccc3

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page