Skip to main content

Write to an indexed sequence of files using the standard Python file API

Project description

File sequences

A FileSequence allows you to write to multiple files using standard Python file descriptor read / write functionality.

You specify the file size limit and naming scheme when opening the sequence, but the library handles creating new files as needed.

Each call to the opened FileSequence’s write function will potentially create a new file, if the chunk you want to write will push the file over the limit. So if you want to only split files on newlines, call write() once for each line. If you want behavior more like BSD’s split command, you can write one byte at a time. Though, at that rate, split is probably the better choice.

Installation PyPI version

pip install filesequence


You can simply use a FileSequence object as if it were a file.

  • returns a FileSequence object.
  • my_file_sequence.write(line) takes a line and writes it to the next available file.

Note that FileSequence requires with wrapping, as opposed to the Python built-in open():

import filesequence

filenames = filesequence.interpolator('numbers-%02d.txt', xrange(1000))

with, 1000000) as out:
    for a in xrange(1000):
        for b in xrange(1000):
            out.write('# %d * %d = %d\n' % (a, b, a * b))

Now you have a huge multiplication table in 20 different files that are 1MB or less! So awesome!

Want to keep going?

filenames = filesequence.interpolator('numbers-%02d.txt', xrange(1000))

with, 1000000, 'a') as out:

The ‘a’ flag will make the sequence jump to the last existing file, and start writing from there.


A filesequence script will be installed to your PATH. This script reads STDIN line by line and command line arguments for the filename pattern and filesize limit (see filesequence --help), and writes out a series of files of at most that filesize and without breaking any lines.

$ filesequence --help

usage: [-h] [--limit LIMIT] [--pattern PATTERN] [--version]

Write STDIN into a sequence of files, splitting only at newlines

optional arguments:
  -h, --help         show this help message and exit
  --limit LIMIT      Maximum bytes per file (default: 50000000)
  --pattern PATTERN  Filename string pattern: generate filenames in sequence
                     by interpolating `pattern %`
                     (default: split.%02d)
  --version          show program's version number and exit


  • Support reading (flags r and r+).


This package is published to PyPI at

Typical publish process:

  1. pandoc -o README.rst
  2. If needed, git commit ...
  3. npm version patch
  4. git push
  5. python register sdist upload


Continuous integration:

Travis CI Build Status

Or run tests locally (after installing):



Copyright (c) 2013 Christopher Brown. MIT Licensed.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

filesequence-0.1.12.tar.gz (6.4 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page