pysplitter 0.0.16

A python package that splits large files into smaller chunks.

Tutorial using the pysplit package

pysplit is a Python package used for splitting large files into smaller chunks.

Currently the default chunk size is 100MB. This size was chosen to work around GitHub's upload file size limit.

Install the latest version of pysplitter

Run the cell below to ensure you have the latest version of pysplitter installed on your machine.

# !pip install --upgrade _pysplitter

Import required packages

import pysplitter as pysp
import numpy as np

Import helpful packages

import sys
import os

Create a numpy array that will exceed 100MB when saved to disk.

The numeric values of the data are not important. Random values were used for convenience only.

dim = 250
num = int(dim * dim * dim)
x = np.random.normal(size=num).reshape(dim, dim, dim)
x.shape

(250, 250, 250)

Save numpy array to disk and list directory contents

np.save('x.npy', x)

os.listdir()

['.ipynb_checkpoints',
 '1-split-unsplit-tutorial.ipynb',
 'x.npy']

Display size of file on disk

size = os.path.getsize('x.npy')
print(f'{size / 1e6} MB')

125.000128 MB

As many people may know, GitHub will not allow files exceeding 100 MB to be uploaded.

Use the commands below to split the original (and too large) file into multiple .split files.

Currently the default split size is <= 100 MB, but this may become a variable paramter in furture distributions.

os.listdir()

['.ipynb_checkpoints',
 '1-split-unsplit-tutorial.ipynb',
 'x(unsplit).npy',
 'x.npy']

src = 'x.npy'
pysp.split(src)

2 file(s) written.

Check file size of the two chunks that were just written.

os.listdir()

['.ipynb_checkpoints',
 '1-split-unsplit-tutorial.ipynb',
 'x(unsplit).npy',
 'x.npy',
 'x0000.npy.split',
 'x0001.npy.split']

print(os.path.getsize('x0000.npy.split') / 1e6, 'MB')

100.0 MB

print(os.path.getsize('x0001.npy.split') / 1e6, 'MB')

25.000128 MB

As is clearly shown from the output of the above cells, both chunks are <= 100MB. This means that this data can now pushed to GitHub as any other file would.

Recombine the data chunks back into a single file

search_pattern = './x*.split'
dst = '.'
pysp.unsplit(search_pattern, dst, validate=True, orig_src=src)

File reconstructed without loss: True

os.listdir()

['.ipynb_checkpoints',
 '1-split-unsplit-tutorial.ipynb',
 'x(unsplit).npy',
 'x.npy',
 'x0000.npy.split',
 'x0001.npy.split']

x_unsplit = np.load('x(unsplit).npy')
x_unsplit.shape

(250, 250, 250)

Show that the manipulated data x_unsplit is the same as the original data x.

np.allclose(x, x_unsplit)

True