Skip to main content

Treat lines of a file as elements of a set

Project description

Introduction

Lines is a simple program that allows you to manipulate lines of a file as if they were members of a set. It also provides a few other useful functions to analyse such files.

Set operations

Given two files

file1 containing

a b c d

and file2 with

c d e f

It’s possible to do things like

  • Unions

lines -u file1 file2

gives

a b c d e f

  • Intersections

lines -i file1 file2

gives

c d

  • Difference (All elements in file1 that are not in file2).

lines -d file1 file2

gives

a b

  • Symmetric difference (All elements present in only one of the

    sets).

lines -s file1 file2

gives

a b e f

Other Operations

These are a few other operations which I’ve found useful

  • Squeeze blanks

This operation squeezes out the blank lines in a file.

So, If you run lines –squeeze file1 where file1 looks like this

a b c

d

f

You’d get

a b c d f

  • Identify Patterns

This partitions the elements of the set into subsets all of whose members have an upper bound on the levenshtein distance from each other. This is useful to identify patterns in the input file.

So, if I have a file examples/f6 that looks like this

Archive.001-of-020.part Archive.002-of-020.part Archive.003-of-020.part Archive.004-of-020.part Archive.005-of-020.part Archive.006-of-020.part Archive.007-of-020.part .Archive.008-of-020.part.zbnrw Archive.009-of-020.part Archive.010-of-020.part Archive.011-of-020.part Archive.012-of-020.part Archive.013-of-020.part Archive.014-of-020.part Archive.015-of-020.part Archive.016-of-020.part Archive.017-of-020.part Archive.018-of-020.part Archive.019-of-020.part Archive.020-of-020.part

I can run python lines.py –patterns -l 5 examples/f6 and get

19 elements 1 elements - {‘.Archive.008-of-020.part.zbnrw’}

The -l 5 is to set the upper bound on the levenshtein distance to 5. The -p option allows us to specify an “outlier percentage”. If the number of elements in a subset is below this, it will print all the elements of the subset. This is useful to see the items that don’t match the general pattern in the file.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lines-0.1.0-a.tar.gz (2.6 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page