file-utils 0.1.3

This is a python library to parse files, it's giving tools to easily read a file with efficiency. It's based on linux commands like grep, sed, cat, head, tail and tested with them.

file utils

Table of contents

• Intro
• Installation
  • python
  • rust
• Before starting
• Arguments-explaination

Examples:

• WithEOL: python:

  • Example-file
  • Examples:
    • Example-simple-head
    • Example-simple-tail
    • Example-simple-between
    • Example-simple-parse
    • Example-simple-count_lines
    • Example-remove_empty_string
    • Example-regex_keep
    • Example-regex_pass
    • Example-restrict

• WithCustomDelims: python:

  • How to use it?
  • What delim can be used?
  • With more than one delimiter?

• How to use the rust crate?

• Python class

• Rust Structure

• Structure

Intro

This package allows to read/parse a file in python. When should we use this package? If your file is really big (> 100 000 lines), because if you want to parse a file in python you'll write:

f = open("my_file", "r")
buffer: str = f.read()
...

or:

f = open("my_file", "r")
for line in f.readlines():
    ...

• With the first one, there is a memory issue because you must save the entire file into a buffer.
• With the second one, there is a time issue because a loop can be very slow in python.

So, this package gives tools to easily read a file with efficiently. It's based on Linux tools like grep, sed, cat, head, tail and tested with them.
WithEOL class as the same memory problem as the first example. If you want to resolve it, you must use WithCustomDelims with the "\n" delimiter.
So, why I keep WithEOL?
WithEOL is helping me to test the code, it's using a built in rust function and I'm using it as a reference to compare with WithCustomDelims.

Installation

python

With pypi:

pip install file-utils

From source:

maturin develop

rust

cargo add file_utils

Before-starting

This package is ASCII/UTF-8 compliant, all others encoded files will not work...

Arguments-explaination

• path: the path to the file
• remove_empty_string: ignore the empty string "[ ]*"
• n: get n lines with tail/head
• n1: the beginning line to take with between
• n2: the last line to take with between
• restrict: if enable, if we have last N lines, it just keep the regex in those lines. If not enable, it takes last N regex

with regex:

• regex_keep: list of regex to keep
• regex_pass: list of regex to pass/ignore

WithEOL-python

Example-file

We will use this example file test.txt

With cat -e test.txt:

[Warning]:Entity not found$
[Error]:Unable to recover data$
[Info]:Segfault$
[Warning]:Indentation$
[Error]:Memory leaks$
[Info]:Entity not found$
[Warning]:Unable to recover data$
  $
[Error]:Segfault$
[Info]:Indentation$
[Warning]:Memory leaks$

Example-simple-head-python

1\ Simple head (can be change to tail) Code:

import file_utils_lib

path: str = "my_path_to_file"
n: int = 2 # Number of lines to read

try:
    head: list = file_utils_lib.WithEOL.head(path=path, n=n)
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data']

Example-simple-tail-python

Code:

import file_utils_lib

path: str = "my_path_to_file"
n: int = 2 # Number of lines to read

try:
    tail: list = file_utils_lib.WithEOL.tail(path=path, n=n)
    print(tail)
except:
    print("Unable to open/read the file")

Stdout:

['[Info]:Indentation', '[Warning]:Memory leaks']

Example-simple-between-python

Code:

import file_utils_lib

path: str = "my_path_to_file"
n1: int = 2 # First line to read
n2: int = 4 # Last line to read

try:
    between: list = file_utils_lib.WithEOL.between(path=path, n1=n1, n2=n2)
    print(between)
except:
    print("Unable to open/read the file")

Stdout:

['[Error]:Unable to recover data', '[Info]:Segfault', '[Warning]:Indentation']

Example-simple-parse-python

Code:

import file_utils_lib

path: str = "my_path_to_file"

try:
    parse: list = file_utils_lib.WithEOL.parse(path=path)
    print(parse)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data', '[Info]:Segfault', '[Warning]:Indentation', '[Error]:Memory leaks', '[Info]:Entity not found', '[Warning]:Unable to recover data', '  ', '[Error]:Segfault', '[Info]:Indentation', '[Warning]:Memory leaks']

Example-simple-count_lines-python

Code:

import file_utils_lib

path: str = "my_path_to_file"

try:
    count: list = file_utils_lib.WithEOL.count_lines(path=path)
    print(count)
except:
    print("Unable to open/read the file")

Stdout:

11

Example-remove_empty_string-python

With remove_empty_string enable:
Code:

import file_utils_lib

path: str = "my_path_to_file"
n: int = 4 # First line to read

try:
    tail: list = file_utils_lib.WithEOL.tail(path=path, n=n, remove_empty_string=True)
    print(tail)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Unable to recover data', '[Error]:Segfault', '[Info]:Indentation', '[Warning]:Memory leaks']

With remove_empty_string disable (default option):
Code:

import file_utils_lib

path: str = "my_path_to_file"
n: int = 4 # First line to read

try:
    tail: list = file_utils_lib.WithEOL.tail(path=path, n=n, remove_empty_string=False)
    print(tail)
except:
    print("Unable to open/read the file")

Stdout:

['  ', '[Error]:Segfault', '[Info]:Indentation', '[Warning]:Memory leaks']

Example-regex_keep-python

Code:

import file_utils_lib

path: str = "my_path_to_file"
n: int = 4 # First line to read

try:
    head: list = file_utils_lib.WithEOL.head(path=path, n=n, remove_empty_string=False, regex_keep=["\[Warning\]:*", "\[Error\]:*"])
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data', '[Warning]:Indentation']

Why there is just 3 elements instead of 4? You should look at the restrict option

Example-regex_pass-python

Code:

import file_utils_lib

path: str = "my_path_to_file"
n: int = 4 # First line to read

try:
    head: list = file_utils_lib.WithEOL.head(path=path, n=n, remove_empty_string=False, regex_pass=["\[Warning\]:*", "\[Error\]:*"])
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Info]:Segfault']

Why there is just 3 elements instead of 4? You should look at the restrict option

Example-restrict-python

With restrict disable:
Code:

import file_utils_lib

path: str = "my_path_to_file"
n: int = 4 # First line to read

try:
    head: list = file_utils_lib.WithEOL.head(path=path, n=4, remove_empty_string=False, regex_keep=["\[Warning\]:*", "\[Error\]:*"], restrict=False)
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data', '[Warning]:Indentation', '[Error]:Memory leaks']

With restrict enbale(default):
Code:

import file_utils_lib

path: str = "my_path_to_file"
n: int = 4 # First line to read

try:
    head: list = file_utils_lib.WithEOL.head(path=path, n=4, remove_empty_string=False, regex_keep=["\[Warning\]:*", "\[Error\]:*"], restrict=True)
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data', '[Warning]:Indentation']

WithCustomDelims-python

How-to-use-it-python

It it like WithEOL but with a list of custom delimiter. For example:

import file_utils_lib

path: str = "my_path_to_file"
n: int = 2 # Number of lines to read

try:
    head: list = file_utils_lib.WithEOL.head(path=path, n=n)
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data']

has the same behavious as

import file_utils_lib

path: str = "my_path_to_file"
n: int = 2 # Number of lines to read

try:
    head: list = file_utils_lib.WithCustomDelims.head(path=path, n=n, delimiter=['\n])
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data']

So, you use it as same as WithEOL but with a list of custom delimiter.

What-delim-can-be-used

All string can be used like:

• ";"
• "abc"
• "éà"
• ::
• "小六号"
• "毫"

With-more-than-one-delimiter

If my file contains:

;À ;la ;;
pêche éèaux moules, @moules, ::小六号moules::Je n'veux小六号 plus ::y 
aller éèmaman小六号

We'll have with ";", "\n", "éè", "@", "小六号", "::"

import file_utils_lib

path: str = "my_path_to_file"

try:
    parse: list = file_utils_lib.WithCustomDelims.parse(path=path, delimiter=[";", "\n", "éè", "@", "::"])
    print(parse)
except:
    print("Unable to open/read the file")

Stdout

['', 'À ', 'la ', '', '', 'pêche ', 'aux moules, ', 'moules, ', '', 'moules', "Je n'veux", ' plus ', 'y ', 'aller ', 'maman', '']

How-to-use-the-rust-crate?

You must import the library with

use file_utils_lib::with_custom_delims::WithCustomDelims;

or

use file_utils_lib::with_eol::WithEOL;

Then, you can use the same functions as python because there are the same behavious.
Example:

use file_utils_lib::with_custom_delims::WithCustomDelims;

fn main() {
    let mut delimiters: Vec<String> = Vec::new();
    delimiters.push("\n".to_string());
    let n: usize = 10;
    let res: Vec<String> = WithCustomDelims::head(
        "my path".to_string(),
        n,
        delimiters,
        false,
        Vec::new(),
        Vec::new(),
        true,
        1024,
    );
}

has the same behaviour as

import file_utils_lib

path: str = "my_path_to_file"
n: int = 2 # Number of lines to read

try:
    head: list = file_utils_lib.WithEOL.head(path=path, n=n)
    print(head)
except:
    print("Unable to open/read the file")

Python-class

If we translate the rust into python, we'll have:

class WithEOL:
    # head: Read the n first lines
    # if n > (numbers of lines in the file) => return the whole file
    def head(path: str, n: int, \
                remove_empty_string: bool = False, \
                regex_keep: list = [] \
                regex_pass: list = [] \
                restrict: bool = True):
        ...

    # between: Read the lines [n1, n2]
    # if n1 > n2 => return an empty list
    # if n1 > (numbers of lines in the file) => return an empty list
    def between(path: str, n1: int, n2: int \
                remove_empty_string: bool = False, \
                regex_keep: list = [] \
                regex_pass: list = [] \
                restrict: bool = True):
        ...
    
    # tail: Read the n last lines
    # if n > (numbers of lines in the file) => return the whole file
    def tail(path: str, n: int, \
                remove_empty_string: bool = False, \
                regex_keep: list = [] \
                regex_pass: list = [] \
                restrict: bool = True):
        ...
    
    # parse: Read the whole file
    def parse(path: str, \ 
                remove_empty_string: bool = False \
                regex_keep: list = [] \
                regex_pass: list = []):
        ...

    # Count the number of lines
    def count_lines(path: str \
                    remove_empty_string: bool = False, \
                    regex_keep: list = [] \
                    regex_pass: list = []):
        ...

class WithCustomDelims:
    # head: Read the n first lines
    # if n > (numbers of lines in the file) => return the whole file
    def head(path: str, n: int, delimiter: list \
                remove_empty_string: bool = False, \
                regex_keep: list = [] \
                regex_pass: list = [] \
                restrict: bool = True \
                buffer_size: int = 1024):
        ...

    # between: Read the lines [n1, n2]
    # if n1 > n2 => return an empty list
    # if n1 > (numbers of lines in the file) => return an empty list
    def between(path: str, n1: int, n2: int, delimiter: list \
                remove_empty_string: bool = False, \
                regex_keep: list = [] \
                regex_pass: list = [] \
                restrict: bool = True \
                buffer_size: int = 1024):
        ...
    
    # tail: Read the n last lines
    # if n > (numbers of lines in the file) => return the whole file
    def tail(path: str, n: int, delimiter: list \
                remove_empty_string: bool = False, \
                regex_keep: list = [] \
                regex_pass: list = [] \
                restrict: bool = True \
                buffer_size: int = 1024):
        ...
    
    # parse: Read the whole file
    def parse(path: str, delimiter: list \
                remove_empty_string: bool = False \
                regex_keep: list = [] \
                regex_pass: list = [] \
                buffer_size: int = 1024):
        ...

    # Count the number of lines
    def count_lines(path: str, delimiter: list \
                    remove_empty_string: bool = False, \
                    regex_keep: list = [] \
                    regex_pass: list = [] \
                    buffer_size: int = 1024):
        ...

Rust-Structure

Take a look at https://docs.rs/file_utils/latest/file_utils_lib/

Structure

• src/: all sources files
• tests/: all tests for rust
• tests_files/: all files used for tests
• tests_python/: a python script to test