pnu-adsv 1.0.1

Analyze delimiter-separated values files

Installation

Depending on if you want only this tool, the full set of PNU tools, or PNU plus a selection of additional third-parties tools, use one of these commands:

pip install pnu-adsv
pip install PNU
pip install pytnix

ADSV(1)

NAME

adsv - Analyze delimiter-separated values files

SYNOPSIS

adsv [-d|--delimiter CHAR] [-e|--encoding STRING] [-f|--fields LIST] [-F|--flatten] [-h|--hide INT] [-m|--min INT] [-M|--max INT] [-t|--top INT] [--debug] [--help|-?] [--version] [--] filename [...]

DESCRIPTION

The adsv utility analyzes delimiter-separated values files, such as Comma-Separated Values .csv or Tab-Separated Values .tsv files, and either prints information about their structure and the data in each of their fields, or prints a selection of fields in the order requested.

The information gathered are:

• for the file:
  • the character set encoding
  • the CSV dialect (characters used for delimiting, quoting, escaping or lines terminating. Plus the use or not of double quoting)
  • the presence or not of a headers line
  • the number of lines and fields
• for each field:
  • its number and header
  • the number of distinct values
  • the values type (strings, integers, floating numbers, complex numbers, date and time (whatever their format))
  • the values by descending count
  • the values range by ascending order using the detected type (useful for numbers and dates)

When analyzing a DSV dataset, this allows for a quick and automated way of getting global information about the contents, and explore any oddities...

There are options:

• to control and limit what is printed (-h|--hide, -m|--min, -M|--max and -t|--top),
• to avoid (or correct) the detection of the character set encoding and delimiter (-d|--delimiter, -e|--encoding):
  • the character set detection can take a long time with big files, so if you know that the file is in "Windows-1252" or "utf-8" encoding, it's quicker to say it...

If you use the -f|--fields option, you'll skip printing the file analysis, and instead print the selected fields in the order requested, using the detected delimiting, quoting and escaping characters.

If you encounter multi-lines fields and want to "flatten" them to single lines, you can use the -F|--flatten option for that.

OPTIONS

Options	Use
-d|--delimiter CHAR	Specify delimiter to be CHAR
-e|--encoding STRING	Specify charset encoding to be STRING (because detecting encoding can take a long time!)
-f|--fields LIST	Extract LISTed fields values in given order (ex: 6,2-4,1 with fields numbered from 1)
-F|--flatten	Make multi-lines fields single line
-h|--hide INT	Hide the display of distinct values above INT % (default is 20%)
-m|--min INT	Only display distinct values whose count >= INT (default is to display all distinct values)
-M|--max INT	Only display INT lines of distinct values (default is to display all distinct values, within the hide limit)
-t|--top INT	Only display the top/bottom INT lines of values (default is to display the 5 bottom and top lines)
--debug	Enable debug mode
--help|-?	Print usage and a short help message and exit
--version	Print version and exit
--	Options processing terminator

ENVIRONMENT

The ADSV_DEBUG environment variable can also be set to any value to enable debug mode.

EXIT STATUS

The adsv utility exits 0 on success, and >0 if an error occurs.

SEE ALSO

cut(1), file(1)

STANDARDS

The adsv utility is not a standard UNIX command.

This implementation tries to follow the PEP 8 style guide for Python code.

The DSV dialects that can be handled are those compatible with RFC 4180: Common Format and MIME Type for Comma-Separated Values (CSV) Files.

PORTABILITY

Tested OK under Windows.

HISTORY

This implementation was made for the PNU project.

I do this kind of analysis with each dataset I have to work with. Last time I did that, I decided that it was about time to fully automate the process, especially as I was working with fields containing multi-lines values...

LICENSE

It is available under the 3-clause BSD license.

AUTHORS

Hubert Tournier

CAVEATS

Using "Sep=X" as a first line in order to set the X character as a delimiter is not supported.

There is no support either for potential commented lines inside the data (for example, with /etc/passwd files under Unix), but it's not part of any recognized DSV dialect anyway.