dalimil 2.0.1

CLI to archive files into time related containers (directories or archives).

Dalimil

archive files into time related containers (directories or archives).

Have a directory of files. In this case they have datetime encoded in file name:

$ ls
2017-09-27T00_27_05Z8H47_U.xml
2017-09-27T00_43_00ZLJQASR.xml
2017-09-27T00_46_41ZPMZOpb.xml
2017-09-27T00_50_07Z7KiJAe.xml
2017-09-27T00_50_17ZXj7pCs.xml
2017-09-27T01_03_09Z_LuWzf.xml
2017-09-27T01_07_27ZE_Mo1_.xml
2017-09-27T01_12_22Zw1A_di.xml
2017-09-27T01_15_32ZsH7mLr.xml
2017-09-27T01_31_26ZJnFw8F.xml
...

Let dalimil archive the files:

$ dalimil -a move2zip *.xml
...
...
...
2017-10-02 09:53:02.412068 - Dalimil completed his Chronicle.

Default time period is one day and move2zip is archiving one time period into one zip file put into directory tree grouped per year and month:

$ tree archive
archive
├── year-2016
│   ├── month-08
│   │   ├── 2016-08-24.zip
│   │   ├── .....
│   │   └── 2016-08-31.zip
│   ├── month-09 ...
│   ├── month-10 ...
│   ├── month-11 ...
│   └── month-12 ...
└── year-2017
    ├── month-01 ...
    ├── month-02 ...
    ├── month-03 ...
    ├── month-04 ...
    ├── month-05 ...
    ├── month-06 ...
    ├── month-07 ...
    ├── month-08 ...
    ├── month-09
    │   └── .....
    └── month-10
        └── 2017-10-01.zip

move2zip action has packed all past periods (in this case a day) into daily zip archives, put them into archive tree and removed original files from source directory.

Files for current period (by default today) are left untouched in source directory.

Prerequisities

Python 2.6, 2.7, 3.3, 3.4, 3.5 or 3.6

Runs on Windows as well as on Linux.

Warning

Files can be deleted - Use it at your own risk!!!

Some actions are deleting files (after they are processed), so be aware of it.

The tool was carefully designed and is used regularly for couple of years without problems so it shall work well.

But it is a software created by human being.

Install

Use pip:

$ pip install dalimil

After that, command dalimil shall be available.

Usage

$ dalimil -h

usage: dalimil [-h]
               [-action {list,move2dir,move2zip,move2targz,copy2dir,copy2zip,copy2targz}]
               [-destination DESTINATION] [-time {modified,pattern}]
               [-pattern TIME_DETECTION_PATTERN] [-incomplete]
               source [source ...]

dalimil tool organizes files into time related containers (directories or archives).
Note: This command never starts anything by itself, it runs only once per call.

So called Dalimil wrote Chronicle of Dalimil, sorting past and current events.
See http://en.wikipedia.org/wiki/Chronicle_of_Dalimil

positional arguments:
  source                Unix shell pattern for selecting files to archive.
                        (defaults to all files in current dir)

optional arguments:
  -h, --help            show this help message and exit
  -action {list,move2dir,move2zip,move2targz,copy2dir,copy2zip,copy2targz}
                        Defines action to do with files organized into time
                        related containers. (default: list)
  -destination DESTINATION
                        Time formated pattern for resulting container
                        (default: archive/year-%Y/month-%m/%Y-%m-%d.zip)
  -time {modified,pattern}
                        Method, how time of file is detected, options:
                        modified, pattern (default: modified)
  -pattern TIME_DETECTION_PATTERN
                        Pattern, detecting time from filename. Effective only,
                        if -time_detection_method is "pattern". Path is
                        ignored, first part of filename must fit, useless end
                        of filename can be omitted. (default:
                        %Y-%m-%dT%H_%M_%S))
  -incomplete           Allows creation of containers for periods, which are
                        not yet completed (default: False))

Files are selected using Unix shell like syntax using *, ?, [seq] and [!seq]
Finally, files are placed container, which is archive file or end leaf directory.
Warning: File selection pattern can select files from multiple directories.
If final container rejects storing duplicate names, duplicates are skipped.

Time is detected from file modification or creation time, or decoded from filename.

Resulting containers are defined by time formating pattern.

Time formating patters for target path and container name:
Defines path and file name, which can be created from related file time.
    %c Locale's appropriate date and time representation.
    %d Day of the month as a decimal number [01,31].
    %f Microsecond as a decimal number [0,999999], zero-padded on the left
    %H Hour (24-hour clock) as a decimal number [00,23].
    %j Day of the year as a decimal number [001,366].
    %m Month as a decimal number [01,12].
    %M Minute as a decimal number [00,59].
    %S Second as a decimal number [00,61].
    %U Week number of the year (Sunday as the first day of the week) as a decimal number [00,53]. All days in a new year preceding the first Sunday are considered to be in week 0.
    %w Weekday as a decimal number [0(Sunday),6].
    %W Week number of the year (Monday as the first day of the week) as a decimal number [00,53]. All days in a new year preceding the first Monday are considered to be in week 0.
    %y Year without century as a decimal number [00,99].
    %Y Year with century as a decimal number.
    %z UTC offset in the form +HHMM or -HHMM (empty string if the the object is naive).
    %Z Time zone name (empty string if the object is naive).
For more and details see bottom of page http://docs.python.org/library/datetime.html
Samples: pattern => resulting path + archive name:
    "archive/%Y-%m-%dT%H.zip" => "archive/2010-02-28T13.zip"
    "archive/%Y/%m/%d.zip" => "archive/2010/02/28.zip"
    "archive/%Y/week-%W.zip" => "archive/2010/week-10.zip"
default value is:
    "archive/year-%Y/month-%m/%Y-%m-%d.zip" => "archive/year-2010/month-08/2010-08-28.zip"

Containers contain flat structure without deeper directory tree.
Source files can be finally deleted or left as they are.
Use action list (default) to see expected result without endangering files.

Existing containers are never touched, if they are found, *_1.* etc. is used.

Reading command line parameters from file: write arguments into text file,
each prefix and each value on separate lines like
    ------(quotation of my.cfg start)
    -action
    movetozip
    D:\my files with spaces\data\2010-0[789]\*.xml
    E:/other/location/data\2010-0[789]\*.xml
    (quotation of my.cfg end)------
Then from command line
    dalimil -incomplete @my.cfg
will read it.
Mixing command line arguments and others from file(s) is possible.

Examples:
Dry test of archiving *.xml from current folder without touching the files
    dalimil *.xml

Move the *.xml fles into subdir archive/year-2010/month-08/2010-08-28.zip etc.
Current period are skipped
    dalimil -a move2zip *.xml

Move there all files, including current period
    dalimil -a move2zip -incomplete *.xml

Copy the *.xml files into dir structure without zipping (dirs keep the .zip extension)
    dalimil -a copy2dir *.xml

Move the *.xml files into dir structure of style archive/year-2010/month-08/2010-08-28
    dalimil -a move2dir -d archive/year-%Y/month-%m/%Y-%m-%d *.xml

Move to archives, detecting time of files from file names
Expecting file names notes-201010251325_abc.xml
    dalimil -t pattern -p notes-%Y%m%d%H%M -a move2zip *.xml