scolp 0.1.5

Streaming Column Printer

Scolp

Introduction

Scolp is Streaming Column Printer for Python 3.6 or later.

Scolp let you easily pretty-print masses of tabular data in a streaming fashion - each value is printed when available, without waiting for end of data. It is perfect for apps that need to print progress reports in columns.

Main features:

• Auto-adjusting column width according to the largest value so far or column header width.

• Control verbosity of printing by:

  • printing 1 of each n rows
  • printing no more than 1 row per n seconds

• Control format of printed values by:

  • setting global defaults
  • setting defaults per variable type (int, float, str, datetime)
  • setting explicit formatting per column

• Control alignment of printed values:

  • left
  • right
  • center
  • auto: numbers to the right, strings or other types to the left.

• Control cosmetics of columns (initial width, padding fill char, alignment, and more..) by:

  • setting global defaults
  • setting explicit formatting per column

• Control column title printing style:

  • Inline in each row
  • As headers, repeating each n rows

• Easily print row count or time since execution started without need to keep track of those values yourself.

Examples

Example 1

Lets start with a simple country statistics output using default settings:

import scolp

scolper = scolp.Scolp()
scolper.config.add_columns("country", "population (mil)", "capital city", "life expectancy (female)",
                           "life expectancy (male)", "fertility rate")
scolper.print("Netherlands", 16.81, "Amsterdam", 83, 79, 1.5,
              "China", 1350.0, "Beijing", 76, 72, 1.8,
              "Israel", 7.71, "Jerusalem", 84, 80, 2.7,
              "Nigeria")
scolper.print(174.51)

Output:

(Note how column width is auto adjusting, line breaks are printed automatically after last column, and each value is printed immediately without waiting for end of row)

country |population (mil)|capital city|life expectancy (female)|life expectancy (male)|fertility rate
--------|----------------|------------|------------------------|----------------------|--------------
Netherlands|          16.810|Amsterdam   |                      83|                    79|         1.500

country    |population (mil)|capital city|life expectancy (female)|life expectancy (male)|fertility rate
-----------|----------------|------------|------------------------|----------------------|--------------
China      |       1,350.000|Beijing     |                      76|                    72|         1.800
Israel     |           7.710|Jerusalem   |                      84|                    80|         2.700
Nigeria    |         174.510|

Example 2

Lets build a program that find prime numbers. We will print the count of primes we found so far and the last prime found.

import datetime, scolp

def is_prime(num):
    return 2 in [num, 2 ** num % num]

scolper = scolp.Scolp()
scolper.config.add_columns("time", "elapsed", "inspected_count", "prime_count", "last", "progress %")
scolper.config.output_each_n_seconds = 1

prime_count = 0
last_prime = None
i = 9_999_800
target_count = 30
while prime_count < target_count:
    if is_prime(i):
        last_prime = i
        prime_count += 1
    progress = prime_count / target_count * 100
    scolper.print(datetime.datetime.now(), scolper.elapsed_since_init(),
                  scolper.row_index + 1, prime_count, last_prime, progress)
    i += 1

Output:

(Note how the header repeats, the column width auto-expanding and the numbers are aligned to the right)

time    |elapsed |inspected_count|prime_count|last    |progress %
--------|--------|---------------|-----------|--------|----------
2019-06-05 11:49:31.271191|0:00:00 |              1|          0|None    |     0.000

time                      |elapsed |inspected_count|prime_count|last    |progress %
--------------------------|--------|---------------|-----------|--------|----------
2019-06-05 11:49:32.306225|0:00:01 |             27|          1|9,999,823|     3.333
2019-06-05 11:49:33.325694|0:00:02 |             53|          1|9,999,823|     3.333
2019-06-05 11:49:34.341678|0:00:03 |             79|          3|9,999,877|    10.000
2019-06-05 11:49:35.378966|0:00:04 |            105|          6|9,999,901|    20.000
2019-06-05 11:49:36.399298|0:00:05 |            131|          8|9,999,929|    26.667
2019-06-05 11:49:37.415522|0:00:06 |            157|         11|9,999,943|    36.667
2019-06-05 11:49:38.450551|0:00:07 |            183|         13|9,999,973|    43.333
2019-06-05 11:49:39.478987|0:00:08 |            209|         14|9,999,991|    46.667
2019-06-05 11:49:40.485409|0:00:09 |            233|         15|10,000,019|    50.000

time                      |elapsed |inspected_count|prime_count|last      |progress %
--------------------------|--------|---------------|-----------|----------|----------
2019-06-05 11:49:41.508298|0:00:10 |            259|         15|10,000,019|    50.000
2019-06-05 11:49:42.543115|0:00:11 |            283|         16|10,000,079|    53.333
2019-06-05 11:49:43.555733|0:00:12 |            306|         17|10,000,103|    56.667
2019-06-05 11:49:44.572379|0:00:13 |            328|         18|10,000,121|    60.000
2019-06-05 11:49:45.574066|0:00:14 |            349|         20|10,000,141|    66.667
2019-06-05 11:49:46.583462|0:00:15 |            372|         21|10,000,169|    70.000
2019-06-05 11:49:47.594724|0:00:16 |            396|         22|10,000,189|    73.333
2019-06-05 11:49:48.639124|0:00:17 |            420|         22|10,000,189|    73.333
2019-06-05 11:49:49.661211|0:00:18 |            441|         24|10,000,229|    80.000
2019-06-05 11:49:50.691037|0:00:19 |            463|         27|10,000,261|    90.000

time                      |elapsed |inspected_count|prime_count|last      |progress %
--------------------------|--------|---------------|-----------|----------|----------
2019-06-05 11:49:51.721844|0:00:20 |            487|         28|10,000,271|    93.333
2019-06-05 11:49:52.733437|0:00:22 |            510|         29|10,000,303|    96.667
2019-06-05 11:49:53.750463|0:00:23 |            534|         29|10,000,303|    96.667

Example 3

Now, lets change the code of the previous example to add a bit of custom formatting:

scolper = scolp.Scolp()
scolper.config.add_column("time", width=20)
scolper.config.add_columns("elapsed",
                           "inspected_count",
                           "prime_count")
scolper.config.add_column("last", width=11)
scolper.config.add_column("progress", fmt="{:.1%}")
scolper.config.output_each_n_seconds = 1
scolper.config.header_repeat_row_count_first = 0
scolper.config.default_column.column_separator = " "
scolper.config.default_column.type_to_format[datetime.datetime] = "{:%Y-%m-%d %H:%M:%S}"

prime_count = 0
last_prime = None
i = 9_999_800
target_count = 30
while prime_count < target_count:
    if is_prime(i):
        last_prime = i
        prime_count += 1
    progress = prime_count / target_count
    scolper.print(datetime.datetime.now(), scolper.elapsed_since_init(),
                  scolper.row_index + 1, prime_count, last_prime, progress)
    i += 1

Output:

time                 elapsed  inspected_count prime_count last        progress
-------------------- -------- --------------- ----------- ----------- --------
2019-06-05 11:54:46  0:00:00                1           0 None            0.0%
2019-06-05 11:54:47  0:00:01               23           0 None            0.0%
2019-06-05 11:54:48  0:00:02               45           1   9,999,823     3.3%
2019-06-05 11:54:49  0:00:03               67           2   9,999,863     6.7%
2019-06-05 11:54:50  0:00:04               90           5   9,999,889    16.7%
2019-06-05 11:54:51  0:00:05              115           7   9,999,907    23.3%
2019-06-05 11:54:52  0:00:06              139          10   9,999,937    33.3%
2019-06-05 11:54:53  0:00:07              164          11   9,999,943    36.7%
2019-06-05 11:54:54  0:00:08              188          13   9,999,973    43.3%
2019-06-05 11:54:55  0:00:09              212          14   9,999,991    46.7%

time                 elapsed  inspected_count prime_count last        progress
-------------------- -------- --------------- ----------- ----------- --------
2019-06-05 11:54:56  0:00:10              237          15  10,000,019    50.0%
2019-06-05 11:54:57  0:00:11              261          15  10,000,019    50.0%
2019-06-05 11:54:58  0:00:12              284          16  10,000,079    53.3%
2019-06-05 11:54:59  0:00:13              308          17  10,000,103    56.7%
2019-06-05 11:55:00  0:00:14              331          18  10,000,121    60.0%
2019-06-05 11:55:01  0:00:15              355          20  10,000,141    66.7%
2019-06-05 11:55:02  0:00:16              379          21  10,000,169    70.0%
2019-06-05 11:55:03  0:00:17              403          22  10,000,189    73.3%
2019-06-05 11:55:04  0:00:18              426          23  10,000,223    76.7%
2019-06-05 11:55:05  0:00:20              448          25  10,000,247    83.3%

time                 elapsed  inspected_count prime_count last        progress
-------------------- -------- --------------- ----------- ----------- --------
2019-06-05 11:55:06  0:00:21              471          27  10,000,261    90.0%
2019-06-05 11:55:07  0:00:22              493          28  10,000,271    93.3%
2019-06-05 11:55:08  0:00:23              516          29  10,000,303    96.7%
2019-06-05 11:55:09  0:00:24              539          29  10,000,303    96.7%

Example 4

Lets build an HTTP big-file downloader.

import datetime, urllib3, scolp

url = "http://speedtest.tele2.net/100MB.zip"
path = "downloaded.tmp"
chunk_size_bytes = 1000

scolp_cfg = scolp.Config()
scolp_cfg.add_column("time", fmt="{:%H:%M:%S}")
scolp_cfg.add_column("elapsed")
scolp_cfg.add_column("downloaded", width=16, fmt="{:,} B")
scolp_cfg.add_column("speed", width=14, pad_align=scolp.Alignment.RIGHT, type_to_format={float: "{:,.1f} kB/s"})

scolp_cfg.output_each_n_seconds = 1
scolp_cfg.title_mode = scolp.TitleMode.INLINE
scolp_cfg.default_column.column_separator = "  |  "

scolper = scolp.Scolp(scolp_cfg)

http = urllib3.PoolManager()
r = http.request('GET', url, preload_content=False)

dl_bytes = 0


def progress_update():
    elapsed_sec = scolper.elapsed_since_init().total_seconds()
    speed_kbps = dl_bytes / elapsed_sec / 1000 if elapsed_sec > 0 else "unknown"
    scolper.print(datetime.datetime.now(), scolper.elapsed_since_init(), dl_bytes, speed_kbps)


with open(path, 'wb') as out:
    while True:
        data = r.read(chunk_size_bytes)
        if not data:
            break
        out.write(data)
        dl_bytes += len(data)
        progress_update()

scolper.force_print_next_row()
progress_update()
r.release_conn()

Output:

time=14:30:11  |  elapsed=0:00:00   |  downloaded=         1,000 B  |  speed=       unknown
time=14:30:12  |  elapsed=0:00:01   |  downloaded=       801,000 B  |  speed=    801.0 kB/s
time=14:30:13  |  elapsed=0:00:02   |  downloaded=     1,743,000 B  |  speed=    871.5 kB/s
time=14:30:14  |  elapsed=0:00:03   |  downloaded=     2,758,000 B  |  speed=    919.3 kB/s
time=14:30:15  |  elapsed=0:00:04   |  downloaded=     3,779,000 B  |  speed=    944.8 kB/s
time=14:30:16  |  elapsed=0:00:05   |  downloaded=     4,794,000 B  |  speed=    958.8 kB/s
time=14:30:17  |  elapsed=0:00:06   |  downloaded=     5,809,000 B  |  speed=    968.2 kB/s
time=14:30:18  |  elapsed=0:00:07   |  downloaded=     6,824,000 B  |  speed=    974.9 kB/s
time=14:30:19  |  elapsed=0:00:08   |  downloaded=     7,839,000 B  |  speed=    979.9 kB/s
time=14:30:20  |  elapsed=0:00:09   |  downloaded=     8,857,000 B  |  speed=    984.1 kB/s
time=14:30:21  |  elapsed=0:00:10   |  downloaded=     9,799,000 B  |  speed=    979.9 kB/s
time=14:30:22  |  elapsed=0:00:11   |  downloaded=    10,814,000 B  |  speed=    983.1 kB/s
time=14:30:23  |  elapsed=0:00:12   |  downloaded=    11,838,000 B  |  speed=    986.5 kB/s
time=14:30:24  |  elapsed=0:00:13   |  downloaded=    12,855,000 B  |  speed=    988.8 kB/s
time=14:30:25  |  elapsed=0:00:14   |  downloaded=    13,870,000 B  |  speed=    990.7 kB/s
time=14:30:26  |  elapsed=0:00:15   |  downloaded=    14,891,000 B  |  speed=    992.7 kB/s
time=14:30:27  |  elapsed=0:00:16   |  downloaded=    15,906,000 B  |  speed=    994.1 kB/s
time=14:30:28  |  elapsed=0:00:18   |  downloaded=    25,600,000 B  |  speed=  1,422.2 kB/s
time=14:30:29  |  elapsed=0:00:19   |  downloaded=    37,146,000 B  |  speed=  1,955.1 kB/s
time=14:30:30  |  elapsed=0:00:20   |  downloaded=    47,847,000 B  |  speed=  2,392.3 kB/s
time=14:30:31  |  elapsed=0:00:21   |  downloaded=    60,962,000 B  |  speed=  2,903.0 kB/s
time=14:30:32  |  elapsed=0:00:22   |  downloaded=    72,931,000 B  |  speed=  3,315.0 kB/s
time=14:30:33  |  elapsed=0:00:23   |  downloaded=    85,094,000 B  |  speed=  3,699.7 kB/s
time=14:30:34  |  elapsed=0:00:24   |  downloaded=   104,857,600 B  |  speed=  4,369.1 kB/s

Requirements

Scolp has no 3rd party requirements other than Python 3.6 or later.

Getting Started

Scolp is available via PyPi and can be installed using:

pip install scolp.

Todo

• Document public API of the library
• Support colors