Skip to main content

adding tiny resume function, for csv reading iterator

Project description

cssresumable

Adding tiny resume function for your csv reading iterator.

examples

00main.py

import sys
import os
import random
from io import StringIO
from csvresumable import DictReader

data = """\
id,name
1,foo
2,bar
3,boo
"""

random.seed(int(os.environ.get("SEED", "0")))
for row in DictReader(StringIO(data)):
    n = random.random()
    if n < 0.5:
        print("fail {}".format(n), file=sys.stderr)
        sys.exit(0)
    print("\t", row["name"])
print("finished")

resuming until success.

SEED=0 python 00main.py
fail 0.420571580830845
	 foo
	 bar
SEED=1 RESUME=1 python 00main.py
fail 0.13436424411240122
SEED=2 RESUME=1 python 00main.py
	 boo
finished
SEED=5 python 00main.py
	 foo
	 bar
	 boo
finished

or setting with resume option (run this script, such as python 01main.py --resume)

--- 00main.py	2018-06-16 04:19:51.573578443 +0900
+++ 01main.py	2018-06-16 04:20:04.760554103 +0900
@@ -4,17 +4,23 @@
 from io import StringIO
 from csvresumable import DictReader

-data = """\
+data = """
 id,name
 1,foo
 2,bar
 3,boo
 """

+import argparse
+parser = argparse.ArgumentParser()
+parser.add_argument("--seed", default=0, type=int)
+parser.add_argument("--resume", action="store_true")
+args = parser.parse_args()
+
 random.seed(int(os.environ.get("SEED", "0")))
-for row in DictReader(StringIO(data)):
+for row in DictReader(StringIO(data), resume=args.resume):
     n = random.random()
-    if n < 0.5:
+    if n > 0.8:
         print("fail {}".format(n), file=sys.stderr)
         sys.exit(0)
     print("\t", row["name"])

with multi files

from csvresumable import DictReader

files = ["a.csv", "b.csv"]
for f in files:
    r = DictReader(f)
    for row in r:
        do_something(row)

more

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
csvresumable-0.0.3.post2-py2.py3-none-any.whl (6.4 kB) Copy SHA256 hash SHA256 Wheel py2.py3
csvresumable-0.0.3.post2.tar.gz (4.8 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page