title-grabber-cristianrasch 0.2.0

Grabs page & article titles from lists of URLs contained in files passed in as arguments

title_grabber

Usage instructions

• Just feed it 1 or more files containing URLs (1 per line)

python -m title_grabber /abs/path/2/urls1.csv rel/path/2/urls2.csv

• Optionally, change the output file:

python -m title_grabber -o output.csv /abs/path/2/urls1.csv rel/path/2/urls2.csv

• See all available config options:

python -m title_grabber -h

usage: title_grabber [-h] [-o OUT_FILE] [--connect-timeout TIMEOUT]
                     [--read-timeout TIMEOUT] [-r RETRIES] [-t THREADS] [-d]
                     [FILES [FILES ...]]

positional arguments:
  FILES                 1 or more CSV files containing URLs (1 per line)

optional arguments:
  -h, --help            show this help message and exit
  -o OUT_FILE, --output OUT_FILE
                        Output file (defaults to out.csv)
  --connect-timeout TIMEOUT
                        HTTP connect timeout. Defaults to the value of the
                        CONNECT_TIMEOUT env var or 10
  --read-timeout TIMEOUT
                        HTTP read timeout. Defaults to the value of the
                        READ_TIMEOUT env var or 15
  --max-redirects REDIRECTS
                        Max. # of HTTP redirects to follow. Defaults to the
                        value of the MAX_REDIRECTS env var or 5
  -r RETRIES, --max-retries RETRIES
                        Max. # of times to retry failed HTTP reqs. Defaults to
                        the value of the MAX_RETRIES env var or 3
  -t THREADS, --max-threads THREADS
                        Max. # of threads to use. Defaults to the value of the
                        MAX_THREADS env var or the # of logical processors in
                        the system (8)
  -d, --debug           Log to STDOUT instead of to a file in the CWD.
                        Defaults to the value of the DEBUG env var or False
  -V, --version         Print program version and exit

dev setup instructions

1. Clone the project

git clone [e-mail]:cristianrasch/title_grabber.git

2. Create a new virtual environment for it

cd title_grabber && python3 -m venv venv

3. Install its dependencies

pip install -r requirements.txt

4. Run the test suite to make sure everything is set up OK

python -m unittest discover -v -s title_grabber/tests/