Skip to main content

Tool for bulk harvests of digitised newspaper articles from Trove

Project description

trove-newspaper-harvester

No installation required!

If you want to use the harvester without installing anything, just head over to the Trove Newspaper Harvester section in my GLAM Workbench.

Installation

pip install trove-newspaper-harvester

Use as a library

from trove-newspaper-harvester.core import prepare_query, Harvester

Generate a set of query parameters using prepare_query.

my_query = "https://trove.nla.gov.au/search/category/newspapers?keyword=wragge"
my_api_key = "mYSecREtkEy"

my_query_params = prepare_query(query=my_query, key=my_api_key)

Initialise the Harvester with your query parameters.

harvester = Harvester(query_params=my_query_params)

Start the harvest!

harvester.harvest()

If the harvest fails just run Harvester.harvest again.

See the core module documentation for more options and examples.

Use as a command-line tool

Before you do any harvesting you need to get yourself a Trove API key.

There are three basic commands:

  • start – start a new harvest
  • restart – restart a stalled harvest
  • report – view harvest details

Start a harvest

To start a new harvest you can just do:

troveharvester start "[Trove query]" [Trove API key]

The Trove query can either be a url copied and pasted from a search in the Trove web interface, or a Trove API query url constructed using something like the Trove API Console. Enclose the url in double quotes.

See the CLI module documentation for more details.


Created by Tim Sherratt for the GLAM Workbench. Support this project by becoming a GitHub sponsor.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

trove-newspaper-harvester-0.6.0.tar.gz (12.5 kB view hashes)

Uploaded Source

Built Distribution

trove_newspaper_harvester-0.6.0-py3-none-any.whl (11.5 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page