Skip to main content

CORD 19 tools and utilities

Project description

COVID-19 Data Tools

Tools for making COVID 19 data slightly easier for everyone! If you A) think something would be useful in your research or B) have some helpful code to contribute, make an issue or PR ASAP so we can get your code shared!

Installation

pip install cord-19-tools

Demo

Demonstration Notebook on colab

Downloading the data

To download and extract the data, use the download function:

import cotools
from pprint import pprint

cotools.download(dir="data")

For now this just downloads the data from the CORD-19 dataset, metadata is not included (will be by end of day), extracts all the tarfiles, and places them in a directory

The Paperset class

This is a class for lazily loading papers from the CORD-19 dataset.

# no `/` at the end please!
data = cotools.Paperset("data/comm_use_subset")

# indexes with ints
pprint(data[0])
# returns a dict

# and slices!
pprint(data[:2])
# returns a list of dicts


print(len(data))

# takes about 5gb in memory
alldata = data[:]

Lets talk for a bit about how it works, and why it doesnt take a gigantic amount of memory. The files are not actually loaded into python until the data is indexed. Upon indexing, the files at those indexes are read into python, resulting in a list of dictionaries. This means you can still contribute while working on a low resource system.

Getting text and abstracts

For text, there is the text function, which returns the text from a single document, the texts function, which returns the text from multiple documents, and the Paperset.texts() function, which gets the text from all documents:

print(cotools.text(data[0]))
print(cotools.texts(data[12:18]))

alltext = data.texts()
# alltext = cotools.texts(alldata)

For abstracts, we have a similar API:

print(cotools.abstract(data[0]))
print(cotools.abstracts(data[12:18]))

allabs = data.abstracts()
# allabs = cotools.abstracts(alldata)

Manipulating

You can also manipulate the documents with the Paperset.apply method:

keys = comm_use.apply(lambda x: list(x.keys()))
# then lets combine them into a set
print(set(sum(keys, [])))

Hopkins data

The Hopkins data can be loaded with load_hopkins. It loads three dicts, each containing data from the hopkins dataset:

confirmed, deaths = cotools.load_hopkins()

TODO

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for cord-19-tools, version 0.1.0
Filename, size File type Python version Upload date Hashes
Filename, size cord_19_tools-0.1.0-py3-none-any.whl (5.9 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size cord-19-tools-0.1.0.tar.gz (2.4 MB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page