Skip to main content

A tool to extract and format academic data from Web of Science and Crossref

Project description

COSC425-DATA

A repository which implements data collection of a University's academic research articles within a given time period and classifies them into categories defined by the NSF PhD research focus areas taxonomy then provides:

  • Data on an article level
  • Data on individual authors
  • Data on category level

Currently the data is outputted in JSON format. There exists a script for converting the JSON to an Excel file but is currently somewhat finnicky.

A more thorough offline file formatting will be implemented in the future.

How to install

For non-development

  1. Install the package pip install academic-metrics

  2. Create a .env file in the root directory and add your OpenAI API key: OPENAI_API_KEY=<your_openai_api_key>

  3. Create a script run_pipeline.py in the root directory and add the following:

    from academic_metrics.runners.pipeline import PipelineRunner
    
    runner = PipelineRunner(ai_api_key=os.getenv("OPENAI_API_KEY"))
    runner.run_pipeline()
    

For development

  1. Clone the repository:
    • HTTPS: git clone https://github.com/SpencerPresley/COSC425-DATA.git
    • SSH: git clone git@github.com:SpencerPresley/COSC425-DATA.git
  2. Navigate into the project root directory cd COSC425-DATA and run the setup script python setup_environment.py:
    • This will install the academic_metrics package in editable mode and configure the pre-commit in .git/hooks
    • The git hook will format the code on commit using black

Note

As of 11/9/2024 the pipeline runs off input files in src/academic_metrics/data/core/input_files

Shortly integration of the crossref API code will be made in academic_metrics/runners/pipeline.py so that you can pass in your school name, data range, etc. to get your own data outputted.

Integration for writing to a mongoDB database is currently implemented only for our use case, future integration will allow two modes:

  1. Offline output files to src/academic_metrics/data/core/output_files
    • In this mode the API for crossref will still work but the output files will be saved locally rather to a database.
  2. Database support. For this you will have to create a .env file in the root directory and add the following:
    • MONGO_URI=<your_mongo_uri>
    • MONGO_DB_NAME=<your_mongo_db_name>
    • MONGO_COLLECTION_NAME=<your_mongo_collection_name>

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

academic_metrics-0.4.0a0.tar.gz (96.0 kB view details)

Uploaded Source

Built Distribution

academic_metrics-0.4.0a0-py3-none-any.whl (112.5 kB view details)

Uploaded Python 3

File details

Details for the file academic_metrics-0.4.0a0.tar.gz.

File metadata

  • Download URL: academic_metrics-0.4.0a0.tar.gz
  • Upload date:
  • Size: 96.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.20

File hashes

Hashes for academic_metrics-0.4.0a0.tar.gz
Algorithm Hash digest
SHA256 2e07e4b2c493a1ccb7bc55e22d8bf544f404db01e2abdd1219a4f9684a23e224
MD5 024b51c237269a21f52748650204bb2f
BLAKE2b-256 343f57f9ee07c164dc87cff856c0aa68578ada07f0658a63058e1f3ce5f1946a

See more details on using hashes here.

File details

Details for the file academic_metrics-0.4.0a0-py3-none-any.whl.

File metadata

File hashes

Hashes for academic_metrics-0.4.0a0-py3-none-any.whl
Algorithm Hash digest
SHA256 7fdf49bcc19a1f78fbfb507f68f6a4010f7302f7b0aa42d5bf42cc884f6a9337
MD5 93c00a87f1ec9c097748ee728cd8005b
BLAKE2b-256 dad388db95633977f5308418bb0d880865dfbb13f8dc5b0a38d12cd84d4e07fb

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page