Skip to main content

Scrape NHS Conditions

Build Code style: black License: MIT

This package uses the NHS Website Developer portal and Scrapy to pull down the text content of the NHS Conditions website into text files for downstream use by data science projects.

This is a simplified version of the work found here: https://github.com/nhsx/language-corpus-tools

Contact

This repository is maintained by NHS England Data Science Team.

To contact us raise an issue on Github or via email.

See our (and our colleagues') other work here:

Description

There is a need for easy access to the text content of NHS Conditions, particularly given the useful work by CogStack in creating lists of NHS Conditions questions and answers.

The NHS Developer API is very useful, but requires some setup and training to use - overkill if all a data science project needs is the NHS Conditions text. Additionally, the outputs of the API need further processing to get just the textual components of each page.

This package aims to make this whole process easier, requiring the user to simply run:

  • run_nhs_conditions_scraper: to extract the HTML for each page
  • process_nhs_conditions_json: to extract the text for each page into txt files

An example of how these are used can be see in the scrape_nhs_conditions.ipynb notebook

Prerequisites

If applicable, list the items a user needs to be able to use your repository, such as a certain version of a programming language. It can be useful to link to documentation on how to install these items.

  • Python (> 3.0)

Getting Started

Tell the user how to get started (using a numbered list can be helpful). List one action per step with example code if possible.

  1. Clone the repository. To learn about what this means, and how to use Git, see the Git guide.
git clone <insert URL>
  1. Set up your environment using pip. For more information on how to use virtual environments and why they are important see the virtual environments guide.

Using pip

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt

For Visual Studio Code it is necessary that you change your default interpreter to the virtual environment you just created .venv. To do this use the shortcut Ctrl-Shift-P, search for Python: Select interpreter and select .venv from the list.

Project structure

Provide the user with an outline of your repository structure. This template is primarily designed for publications teams at NHS England. Projects with different requirements (e.g. more complex documentation and modelling) should look to DrivenData's cookiecutter project structure, as well as our Community of Practice for guidance.

|   .gitignore                        <- Files (& file types) automatically removed from version control for security purposes
|   config.toml                       <- Configuration file with parameters we want to be able to change (e.g. date)
|   requirements.txt                  <- Requirements for reproducing the analysis environment 
|   pyproject.toml                    <- Configuration file containing package build information
|   LICENSE                           <- License info for public distribution
|   README.md                         <- Quick start guide / explanation of your project
|
|   scrape_nhs_conditions.ipynb       <- Shows how to use the main functions to scrape NHS Conditions.     
|
+---src                               <- Contains project's codebase.
|   |       __init__.py               <- Makes the functions folder an importable Python module
|   |
|   +---utils                     <- Scripts relating to configuration and handling data connections e.g. importing data, writing to a database etc.
|   |       __init__.py               <- Makes the functions folder an importable Python module
|   | 
|   +---data_ingestion                <- Scripts with modules containing functions to preprocess read data i.e. perform validation/data quality checks, other preprocessing etc.
|   |       __init__.py               <- Makes the functions folder an importable Python module
|   |       simple_nhs_conditions_scrap.py <- Scrapes the HTML down from the NHS Conditions website.
|   |
|   +---processing                    <- Scripts with modules containing functions to process data i.e. clean and derive new fields
|   |       __init__.py               <- Makes the functions folder an importable Python module
|   |       process_html.py           <- processes the HTML files to make text files 
|   |
|   +---data_exports
|   |       __init__.py               <- Makes the functions folder an importable Python module
|   |
|
+---tests
|   |       __init__.py               <- Makes the functions folder an importable Python module
|   |
|   +---backtests                     <- Comparison tests for the old and new pipeline's outputs
|   |       __init__.py               <- Makes the functions folder an importable Python module
|   |
|   +---unittests                     <- Tests for the functional outputs of Python code
|   |       test_simple_nhs_conditions_scrape.py
|   |       __init__.py               <- Makes the functions folder an importable Python module

Licence

The LICENCE file will need to be updated with the correct year and owner

Unless stated otherwise, the codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.

Any HTML or Markdown documentation is © Crown copyright and available under the terms of the Open Government 3.0 licence.

Acknowledgements

Metadata

Release files for scrape-nhs-conditions 1.0.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scrape-nhs-conditions 1.0.4
File Size Uploaded Provenance Actions
scrape_nhs_conditions-1.0.4.tar.gz 11.7 kB Without provenance Details for scrape_nhs_conditions-1.0.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scrape-nhs-conditions 1.0.4
File Interpreter ABI Platform Actions
scrape_nhs_conditions-1.0.4-py3-none-any.whl Python 3 none any Details for scrape_nhs_conditions-1.0.4-py3-none-any.whl

Total release size: 21.4 kB

Release files / scrape_nhs_conditions-1.0.4.tar.gz

Download URL scrape_nhs_conditions-1.0.4.tar.gz
Size 11.7 kB
Tags Source
SHA-256 checksum
How to use checksums
5e1ac4f2bf05ad8be50ea2f25c97d7e81a6cbcddcfca4dcc01b06df0bd837ffd
BLAKE2b-256 checksum
How to use checksums
6176c71568a8ea75c3e5bfb0f15f18bdbf572276c827a35d31b78d1dbf58367a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.11.5

Release files / scrape_nhs_conditions-1.0.4-py3-none-any.whl

Download URL scrape_nhs_conditions-1.0.4-py3-none-any.whl
Size 9.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
517bed9a9485eba92ae80c8c49c16b03e64173366745d8e8ca0ae8dedca5206f
BLAKE2b-256 checksum
How to use checksums
8014e529964f8d2dc1d6f1b9566f9e2fbfdb1139ebfcc9fc6ffe87c56d29174a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.11.5

Security for scrape-nhs-conditions 1.0.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page