Skip to main content

Create a SQLite database containing data pulled from Hacker News

Project description

hacker-news-to-sqlite

PyPI Changelog Tests License

Create a SQLite database containing data fetched from Hacker News.

How to install

$ pip install hacker-news-to-sqlite

Usage

$ hacker-news-to-sqlite user hacker-news.db your-username
Importing items:  37%|███████████                        | 845/2297 [05:09<11:02,  2.19it/s]

Imports all of your Hacker News submissions and comments into a SQLite database called hacker-news.db.

$ hacker-news-to-sqlite trees hacker-news.db 22640038 22643218

Fetches the entire comments tree in which any of those content IDs appears.

Browsing your data with Datasette

You can use Datasette to browse your data. Install Datasette like this:

$ pip install datasette

Now run it against your hacker-news.db file like so:

$ datasette hacker-news.db

Visit https://localhost:8001/ to search and explore your data.

You can improve the display of your data usinng the datasette-render-timestamps and datasette-render-html plugins. Install them like this:

$ pip install datasette-render-timestamps datasette-render-html

Now save the following configuration in a file called metadata.json:

{
    "databases": {
        "hacker-news": {
            "tables": {
                "items": {
                    "plugins": {
                        "datasette-render-html": {
                            "columns": [
                                "text"
                            ]
                        },
                        "datasette-render-timestamps": {
                            "columns": [
                                "time"
                            ]
                        }
                    }
                },
                "users": {
                    "plugins": {
                        "datasette-render-timestamps": {
                            "columns": [
                                "created"
                            ]
                        }
                    }
                }
            }
        }
    }
}

Run Datasette like this:

$ datasette -m metadata.json hacker-news.db

The timestamp columns will now be rendered as human-readable dates, and any HTML in your posts will be displayed as rendered HTML.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

hacker-news-to-sqlite-0.4.tar.gz (3.9 kB view details)

Uploaded Source

Built Distribution

hacker_news_to_sqlite-0.4-py3-none-any.whl (8.4 kB view details)

Uploaded Python 3

File details

Details for the file hacker-news-to-sqlite-0.4.tar.gz.

File metadata

  • Download URL: hacker-news-to-sqlite-0.4.tar.gz
  • Upload date:
  • Size: 3.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.9.2

File hashes

Hashes for hacker-news-to-sqlite-0.4.tar.gz
Algorithm Hash digest
SHA256 efcf4290ed2ea216004a2590bebb84954fc667a193301f7eae2eb8d2515769a2
MD5 e2003fc212a477d7c63f8a97c05ba9e6
BLAKE2b-256 885882302c7d9e7d29fb5860d778b3d50294bf70d9ff109a562b900700ca4f13

See more details on using hashes here.

File details

Details for the file hacker_news_to_sqlite-0.4-py3-none-any.whl.

File metadata

  • Download URL: hacker_news_to_sqlite-0.4-py3-none-any.whl
  • Upload date:
  • Size: 8.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.9.2

File hashes

Hashes for hacker_news_to_sqlite-0.4-py3-none-any.whl
Algorithm Hash digest
SHA256 779dce2993c8de7fd5f31a3e2f033807f89b0113b0a018410537c711e3c4ab9c
MD5 8b334ee510dd9e129423416494f2b895
BLAKE2b-256 89d77c4aab56f27a0d465442f41b27074814bdd6f7b75f6c47fe3f5cb93b9ba1

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page