Skip to main content

Software Heritage loader for npm packages

Project description

swh-loader-npm

Software Heritage loader to ingest npm packages into the archive.

What does the loader do?

The npm loader visits and loads a npm package [1].

Each visit will result in:

  • 1 snapshot (which targets n revisions ; 1 per package release version)
  • 1 revision (which targets 1 directory ; the package release version uncompressed)

[1] https://docs.npmjs.com/about-packages-and-modules

First visit

Given a npm package (origin), the loader, for the first visit:

  • retrieves information for the given package (notably released versions)
  • then for each associated released version:
    • retrieves the associated tarball (with checks)
    • uncompresses locally the archive
    • computes the hashes of the uncompressed directory
    • then creates a revision (using package.json metadata file) targeting such directory
  • finally, creates a snapshot targeting all seen revisions (uncompressed npm package released versions and metadata).

Next visit

The loader starts by checking if something changed since the last visit. If nothing changed, the visit's snapshot is left unchanged. The new visit targets the same snapshot.

If something changed, the already seen package release versions are skipped. Only the new ones are loaded. In the end, the loader creates a new snapshot based on the previous one. Thus, the new snapshot targets both the old and new package release versions.

Development

Configuration file

Location

Either:

  • /etc/softwareheritage/loader/npm.yml
  • ~/.config/swh/loader/npm.yml

Configuration sample

storage:
  cls: remote
  args:
    url: http://localhost:5002/

debug: false

Local run

The built-in command-line will run the loader for a specified npm package.

For instance, to load jquery:

$ python3 -m swh.loader.npm.loader jquery

If you need more control, you can use the loader directly. It expects three arguments:

  • package_name (required): a npm package name
  • package_url (optional): URL of the npm package description (human-readable html page) that will be used as the associated origin URL in the archive
  • project_metadata_url (optional): URL of the npm package metadata information (machine-parsable JSON document)
import logging

from urllib.parse import quote

from swh.loader.npm.loader import NpmLoader

logging.basicConfig(level=logging.DEBUG)

package_name='webpack'

NpmLoader().load(package_name,
                 'https://www.npmjs.com/package/%s/' % package_name,
                 'https://replicate.npmjs.com/%s/' % quote(package_name, safe=''))

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

swh.loader.npm-0.0.7.tar.gz (164.5 kB view details)

Uploaded Source

Built Distribution

swh.loader.npm-0.0.7-py3-none-any.whl (185.3 kB view details)

Uploaded Python 3

File details

Details for the file swh.loader.npm-0.0.7.tar.gz.

File metadata

  • Download URL: swh.loader.npm-0.0.7.tar.gz
  • Upload date:
  • Size: 164.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.20.1 setuptools/40.6.2 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/3.5.3

File hashes

Hashes for swh.loader.npm-0.0.7.tar.gz
Algorithm Hash digest
SHA256 1d9ab82a26d33be5cd2ac672e4b759c598c2d456af25ad6a98839ee74b333c4d
MD5 8034e8128b5e80585a51680da3bcf84e
BLAKE2b-256 46c9bd1492a819f69476fef211c293c720511395db83071a9b77870ad82423d8

See more details on using hashes here.

File details

Details for the file swh.loader.npm-0.0.7-py3-none-any.whl.

File metadata

  • Download URL: swh.loader.npm-0.0.7-py3-none-any.whl
  • Upload date:
  • Size: 185.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.20.1 setuptools/40.6.2 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/3.5.3

File hashes

Hashes for swh.loader.npm-0.0.7-py3-none-any.whl
Algorithm Hash digest
SHA256 6e86879a8fc8fa6ba38bc618e485914495fe60cdd83696281b04bdaa1dcb3037
MD5 b80cc2fcbb9397e096176bcdd28dfb65
BLAKE2b-256 662fe6aae974fbf5a2bcae27110b4dfb0e055c6280750135a5b949abba8c3679

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page