Dependency Observatory Scanner: a scanner for software packages and dependencies
Project description
find-package-rugaru
find-package-rugaru
finds open source dependent packages in a git
repository and tests and flags suspicious open source packages (like
the legendary rugaru).
NB: this project is in an alpha state and its APIs are not stable
Scanner
Installation
Requirements
- docker >=18.06.3
- python 3.8+ and pip
- jq
- system packages to build psycopg2 (e.g.
build-essential libpq-dev
on debian buster)
After installing the above requirements:
$ git clone https://github.com/mozilla-services/find-package-rugaru.git
$ cd find-package-rugaru
$ make install install-dev
$ docker pull mozilla/dependencyscan
$ docker pull postgres:12
Example Usage
- start a local postgres database:
$ make start-db
- run one or more of the analysis scripts:
./bin/analyze_package.sh <package_name> [<package_version>]
./bin/analyze_repo.sh <repository_url>
$ ./bin/analyze_package.sh lodash 4.17.15
analyzing lodash@4.17.15 saving intermediate results to /tmp/dep-obs.g7mNNBaLyVjR
...
2020-02-27 17:31:31,900 - fpr - INFO - pipeline finished
null
2020-02-27 17:31:32,403 - fpr - INFO - pipeline finished
$
or if you have more time to scan all git tags of the lodash repo:
./bin/analyze_repo.sh https://github.com/lodash/lodash.git
analyzing tags of https://github.com/lodash/lodash.git saving intermediate results to /tmp/dep-obs.5pvSrfbn6Nox
...
Check the source of the scripts to find additional configuration via environment variables.
- Inspect the results in the local database:
make db-shell
PGPASSWORD=postgres psql -U postgres -h localhost -p 5432 dependency_observatory
psql (12.2 (Ubuntu 12.2-2.pgdg18.04+1), server 12.1 (Debian 12.1-1.pgdg100+1))
Type "help" for help.
dependency_observatory=# \x on
Expanded display is on.
dependency_observatory=# SELECT * FROM package_versions WHERE name = 'lodash' ORDER BY inserted_at DESC;
-[ RECORD 1 ]-------------------------------------------------------
id | 102
name | lodash
version | 4.17.15
language | node
url | https://registry.npmjs.org/lodash/-/lodash-4.17.15.tgz
repo_url |
repo_commit |
inserted_at | 2020-02-26 17:12:47.373348
updated_at |
Pipelines
The scripts are composed of components called pipelines (for lack of a
better term). For example analyze_package.sh
will:
- fetch information about the package from the npm registry
- filter for git refs to clone and, if specified, the matching version
- find dependency manifests or lockfiles (e.g.
package.json
) for each ref in adebian:buster-slim
docker image - run
npm install --save=true
,npm list --json
, andnpm audit --json
in the project root for each ref in anode:10-buster-slim
docker image - postprocess and save the results to the local postgres database
Each individual pipeline can be run on its own. For example the
following find_git_refs
pipeline used in analyze_repo.sh
will find
git tags for the mozilla-services/channelserver
project:
$ echo '{"repo_url": "https://github.com/mozilla-services/channelserver"}' | docker run -i --rm -v /var/run/docker.sock:/var/run/docker.sock --name fpr-test mozilla/dependencyscan python fpr/run_pipeline.py -v find_git_refs
Pipeline API
Note that this interface may be subject to change
Each pipeline:
-
reads and writes JSON lines (basically newline delimited JSON objects)
-
uses the args
-i,--infile
and-o,--outfile
that respectively default to stdin and stdout to allow pipelining -
run as a python asyncio generator
See the design doc for why this interface was chosen.
Adding a pipeline
- copy an existing file from
fpr/pipelines/
- give it a new name in its
Pipeline
model declaration - add it to
fpr/pipelines/__init__.py
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dependency-observatory-scanner-2020.3.25a2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 16c55d111b7daf14cff4ca20a4cf665441cf5c9f7d9e66a891dd3b7e65ebfd3f |
|
MD5 | 1ab67a0fa037e327aa7e7c3c25a28ab9 |
|
BLAKE2b-256 | 5e034d6f493542111b961677e587d8a0dac701cb42e73b59ff6e9c766a5da75e |
Hashes for dependency_observatory_scanner-2020.3.25a2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4417a95749470efec0b9d5de5ed4a85fbb0dddba8d5637e2281355abea9a068a |
|
MD5 | 7d72bd03073e0d9e184f427cd7f2d60f |
|
BLAKE2b-256 | 3ad4ea95db0d7e36a0ddf3d219df771f776ece4608542de7722ee351b9974970 |