Sends JSONL data into an Ed-Fi API
Project description
lightbeam
transmits payloads from JSONL files into an Ed-Fi API.
Table of Contents
- Requirements
- Installation
- Setup
- Usage
- Features
- Design
- Performance
- Limitations
- Changelog
- Contributing
- License
Requirements
Python 3, pip, and connectivity to an Ed-Fi API.
Installation
pip install git+https://github.com/edanalytics/lightbeam.git#egg=lightbeam
Setup
Running the tool requires
- a folder of JSONL files, one for each Ed-Fi Resource and Descriptor to populate
- a YAML configuration file An example YAML configuration is below, followed by documentation of each option.
state_dir: ~/.lighbeam/
data_dir: ./
namespace: ed-fi
edfi_api:
base_url: https://api.schooldistrict.org/v5.3/api
version: 3
mode: year_specific
year: 2021
client_id: yourID
client_secret: yourSecret
connection:
pool_size: 8
timeout: 60
num_retries: 10
backoff_factor: 1.5
retry_statuses: [429, 500, 502, 503, 504]
verify_ssl: True
force_delete: True
log_level: INFO
show_stacktrace: True
- (optional)
state_dir
is where state is stored. The default is~/.lightbeam/
on *nix systems,C:/Users/USER/.lightbeam/
on Windows systems. - (optional) Specify the
data_dir
which contains JSONL files to send to Ed-Fi. The default is./
. The tool will look for files like{Resource}.jsonl
or{Descriptor}.jsonl
in this location, as well as directory-based files like{Resource}/*.jsonl
or{Descriptor}/*.jsonl
. Files with.ndjson
or simply.json
extensions will also be processed. (More info at thendjson
standard page.) - (optional) Specify the
namespace
to use when accessing the Ed-Fi API. The default ised-fi
but others includetpdm
or custom values. - Specify the details of the
edfi_api
to which to connect including- (optional) The
base_url
The default ishttps://localhost/api
(the address of an Ed-Fi API running locally in Docker). - The
version
as one of3
or2
(2
is currently unsupported). - (optional) The
mode
as one ofshared_instance
,sandbox
,district_specific
,year_specific
, orinstance_year_specific
. - (required if
mode
isyear_specific
orinstance_year_specific
) Theyear
used to build the resource URL. The default is the current year. - (required if
mode
isinstance_year_specific
) Theinstance_code
used to build the resource URL. The default is none. - (required) Specify the
client_id
to use when connecting to the Ed-Fi API. - (required) Specify the
client_secret
to use when connecting to the Ed-Fi API.
- (optional) The
- Specify the
connection
parameters to use when making requests to the API including- (optional) The
pool_size
. The default is 8. The optimal setting depends on the Ed-Fi API's capabilities. - (optional) The
timeout
(in seconds) to wait for each connection attempt. The default is60
seconds. - (optional) The
num_retries
to do in case of request failures. The default is10
. - (optional) The
backoff_factor
to use for the exponential backoff. The default is1.5
. - (optional) The
retry_statuses
, that is, the HTTPS response codes to consider as failures to retry. The default is[429, 500, 501, 503, 504]
. - (optional) Whether to
verify_ssl
. The default isTrue
. Set toFalse
when working withlocalhost
APIs or to live dangerously.
- (optional) The
- (optional) Skip the interactive confirmation prompt (for programmatic use) when using the
delete
command. The default isFalse
(prompt). - (optional) Specify a
log_level
for output. Possible values areERROR
: only output errors like missing required sources, invalid references, invalid YAML configuration, etc.WARNING
: output errors and warnings like when the run log is getting longINFO
: all errors and warnings plus basic information about whatearthmover
is doing: start and stop, how many rows were removed by adistinct_rows
orfilter_rows
operation, etc. (This is the defaultlog_level
.)DEBUG
: all output above, plus verbose details about each transformation step, timing, memory usage, and more. (Thislog_level
is recommended for debugging transformations.)
- (optional) Specify whether to show a stacktrace for runtime errors. The default is
False
.
Usage
lightbeam
recognizes several commands:
validate
lightbeam validate -c path/to/config.yaml
You may validate
your JSONL before transmitting it. This checks that the payloads
- are valid JSON
- conform to the structure described in the Swagger documents for resources and descriptors fetched from your API
- contain valid descriptor values (fetched from your API)
- contain unique values for any natural key
This command will not find invalid reference errors, but is helpful for finding payloads that are invalid JSON, are missing required fields, or have other structural issues.
send
lightbeam send -c path/to/config.yaml
Sends your JSONL payloads to your Ed-Fi API.
validate+send
lightbeam validate+send -c path/to/config.yaml
This is a shorthand for sequentially running validate and then send. It can be useful to catching errors in automated pipelines earlier in the validate
step before you actually send
problematic data to your Ed-Fi API.
delete
lightbeam delete -c path/to/config.yaml
Delete payloads by
- determing the natural key (set of required fields) for each endpoint
- iterating through your payloads and looking up each one via a
GET
request to the API filtering for the natural key values - if exactly one result is returned,
DELETE
ing it byid
Payload hashes are also deleted from saved state. Endpoints are processed in reverse-dependency order to prevent delete failures due to data dependencies.
Note that student
resource payloads cannot be deleted since other resources may reference them. (This is an Ed-Fi API restriction.)
Running the delete
command will prompt you to type "yes" to confirm. This confirmation prompt can be disabled (for programmatic use) by specifying force_delete: True
in your YAML.
Other options
See a help message with
lightbeam -h
lightbeam --help
See the tool version with
lightbeam -v
lightbeam --version
Features
This tool includes several special features:
Selectors
Send only a subset of resources or descriptors in your data_dir
using a selector:
lightbeam send -c path/to/config.yaml -s schools,students,studentSchoolAssociations
Environment variable references
In your YAML configuration, you may reference environment variables with ${ENV_VAR}
. This can be useful for passing sensitive data like credentials to lightbeam
, such as
...
edfi_api:
client_id: ${EDFI_API_CLIENT_ID}
client_secret: ${EDFI_API_CLIENT_SECRET}
...
Command-line parameters
Similarly, you can specify parameters via the command line with
lightbeam send -c path/to/config.yaml -p '{"CLIENT_ID":"populated", "CLIENT_SECRET":"populatedSecret"}'
lightbeam send -c path/to/config.yaml --params '{"CLIENT_ID":"populated", "CLIENT_SECRET":"populatedSecret"}'
Command-line parameters override any environment variables of the same name.
State
This tool maintains state about payloads previously dispatched to the Ed-Fi API to avoid repeatedly resending the same payloads. This is done by maintaining a pickled Python dictionary of payload hashes for each Ed-Fi resource and descriptor, together with a timestamp and HTTP status code of the last response. The files are located in the config file's state_dir
and have names like {resource}.dat
or {descriptor}.dat
.
By default, only new, never-before-seen payloads are sent.
You may choose to resend payloads last sent before timestamp using the -t
or --older-than
command-line flag:
lightbeam send -c path/to/lightbeam.yaml -t 2020-12-25T00:00:00
lightbeam send -c path/to/lightbeam.yaml --older-than 2020-12-25T00:00:00
Or you may choose to resend payloads last sent after timestamp using the -n
or --newer-than
command-line flag:
lightbeam send -c path/to/lightbeam.yaml -n 2020-12-25T00:00:00
lightbeam send -c path/to/lightbeam.yaml --newer-than 2020-12-25T00:00:00
Or you may choose to resend payloads that returned a certain HTTP status code(s) on the last send using the -r
or --retry-status-codes
command-line flag:
lightbeam send -c path/to/lightbeam.yaml -r 200,201
lightbeam send -c path/to/lightbeam.yaml --retry-status-codes 200,201
These three options may be composed; lightbeam
will resend payloads that match any conditions (logical OR).
Finally, you can ignore prior state and resend all payloads using the -f
or --force
flag:
lightbeam send -c path/to/lightbeam.yaml -f
lightbeam send -c path/to/lightbeam.yaml --force
Cache
To reduce runtime, lightbeam
caches the resource and descriptor Swagger docs it fetches from your Ed-Fi API as well as the descriptor values for up to a month. This way, the data does not have to be re-loaded from your API on every run. The cached files are stored in the cache
directory within your state_dir
. You may run lightbeam
with the -w
or --wipe
flag to clear the cache and force re-fetching the API metadata:
lightbeam send -c path/to/config.yaml -w
lightbeam send -c path/to/config.yaml --wipe
Design
Some details of the design of this tool are discussed below.
Resource-dependency ordering
JSONL files are sent to the Ed-Fi API in resource-dependency order, which avoids "missing reference" API errors when populating multiple endpoints.
Performance & Limitations
Tool performance depends on primarily on the performance of the Ed-Fi API, which in turn depends on the compute resources which back it. Typically the bottleneck is write performance to the database backend (SQL server or Postgres). If you use lightbeam
to ingest a large amount of data into an Ed-Fi API (not the recommended use-case, by the way), consider temporarily scaling up your database backend.
For reference, we have achieved throughput rates in excess of 100 requests/second against an Ed-Fi ODS & API running in Docker on a laptop.
Change log
[2022-09-22] Version 0.0.1 released
Contributing
Bugfixes and new features (such as additional transformation operations) are gratefully accepted via pull requests here on GitHub.
Contributions
- Cover image created with DALL • E mini
License
See License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for lightbeam-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6432ed125bde12f1ff9ada7c37106bdc566f738ca1a824cddf881b264b60ad26 |
|
MD5 | 99af3f96c39fad9e7d4f60580068eb49 |
|
BLAKE2b-256 | a074382a5e847712230ea81d1c9afe0212752959bb01a79cdfe98418c8c7c364 |