Kudaf Data Consumer Jupyter server extension for JupyterLab. It provides the basic infrastructure to interact with the Kudaf system
Project description
Kudaf Analytics platform on JupyterLab
This analytics platform is intended to be used by a Kudaf Data Consumer. It provides the basic infrastructure to interact with the Kudaf system, namely to:
- OAuth2 Feide authorization and authentication to receive a JWT (JSON Web Token).
- This JWT will allow access to the granted data variables on the different datasources.
- Finally, the user will be able to retrieve the granted data using the notebook and perform analytics on it.
On JupyterLab
From the JupyterLab Documentation:
JupyterLab is a highly extensible, feature-rich notebook authoring
application and editing environment, and is a part of Project Jupyter,
a large umbrella project centered around the goal of providing tools
(and standards) for interactive computing with computational notebooks.
A computational notebook is a shareable document that combines computer
code, plain language descriptions, data, rich visualizations like 3D
models, charts, graphs and figures, and interactive controls. A notebook,
along with an editor like JupyterLab, provides a fast interactive
environment for prototyping and explaining code, exploring and
visualizing data, and sharing ideas with others.
Click on the link for the documentation on how to Get Started with Jupyterlab.
Local installation instructions (Linux/Mac)
Make sure Python3 is installed on your computer (versions from 3.8 up to 3.11 should work fine)
$ python3 --version
Navigate to the folder chosen to contain this project
$ cd path/to/desired/folder
Create a Python virtual environment and activate it
$ python3 -m venv .venv
This created the virtualenv under the hidden folder .venv
Activate it with:
$ source .venv/bin/activate
Install Kudaf Datasource Tools and other required Python packages
$ pip install kudaf_jupyter_server_extension
Enable the KUDAF extension
$ jupyter server extension enable kudaf_extension
JupyterLab operation
Launch the JupyterLab instance
Navigate to the project directory and activate the virtual environment (if not already activated):
$ source .venv/bin/activate
Start-up the Jupyter server
$ jupyter lab
You should see output very much like this:
[I 2023-06-16 12:05:45.548 ServerApp] Jupyter Server 2.6.0 is running at:
[I 2023-06-16 12:05:45.549 ServerApp] http://localhost:8888/lab?token=db15ff4d84eebd80cd45ac1b03921a2f1e13d6de4f874246
[I 2023-06-16 12:05:45.549 ServerApp] http://127.0.0.1:8888/lab?token=db15ff4d84eebd80cd45ac1b03921a2f1e13d6de4f874246
[I 2023-06-16 12:05:45.549 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2023-06-16 12:05:45.580 ServerApp]
To access the server, open this file in a browser:
file:///home/daniel/.local/share/jupyter/runtime/jpserver-28857-open.html
Or copy and paste one of these URLs:
http://localhost:8888/lab?token=db15ff4d84eebd80cd45ac1b03921a2f1e13d6de4f874246
http://127.0.0.1:8888/lab?token=db15ff4d84eebd80cd45ac1b03921a2f1e13d6de4f874246
This should also automatically open a new Browser window pointing to that local Jupyter server at http://localhost:8888/lab.
Kudaf pre-requisites to be able to obtain data
The Kudaf front-end provides the entry-point for a Kudaf user to:
- Log into Feide.
- Browse the Catalog to see what data is available.
- Select the desired Variables: NOTE: Not all Variables in Kudaf have (at this point) data to be downloaded
- Finish the data-shopping by entering the Project details.
Now this data request still needs to be Approved.:
- This can be achieved using this Kudaf Core Backend API endpoint,
- for which an
admin
scope JWT token is required.
Only those approved variables that have defined data-download URLs will be made available to download here.
Below is a list of some of the variables for which data is available:
- Sikt datasource:
Antall Feide-pålogginger per tjeneste og hos en gitt FeideOrg og grunnskole
** (internal Kudaf variable name: FEIDESTATS_LOGINS_FORG_SCHOOL_SP_ACCUM)Feide organisasjonsnavn
(FEIDE_ORG_NAVN)Feide-organisasjons organisasjonsnummer
(FEIDE_ORG_BRREG_ORG)Feide organisasjons-ID knyttet til en gitt Feide-tjenesteleverandør
(FEIDE_SP_FEIDE_ORG)Tjenesteleverandørsnavn
(FEIDE_SP_NAVN)Tjenesteleverandørsbeskrivelse
(FEIDE_SP_BESKRIVELSE)
- Brreg datasource:
Organisasjons navn i Enhetsregisteret
(BRREG_ENHETSNAVN)En organisasjonenhets kommunetilhørighet
(BRREG_ORG_KOMMUNENR)En organisasjonenhets fylketilhørighet
(BRREG_ORG_FYLKESNR)
- SSB datasource:
Kommunesnavn
(SSB_KOMMUNENAVN)Fylkesnavn
(SSB_FYLKESNAVN)Fylkeskommunenavn
(SSB_FYLKESKOMMUNENAVN)En Kommuneenhets fylketilhørighet
(SSB_KOMMUNENR_FYLKESNR)
- NSR datasource:
Skolenivå
(NSR_ORG_LEVEL)Antall skoler i en organisasjon
(NSR_ORG_SCHOOL_COUNT)Antall elever i grunnskole
(NSR_ORG_STUDENT_COUNT)
Once that is done, we can proceed to retrieve the required data.
Run a Notebook to obtain statistics from Feide
On the left-hand side of the browser window, there should be a file tree open showing the various files on the project's directory.
Double-click on the file kudaf.ipynb
, this will open the Notebook.
Notebook Instructions : Downloading data using the Kudaf Panel
- Go to the first code cell and enter the following code:
from kudaf_extension import kudaf_notebook
kudaf_notebook.run()
- Execute the code cell (
Ctrl-Enter
) to enable the KUDAF extension. - An orange button
Enable Kudaf
will appear on the right-side of the Top Bar -> Click on it - A new browser tab will open to prompt for the Feide Login
- After the login is complete, the tab will become blank -> Close it
- Go back to the JupyterLab tab, the orange button from before is now green, and shows we're logged in
- A new orange button
Granted Variables
appears now on the right-side of the Top Bar -> Click on it - A new Jupyter tab
Kudaf Panel
will appear on the bottom half of the screen (if it appears blank on your browser, change the Zoom level by pressingCtrl +
orCtrl -
)- This shows different Accordions with the User's Projects.
- Under each Project Accordion, another Accordion-style display shows the Datasources and the Variables granted.
- Some Variables may display possible parameter-entry fields, if defined for the variable in question, so the data query can be further refined.
- Click on the (orange) Download button to the right of a Variable to download its data file
- Data files will be saved under Jupyter Lab's
downloads
folder (click on the folder icon on the Jupyter Lab left side panel to toggle display of the available files and folders)
Notebook Instructions : Downloading data using Python code (and the Kudaf Jupyter Server Extension)
Required:
- Name of a Kudaf variable that has been authorized for you
- URL where its data can be downloaded from
A list of the authorized variables for the logged-in user can be found at: Granted Variables
Below is an example Async function call that will download the data file to the 'downloads' folder on the left panel
var_params = {
"variable": "FEIDE_SP_NAVN",
"url": "https://kudaf-feide-stats.paas2.uninett.no/api/v1/fixed/FEIDE_SP_NAVN",
}
result = kudaf_notebook.loop.run_until_complete(kudaf_notebook.fetch_granted_data(var_params))
HAPPY ANALYTICS! :)
Developer installation
Clone this repository locally
Open up a Terminal window and enter the following commands:
$ git clone git@gitlab.sikt.no:kudaf/jupyterlab.git
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for kudaf_jupyter_server_extension-0.2.4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 691eaeba8075de85348d11c53c7e63acfbfb72850a346fc11fdf59cb2e3e9749 |
|
MD5 | e2b51779d4914a84c870e30ae7616433 |
|
BLAKE2b-256 | ddeb9c538fea22b5c940e3cf17d7303df75ff5a36b55706d2a42d79a73734154 |
Hashes for kudaf_jupyter_server_extension-0.2.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5387be0cf39183604f85863ff6849a808b12e78dfe749bc75829f42ed318907d |
|
MD5 | 463f692015069587bad7bdc961dccd4e |
|
BLAKE2b-256 | 8841eb84c074adb3efab6789ca486ef60302e8110fd81222eb19718fa668eca2 |