Shadow Scholar

Table of Contents

• Installation
• Available Scripts
• Getting Access to AWS services
• Adding Your Own Script
• Adding A New Module

Installation

To install from PyPI, simply run:

pip install shadow-scholar

Available Scripts

Each script is launched with shadow <entrypoint_name>. For a full list of all entry points, run shadow -l.

Getting Access to AWS services

To run the scripts that use AWS services, you will need to have access to the following services:

• Athena
• S3

The best way to do so is to obtain AWS credentials (access key and secret key) and set them as environment variables.

Adding Your Own Script (Application Entry Point)

To write your own script for Shadow Scholar, follow these steps:

Step 1: Choose where to add your code in Shadow Scholar. It can either be in an existing module, such as shadow_scholar.collections.athena, or in a new module.

Step 2: Understand that the entry point for your script should be a single function; think of this as the main function.

Step 3: Write your main function. For each argument you expect a user might want to provide from command line, add a corresponding argument to the function. For example:

def my_script(
    arg1: str,
    arg2: int,
    arg3: bool,
    arg4: Optional[str] = None,
):
    # Do something with the arguments
    pass

Step 4: Add the cli from shadow_scholar.cli to your script. This will allow users to run your script from the command line. For example:

from shadow_scholar.cli import cli
from typing import Optional

@cli(
    name="scripts_collection.my_script",
    arguments=...,
    requirements=...,
)
def my_script(
    arg1: str,
    arg2: int,
    arg3: bool,
    arg4: Optional[str] = None,
):
    # Do something with the arguments
    pass

The cli decorator takes three arguments: the name of the script, a list of arguments, and a list of requirements. The name of the script should be the name a user would use to run the script from the command line. In the example above, the user would run the script with shadow scripts_collection.my_script.

Step 5: Add arguments to your script. Each argument should be an instance of shadow_scholar.cli.Argument. For example:

from typing import Optional
from shadow_scholar.cli import Argument, cli

@cli(
    name="scripts_collection.my_script",
    arguments=[
        Argument(
            name="arg1",
            type=str,
            help="This is the first argument",
        ),
        Argument(
            name="arg2",
            type=int,
            help="This is the second argument",
        ),
        Argument(
            name="arg3",
            type=bool,
            help="This is the third argument",
        ),
        Argument(
            name="arg4",
            type=str,
            help="This is the fourth argument",
            default=None,
        ),
    ],
    requirements=...,
)
def my_script(
    arg1: str,
    arg2: int,
    arg3: bool,
    arg4: Optional[str] = None,
):
    # Do something with the arguments
    pass

You should have as many Arguments as you have arguments to your main function.

Step 6: Add requirements to your script. Each requirement should be an in the format used by requirements.txt. When using optional requirements, make sure to wrap them in a with safe_import() statement at the top of your script. For example:

from typing import Optional
from shadow_scholar.cli import Argument, cli, safe_import

with safe_import() as safe:
    # this will not fail if pandas is not installed
    import pandas as pd


@cli(
    name="scripts_collection.my_script",
    arguments=[
        Argument(
            name="arg1",
            type=str,
            help="This is the first argument",
        ),
        Argument(
            name="arg2",
            type=int,
            help="This is the second argument",
        ),
        Argument(
            name="arg3",
            type=bool,
            help="This is the third argument",
        ),
        Argument(
            name="arg4",
            type=str,
            help="This is the fourth argument",
            default=None,
        ),
    ],
    requirements=[
        "pandas>=1.0.0",
    ],
)
def my_script(
    arg1: str,
    arg2: int,
    arg3: bool,
    arg4: Optional[str] = None,
):
    # Do something with the arguments
    pass

Step 7: Import the function in the __init__.py file of the module. For example, if you added your script to shadow_scholar/examples.py, you would add the following to shadow_scholar/__init__.py:

from shadow_scholar.examples import my_script

Adding A New Module (Available to All Scripts/as a Dependency)

Adding a module to Shadow Scholar is as easy as adding a script, but, instead of decorating the main entry-point function with cli, you need to use the @require decorator. For example:

from shadow_scholar.cli import require, safe_import

with safe_import():
    import requests

@require(["requests>=2.0.0"])
def module_function():
    # Do something with requests
    requests.get(...)