Silence 2.0.0

An educational API+Web framework.

---

Introduction

Silence is a framework that allows for a rapid deployment of a RESTful API and a Web application around a relational database. Silence revolves around the concept of projects, which contain everything needed to deploy the previously mentioned elements and can be moved and/or shared between computers or systems with ease.

Silence has been built as an educational and teaching tool for its use in several courses of the Degree in Computer Science at the University of Seville.

Installation

Silence is available in the Python Package Index (PyPI) for Python ≥ 3.6. To install Silence, run:

pip install Silence

Silence also requires a connection to a MySQL/MariaDB database.

Creating a new project

Once Silence has been installed via pip, the silence command becomes available. To create a new project, run silence new <name>, where <name> is the name of the new project.

This will download an example project template that you can adapt to your needs. There are many different templates available. If you wish to download a different one, you can use silence new <name> --template <template-name>.

Alternatively, you can use silence new <name> --blank to download a blank template, or silence new --url <url-to-a-repo> to download a project hosted in the specified repository.

Project templates

We have a variety of different examples implemented using Silence, which we call templates. You can find a list of all available templates using silence list-templates, and download one as shown previously.

Configuring your project

The project settings can be found in settings.py. The available configuration parameters are:

• DEBUG_ENABLED Controls whether debug messages and Flask's debug mode are active (bool, default: False)
• LISTEN_ADDRESS IP address in which the web server will listen to requests (str, default: "127.0.0.1")
• HTTP_PORT Port in which the web server will listen to requests (int, default: 8080)
• SQL_SCRIPTS Sequence of files inside the sql/ folder to run when the silence createdb command is issued (list[str], default: [])
• API_PREFIX URL prefix for API requests (str, default: /api, do not set empty)
• DB_CONN Connection details to the MySQL/MariaDB database
  • host IP or name of the server (str, default: '127.0.0.1')
  • port Port the SQL server is listening to (int, default: 3306)
  • username User in the SQL server to use (str, default: default_username)
  • password Password for the previous user (str, default: default_password)
  • database Name of the schema to use, the user should have privileges to create and destroy tables in this schema (str, default: default_database)
• RUN_API Deploy the API endpoints (bool, default: True)
• RUN_WEB Deploy the static web server (bool, default: True)
• ENABLE_LOGIN Enables the /login endpoint (bool, default: True)
• ENABLE_REGISTER Enables the /register endpoint (bool, default: True)
• ENABLE_SUMMARY Enables the API summary endpoint (GET API_PREFIX) (bool, default: True)
• SHOW_ENDPOINT_LIST Controls whether the list of all available endpoints is displayed when using silence run (bool, default: True)
• COLORED_OUTPUT Enables colors in the console output (bool, default: True)
• DECIMALS_AS_STRINGS Controls whether Decimal types are serialized as str instead of float (bool, default: False)
• USER_AUTH_DATA Configures which information to use for login and register
  • table Name of the table containing your users (str, default: users)
  • identifier Column of this table containing the unique identifiers used for login (str, default: username)
  • password Column of this table containing the hashed passwords (str, default: password)
  • role Column of this table containing the role of the user (optional, str, default: role)
• DEFAULT_ROLE_REGISTER Role to assign to the users that register via the /register endpoint (str, default: None)
• SECRET_KEY Random string used for signing session tokens and Flask security. Generated automatically upon project creation when using silence new. No default is provided and not setting one will result in an error.
• HTTP_CACHE_TIME Sets the max-age value in the Cache-Control HTTP header for static files sent by the server. In practice, this controls for how long these files are cached by the web browser. (int, default: 0 for development purposes)
• MAX_TOKEN_AGE Time in seconds during which a session token is valid after it has been issued (int, default: 86400)
• CHECK_FOR_UPDATES Whether to check for new Silence versions when using silence run (bool, default: True)
• ENABLE_ENDPOINT_AUTO_GENERATION Allows for the use of the silence createapi (bool, default:True)

Creating the database

Silence provides the silence createdb command to automatically execute any number of SQL scripts to create your database and/or set it to a controlled initial state.

To use this command, put the .sql files that you want it to run inside the project's sql/ folder, and change the SQL_SCRIPTS configuration parameter accordingly in settings.py to reflect which files you want it to run and in which order, for example:

SQL_SCRIPTS = ["create_tables.sql", "populate_database.sql"]

Once this has been configured, silence createdb will execute the specified files in the desired order, using the connection and database/schema specified in DB_CONN.

Defining your API endpoints

The Silence philosophy is that API endpoints are wrappers around SQL operations, and thus they can be defined in a relatively simple manner. As an example, we will demonstrate how to create basic CRUD endpoints for a Department table whose columns are (departmentId, name, city).

We begin by creating a department.json file inside the project's endpoints/ folder. The name of this file is not relevant, and all files inside the endpoints/ folder containing endpoint declarations will be automatically detected and imported.

We setup a simple object containing all of the endpoints with the following structure: the keys must contain at least route, method, sql and can optionally contain auth_required, allowed_roles, description, request_body_params

GET operations

We define an endpoint to retrieve all existing Departments like this:

"getAll":{
  "route": "/departments",
  "method": "GET",
  "sql": "SELECT * FROM departments",
  "description": "Gets all departments"
}

Note that the endpoint route is defined without the global /api prefix, it is automatically added later by Silence. In this case, whenever anyone performs an HTTP GET request against /api/departments, the SQL query will be executed and the results will be displayed as JSON.

GET /api/departments

(200 OK)
[
  {
    "city": "C\u00e1diz",
    "departmentId": 1,
    "name": "Arte"
  },
  {
    "city": null,
    "departmentId": 2,
    "name": "Historia"
  },
  {
    "city": "Sevilla",
    "departmentId": 3,
    "name": "Inform\u00e1tica"
  }
]

URL parameters can be defined and passed on to the SQL query. For example, we can also define an endpoint to retrieve a certain Department by its departmentId:

"getById":{
  "route": "/departments/$departmentId",
  "method": "GET",
  "sql": "SELECT * FROM departments WHERE departmentId = $departmentId",
  "description": "Gets a departments with corresponding primary key"
},

In this case, the URL parameter is defined using the $param syntax so Silence knows that it is a variable part of the URL pattern. It is passed to the SQL query using the same syntax by specifying the same parameter name. You can capture and pass as many parameters as you want, in any order, in this manner. Silence checks that all parameters in the SQL query can be obtained through the URL pattern.

All parameters in a SQL query, whether received via URL pattern or request body, are passed in a safe manner using SQL placeholders.

A user may thus request a specific Department by its ID using this endpoint:

GET /api/departments/3

(200 OK)
[
  {
    "city": "Sevilla",
    "departmentId": 3,
    "name": "Inform\u00e1tica"
  }
]

Endpoints whose routes end with a URL parameter automatically return a 404 HTTP code if the SQL query returns an empty result:

GET /api/departments/-1

(404 NOT FOUND)
{
    "code": 404,
    "message": "Not found"
}

POST/PUT operations

Editing data through POST and PUT requests follows the same guidelines, however, most commonly they will receive aditional parameters through the HTTP request body.

An endpoint to create a new Department is associated with a SQL INSERT operation:

"create":{
  "route": "/departments",
  "method": "POST",
  "sql": "INSERT INTO departments(name, city) VALUES ($name, $city)",
  "auth_required": true,
  "description": "creates a new departments",
  "request_body_params": [
    "name",
    "city"
  ]
}

Note that, in this case, the SQL query expects the parameters $name and $city, but they are not defined in the URL pattern. Instead, they are expected in the HTTP POST body. We declare this by adding them in the request_body_params list.

By doing so, Silence knows that you're expecting to receive them in the body of the received HTTP request and can check your SQL query for unexpected parameters.

Silence can extract parameters from request bodies encoded in application/x-www-form-urlencoded or application/json:

POST /api/departments
Content-Type: application/json
{"name": "New department", "city": "Seville"}

(200 OK)
{
  "lastId": 4
}

For SQL operations other than SELECT, Silence returns the ID of the last modified row (in this case, it represents the ID assigned to the newly created resource).

PUT requests are similar and combine both URL and body parameters, as udpate requests are aimed towards an already existing resource:

"update":{
  "route": "/departments/$departmentId",
  "method": "PUT",
  "sql": "UPDATE departments SET name = $name, city = $city WHERE departmentId = $departmentId",
  "auth_required": true,
  "description": "updates an existing departments with corresponding primary key",
  "request_body_params": [
    "name",
    "city"
  ]
}

Silence makes sure that all parameters included in your SQL string can be obtained from either the request URL (declared in your route pattern) or the request body (declared in your function arguments).

DELETE operations

An example of an endpoint to delete a Department by its departmentId is as follows:

"delete":{
  "route": "/departments/$departmentId",
  "method": "DELETE",
  "sql": "DELETE FROM departments WHERE departmentId = $departmentId",
  "auth_required": true,
  "description": "deletes an existing departments with corresponding primary key"
} 

In this case only the URL parameter is needed, and thus nothing is validated in the associated function.

Default endpoints and utilities

Unless explicitly disabled in the project's settings, Silence provides endpoints to register a new user and log in with an existing one. Additionaly, Silence enhances user-defined endpoints with some additional utilities:

Summary endpoint

A HTTP GET request to the API prefix will result in a list of the currently enabled endpoints, along with their descriptions if they have been provided:

GET /api

(200 OK)
[
  {
    "desc": "Returns the data regarding the API endpoints",
    "method": "GET",
    "route": "/api"
  },
  {
    "desc": "Starts a new session, returning a session token and the user data if the login is successful",
    "method": "POST",
    "route": "/api/login"
  },
  {
    "desc": "Creates a new user, returning a session token and the user data if the register is successful",
    "method": "POST",
    "route": "/api/register"
  },
  {
    "description": "Shows all departments",
    "method": "GET",
    "route": "/api/departments"
  },
  {
    "description": "Shows one department by ID",
    "method": "GET",
    "route": "/api/departments/$departmentId"
  },
  {
    "description": "Creates a new department",
    "method": "POST",
    "route": "/api/departments"
  },
  {
    "description": "Updates an existing department",
    "method": "PUT",
    "route": "/api/departments/$departmentId"
  },
  {
    "description": "Removes an existing department",
    "method": "DELETE",
    "route": "/api/departments/$departmentId"
  }
]

/register endpoint

If ENABLE_REGISTER is enabled, which it is by default, Silence will deploy a /register endpoint to enable users to register in your application. This endpoint must be accessed through a HTTP POST request, including the desired fields in the request body, and it takes care of:

• Ensuring that the request contains at least the identifier and password fields specified in USER_AUTH_DATA.identifier and USER_AUTH_DATA.password
• Ensuring that the identifier does not exist already in the users table specified in USER_AUTH_DATA.table
• Hashing the submitted password
• Starting a new session if the register is successful, returning a session token and the user data

For example, let us assume that you have a Users table containing the columns (userId, username, password, email) and that you have set up your project to use username as the identifier field and password as the password field:

USER_AUTH_DATA = {
    "table": "Users",
    "identifier": "username",
    "password": "password",
}

Then, in order to register a new user, you need to submit a POST request to /register with the username, password and any additional fields that you wish to insert:

POST /api/register
Content-Type: application/json
{"username": "new_user", "password": "123456", "email": "[e-mail]"}

(200 OK)
{
  "sessionToken": ".eJwl[...]7qMo",
  "user": {
    "email": "[e-mail]",
    "userId": 5,
    "username": "new_user"
  }
}

You have successfully registered, and Silence returns a session token under the response field "sessionToken" and the currently logged in user data under "user".

Future attempts to register with the same identifier will be met with an error:

POST /api/register
Content-Type: application/json
{"username": "new_user", "password": "123456", "email": "[e-mail]"}

(400 BAD REQUEST)
{
  "code": 400,
  "message": "There already exists another user with that username"
}

Please note that, in order to be flexible and support any possible table, /register assumes that all fields received in the request exist in the users table, and it will try to insert values in them. Any received fields that have no matching columns in the specified table will be ignored.

/login endpoint

If ENABLE_LOGIN is enabled, which it is by default, Silence will deploy a /login endpoint to enable users to log in in your application. This endpoint must be accessed through a HTTP POST request, including the identifier and password fields in the request body.

Like /register, the /login endpoint uses the table and columns specified in the USER_AUTH_DATA setting and expects to receive the identifier and password fields in the POST body. /login receives a clear-text password and compares it against the hashed version in the database:

POST /api/login
Content-Type: application/json
{"username": "new_user", "password": "123456"}

(200 OK)
{
  "sessionToken": ".eJwl[...]WKXc",
  "user": {
    "email": "[e-mail]",
    "userId": 5,
    "username": "new_user"
  }
}

The response data is the same as in /register.

Restricting endpoints to logged users

All endpoint declarations accept an optional auth_required argument that can be set to true if the endpoint is intended to be used only by logged users:

"create":{
  "route": "/departments",
  "method": "POST",
  "sql": "INSERT INTO departments(name, city) VALUES ($name, $city)",
  "auth_required": true, <<======
  "description": "creates a new departments",
  "request_body_params": [
    "name",
    "city"
  ]
}

When an endpoint is protected in this manner, the user has to prove that they have a current session by sending their session token as an HTTP header, under the key Token:

GET /api/departments
Token: .eJwl[...]WKXc

(200 OK)
[
  {
    "city": "C\u00e1diz",
    "departmentId": 1,
    "name": "Arte"
  },
  {
    "city": null,
    "departmentId": 2,
    "name": "Historia"
  },
  {
    "city": "Sevilla",
    "departmentId": 3,
    "name": "Inform\u00e1tica"
  }
]

Session tokens are provided by the /register and /login endpoints under the "sessionToken" response field, and they remain valid for the duration specified in the MAX_TOKEN_AGE setting. If the user tries to access a restricted endpoint without providing a valid session token, the server responds with a 401 HTTP code:

GET /api/departments
(No "Token" header)

(401 UNAUTHORIZED)
{
  "code": 401,
  "message": "Unauthorized"
}

URL query parameters in GET requests

Silence provides all user-defined GET endpoints with automatic filtering, paging and ordering via URL query params:

• _sort determines the field by which the results will be ordered (default: order determined by the database)
• _order determines the sorting order ("asc" or "desc", default: "asc")
• _limit determines the maximum amount of results to show (default: no limit)
• _page determines the page of results to show (to be combined with _limit for easy paging, default: 0)
• All other query parameters will be interpreted as a field that has to be filtered on

For example:

Get all departments whose city is Sevilla:

GET /api/departments?city=Sevilla

(200 OK)
[
  {
    "city": "Sevilla",
    "departmentId": 3,
    "name": "Inform\u00e1tica"
  }
]

Get all departments whose city is Cádiz AND their name is Arte:

GET /api/departments?city=Sevilla&name=Arte

(200 OK)
[
  {
    "city": "C\u00e1diz",
    "departmentId": 1,
    "name": "Arte"
  }
]

Get all departments ordered by decreasing departmentId:

GET /api/departments?_sort=departmentId&_order=desc

(200 OK)
[
    {
        "city": "Sevilla",
        "departmentId": 3,
        "name": "Inform\u00e1tica"
    },
    {
        "city": null,
        "departmentId": 2,
        "name": "Historia"
    },
    {
        "city": "C\u00e1diz",
        "departmentId": 1,
        "name": "Arte"
    }
]

Get the second page of the previous query, with 2 results per page:

GET /api/departments?_sort=departmentId&_order=desc&_limit=2&_page=1

(200 OK)
[
  {
    "city": "C\u00e1diz",
    "departmentId": 1,
    "name": "Arte"
  }
]

These parameters can be combined in any way and work for all GET endpoints.

Static web server

Silence also serves as a web server for static files (unless explicitly disabled via the RUN_WEB setting). The web/ folder inside your project is the web root, and thus you can place your web application there to be deployed by Silence.

The web server has no prefix. Accessing http://<address>/ will hit the index.html file located in the root of the web/ folder. Any subfolders will work as expected, with the only exception of a route that creates a conflict with the API prefix (for example, if your API prefix is /api, do not create an api/ folder directly in the root of web/).

Endpoint and api files auto generation

Silence also offers a tool to generate basic CRUD operations for all entities that have been declared in the .sql files. By running the silence createapi silence will create a /endpoints/default folder and populate it with a .json file for each of the entities in the database, it will also check if a particular endpoint was declared in the base folder by comparing its route and method parameters and refrain from creating them if found.

This tool is avaliable by default unless explicitly turned off in setting by setting ENABLE_ENDPOINT_AUTO_GENERATION to false.

Running the server

Once you have configured your project, database and defined your endpoints, you can launch the web server with:

silence run

Access logs, debug messages (if allowed) and uncontained exceptions will be logged directly to the console.

Changelog

See CHANGELOG.md

Contributions

All contributions are welcome provided that they follow our Code of Conduct.

We keep a TO-DO with the changes that we'd like to implement in the future. Feel free to open an issue if you need clarifications with any of its items.

Disclaimer

Silence has been designed and built with educational-only use in mind. It makes no specific efforts to ensure efficiency, security, or fitness to purposes other than educational ones. We have not tested Silence for its use in a production environment of any kind.

License

MIT License