s3app 0.8.3

TODO

YOTRON is a consultancy company which is focused on DevOps, Cloudmanagement and Data Management with NOSQL and SQL-Databases. Visit us on www.yotron.de ement with NOSQL and SQL-Databases. Visit us on www.yotron.de

S3App

S3 (Simple Storage Solution) is a file storage services which is part of Cloud solutions. It is known for its scalability, data availability, security, performance and the ease to connect storage clients to it. Introduced by Amazon in AWS, other provider of managed S3 are available just like software solutions to set up an private S3-solution like (Ceph or Cloudian).

S3App simplifies the access to a S3Buckets with a provider independent web based frontend which allows the visualizing and the management of the content of S3 buckets with an S3 provider independent web application, ....

For further information and the manual, please see s3app/manual

Installation

We provide two installation methods. A native installation on any OS which supports Python3 and a installation in Kubernetes via HELM. Per default, S3App runs on a sqlite database. This is recommended only for testing on a standalone installation not in production with more nodes.

In production, you should use a PostgreSQL database for a better data persistency and to allow a HA setup with more S3App nodes.

The S3App has no TLS-termination ("https"). We recommend to use a Reverse Proxy like NGINX or Apache Web Server in front of S3App Web App for TLS termination.

The HELM project of S3App for Kubernetes contains all needed and recommended components. You can optionally install a NGINX Reverse Proxy and a PostgreSQL database with the project, but you can also use separated applications.

Native

Prerequisites

• a clone of this GitHub-project
• Python3
• Python-pip for dependency installation
• recommended: python3-venv to create a virtual environment for Python3
• recommended: Waitress python WSGI production ready web-server
• recommended: PostgreSQL database for the S3App metadata
• recommended: A reverse proxy with TLS termination (e.g. NGINX).

S3App supports all current Python3 versions. It is tested with Python 3.9.x and 3.10.x

To check which version is running:

vagrant@bullseye:~$ python3 -V
Python 3.9.2

Pip and python3-venv are available in the main OS distribution.

For Debian/Ubuntu: sudo apt-get install python3-pip python3-venv

For CentOS: sudo yum install python3-pip python3-venv

Installation

There are a lot of variants how to start a Python web project. We use a Python virtual environment for setting up S3App on a virtual environment. Depend on your environment and knowledge in Python the process can be different.

1. Create a virtual environment of Python

   1. Create a virtual environment for S3App

      vagrant@bullseye:~$ python3 -m venv s3app
      

   2. Activate your virtual environment

      This steps is to repeat every time you want to restart S3App.

      vagrant@bullseye:~$ source s3app/bin/activate
      (s3app) vagrant@bullseye:~$
      

2. Install all Python dependencies needed for the project:

   (s3app) vagrant@bullseye:~$ pip install s3app --upgrade
   

3. Start S3App server (testing)

   (s3app) vagrant@bullseye:~$ flask --app s3app run --host=0.0.0.0 --port=8080
   The APP is running in DEBUG mode.
   2022-10-06 15:21:07,382:INFO:flask_appbuilder.base:Registering class S3View on menu 
   2022-10-06 15:21:07,383:INFO:flask_appbuilder.baseviews:Registering route /s3/<string:prefixUrl> ('GET',)
   2022-10-06 15:21:07,383:INFO:flask_appbuilder.baseviews:Registering route /s3/ ('GET',)
   ...
   2022-10-06 15:21:07,793:INFO:flask_appbuilder.base:Registering class S3IndexView on menu 
   2022-10-06 15:21:07,793:INFO:flask_appbuilder.baseviews:Registering route / ('GET',)
   * Serving Flask app 'webapp'
   * Debug mode: off
   022-10-06 15:21:07,797:INFO:werkzeug:WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
   * Running on all addresses (0.0.0.0)
     * Running on http://127.0.0.1:8080
     * Running on http://192.168.56.106:8080
   

4. Open Browser and start S3App with one of the IPs mentioned above:

   

   Your will automatically redirected to http://x.x.x.x:8080/login

5. Login as a admin user

   The default credentials for the admin user are:

   Username: admin

   Password: admin

   

Installation with waitress

For production waitress as a WSGI web server for Python is recommended. The steps above are the same.

1. Install waitress WSGI web server

   There are different ways to install waitress. Via OS packaging like yum (CentOS) or apt-get (Debian, Ubuntu) or via Pip. As an example, you can install waitress on Debian-Bullseye with:

   sudo apt-get install python3-waitress
   

2. Start S3App with waitress

   waitress-serve --host=0.0.0.0 --port=8080 S3App:app
   

HELM

You find the installation instructions for Kubernetes here

Manual

The manual for S3App you find here

Configuration

The dotenv file and S3APP_DOTENV_FILE variable

You can customize S3App with a few parameters. All parameters must be set in a config file with a name like .s3app. The name can be chosen freely and you can put the file in a folder of your choice but the file must be reachable by the S3App App.

Additionally, you need a environment variable S3APP_CONF_FILE with the path to your config file.

For example, for Linux you can set the environment variable with:

export S3APP_CONF_FILE=/etc/s3app/.s3app

The config file can contain the following parameters:

S3APP_APP_NAME = "My Fancy S3 App"
S3APP_APP_ICON = "https://myserver.com/my_own_logo.svg"
S3APP_LOG_LEVEL = "debug"
S3APP_LOG_FILE = ./s3app.log 
S3APP_DB_TYPE = "postgresql"
S3APP_PG_DB_HOST = "k8s-1"
S3APP_PG_DB_NAME = "s3app"
S3APP_PG_DB_PORT = 5432
S3APP_PG_DB_USER_PW = "s3app"
S3APP_PG_DB_USER_NAME = "s3app"
S3APP_SECRET_KEY = "thisIsMyHiddenSecretKey"

The parameter of the config file

name	example	description	possible values	default
S3APP_APP_NAME	My Fancy S3 App	The name of your S3App. It is display in the head of your application.
S3APP_APP_ICON	https://myserver.com/my_own_logo.svg	The icon of your S3App as a html link. It is display in the head of your application.
S3APP_LOG_LEVEL	info	LogLevel for the Logging. Can be	fatal, error, warning, info, debug	info
S3APP_LOG_FILE	/var/log/s3app.log	When set logs are written to the mentioned file. If not set it is written to console (stdout).		not set
S3APP_DB_TYPE	postgresql	Type of database to use. Can be a SQLite or a PostgreSQL Database	sqlite, postgres	sqlite
S3APP_PG_DB_HOST	k8s-1.mydatabse.com	Only if db-type postgres: Host name of the PostresSQL server.
S3APP_PG_DB_PORT	5432	Only if db-type postgres: Port number of the PostresSQL server.		5432
S3APP_PG_DB_NAME	s3app	Only if db-type postgres: Name of the PostgreSQL Database.
S3APP_PG_DB_USER_NAME	s3app	Only if db-type postgres: Username to authenticate against the PostgreSQL Database
S3APP_PG_DB_USER_PW	s3app	Only if db-type postgres: Password to authenticate against the PostgreSQL Database
S3APP_SECRET_KEY	thisIsMyHiddenSecretKey	A key which used to sign session cookies for protection against cookie data tampering. In production please change it.