ETL Lib
This is a simple package built for Portobello Shop integrations, and now is available as ETL lib.
Can be used with several services, like:
- Oracle
- Salesforce
- MSSQL Server
- GSuite
- Microvix
- ODBC Drivers: PYODBC and SQLALCHEMY
- And any REST services ...
Comming soon:
- RabbitMQ
- Kafka
- OpenVault (Credentials)
- ...
Basic Usage
Rewrite ETL ABC class
Use inheritance and rewrite the three abstract methods (even if you don't use it):
from etl_pbshop import ETL, Connectors
class MyETL(ETL):
def __init__(self):
self.config = MyConfiguration()
self.log = self.config.log
self.connector = Connectors(
config=self.config,
log=self.log,
services={'google': 'PROD', "salesforce": "PROD"}
)
def extract(self):
pass # extract the needed data using Connectors
def transform(self):
pass # do some transformations
def load(self):
pass # upload your transformations
Main caller:
On the main caller, you can simply:
if __name__ == '__main__':
etl = MyETL()
etl.config.start()
try:
etl.run()
exit(0)
except Exception as e:
etl.get_error(f"ERROR on main: {str(e)}")
raise e
finally:
etl.config.finish()
Rewriting default ConfigModel:
In another class, you can rewrite the ConfigModel class and input your values:
from etl_pbshop.config_model import ConfigModel
class MyConfiguration(ConfigModel):
def __init__(self):
super().__init__(__file__)
self.SQL_QUERIES = {
"produtos": "SELECT * FROM Products2"
}
self.INTEGRATION_SPREADSHEET_ID = '654asd5as1das5d165a4d68'
Environment variables:
You can put your environment variables to use the Connectors class.
In a .env file, set it in groups, like:
INTEGRACAO_ORACLE_ID_1=1
INTEGRACAO_ORACLE_TOKEN_VALUE_1=asd1234asd==
INTEGRACAO_ORACLE_TOKEN_URL_1=http://example.com/token
INTEGRACAO_ORACLE_METHOD_URL_1=http://example.com/method_to_post
INTEGRACAO_ORACLE_USER_1=asd..
INTEGRACAO_ORACLE_PASS_1=qwerty..
INTEGRACAO_ORACLE_ENVIRONMENT_1=DEV
INTEGRACAO_MICROVIX_ID_1=1
INTEGRACAO_MICROVIX_TOKEN_VALUE_1=1234-123-123-1234
INTEGRACAO_MICROVIX_METHOD_URL_1=https://webapi.microvix.com.br/1.0/api/method
INTEGRACAO_MICROVIX_USER_1=user
INTEGRACAO_MICROVIX_PASS_1=pass
INTEGRACAO_MICROVIX_ENVIRONMENT_1=PROD
INTEGRACAO_SALESFORCE_ID_1=1
INTEGRACAO_SALESFORCE_TOKEN_VALUE_1=qwerty
INTEGRACAO_SALESFORCE_METHOD_URL_1=https://login.salesforce.com/services/Soap/u/47.0
INTEGRACAO_SALESFORCE_USER_1=username@example.com
INTEGRACAO_SALESFORCE_PASS_1=myPass
INTEGRACAO_SALESFORCE_ENVIRONMENT_1=PROD
You only need to keep the pattern: INTEGRACAO_<service_name>_<key_name>_<group_id>
More info, please contact: daniel.camargo@portobelloshop.com.br
1 maintainer
Release files for etl-pbshop 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| etl-pbshop-0.3.0.tar.gz | 15.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| etl_pbshop-0.3.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 32.2 kB
Release files / etl-pbshop-0.3.0.tar.gz
| Download URL | etl-pbshop-0.3.0.tar.gz |
|---|---|
| Size | 15.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f494670530f610f272bc5cfcc19edc9c681c2bc2237bd6a69c9f42077971fb65
|
|
BLAKE2b-256 checksum How to use checksums |
7ccc80ae260f28bfad97ecc3d0dc4a22b3d14d80537a19926e8ef14a0607c378
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5
|
Release files / etl_pbshop-0.3.0-py3-none-any.whl
| Download URL | etl_pbshop-0.3.0-py3-none-any.whl |
|---|---|
| Size | 16.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
64621b8c4cf052538422a358f9b7ba95cb5e2014bec86ec1c30a270aab8b4e87
|
|
BLAKE2b-256 checksum How to use checksums |
feb5d8e9d4911a1673954e166f7c8014dff04c866fd7416f9c2d6c92053fd73b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5
|
