Library for retrieving and uploading to database data from amocrm API
Project description
Amocrm custom library
- Retrieving data from amocrm amocrm.api (v2 and v4).
- Uploading data to database.
Supported databases:
- clickhouse
- vertica
Usage
pip3 install amocrm-talenttech-oss
Retrieving data from API:
import datetime
from api.api_loader_amocrm_v4 import AmocrmApiLoader as ApiLoaderV4
date_modified_from = datetime.datetime.now() - datetime.timedelta(days=1) #
args_api = {
"amocrm_api_url": "https://<NAMESPACE>.amocrm.ru/amocrm.api/v4/<ENTITY>?page={page}&limit={limit}",
"AUTH_URL":"https://<NAMESPACE>.amocrm.ru/oauth2/access_token",
"CLIENT_SECRET":"xxxx",
"CLIENT_ID": "xxx-xxx-xxxx-xxxx-xxxxxxx",
"REDIRECT_URL":"https://xxxx/xx"
}
args_s3 = {
"aws_access_key_id": <S3_ACCESS_KEY>,
"aws_secret_access_key": <S3_ACCESS_SECRET>,
"endpoint_url": <S3_ENDPOINT_URL>,
"bucket": <S3_BUCKET>
}
api_loader = ApiLoaderV4(
entity=<ENTITY>, # leads/tasks/companies or e.t.c
s3_path=<S3_PATH>, # this s3 folder should contain retrieved files
s3_token_path=<S3_TOKEN_PATH>, # directory for tokens
args_s3=args_s3,
args_api=args_api,
date_modified_from=date_modified_from, #parameters is optional, if want load only updated records from date
with_offset=True,
batch_api=500
)
api_loader.auth(<CODE_AUTH>) #call it if you need to create or regenerate refresh token the first time
api_loader.extract()
Uploading data to vertica:
from db.vertica_uploader import UploaderDB as VerticaUploaderDB
sql_credentials = {
"database": <DATABASE>,
"schema": <SCHEMA>
"user": <VERTICA_WRITE_USER>,
"host": <VERTICA_HOST>,
"port": <VERTICA_PORT>,
"password": <VERTICA_WRITE_PASSWORD>,
"vertica_configs": <VERTICA_CONFIGS>,
}
args_s3 = {
"aws_access_key_id": <S3_ACCESS_KEY>,
"aws_secret_access_key": <S3_ACCESS_SECRET>,
"endpoint_url": <S3_ENDPOINT_URL>,
"bucket": <S3_BUCKET>
}
db_uploader = VerticaUploaderDB(
args_s3=args_s3,
s3_path=s3_path,
sql_credentials=sql_credentials,
entity=<ENTITY>,
table_name=<TABLE_NAME>,
json_columns=[<COLUM_JSON_1>, <COLUM_JSON_2>]
)
db_uploader.load_s3_to_db()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for amocrm-talenttech-oss-0.0.6.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5466b67b323f1880c555b29a5d5ca102ddda89371e572f05272bb212b6d9a376 |
|
MD5 | 63ab8858d55eb996d8eaec3d2662ead9 |
|
BLAKE2b-256 | 9e06a595d9760f1e3aad2716d44519ac56e7a5ccc85b1c2a6df4e0bf4e3c290a |
Close
Hashes for amocrm_talenttech_oss-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e8e78bae693df698c8cf504a620361c823c6ac8a1019b12e5dfa4dfb9722c39c |
|
MD5 | c7440a1cf05882eb9ce3032cf15cb1e0 |
|
BLAKE2b-256 | 0c37bbd8807148a9e4f0ff4b3a9a70c4a389cca9a2f51928e1ed534d5fc0e6e5 |