th2 gRPC crawler data processor library (0.2.1)

This project contains the gRPC interface to implement if you want to create your own crawler data processor.

The crawler data processor work with the crawler

How to transform template

Create a directory with the same name as project name (use underscores instead of dashes) under src/main/proto directory (remove other files and directories if they exist).
Place your custom .proto files in created directory. Pay attention to package specifier and import statements.
Edit release_version and vcs_url properties in gradle.properties file.
Edit rootProject.name variable in settings.gradle file. This will be the name of Java package.
Edit package_info.json file in order to specify name and version for Python package (create file if it's absent).
Edit parameters of setup.py in setup function invocation such as: author, author_email, url. Do not edit the others.
Edit README.md file according to the new project.

Note that the name of created directory under src/main/proto directory is used in Python (it's a package name).

How to maintain project

Make your changes.
Up version of Java package in gradle.properties file.
Up version of Python package in package_info.json file.
Commit everything.

How to run project

Java

If you wish to manually create and publish package for Java, run these command:

gradle --no-daemon clean build publish artifactoryPublish \
       -Pbintray_user=${BINTRAY_USER} \
       -Pbintray_key=${BINTRAY_KEY}

BINTRAY_USER and BINTRAY_KEY are parameters for publishing.

Python

If you wish to manually create and publish package for Python:

Generate services by gradle:
```
   gradle --no-daemon clean generateProto
```
You can find the generated files by following path: src/gen/main/services/python

Generate code from .proto files and publish everything:

pip install -r requirements.txt
python setup.py generate
python setup.py sdist
twine upload --repository-url ${PYPI_REPOSITORY_URL} --username ${PYPI_USER} --password ${PYPI_PASSWORD} dist/*

PYPI_REPOSITORY_URL, PYPI_USER and PYPI_PASSWORD are parameters for publishing.

Changes:

v0.2.1

max_message_size in DataProcessorInfo allows telling Crawler send messages not greater than maxMessageSize

v0.2.0 (Breaking changes)

Braking:

Use list of MessageID instead of mapping between session and MessageID. User now will have to specify MessageID for both directions in the response if he or she needs to set a checkpoint. The list should contain a single MessageID for each pair alias + direction. If more than one is found the last one (according to their sequences) will be taken.
The rpc methods was renamed according to the Protobuf naming convention (uses PascalCase).
The event and message IDs from the response to connect method are removed because this functionality requires additional improvements on Crawler's side.

Added:

New method that will be invoked by the crawler each time the new interval is started.

Metadata

Release files for th2-grpc-crawler-data-processor 0.2.0.dev1769603491

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for th2-grpc-crawler-data-processor 0.2.0.dev1769603491
File	Size	Uploaded
th2_grpc_crawler_data_processor-0.2.0.dev1769603491.tar.gz	10.3 kB	Jan 30, 2022	Details

Release files / th2_grpc_crawler_data_processor-0.2.0.dev1769603491.tar.gz

Download URL	th2_grpc_crawler_data_processor-0.2.0.dev1769603491.tar.gz
Size	10.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`fdf30e1028e28c9c2cbb8c501feb7909899597a7bf418ac6c1b5002cd267b8d2`
BLAKE2b-256 checksum How to use checksums	`eddf681aa1ac019efa76ab432dc785dd2e7fcbbf0062ea05c09c2eb69a407e06`
Upload date	Jan 30, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.7.1 importlib_metadata/4.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.10`