recon-lw 3.0.0.dev8944172662

recon_lw

Recon-lw

Table Of Contents

• Overview
• Module structure
• Recon-lw Building Blocks
• Example

Overview

This library allows user to:

1. Match message streams by using different rules
2. Process found matches to check if they have any discrepancies
3. Publish events on found discrepencies, draw statistics reports and categorise errors.

Module structure

Core

Contains core classes and utility functions that are used in matching/interpretation reconciliation phases.

• recon_lw/core/rule - Defines rule objects which contains everything is needed for recon execution:
  • collector function/object
  • flush function/object
  • key functions/object
  • matching function/object
• recon_lw/core/type - Defines types used in other entities
• recon_lw/utility - Defines recon_lw core utility functions:
  • getting stream from file
  • common state manipulations
  • streams syncing
  • batch extraction

Interpretation

Contains interfaces, classes and utility functions that are used to interpret matching results.

• recon_lw/interpretation/adapter - Adapter definition and implementations.
• recon_lw/interpretation/check_rule - Check rules. Basic field comparison functions/objects.
• recon_lw/interpretation/condition - Condition. Common condition functions/objects.
• recon_lw/interpretation/converter - Converter. Common field converter functions/objects.
• recon_lw/interpretation/field_checker/ - Field checker. Basic comparison functions.
• recon_lw/interpretation/field_extractor/ - Field extractor. Common functions/objects that allows to extract fields from messages in different ways.
• recon_lw/interpretation/filter/ - Filter. Common filters functions/objects to filter message streams.
• recon_lw/interpretation/interpretation_functions - Interpretation function. Basic functions to interpret matching messages and publish recon events.
  • recon_lw/interpretation/interpretation_functions/event_enhancement - Common event enhancement functions/objects that allows to add additional information to recon events.
  • recon_lw/interpretation/interpretation_functions/event_handling_strategy - Basic event handling strategies ( how to construct recon event on certain condition ) for miss and match events.
  • recon_lw/interpretation/interpretation_functions/event_name_provider - Common event name provider definitions. Describes how to provide name for certain event type.

Matching

Contains interfaces, classes and utility functions that are used to match message streams.

• recon_lw/matching/collect_matcher - Common collector functions.
• recon_lw/matching/flush_function - Common flush functions
• recon_lw/matching/key_functions - Common key functions
• recon_lw/matching/old - Old collector, flush, matching functions to be backward compatible with older recon versions.
• recon_lw/stream_matcher/ - Common stream matching functions. Different stream matching algorithms.

Reporting

Defines common report viewers, match and miss error categorisers.

• recon_lw/reporting/coverage/viewer - Fields coverage report. Fields covered by recon.
• recon_lw/reporting/known_issues - [Known issues][#known-issues]. Defines data classes for known issues that are used in categorisation.
• recon_lw/reporting/match_diff
  • categorizer - Defines common match diff categorizer which categorises messages that were matched but some fields aren't equal.
  • viewer - Defines common match diff report viewer which displays found match diff categories.
• recon_lw/reporting/missing_messages
  • categorizer - Defines basic miss_categoriser which categorises missed messages.
  • viewer - Defines basic miss category viewer which shows table after categorisation.
• recon_lw/reporting/recon_context - Defines recon context entity that is used in reporting classes to get recon events and to know how to extract fields from messages/events.
• recon_lw/reporting/stats - Defines data classes where recon run statistics is collected.

Recon-lw Building Blocks

Entrypoint

Recon execution can be started by calling entrypoint execute_standalone function:

def execute_standalone(message_pickle_path: Optional[str], sessions_list: Optional[List[str]], result_events_path: Optional[str],
                       rules: Dict[str, Dict[str, Union[Dict[str, Any], AbstractRule]]],
                       data_objects: List[Data]=None,
                       buffer_len=100):
  """
  params:
    :param message_pickle_path - optional path to directory with pickle file that contains th2 messages
    :param rules - a bunch of rules describing how to reconcile multiple streams.
    :param sessions_list - optional list of sessions that should be readed from pickle files
    :param result_events_path - path to directory where recon events should be published.
    :param data_objects - optional list of data objects.
    :param buffer_len - how many messages should be retrieved from data objects list in every iteration. 
  """
...

General execution flow

Common recon execution consist of the following steps:

1. Retrieving batch of messages from data objects
2. Calling collector function on this batch.
3. Calling matcher function on this batch to find new matches.
4. Calling flush matcher to filter out finalized message matches.
5. Callint interpretation function to generate recon events on found matches.

ReconMatcher

• Interface: here.
• Usage: This entity allows user to define streams matching logic.

Implementations:

• OneToMany
  • This matcher matches one message from first stream with multiple messages from another stream.
• PairOne
  • This matcher matches two messages from first stream with one message from second stream.

Matching key extractor

• Interface: link
• Usage: This entity defines how key from message should be extracted. Extractors are used in matchers to extract key in matchers to use extracted keys for messages matching.

Implementations:

• SeparatorKeyExtractor: this key extractor extracts key from message based on list of fields which combines key and then join values using provided separator.

Key Function

• Interface: link
• Usage: This entity is used to filter messages to match from stream and to extract key from messages that passed through filter.

Implementations:

• SimpleOrigKeyFunction
• SimpleCopyKeyFunction

Flush function

• Interface: link
• Usage: This entity describes what should be done with found matches between messages streams:

Implementations:

• DefaultFlushFunction: this flush function collects messages that were matched and passes them into interpretation function which generates events and publish received events.

Collect Matcher

• Interface: link
• Usage: This entity describes how and in which order we want to process incoming stream messages batches.

Implementations:

• DefaultCollectMatcher: This implementation just calls stream matcher implementation delegated to it on every new stream batch.

Adapter

• Interface: link
• Usage: The adapter component serves as a crucial intermediary in standardizing data from diverse streams into a uniform format. Its primary purpose is to facilitate seamless comparison across these streams by abstracting away the need for bespoke comparison implementations tailored to each stream pairing. This abstraction minimizes redundancy and enhances efficiency in data processing workflows.

Adapter Context

Adapter context is accessible from adapter. Adapter context can be used to get/put some data into cache by other building blocks or adapter itself.

Implementation

• SimpleAdapter - this is basic adapter. This adapter uses passed by user mapping from common field name to extractor which defines how field should be extracted from message.

• CompoundAdapter - this adapter allows user to use different Adapters for different scenarios. There is multiple extractors that are chosen based on conditions

Extractor

• Interface: link
• Usage: This building block is used in adapter to define how to get certain field from message.

Implementations:

• AnyValExtractor: returns AnyVal class instance. AnyVal class will return true when compared with any other value except for NOT_EXTRACTED value.
• SimpleCacheExtractor: returns field value by field name if exists. If not cache is checked. Cache is filled when field value by field name can be extracted.
• CacheFillWithConditionExtractor: updates the cache only if condition is true.
• ConditionExtractor: extracts value with one extractor if certain condition is met and uses another extractor if condition isn't met.
• ConditionMaskExtractor: masks value if condition is met and returns value if condition isn't met
• ConstantExtractor: returns same value every time certain field extraction is issued.
• NEConstantExtractor: returns NOT_EXTRACTED value every time certain field extraction is issued.
• BasicConverterExtractor: extracts field value with base extractor and then applies converter to it.
• ChainConverterExtractor: extracts field value with base extractor and then applies chain of converts to extracted value.
• BasicDictExtractor: extracts field value by field name from message dictionary without permutations.
• ListAggregationExtractor: TODO
• OneOfExtractor: extracts values for different field names using different extractors and the first one that is not null is returned to issuing code.
• SimpleRefDataFieldExtractor: extracts field value from message and gets refdata value corresponding to field value.

Filter

• Interface: link
• Usage: Defines streams filtering rules.

Implementations:

• SessionAliasFilter - enables filtering by session alias.
• MessageTypeFilter - enables filtering by message type.
• FunctionFilter - enables filtering by custom user function.
• FilterChain - enables multifilters filtering.
• FieldFilter - enables filtering by certain field whitelisted values.
• DummyFilter - every message isn't filtered.
• AmendRejectFilter - rejects messages with certain reject texts/codes

FieldChecker

• Interface: link
• Usage: Accepts two messages and returns iterator with comparison results comparison being made. This can be used in interpretation function to mark discrepancies.

Implementations:

• SimpleFieldChecker: Accepts list of fields that needs to be compared and comparison strategy for each field and executes comparison field by field.

CheckRule

• Interface: link
• Usage: Defines comparison logic for field inside messages from different streams.

Implementations:

• IAdapterFieldCheckRule: Abstract check rule which uses adapters inside comparison method
• EqualFieldCheckRule: check rule based on IAdapterFieldCheckRule which extracts field value from messages using adapters and then compares then applying simple equality check.

FieldCheckResult

Data class which holds comparison result related informations such as: left stream field value, right stream field value, field name, comparison result and comment

Condition

• Interface: link
• Usage: Helper entity which can be used in other entities implementations. Accepts message and executes some condition under it.

Implementation:

• [FunctionCondition]: Accept user function which executes condition on message.

Converter

• Interface: link
• Usage: Helper entity which can be used in other entities implementations. Accepts one value and returns another value based on internal logic.

Implementations:

• BooleanConverter: Accepts value and maps it to boolean based on user provided mapping
• ChainConverter: Accepts value and applies list of convertes to it one by one.
• FirstNonNullChainConverter: Applies converters to value until first non NOT_EXTRACTED value.
• ConditionConverter: Applies one converter if condition is met and applies another converter if condition isn't being met.
• ConstantConverter: Applies no convertations.
• DateTimeConverter: Takes value and casts it to datetime based on formatting string.
• DateConverter: Takes value and casts it to date based on formatting string.
• DictPathConverter: Accepts dictionary field value and gets value under specified path inside dictionary.
• EmptyStringConverter: Returns empty string for each value.
• FunctionConverter: Accepts user converter function and applies it to value.
• LengthConverter: Accepts value with length attribute and returns its length.
• MappingConverter: Accepts value and user map and returns mapped value for input value.
• IndexListConverter: Extracts certain list element based on user index function.
• RegexConverter: Applies regex to value and returns matches list.
• TypeConverter: Casts value to selected type.

Interpretation Function Provider

• Interface: link
• Usage: This entity defines logic for matched messages interpretation.

Implementations:

• BasicInterpretationFunctionProvider: categorises matched message by four categories: match, match with diff, miss left, miss right. Generates events for different categories using different strategies for match and miss categories. Returns list of events.

Event Name Provider

• Interface: link
• Usage: Allows user to define how event names for different match categories should be constructed

Implementations:

• SimpleMatchEventHandlingStrategy: defines default event names

EventHandlingStrategy

• Interface: link
• Usage: Describes how event should be constructed.

Implementations:

• SimpleMatchEventHandlingStrategy: compares fields values using field checker and publishes diff into event. Applies event enhancements.
• SimpleMissEventHandlingStrategy : publishes event with event type ReconEventBasicMissLeft or ReconEventMissRight based on is_copy parameter. Applies event enhancement functions.

Reporting

These entities are used to give some stats on events gathered during reconcillation.

Coverage

• Implementation: link
• Usage: Displays Recon metadata object in form of the table where covered fields and descriptions are shown.

Known Issues

• Implementation: link
• Usage: Data class to store all known issues that were found during previous recon run.

Match Diff

Match Diff Categoriser

• Interface: link
• Usage: Defines how category names should be extracted from recon events.

Implementations:

• BasicErrorCategoriser - describes simple error categoriser which collects categories basing on category extractor strategy which desribes the way to extract category name for different event types:
  • miss left
  • miss right
  • match
  • match diff

Match Diff Report Viewer

• Implementation: link
• Usage: Displays found error categories and examples for them in form of comparison table.

Match Diff Color provider

• Interface: link
• Usage: Allows to define color for different categories types during report displaying.

Match Diff Content provider

• Interface: link
• Usage: Allows to define how content for example should be extracted.

Match Diff Style Provider

• Interface: link
• Usage: Allows to define to provide different css styles for html report table.

Implementations:

• DefaultStyleProvider - defines default styles.

Missing Messages

Missing Messages Categoriser

• Interface: link
• Usage: Allows to define how to extract category from recon event.

Implementations:

• SimpleMissesCategorizer - defines straightforward categorisation based on miss category rules

Missing Messages Rule

• Interface: link
• Usage: Defines events filter to filter events that falls into issue from rule or not.

Missing Messages Viewer

• Implementation: link
• Usage: Defines basic miss categories table html viewer.

Example

Example featuring main functionality of the library can be found here