neoaria-commons 0.1.9

neoaria commons module for neoaria-backend-generator

neoaria-commons

This software is one of the systems that form the basis of software produced by NeoAria Co., Ltd.

It mainly aims to provide the foundation for the back-end system, and among the foundations, it focuses on implementing and operating the source system (data, events, search, web socket).

Quick start

It can be installed through pip application.

• If installing from scratch :: pip install neoaria-commons

• In case of update :: pip install --upgrade neoaria-commons

How to use

First, after installation, access must be possible through internal Python.

DataSource

The data source class is a class that focuses on integrating and using various data sources. This class is closely related to the configuration and use of MongoDB, MariaDB, Kafka, Solr, and WebSocket, and the initial configuration values ​​must be defined by the system using this class.

Data sources have a total of three types. Fundamentally, it is divided into DataSource, which manages the state of data, EventSource, which processes data events, and SearchSource, which adjusts data to be searched efficiently.\

DataSource

• MongoDBDataSource

This is a source that is primarily intended to store and update the state of a specific object, and is the main database for the system currently being built in NeoAria.

• MariaDBDataSource

Although this source is not currently in active use, it is also a source used for the main purpose of creating and updating the state of data.

EventSource

• KafkaDataSource

This source is for event processing between external systems and will be the only system that can be used at present.

• RabbitMQDataSource

Support will be provided later.

SearchSource

• SolrDataSource

It is a data source that searches data by registering a schema, a series of data models, with the main role of searching through data indexing. It plays an important role in separating and reflecting the existing editing model and service model.

Business

Based on the actual business model, it is composed of BusinessParameter, BusinessResponse, and BusinessCommon and forms a common business.

BusinessParamter

It is a class that transmits parameters for executing a business, and its main purpose is to transmit an object with a Key-Value structure by inheriting BaseModel.

BusinessResponse

Business is a program that ultimately produces results. The main purpose of BusinessResponse is to contain and deliver the results. Like BusinessPrarameter, this also aims to convey values ​​centered on Key-Value.

BusinessCommon

As a common class for all businesses, all businesses must inherit and implement this class. Contains a list of most implementations.

Version History

0.1.0 :

Middleware Configuration

This project now supports middleware configuration through a centralized settings file. The available middleware options include:

• Logging: Customize logging behavior to track and store application logs.
• JWT (JSON Web Token): Configure JWT for secure token-based authentication.
• Gzip Response: Enable gzip compression for responses to improve performance.
• CORS (Cross-Origin Resource Sharing): Set up CORS policies to control resource sharing between different domains.
• System Preformance Monitoring for Prometheus: Integrate with Prometheus for system monitoring and metrics collection.

To configure these middleware options, please refer to the settings in the config/middleware.yaml file. Each section of the file corresponds to one of the middleware components, and you can modify the parameters according to your requirements.

The code to use is provided below.

from app.routers import router
from neoaria.middleware import setupMiddleware

app = createFastAPI()
app.include_router(router.router)

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)

And to use it this way, specify it in a similar format in the config/middleware.yaml file within your project folder as follows.

• The latest version adds the ability to send logs through kafka.

logging:
  enabled: True
  level: 'DEBUG' # DEBUG, INFO, WARNING, ERROR, CRITICAL
  include_headers: True # True, False
  type: kafka # http, syslog, kafka
  format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
  servers:
    http:
      host: localhost
      port: 8080
      endpoint: /log/create
    syslog:
      host: localhost
      port: 514
    kafka:
      producer:
        bootstrap.servers: '192.168.1.100:9091,192.168.1.100:9092,192.168.1.100:9093,192.168.1.100:9094,192.168.1.100:9095'
        client.id: 'token-manager'
        retries: 5
      topic: 'rms.log'
  
jwt:
  enabled: True
  secret_key: '******'
  algorithms: HS256
  targets:
    - /api/v1/*

  
gzip-response:
  enabled: True
  minimum_size: 1024

cors:
  enabled: True
  allow_credentials: True
  allow_origins: ['*']
  allow_methods: ['*']
  allow_headers: ['*']

performance-monitor:
  enabled: True

and source.yaml

mongodb_001:
  type: 'mongodb'
  data:
    id:  'mongodb_001'  
    uri: 'mongodb://user:pass@ip:port'
    database: 'token'
    collection: 'white_list'

mongodb_002:
  type: 'mongodb'
  data:
    id:  'mongodb_002'  
    uri: 'mongodb://user:pass@ip:port'
    database: 'token'
    collection: 'black_list'

kafka_001:
  type: 'kafka'
  data:
    id: 'kafka_001'
    consumer:
      settings:
        bootstrap.servers: 'ip1:port1,ip2:port2'
        group.id: 'kafka.group.id'
        auto.offset.reset: 'earliest'
        enable.auto.commit: True
      topics: 
        - 'kafka.topic.1'
    producer:
      settings:
        bootstrap.servers: 'ip1:port1,ip2:port2'
      topics: 
        - kafka.topic.1
        - kafka.topic.2

solr_001:
  type: 'solr'
  data:
    id: 'solr_001'
    url: 'http://ip:port/solr/collection_name'

websocket_001:
  type: 'websocket'
  data:
    id: 'websocket_001'

Common model information

Common models have been added for shared use. The common models are as follows:

1.	Package Name: neoaria.models.rms
2.	class DeployStatus(Enum): Enum representing deployment status
3.	class ElementStatus(Enum): Enum representing approval status
4.	class Language(Enum): Enum representing types of translation languages
5.	class FormType(Enum): Enum representing types of documents
6.	class ContentsType(Enum): Enum representing types of content
7.	class ImageContentsType(Enum): Enum representing types of images
8.	class TranslationStatus(Enum): Enum representing translation status
9.	class RmsAccount(BaseModel): Class representing account information
10.	class RmsDepartment(BaseModel): Class representing department information
11.	class RmsOrganization(BaseModel): Class representing organization information