insyt-secure 0.3.3

A proprietary service for secure database and API interactions

Insyt Secure

A proprietary service for securely interacting with databases, APIs, and other services within a secure environment. Visit https://insyt.co for more information.

Quick Start

Installation

# Basic installation (core functionality only)
pip install insyt-secure

Advanced Installation Options

Insyt Secure provides flexible installation options to include only the dependencies you need:

# Install with PostgreSQL support
pip install "insyt-secure[postgres]"

# Install with MongoDB support
pip install "insyt-secure[mongodb]"

# Install with multiple database support
pip install "insyt-secure[postgres,mongodb,redis]"

# Install with vector database support
pip install "insyt-secure[pinecone]" # or any other vector DB

# Install with cloud provider support
pip install "insyt-secure[aws]" # or azure

# Install with messaging system support
pip install "insyt-secure[kafka]" # or rabbitmq, pulsar

# Complete installation with all dependencies
pip install "insyt-secure[all]"

Available extension categories:

• SQL databases: postgres, mysql, mssql, oracle, clickhouse, snowflake, duckdb
• NoSQL databases: mongodb, redis, cassandra, neo4j, elasticsearch, couchdb
• Vector databases: pinecone, qdrant, milvus, weaviate, chroma, faiss
• Cloud services: aws, azure
• Messaging systems: kafka, pulsar, rabbitmq

Broader categories are also available: rdbms, nosql, vector, cloud, messaging

Basic Usage

# Run with a single project
insyt-secure --projects "your-project-id-123:your-api-key-xyz"

Multi-Project Support

Insyt Secure supports managing one or more projects simultaneously:

# Run with multiple projects
insyt-secure --projects "project-id-1:api-key-1,project-id-2:api-key-2"

Each project connection is managed independently, allowing for simultaneous interactions with multiple Insyt projects, each with its own topic subscriptions and credential management.

Getting Help

# View all available options and examples
insyt-secure --help

The help command provides detailed information about all parameters, their defaults, and usage examples directly in your terminal.

Advanced Options

# Run with all options (for a single project)
insyt-secure \
  --projects "your-project-id-123:your-api-key-xyz" \
  --max-workers 10 \
  --execution-timeout 60 \
  --allowed-ips "192.168.1.1,10.0.0.1:3456"

Logging Options

By default, logs are user-friendly and redact sensitive information. You can customize logging behavior:

# Enable more detailed debug logs
insyt-secure --projects "your-project-id:your-api-key" --debug

# Show more verbose logs including from third-party libraries
insyt-secure --projects "your-project-id:your-api-key" --verbose

# Output logs in JSON format (useful for log processing systems)
insyt-secure --projects "your-project-id:your-api-key" --json-logs

# Disable redaction of sensitive information (not recommended for production)
insyt-secure --projects "your-project-id:your-api-key" --show-sensitive

You can also control the log level via environment variables:

# Set log level using environment variable
INSYT_LOG_LEVEL=DEBUG insyt-secure --projects "your-project-id:your-api-key"

# Configure projects using environment variable (optional alternative to --projects)
INSYT_PROJECTS="project-id-1:api-key-1,project-id-2:api-key-2" insyt-secure

Cross-Platform Compatibility

Insyt Secure is designed to work seamlessly on all major platforms:

• Windows: Fully supported natively, no additional configuration needed
• macOS: Fully supported
• Linux: Fully supported

The service uses paho-mqtt with a platform-agnostic implementation to ensure consistent behavior across all operating systems.

Command Line Parameters

Parameter	Description	Default
--projects	Comma-separated list of project-id:api-key pairs for managing one or more projects	-
--max-workers	Maximum number of concurrent code executions per project	5
--execution-timeout	Default maximum execution time in seconds per code snippet	30
--allowed-ips	Comma-separated list of allowed IPs/hostnames	All allowed
--verbose	Enable more verbose logs	False
--debug	Enable debug level logging	False
--json-logs	Output logs in JSON format	False
--show-sensitive	Show sensitive information in logs	False

Credentials Management

This service automatically retrieves and manages connection credentials:

1. When the service starts, it gets credentials from the Insyt API for each configured project
2. If the connection drops or credentials expire, it automatically requests new credentials
3. Each project's credentials are managed independently

Project Management

Insyt Secure supports managing one or more projects simultaneously:

Configuration Options

You can configure projects in two ways:

1. Command line parameter:

   insyt-secure --projects "project-id-1:api-key-1,project-id-2:api-key-2"
   

2. Environment variable:

   export INSYT_PROJECTS="project-id-1:api-key-1,project-id-2:api-key-2"
   insyt-secure
   

Benefits of Multi-Project Mode

• Resource Efficiency: All projects share a single process but maintain independent connections
• Simplified Management: Manage multiple projects from a single service instance
• Independent Reconnection: Each project can reconnect independently if its connection fails
• Shared DNS Cache: All projects benefit from the DNS cache for improved resilience

API Usage for Multiple Projects

You can also use the project management programmatically:

from insyt_secure.project_manager import ProjectManager
import asyncio

async def run_projects():
    manager = ProjectManager()
    
    # Add one or more projects
    await manager.add_project("project-id-1", "api-key-1", "https://api.account.insyt.co/")
    await manager.add_project("project-id-2", "api-key-2", "https://api.account.insyt.co/")
    
    # Start the manager
    await manager.start()
    
    # Keep the service running
    while True:
        await asyncio.sleep(60)

if __name__ == "__main__":
    asyncio.run(run_projects())

Use Cases

Data Processing Microservice

Perfect for running data transformation code that connects to various data sources:

insyt-secure --projects "your-project-id:your-api-key" --max-workers 15

Secure Environment for Code Testing

Create a sandboxed environment with restricted network access:

insyt-secure --projects "your-project-id:your-api-key" \
  --allowed-ips "10.0.0.1,192.168.1.100" --execution-timeout 20

Multi-Project Service

Run a single service that executes code for multiple projects:

insyt-secure --projects "PRJ-A235466:api-key-1,PRJ-A235477:api-key-2,ai-models:api-key-3" \
  --max-workers 10 --execution-timeout 60

Containerized Deployment

docker run -d --name insyt-secure \
  insyt-secure insyt-secure \
  --projects "your-project-id:your-api-key"

System Requirements and Dependencies

Insyt Secure is designed with a modular dependency structure to minimize installation size and resource usage. Below is a breakdown of what's included in each installation option:

Core Dependencies (included in base install)

The base installation includes:

• HTTP client capabilities via httpx
• MQTT connectivity via paho-mqtt
• Basic data science libraries: NumPy, Pandas, SciPy
• JSON and date handling utilities

Optional Dependencies

SQL Database Connectors

• postgres: High-performance PostgreSQL client (asyncpg)
• mysql: MySQL client libraries
• mssql: Microsoft SQL Server connectivity
• oracle: Oracle database connectivity
• clickhouse: ClickHouse analytics database client
• snowflake: Snowflake data warehouse client
• duckdb: Embedded analytical database

NoSQL Database Connectors

• mongodb: MongoDB client with async support
• redis: Redis client
• cassandra: Apache Cassandra and ScyllaDB clients
• neo4j: Neo4j graph database client
• elasticsearch: Elasticsearch search engine client
• couchdb: CouchDB document database client

Vector Database Connectors

• pinecone: Pinecone vector database client
• qdrant: Qdrant vector search engine client
• milvus: Milvus vector database client
• weaviate: Weaviate vector search engine
• chroma: ChromaDB for AI embeddings
• faiss: Facebook AI Similarity Search

Cloud Services

• aws: AWS SDK (boto3) with S3, Dynamo, etc.
• azure: Azure clients for Cosmos DB, Blob Storage, etc.

Messaging Systems

• kafka: Apache Kafka client
• pulsar: Apache Pulsar client
• rabbitmq: RabbitMQ client

Performance Considerations

The base installation already includes the core ML libraries (numpy, pandas, etc.). If you're installing on a resource-constrained environment, consider using only the specific connector extensions you need rather than the broader categories.

For production deployments, we recommend specifying exact dependencies rather than using broader categories:

# Good (minimal dependencies)
pip install "insyt-secure[postgres,redis]"

# Less efficient (pulls in many unused dependencies)
pip install "insyt-secure[rdbms,nosql]"

Platform-Specific Installation

Insyt Secure is designed to work on all major platforms without modification:

Windows

# Install on Windows
pip install insyt-secure

# Run (in PowerShell or Command Prompt)
insyt-secure --projects "your-project-id-123:your-api-key-xyz"

macOS/Linux

# Install on macOS/Linux
pip install insyt-secure

# Run
insyt-secure --projects "your-project-id-123:your-api-key-xyz"

Docker

# Create a simple Dockerfile
echo 'FROM python:3.10-slim
RUN pip install insyt-secure
ENTRYPOINT ["insyt-secure"]' > Dockerfile

# Build the Docker image
docker build -t insyt-secure .

# Run in Docker
docker run insyt-secure --projects "your-project-id-123:your-api-key-xyz"

Platform-Specific Considerations

• Windows: Works natively without WSL, using a cross-platform MQTT implementation
• MacOS/Linux: All features fully supported
• Docker: Ideal for deployment in containerized environments

Installing on a Fresh Ubuntu Instance

Follow these steps to set up insyt-secure on a fresh Ubuntu installation:

# Step 1: Install required packages
sudo apt update
sudo apt install python3-pip
sudo apt install python3-venv python3-full

# Step 2: Create a directory for your project (optional)
mkdir myproject
cd myproject

# Step 3: Create the virtual environment
python3 -m venv venv

# Step 4: Activate the virtual environment
source venv/bin/activate
# Your command prompt should change to show the virtual environment name
# Example: (venv) ubuntu@ip-100-1-11-111:~/myproject$

# Step 5: Install insyt-secure with desired extensions
pip install insyt-secure[mysql,postgres]
pip install python-dotenv

Notes:

1. To run the service, use the following command:

insyt-secure --projects "your-project-id-123:your-api-key-xyz"

or run the service in the background:

nohup insyt-secure --projects "your-project-id-123:your-api-key-xyz" &

2. Remember to activate the virtual environment each time you want to use this package in a new terminal session.

source venv/bin/activate

3. To deactivate the virtual environment when you're done:

deactivate

4. To stop the service, use the following command:

pkill -f insyt-secure

5. To check the status of the service, use the following command:

ps aux | grep insyt-secure