sekoia-event-exporter 0.5.0

Export search job results from Sekoia.io API

Sekoia.io Event Exporter

A Python CLI tool to export search job results from the Sekoia.io API.

Features

• Secure by default: All exports are automatically encrypted with SSE-C
• Export search job results with a single command
• Automatic file download when exports complete
• Monitor export progress with real-time updates and ETA
• Check status of existing export tasks
• Configurable polling intervals and timeouts
• Custom S3 bucket support with full encryption control

Installation

From PyPI

The easiest way to install is from PyPI:

pip install sekoia-event-exporter

Or using uv for faster installation:

uv pip install sekoia-event-exporter

From Source

If you want to install from source or contribute to the project:

# Clone the repository
git clone https://github.com/sekoia-io/sekoia-event-exporter.git
cd sekoia-event-exporter

# Install with uv (recommended)
uv sync

# Or with pip
pip install .

For Development

# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh  # macOS/Linux
# or: powershell -c "irm https://astral.sh/uv/install.ps1 | iex"  # Windows

# Install with all development dependencies
uv sync --all-extras

# Or using pip
pip install -e ".[dev]"

Configuration

API Key

The tool requires a Sekoia.io API key to authenticate requests. Set it as an environment variable:

export API_KEY="your-sekoia-api-key-here"

Alternatively, you can create a .env file in your working directory:

API_KEY=your-sekoia-api-key-here

Required Permission:

The API key only needs the Massive export of events permission. When creating your API key in Sekoia.io, you can limit its scope to this single permission following the principle of least privilege.

Region Configuration

Sekoia.io operates multiple regional instances. By default, the tool uses the European region (api.sekoia.io). To use a different region, you can configure the API host in one of three ways:

Option 1: Environment Variable

export API_HOST="api.usa1.sekoia.io"  # For USA1 region

Or in your .env file:

API_HOST=api.usa1.sekoia.io

Option 2: Command-Line Argument

sekoia-event-export export <job_uuid> --api-host api.usa1.sekoia.io
sekoia-event-export status <task_uuid> --api-host api.usa1.sekoia.io

Option 3: Default

If neither is specified, the tool uses api.sekoia.io (European region).

For the complete list of available regions and their API endpoints, see the Sekoia.io Regions Documentation.

Custom S3 Configuration and SSE-C Encryption

⚠️ IMPORTANT: Exports are encrypted by default!

All exports are automatically encrypted with SSE-C (Server-Side Encryption with Customer-Provided Keys) using an auto-generated encryption key. You must save this key to download your export later.

You can also configure custom S3 settings for your exports:

• Export to your own S3 bucket
• Use your own SSE-C encryption key
• Customize S3 endpoint, region, and credentials

Default Behavior: Auto-Generated Encryption

By default, the tool generates a random 256-bit encryption key for each export:

# Encryption is automatic - the tool will display the generated key
sekoia-event-export export <job_uuid>

Output:

================================================================================
⚠️  SSE-C ENCRYPTION KEY AUTO-GENERATED
================================================================================
Encryption Key: dGhpc2lzYW5leGFtcGxla2V5Zm9ydGVzdGluZzEyMzQ1Njc4

⚠️  IMPORTANT: Save this key securely!
   You will need it to download this export later.
   If you lose this key, you will NOT be able to decrypt your data.
================================================================================

Using Your Own Encryption Key

To use a specific key instead of auto-generation:

# Generate your own key
export S3_SSE_C_KEY=$(openssl rand -base64 32)
echo "Your encryption key (save this securely): $S3_SSE_C_KEY"

# Use it for export
sekoia-event-export export <job_uuid>

# Or provide it directly
sekoia-event-export export <job_uuid> --s3-sse-c-key "<your-base64-encoded-key>"

Disabling Encryption

To disable SSE-C encryption (not recommended):

sekoia-event-export export <job_uuid> --no-sse-c

Full S3 Configuration

For complete control over S3 settings, you can configure multiple parameters:

# Using environment variables
export S3_BUCKET="my-export-bucket"
export S3_PREFIX="exports/"
export S3_ACCESS_KEY_ID="<access-key>"
export S3_SECRET_ACCESS_KEY="<secret-key>"
export S3_ENDPOINT_URL="https://s3.amazonaws.com"
export S3_REGION_NAME="us-east-1"
export S3_SSE_C_KEY="<base64-key>"

sekoia-event-export export <job_uuid>

Or using command-line arguments:

sekoia-event-export export <job_uuid> \
  --s3-bucket my-export-bucket \
  --s3-prefix exports/ \
  --s3-access-key <access-key> \
  --s3-secret-key <secret-key> \
  --s3-endpoint https://s3.amazonaws.com \
  --s3-region us-east-1 \
  --s3-sse-c-key <base64-key>

Automatic Download with SSE-C

When using SSE-C encryption, the tool automatically includes the required encryption headers when downloading. The same key used for export is automatically applied to the download.

If you need to manually download later with curl:

curl "<download-url>" \
  -H "x-amz-server-side-encryption-customer-algorithm: AES256" \
  -H "x-amz-server-side-encryption-customer-key: <base64-key>" \
  -H "x-amz-server-side-encryption-customer-key-MD5: <base64-md5>" \
  -o export.json.gz

Generating SSE-C Keys

SSE-C requires a 256-bit (32-byte) encryption key encoded in base64. Here are several ways to generate one:

Using OpenSSL (recommended):

# Generate a random 256-bit key and encode to base64
openssl rand -base64 32

Using Python:

python3 -c "import os, base64; print(base64.b64encode(os.urandom(32)).decode())"

Using /dev/urandom (Linux/macOS):

head -c 32 /dev/urandom | base64

Example output:

dGhpc2lzYW5leGFtcGxla2V5Zm9ydGVzdGluZzEyMzQ1Njc4

Important: Store this key securely! You'll need the same key to download encrypted exports. If you lose the key, you won't be able to decrypt your data.

Usage

Export Search Job Results

Trigger an export for a specific search job, monitor its progress, and automatically download the file:

sekoia-event-export export <job_uuid>

Example:

sekoia-event-export export 550e8400-e29b-41d4-a716-446655440000

The export will be automatically downloaded when ready. The file is saved as export_YYYYMMDD_HHMMSS.json.gz by default.

Specify fields to export:

By default, only message and timestamp fields are exported to avoid exporting unnecessary data. You can specify additional fields:

# Export specific fields
sekoia-event-export export <job_uuid> --fields "message,timestamp,source.ip,destination.ip"

# Or using environment variable
export EXPORT_FIELDS="message,timestamp,source.ip,user.name"
sekoia-event-export export <job_uuid>

Custom output filename:

sekoia-event-export export <job_uuid> --output my-export.json.gz
# or using short form
sekoia-event-export export <job_uuid> -o my-export.json.gz

Disable auto-download (just get the URL):

sekoia-event-export export <job_uuid> --no-download

With custom polling interval and timeout:

sekoia-event-export export <job_uuid> --interval 5 --max-wait 3600

Check Export Status

Check the current status of an export task:

sekoia-event-export status <task_uuid>

Example:

sekoia-event-export status 660e8400-e29b-41d4-a716-446655440001

The status command shows:

• Current export status (RUNNING, FINISHED, FAILED, etc.)
• Progress percentage and event count
• Download URL (when export is complete)

Note: The status command performs a single status check and does NOT download the file. Use the download command to download completed exports.

Download Export

Download a completed export:

sekoia-event-export download <task_uuid>

Example:

sekoia-event-export download 660e8400-e29b-41d4-a716-446655440001

Important: If the export was encrypted (default behavior), you must provide the same encryption key used during export:

# Provide the key that was displayed when you created the export
sekoia-event-export download <task_uuid> --s3-sse-c-key "<base64-key>"

Or set it as an environment variable:

export S3_SSE_C_KEY="<base64-key-from-export>"
sekoia-event-export download <task_uuid>

Custom output filename:

sekoia-event-export download <task_uuid> --output my-export.json.gz
# or using short form
sekoia-event-export download <task_uuid> -o my-export.json.gz

Command Options

export command

Basic Options:

• job_uuid (required): The UUID of the search job to export
• --api-host: API host to use (overrides API_HOST env var, default: api.sekoia.io)
• --interval: Polling interval in seconds (default: 2)
• --max-wait: Maximum wait time in seconds (default: no limit)
• --no-download: Don't download the file, just print the URL
• --output, -o: Output filename for the downloaded file (default: export_YYYYMMDD_HHMMSS.json.gz)
• --fields: Comma-separated list of fields to export (default: message,timestamp, overrides EXPORT_FIELDS env var)

S3 Configuration:

• --s3-bucket: S3 bucket name (overrides S3_BUCKET env var)
• --s3-prefix: S3 key prefix (overrides S3_PREFIX env var)
• --s3-access-key: S3 access key ID (overrides S3_ACCESS_KEY_ID env var)
• --s3-secret-key: S3 secret access key (overrides S3_SECRET_ACCESS_KEY env var)
• --s3-endpoint: S3 endpoint URL (overrides S3_ENDPOINT_URL env var)
• --s3-region: S3 region name (overrides S3_REGION_NAME env var)

SSE-C Encryption (enabled by default):

• --no-sse-c: Disable SSE-C encryption (encryption is enabled by default)
• --s3-sse-c-key: SSE-C encryption key, base64 encoded (auto-generated if not provided, overrides S3_SSE_C_KEY env var)
• --s3-sse-c-key-md5: SSE-C key MD5, base64 encoded (auto-computed if not provided)
• --s3-sse-c-algorithm: SSE-C algorithm (default: AES256)

status command

Basic Options:

• task_uuid (required): The UUID of the export task to check
• --api-host: API host to use (overrides API_HOST env var, default: api.sekoia.io)

Note: The status command performs a single status check and does not download the file. Use the download command to download completed exports.

download command

Basic Options:

• task_uuid (required): The UUID of the export task to download
• --api-host: API host to use (overrides API_HOST env var, default: api.sekoia.io)
• --output, -o: Output filename for the downloaded file (default: export_YYYYMMDD_HHMMSS.json.gz)

SSE-C Encryption (exports are encrypted by default):

• --no-sse-c: Don't use SSE-C headers for download (use only if export was created with --no-sse-c)
• --s3-sse-c-key: SSE-C encryption key used during export, base64 encoded (required if export was encrypted, overrides S3_SSE_C_KEY env var)
• --s3-sse-c-key-md5: SSE-C key MD5, base64 encoded (auto-computed if not provided)
• --s3-sse-c-algorithm: SSE-C algorithm (default: AES256)

Note: The download command does NOT auto-generate keys. You must provide the same key that was used when creating the export.

Progress Monitoring

The export command provides real-time progress updates with visual progress bars:

Sekoia Event Exporter
──────────────────────────────────────────────────
API Host: api.sekoia.io

✓ Export triggered
Task UUID: 660e8400-e29b-41d4-a716-446655440001

Exporting: ████████████░░░░░░░░░░░░░░░░░░ 40.0% | 4,000 / 10,000 events | ⏱ 2m 30s remaining

When the export completes and download starts:

✓ Export ready!

Downloading: export_20260214_143842.json.gz
██████████████████████████████ 100.0% | 15.0 MB / 15.0 MB | 2.5 MB/s
✓ Download complete (15.0 MB in 6s)

The status command shows a snapshot of the current task state:

Sekoia Event Exporter - Status
──────────────────────────────────────────────────
API Host: api.sekoia.io
Task UUID: 660e8400-e29b-41d4-a716-446655440001

Status: RUNNING
Progress: ████████████░░░░░░░░░░░░░░░░░░ 40.0%
Events: 4,000 / 10,000

⏱ Export still in progress
Use the status command again to check progress

If SSE-C encryption is enabled, downloads automatically include the required encryption headers.

Error Handling

The tool provides clear error messages for common issues:

• Missing API Key: Config error: API_KEY environment variable not set.
• Failed Export: Failed to trigger export: 403 Forbidden
• Task Timeout: Timed out after 3600s waiting for task <uuid>
• Task Failed: Task ended with status=FAILED. Details: ...

Exit codes:

• 0: Success
• 1: General error
• 2: Configuration error
• 130: Interrupted by user (Ctrl+C)

Development

Running Tests

uv run pytest

With coverage:

uv run pytest --cov=sekoia_event_exporter --cov-report=html

Code Formatting

# Format code with black
uv run black src/ tests/

# Check formatting without changes
uv run black --check src/ tests/

# Lint with ruff
uv run ruff check src/ tests/

# Fix auto-fixable issues
uv run ruff check --fix src/ tests/

# Type check with mypy
uv run mypy src/

Running All Quality Checks

# Run all checks at once
uv run black --check src/ tests/ && \
uv run ruff check src/ tests/ && \
uv run mypy src/ && \
uv run pytest

Project Structure

sekoia-event-exporter/
├── src/
│   └── sekoia_event_exporter/
│       ├── __init__.py       # Package metadata and exports
│       └── cli.py            # Main CLI implementation
├── tests/
│   └── __init__.py
├── examples/
│   └── .env.example          # Example environment configuration
├── pyproject.toml            # Project metadata and dependencies
├── README.md                 # This file
├── LICENSE                   # MIT License
└── .gitignore               # Git ignore patterns

API Reference

Core Functions

create_http_session() -> requests.Session

Creates an authenticated HTTP session using the API_KEY environment variable.

Raises:

• ConfigError: If API_KEY is not set

trigger_export(job_uuid: str, session: requests.Session) -> str

Triggers an export job for the specified search job UUID.

Parameters:

• job_uuid: UUID of the search job to export
• session: Authenticated requests session

Returns:

• Task UUID for the triggered export

Raises:

• RuntimeError: If the export fails to trigger

poll_status(task_uuid: str, session: requests.Session, interval_s: int = 2, max_wait_s: Optional[int] = None) -> None

Polls the export task status until completion or timeout.

Parameters:

• task_uuid: UUID of the export task
• session: Authenticated requests session
• interval_s: Polling interval in seconds (default: 2)
• max_wait_s: Maximum wait time in seconds (default: None)

Raises:

• TimeoutError: If max_wait_s is exceeded
• RuntimeError: If task fails or is cancelled

Requirements

• Python 3.10 or higher
• requests >= 2.28.0

License

MIT License - see LICENSE file for details.

Support

For issues, questions, or contributions, please visit:

• Issues: https://github.com/sekoia-io/sekoia-event-exporter/issues
• Documentation: https://docs.sekoia.io

Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Run tests and linting
5. Commit your changes (git commit -m 'Add amazing feature')
6. Push to the branch (git push origin feature/amazing-feature)
7. Open a Pull Request

Changelog

See CHANGELOG.md for a detailed history of changes to this project.