moark-ingest 0.1.0

Moses in the Ark - Ingest tool for loading Git bundles in air-gapped networks

⛵ Moses in the Ark - Ingest

Moses in the Ark is a secure air-gap transfer system for Git repositories and CI/CD artifacts.

The Ingest tool loads bundles in air-gapped networks and uploads them to internal Git repositories.

Installation

pip install .
# Or with Terminal UI
pip install ".[ui]"

Requirements: git installed, Python 3.10+.

Usage

Terminal User Interface (TUI)

moark-ingest-ui

Opens an interactive terminal interface for loading bundles. The interface includes:

• S3 Configuration: Configure S3 connection for downloading mapping dictionary
• Bundle Scanner: Auto-scan disk-on-key or specific folders for bundles
• Bundles Table: View all available bundles with details
• Mappings Table: View external → internal repository mappings
• Upload: Automatically upload bundles to the correct internal repository
• History: View recent uploads

Workflow in TUI:

1. Configure S3 (First Time):

   • Enter S3 endpoint URL (e.g., https://s3.internal.company)
   • Enter bucket name
   • Enter access key and secret key (optional)
   • Click "Save & Download Mapping" to download the mapping dictionary

2. Scan for Bundles:

   • Click "Auto Scan" to automatically scan all disk-on-key devices
   • Or enter a specific path and click "Scan Path"
   • The table will display all found bundles

3. Select Bundle:

   • Click on a row in the Bundles table
   • The system will automatically check if a mapping exists for the bundle

4. Upload:

   • Enter username and password for internal Git
   • Click "Upload Bundle"
   • The system will extract the bundle and upload it to the correct internal repository

CLI Usage

Load from bundle file

export MOARK_REMOTE_TEMPLATE="https://{username}:{password}@git.internal/{repo}.git"
export MOARK_REMOTE_USERNAME=alice
export MOARK_REMOTE_PASSWORD=secret
moark-ingest --tar ./dist/project-....tar.gz --profile default

With artifact extraction:

moark-ingest \
  --tar ./dist/project-....tar.gz \
  --profile default \
  --artifacts-output-dir ./extracted-artifacts

Repository Mapping Management

The system supports mapping external repository names to internal names for security and classification purposes.

Mapping CLI

List all mappings:

moark-mapping list
moark-mapping list --profile team-alpha

Add mapping:

moark-mapping add external-repo-name internal/repo/path
moark-mapping add external-repo-name internal/repo/path --profile team-alpha --notes "Main app"

Remove mapping:

moark-mapping remove external-repo-name
moark-mapping remove external-repo-name --profile team-alpha

Validate configuration:

moark-mapping validate
moark-mapping validate --profile team-alpha

Environment Variables

• MOARK_CONFIG_DIR - Configuration directory (default: ~/.moark)
• MOARK_REMOTE_TEMPLATE - URL template for internal remote
• MOARK_REMOTE_USERNAME - Username for internal Git
• MOARK_REMOTE_PASSWORD - Password for internal Git

Configuration Structure

~/.moark/
├── profiles.json           # Profile settings
├── mappings/
│   ├── default.json        # Mappings for default profile
│   ├── team-alpha.json     # Mappings for team alpha
│   └── team-beta.json      # Mappings for team beta
├── history.json            # Ingestion history
├── s3_settings.json        # S3 settings (endpoint, bucket, credentials)
├── mapping-dict.json       # Mapping dictionary downloaded from S3
└── backups/                # Automatic backups

Mapping Dictionary

The system uses a mapping dictionary stored in S3 to convert external repository names (from the internet network) to internal names (in the air-gapped network).

Dictionary File Structure

See example in example-mapping-dict.json:

{
  "_description": "Repository name mapping dictionary",
  "_version": "1.0",
  "mappings": {
    "yossi": {
      "internal_repo": "project-alpha",
      "internal_url": "https://git.internal.company/security/project-alpha.git",
      "description": "Security project - classified",
      "team": "security-team",
      "classification": "secret"
    }
  }
}

S3 Configuration

On first run, configure S3 details:

• Endpoint URL: S3 server address (e.g., https://s3.internal.company)
• Bucket Name: Bucket containing the dictionary file
• Access Key / Secret Key: Authentication credentials (optional if using IAM roles)

Settings are saved in ~/.moark/s3_settings.json and loaded automatically on subsequent runs.

Parameters

Ingest Parameters

• --tar: Bundle file to transfer
• --remote-template: URL template for internal remote (or ENV: MOARK_REMOTE_TEMPLATE)
• --username, --password: Authentication credentials (or ENV: MOARK_REMOTE_USERNAME/MOARK_REMOTE_PASSWORD)
• --profile: Profile to use (default: default)
• --mapping-config: Configuration directory path (or ENV: MOARK_CONFIG_DIR)
• --artifacts-output-dir: Directory for extracting artifacts (optional)

Mapping CLI Parameters

• --profile, -p: Profile name (default: default)
• --config-dir: Configuration directory (or ENV: MOARK_CONFIG_DIR)
• --notes, -n: Notes for mapping (in add command)
• --force, -f: Skip confirmation (in remove command)

Note

This package is intended for developers in the internal (air-gapped) network. It uses a mapping dictionary to convert external repository names to internal names before uploading to internal Git.

For Support

For support or questions, contact: Moshe Eliya