AWS - SharePoint Connector

Provides a simple connector for moving files between AWS S3 and Microsoft SharePoint (via Microsoft Graph API).

Operates in two modes: write_to_s3 (SharePoint -> S3) and write_to_sharepoint (S3 -> SharePoint). Instantiate an engine via create_engine and call copy for each file you want to transfer.

Table of contents

• Architecture and flow
• Configuration
• Prerequisites
• Installation
• How to run
  • Programmatic API
• Error handling and retries
• How to modify or extend
• Troubleshooting
• Security considerations
• Changelog
• License

Architecture and flow

High level process flow

1. Create an engine with create_engine(mode, sp_domain, sp_site, sp_library, s3_bucket)
2. Identify the files you want to move, and their destination
3. For each file, call engine.copy(source, destination, archive_folder="", source_handling="none"):
   • Validates source and destination paths (and archive folder if supplied).
   • Calls engine validation checks before any file movement begins.
   • Download from the source system (SharePoint or S3).
   • Upload to the destination system (S3 or SharePoint).
   • Verify transfer was successful by comparing byte size.
   • For write_to_sharepoint, create missing destination folders automatically.
   • Optionally delete or archive the source file.
4. Returns a Result object confirming the copying was successful
5. Handle errors per file in your calling code.

Transfer Modes

• write_to_s3: Download from SharePoint → Upload to S3
• write_to_sharepoint: Download from S3 → Upload to SharePoint

Core Components

• src/aws_sharepoint_connector/main.py: Public API — create_engine()
• src/aws_sharepoint_connector/config.py: Pydantic models for validated configuration.
• src/aws_sharepoint_connector/engine.py: Abstract transfer logic.
• src/aws_sharepoint_connector/sharepoint.py: SharePoint connector.
• src/aws_sharepoint_connector/s3.py: AWS S3 connector.
• src/aws_sharepoint_connector/auth.py: Azure authentication and Graph utilities.
• src/aws_sharepoint_connector/utils.py: Logger, extension normalisation, path validation, and HTTP retry logic.

Configuration

Configuration is parsed by the config classes in src/aws_sharepoint_connector/config.py which store the S3 bucket, Sharepoint site and library and Azure App secrets

Required Environment Variables

MUST be provided as secrets via airflow from AWS Secrets Manager. DO NOT store as plain text

Required configuration variables

Passed directly to create_engine() from your calling code

create_engine(mode, sp_domain, sp_site, sp_library, s3_bucket)

Passed to the engine's copy method to identify a specific file to move

copy(source, destination, archive_folder="", source_handling="none")

Optional source listing helper on each engine:

list_source_files(search_folders=None, include_ext=None, exclude_ext=None)

Example: SharePoint → S3 (single file)

For a SharePoint file at: https://organisation.sharepoint.com/sites/analytics-site/Documents/reports/2026/daily_report.csv

To copy to s3://my-bucket/path/to/daily_report.csv:

engine = create_engine(
    mode="write_to_s3",
    sp_domain="organisation.sharepoint.com",
    sp_site="analytics-site",
    sp_library="Documents",
    s3_bucket="my-bucket",
)
plans = [
    {
        "source": "reports/2026/daily_report.csv",
        "destination": "path/to/daily_report.csv",
    }
]
for plan in plans:
    result = engine.copy(plan["source"], plan["destination"])

Example: S3 → SharePoint (single file)

To move the same file in the other direction:

engine = create_engine(
    mode="write_to_sharepoint",
    sp_domain="organisation.sharepoint.com",
    sp_site="analytics-site",
    sp_library="Documents",
    s3_bucket="my-bucket",
)
plans = [
    {
        "source": "path/to/daily_report.csv",
        "destination": "reports/2026/daily_report.csv",
    }
]
for plan in plans:
    result = engine.copy(plan["source"], plan["destination"])

Example: archive source after successful transfer

engine = create_engine(
    mode="write_to_sharepoint",
    sp_domain="organisation.sharepoint.com",
    sp_site="analytics-site",
    sp_library="Documents",
    s3_bucket="my-bucket",
)

result = engine.copy(
    source="reports/2026/daily_report.csv",
    destination="processed/daily_report.csv",
    archive_folder="archive/reports/2026",
    source_handling="archive",
)

Prerequisites

Sharepoint site

You will require a Sharepoint site to serve as the source or destination for files. This can be a pre-existing Sharepoint site, though you should be mindful of who will have access to the data.

Azure app registration

An Azure app has to be registered in Entra ID. This will be bespoke to your project and provide the connection to the Sharepoint site and is what the connector will authenticate into via the secret key. To request a new Azure app and have it connected to your Sharepoint site, raise a demand request by following the instructions here. You can do this in terraform against the staff infrastructure authentication services repo (see EM setup for an example), then post to #staff-identity-authentication-services.

The app will require these permissions:

• sites.selected

You will then need to speak to the File and Data Management team, who will grant your app access to the specific sharepoint sites you need access to.

Azure app details & secret

You can view your app registrations here.

Open up the app registration and the tenant ID will be available as Directory (tenant) ID.

The client ID is available as Application (client) ID.

The client secret is available from manage -> certificates and secrets - you may not be able to view it and instead may be sent it when the app is created.

AWS access

If running via airflow, or from within another repo that is running via airflow, then standard AP credentials and access management apply and will grant access to s3.

Technical requirements

• Python 3.13+
• uv for dependency management

Installation

Local install with uv

uv sync --all-groups --all-extras

Package install in another project

If your project uses uv, add this package from the public GitHub repo directly and pin to a specific commit SHA:

uv add "git+https://github.com/ministryofjustice/aws-sharepoint-connector.git@<commit_sha>"

Running tests

uv run python -m pytest                        # all tests with coverage
uv run python -m pytest tests/unit             # unit tests only
uv run python -m pytest tests/e2e              # E2E tests (no real API calls)

How to run

Programmatic API

Import create_engine from the aws_sharepoint_connector package. The Azure secret values must be present as environment variables.

from aws_sharepoint_connector import create_engine

engine = create_engine(
    mode="write_to_s3",
    sp_domain="organisation.sharepoint.com",
    sp_site="analytics-site",
    sp_library="Documents",
    s3_bucket="my-bucket",
)

plans = [
    {
        "source": "reports/2026/daily_report.csv",
        "destination": "path/to/daily_report.csv",
    },
    {
        "source": "reports/2026/summary.csv",
        "destination": "path/to/summary.csv",
    },
]

for plan in plans:
    result = engine.copy(plan["source"], plan["destination"])

You can optionally use the list_source_files methods on the engines to obtain a list of all files in the S3 bucket or SharePoint library. This can be used to programmatically build the list of plans to iterate over.

Example with filters:

source_files = engine.list_source_files(
    search_folders=["reports/2026", "reports/2026/subfolder"],
    include_ext=["csv", ".json"],
    exclude_ext=["tmp"],
)

Error handling and retries

The connector implements robust retry logic to handle transient failures.

Chunk Upload Strategy

For large files, uploads are split into 10 MB chunks:

• Max 5 consecutive failures per chunk before aborting the entire transfer
• Transient errors (429 Too Many Requests, 5xx): retried with exponential backoff
• Permanent errors (4xx excluding 429): immediately raised as ProcessingError without retry
• File pointer reset on every retry to ensure data consistency

Example: If a 50 MB file fails on chunk 3 of 5, the transfer aborts and raises ProcessingError.

HTTP Request Retries

All Graph API and HTTP calls use request_with_retry():

• Max 3 attempts per request
• Retryable errors: 429 Too Many Requests, 5xx Server Errors
• Non-retryable errors: 4xx Client Errors (except 429)
• Exponential backoff between retries

Batch Processing Behavior

Batch iteration is handled by the calling code. The engine processes one file per engine.copy() call and raises ProcessingError on failure. It is the caller's responsibility to decide whether to abort or continue processing remaining files.

How to modify or extend

1) Add a new transfer mode

1. Create a new engine class in src/aws_sharepoint_connector/engine.py implementing:
   • _list_source_files(self, folders: list[str], include_ext: list[str], exclude_ext: list[str]) -> list[str]
   • _download_file(self, source: str) -> bytes
   • _upload_file(self, content: bytes, destination: str, content_size: int) -> None
   • _archive_source_file(self, source: str, archive_folder: str, content_size: int) -> None
   • _delete_source_file(self, source: str) -> None
   • _validate_plan(self, source: str, destination: str) -> None
2. Register the engine in MODE_MAP in src/aws_sharepoint_connector/main.py.
3. Expand the Literal type for mode in create_engine() in src/aws_sharepoint_connector/main.py.
4. Add unit tests for success and failure paths, including validate_plan and source handling (none/delete/archive).

2) Add additional configuration

1. Add a field in SecretConfig (src/aws_sharepoint_connector/config.py).
2. Add validation if needed with a field_validator.
3. Update .env docs and this README.
4. Use the field in connector or engine logic.

Troubleshooting

Common errors and solutions

• Validation failed with N error(s): One or more sources or destinations could not be verified before transfers started. The error message lists every problem - fix all of them before retrying.
• Library 'X' not found on site: Verify sp_library spelling and that the app has SharePoint access to the target site/library
• Source file not found in SharePoint: Verify the file exists at the exact path supplied as source; check case sensitivity
• InvalidPathError: The provided source, destination, or archive path was empty, absolute, or contained unsafe path segments (such as ..).
• Destination folder not found in SharePoint: In write_to_sharepoint mode, missing destination folders are now created automatically; this error usually indicates permissions or object-type conflicts.
• S3 bucket does not exist or S3 object does not exist: Verify bucket name is correct, bucket exists in eu-west-2, and IAM principal has access
• Access denied to S3 bucket/object: Check IAM policy grants s3:GetObject, s3:PutObject, s3:HeadObject, and s3:HeadBucket on the bucket
• AADSTS65001 or Graph auth failures: Verify app permissions and consent in Azure; may need admin consent
• File transfer failed: Max retries exceeded: File chunk upload exceeded 5 consecutive failures; check network stability, S3/SharePoint availability, and file size

Security considerations

• Never commit .env files or secrets: Add .env to .gitignore
• Prefer managed identity: Use workload identity or managed identity in AWS/Azure instead of storing static credentials
• Scope permissions tightly:
  • Azure: Limit app permissions to only what the target site/library requires
  • AWS: Restrict IAM policy to specific bucket and prefix (e.g., arn:aws:s3:::bucket/prefix/*)
• Rotate secrets: Change Azure client secrets every 90 days and update secret manager
• Store secrets securely: Use AWS Secrets Manager, Azure Key Vault, or Kubernetes secrets (never hardcode in env vars)
• Audit access: Monitor S3 CloudTrail and SharePoint audit logs for sensitive data access
• Network isolation: Consider running connector in private network with appropriate egress controls
• Data residency: Ensure S3 bucket and SharePoint site comply with data residency requirements

License

MIT License. See LICENSE.