Overview

A set of helper functions for CSV to Salesforce procedures, with reporting in AWS S3, based in a Django project. The use case is extremely specific, but the helpers should be modular so they can be cherry-picked.

Typical use case:

• Receive an S3 event
• Download the S3 object
• Serialize the file into JSON
• Bulk upsert the JSON data to Salesforce
• Parse the results of the upsert for errors
• Construct a CSV error report
• Move the triggering S3 object to an archive folder
• Push the error report to an error folder in the same bucket
• Push an object to Salesforce that details information about the above execution

High-level Example

Using the Orchestrator class, you can skip manually setting up a lot of the above steps. This class is intended to be subclassed, and should provide plenty of options for overriding methods to better suit your use-case.

Inheriting the Orchestrator

# orchestrator.py
from django.conf import settings
from django_s3_csv_2_sfdc.classes import Orchestrator as BaseOrchestrator


class Orchestrator(BaseOrchestrator):
    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        # This must be defined in the child class because your Salesforce object could be named anything
        self.execution_object_name = "Integration_Execution__c"

    @property
    def execution_sfdc_hash(self):
        # And it could have any number of fields
        return {
            "Number_of_Errors__c": self.error_count,
            "Error_Report__c": self.error_report_link,
            "Data_File__c": self.s3_object_key,
        }

    @property
    def error_report_link(self):
        return f"https://{self.bucket_name}.{settings.AWS_REGION}.amazonaws.com/{self.error_file_s3_key}"

Using the Orchestrator

# biz_logic.py
from django.conf import settings
from django_s3_csv_2_sfdc.classes import SfClient

# import the custom Orchestrator defined above
from .orchestrator import Orchestrator

salesforce = SfClient()
orchestrator = Orchestrator("some/s3/key/file.csv", settings.S3_BUCKET, sf_client=salesforce)

upsert_key = "My_External_ID__c"
accounts_data = [{"Name": "A name", upsert_key: "123"}]
results = salesforce.bulk.Account.upsert(results, upsert_key)

# You'll call log_batch for each batch you upload. This method
# will parse the results in search of errors
orchestrator.log_batch(results, accounts_data, "Account", upsert_key)

# This will create the error report, archive the source s3 file, and push
# the integration object to Salesforce. You'll definitely want to customize
# this by overriding this method or the methods it invokes
orchestrator.automagically_finish_up()

Low-level Example

from django_s3_csv_2_sfdc.csv_helpers import create_error_report
from django_s3_csv_2_sfdc.s3_helpers import download_file, respond_to_s3_event, upload_file
from django_s3_csv_2_sfdc.sfdc_helpers import extract_errors_from_results


# handler for listening to s3 events
def handler(event, context):
    respond_to_s3_event(event, download_and_process)


def download_and_process(s3_object_key, bucket_name):
    download_path = download_file(s3_object_key, bucket_name)

    # This function contains your own biz logic; does not come from this library
    results = serialize_and_push_to_sfdc(download_path)

    sucesses, errors = parse_bulk_upsert_results(results)

    report_path, errors_count = create_error_report([errors])

    upload_file(report_path, bucket_name)

Just take what'cha need!