openmapflow 0.0.2

Creating maps with machine learning models and earth observation data.

OpenMapFlow 🌍

Rapid map creation with machine learning and earth observation data.

Examples: Cropland, Buildings, Maize

• Tutorial
• How it works
• Generating a project
• Adding data
• Training a model
• Creating a map

Tutorial

Colab notebook tutorial demonstrating data exploration, model training, and inference over small region. (video)

Prerequisites:

• Github access token (obtained here)
• Forked OpenMapFlow repository
• Basic Python knowledge

How it works

To create your own maps with OpenMapFlow, you need to

1. Generate your own OpenMapFlow project, this will allow you to:
2. Add your own labeled data
3. Train a model using that labeled data, and
4. Create a map using the trained model.

Generating a project

Prerequisites:

• Github repository - where your project will be stored
• Google/Gmail based account - for accessing Google Drive and Google Cloud
• Google Cloud Project - for deploying Cloud resources for creating a map (additional info)

Once all prerequisites are satisfied, inside your Github repository run:

pip install openmapflow
openmapflow generate

The command will prompt for project configuration such as project name and Google Cloud Project ID. Several prompts will have defaults shown in square brackets. These will be used if nothing is entered.

After all configuration is set, the following project structure will be generated:

<YOUR PROJECT NAME>
│   README.md
│   datasets.py             # Dataset definitions (how labels should be processed)
│   evaluate.py             # Template script for evaluating a model
│   openmapflow.yaml        # Project configuration file
│   train.py                # Template script for training a model
│   
└─── .dvc/                  # https://dvc.org/doc/user-guide/what-is-dvc
│       
└─── .github
│   │
│   └─── workflows          # Github actions
│       │   deploy.yaml     # Automated Google Cloud deployment of trained models
│       │   test.yaml       # Automated integration tests of labeled data
│       
└─── data
    │   raw_labels/                     # User added labels
    │   processed_labels/               # Labels standardized to common format
    │   features/                       # Labels combined with satellite data
    │   compressed_features.tar.gz      # Allows faster features downloads
    │   models/                         # Models trained using features
    |   raw_labels.dvc                  # Reference to a version of raw_labels/
    |   processed_labels.dvc            # Reference to a version of processed_labels/
    │   compressed_features.tar.gz.dvc  # Reference to a version of features/
    │   models.dvc                      # Reference to a version of models/
    

This project contains all the code necessary for: Adding data ➞ Training a model ➞ Creating a map.

Adding data

Prerequisites:

• Generated OpenMapFlow project
• EarthEngine account - for accessing Earth Engine and pulling satellite data
• Raw labels - a file (csv/shp/zip/txt) containing a list of labels and their coordinates (latitude, longitude)

Move raw labels into project:

export RAW_LABEL_DIR=$(openmapflow datapath RAW_LABELS)
mkdir RAW_LABEL_DIR/<my dataset name>
cp -r <path to my raw data files> RAW_LABEL_DIR/<my dataset name>

Add reference to data using a LabeledDataset object in datasets.py, example:

datasets = [
    LabeledDataset(
        dataset="example_dataset",
        country="Togo",
        raw_labels=(
            RawLabels(
                filename="Togo_2019.csv",
                longitude_col="longitude",
                latitude_col="latitude",
                class_prob=lambda df: df["crop"],
                start_year=2019,
            ),
        ),
    ),
    ...
]

Run feature creation:

earthengine authenticate    # For getting new earth observation data
gcloud auth login           # For getting cached earth observation data

openmapflow create-features # Initiatiates or checks progress of features creation
openmapflow datasets        # Shows the status of datasets

dvc commit && dvc push      # Push new data to data version control

git add .
git commit -m'Created new features'
git push

Important: When new data is pushed to the repository a Github action will be run to verify data integrity. This action will pull data using dvc and thereby needs access to remote storage (your Google Drive). To allow the Github action to access the data add a new repository secret (instructions).

• In step 5 of the instructions, name the secret: GDRIVE_CREDENTIALS_DATA
• In step 6, enter the value in .dvc/tmp/gdrive-user-creditnals.json (in your repository)

After this the Github action should successfully run if the data is valid.

Training a model

Prerequisites:

• Generated OpenMapFlow project
• Added labeled data

# Pull in latest data
dvc pull    
tar -xzf $(openmapflow datapath COMPRESSED_FEATURES) -C data

# Set model name, train model, record test metrics
export MODEL_NAME=<YOUR MODEL NAME>              
python train.py --model_name $MODEL_NAME    
python evaluate.py --model_name $MODEL_NAME 

# Push new models to data version control
dvc commit 
dvc push  

# Make a Pull Request to the repository
git checkout -b"$MODEL_NAME"
git add .
git commit -m "$MODEL_NAME"
git push --set-upstream origin "$MODEL_NAME"

Important: When a new model is pushed to the repository a Github action will be run to deploy this model to Google Cloud. To allow the Github action to access Google Cloud add a new repository secret (instructions).

• In step 5 of the instructions, name the secret: GCP_SA_KEY
• In step 6, enter a Google Cloud Service Account key (how to create)

Now after merging the pull request, the model will be deployed to Google Cloud.

Creating a map

Prerequisites:

• Generated OpenMapFlow project
• Added labeled data
• Trained model

Only available through Colab. Cloud Architecture must be deployed using the deploy.yaml Github Action.