urban-worm 0.1.6

Workflow of reproducible multimodal inference for urban environment evaluation.

Urban-WORM

Introduction

Urban-WORM (Workflow Of Reproducible Multimodal Inference) is a user-friendly high-level interface that is designed for adding rich and meaningful captions for crowdsourced data with geotags using multimodal models. Urban-WORM can support the batched analysis of images and sounds for investigating urban environments at scales. The investigation may cover topics about building conditions, street appearance, people's activities, etc.

• Free software: MIT license
• Website/Documentation: https://digital-landscapes.github.io/urbanworm/

Features

• Collect geotagged data (Mapillary street views, Flickr photos, and Freesound audios) via APIs within the proximity of building footprints (or other POIs)
• Calibrate the orientation of the panorama street views to look at given locations
• Filter out personal photo using face detection
• Divide sound recording to multiple clips with given duration
• Support (batched) multiple data input with multimodal models

Installation

1 install the package

The package urban-worm can be installed with pip:

pip install urban-worm

2 Inference with llama.cpp

To run more pre-quantized models with vision capabilities, please install pre-built version of llama.cpp:

# Windows
winget install llama.cpp

# Mac and Linux
brew install llama.cpp

More information about the installation here

More GGUF models can be found at the Hugging Face pages here and here

3 Inference with Ollama client

Please make sure Ollama is installed before using urban-worm if you plan to rely on Ollama

For Linux, users can also install ollama by running in the terminal:

curl -fsSL https://ollama.com/install.sh | sh

For MacOS, users can also install ollama using brew:

brew install ollama

To install brew, run in the terminal:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Windows users should directly install the Ollama client

To install the development version from this repo:

pip install -e git+https://github.com/billbillbilly/urbanworm.git#egg=urban-worm

Usage

from urbanworm.inference.llama import InferenceOllama

data = InferenceOllama(image = 'docs/data/img_1.jpg')
system = '''
    Your answer should be based only on your observation. 
    The format of your response must include answer (yes/True or no/False), explanation (within 50 words)
'''
prompt = '''
    Is there a tree?
'''

data.llm = "hf.co/ggml-org/InternVL3-8B-Instruct-GGUF:Q8_0"
data.schema = {
    "answer": (bool, ...),
    "explanation": (str, ...)
}
data.one_inference(system=system, prompt=prompt)

More examples can be found here.

To do

v0.1.x:

• A module for collecting social media data (Flickr and Freesound)
• A method for inferencing sound recordings

v0.2.x:

• A web UI providing interactive operation and data visualization

Legal Notice

This repository and its content are provided for educational and research purposes only. By using the information and code provided, users acknowledge that they are using the APIs and models at their own risk and agree to comply with any applicable laws and regulations.

Acknowledgements

The package is heavily built on llama.cpp and Ollama. Credit goes to the developers of these projects.

• llama.cpp
• ollama
• ollama-python

The functionality about sourcing and processing GIS data and image processing is built on the following open projects. Credit goes to the developers of these projects.

• GlobalMLBuildingFootprints
• Equirec2Perspec
• Mapillary API
• Flickr API
• Freesound API

The development of this package is supported and inspired by the city of Detroit.