ImportanceScore

ImportanceScore is a framework for selection, ranking, and curation at scale.

It is designed for workflows where you start with a very large dataset but only want the small subset that matters most. In many real-world cases, that requires much more than sorting on a single field. You may need to split records into meaningful categories, enrich them with additional attributes, assign importance scores using either domain knowledge or machine learning, convert those scores into practical rank tiers, and then apply downstream constraints such as spatial separation for map display.

The framework works on generic tabular data, but it is especially well suited to geospatial and OpenStreetMap workflows. In those cases, it can be used to extract features into scoring datasets, rank them by importance, apply separation rules for clearer map output, and feed the results back into downstream GIS or rendering pipelines.

ImportanceScore gives you a repeatable, explainable way to do that. Instead of building one-off scripts for each project, it provides a configurable framework for turning complex ranking logic into a consistent production workflow.

It is especially useful when:

• the source dataset is very large
• importance must be inferred from many attributes
• different categories or regions need separate treatment
• training data is limited or unavailable at first
• the output needs more than a raw score
• the workflow must be repeatable, explainable, and easy to evolve
• the final result must support map display or other spatially constrained outputs

ImportanceScore supports both:

• explicit rule-based scoring, when you do not yet have training data
• supervised machine learning, when you are ready to learn from labeled examples

All scoring logic is externalized into configuration, so the process can be reused, tuned, audited, and scaled across many regions and categories without rewriting custom scripts.

Note: for a detailed usage guide for the GUI, see the Usage Guide.

---

Why You Would Use It

You would use ImportanceScore when your workflow looks something like this:

• you have a database with a very large number of items
• you need to identify only the most important ones
• different kinds of items need to be scored separately
• the source data does not contain all the attributes needed for meaningful ranking
• you do not yet have labeled training data, so initial scoring must come from manual weights and domain knowledge
• after scoring, you need practical outputs such as rank buckets or tiers
• your final output has downstream constraints, such as spatial separation between displayed items on a map
• the full process is complex enough that you need a reusable, easy-to-run framework rather than ad hoc scripts

In other words, ImportanceScore is not just for producing a score. It is for managing the full selection pipeline that turns a large raw dataset into a curated, explainable set of ranked outputs.

---

Features

1. Large-Scale Filtering and Ranking

ImportanceScore is built for the common case where you start with a huge dataset and need to reduce it to the most meaningful subset. It provides a structured pipeline for moving from raw records to ranked outputs that are ready for use.

2. Category-Based Scoring

Different item types often require different logic. A peak, waterfall, placename, fault, or point of interest should not necessarily be scored the same way. ImportanceScore supports category-based processing so you can define separate logic for different pools of data instead of forcing everything into one global ranking.

3. Rule-Based Scoring Without Training Data

Many ranking projects begin before any labeled training set exists. ImportanceScore supports explicit coefficient-based scoring so you can use domain knowledge immediately. This makes it practical to launch a scoring system early, even when machine learning is not yet possible.

4. Data Augmentation and Enrichment

Real-world source databases are often missing key attributes needed for meaningful ranking. ImportanceScore is designed to work with enriched datasets, allowing you to combine source data with additional derived or external attributes before scoring.

5. Rank Buckets and Tiering

Many workflows need more than a raw numeric score. They need practical outputs such as rank groups, display tiers, or selection buckets. ImportanceScore supports that progression from score to usable rank structure.

6. Spatial Separation for Map Outputs

In map-driven workflows, importance alone is not enough. Highly ranked items may still need to be filtered so they do not visually overlap or cluster too closely together. ImportanceScore supports turning ranked results into map-ready outputs by combining importance with separation constraints.

---

Framework Benefits

These are the architectural and operational qualities that make the workflow practical over time.

1. Repeatable and Reproducible

The scoring process is externalized into configuration and follows a structured workflow such as tune -> train -> predict. That makes runs repeatable and suitable for version control, allowing you to archive configuration, training data, and model artifacts together.

2. Prescriptive Directory and File Naming Conventions

ImportanceScore uses a standardized directory structure and file naming convention. This helps keep multi-region and multi-category workflows understandable and reproducible, and allows you to define scoring logic once and apply it broadly.

3. Transparent and Tunable

The system is designed for the iterative loop of score -> explain -> tune. Detailed logging, feature contribution reporting (--explain), and configuration-driven logic make it easier to understand why items scored the way they did and to refine the model over time. A GUI is included to make this iteration faster.

4. Drop-In Model Flexibility

Because the workflow is structured, you can switch scoring methods without redesigning the entire pipeline. A common pattern is to begin with the rule-based Weighted Linear Model, use it to bootstrap labels or validate assumptions, and later move to a more powerful model such as a Random Forest Regressor.

5. Scoring-Specific Preprocessing Features

The tool includes preprocessing features designed specifically for ranking and importance workflows:

• text_weight_scoring: assign bonus points based on keywords
• feature_interactions: combine related features to avoid double-counting
• clip_outliers: cap extreme values using domain-specific thresholds

These help transform messy real-world attributes into more useful scoring inputs.

6. Scalable Across Regions and Categories

Once configured, the same logic can be applied repeatedly across many data segments. This makes the framework suitable for workflows that need to score multiple categories, multiple regions, or both, without rewriting custom code each time.

---

OpenStreetMap Benefits

Although ImportanceScore works on generic tabular data, it also provides optional capabilities that make it especially useful in OpenStreetMap workflows.

1. Extracts OSM Data for Ranking

The suite can extract OpenStreetMap features and selected fields from the OSM database into CSV files that are ready for importance scoring. This provides a clean bridge between the source geospatial database and the ranking pipeline.

2. Supports Spatial Separation

For map-oriented outputs, ranking alone is often not enough. High-value features may still need to be filtered so that selected items are not clustered too tightly together. The suite can apply spatial separation rules as part of the ranking workflow, helping produce cleaner and more legible map outputs.

3. Writes Rank Results Back to the OSM Database

After scoring is complete, the suite can write the resulting rank field back into the OpenStreetMap database. This makes it possible to use the ranked results directly in downstream GIS, rendering, and map-display workflows.

---

In Practice

A common use case looks like this:

1. Extract records from a large database into meaningful categories
2. Enrich those records with additional attributes
3. Apply rule-based or learned scoring logic
4. Generate explainable scores and tier outputs
5. Apply downstream selection rules such as spatial separation
6. Produce a small, curated set of the most important items for the final product

That final product might be a map, a ranked dataset, a display layer, or any workflow where only the top items should survive.

---

Directory Structure

This system uses two key organizing concepts: category and segment/region.

• category: A reusable blueprint for a type of data (e.g., peaks, poi).
• segment/region: A specific subset of data being processed (e.g., uswest, yellowstone).

The project layout separates reusable configurations from segment-specific data:

• config/: (Category-centric) Contains all reusable YAML configuration files. These are named by category (e.g., peaks_model.yml).
• models/: (Category-centric) Stores the final trained .joblib model artifacts, which are also named by category.
• data/: (Segment-centric) source data Holds source data files, which are almost always specific to a segment.
• data/raw/: Input feature and target files.
• build/: (Segment-centric) build files
• build/<segment>/interim/: Intermediate outputs, such as scored files.
• logs/: (Segment-centric) Contains detailed output and explanation files from specific runs.

File Naming Convention

File names are designed to be self-describing:

• Configuration Files: Are always named for the category they configure.
• config/peaks_model.yml
• config/poi_classification.yml
• Data and Log Files: Must be prefixed with their segment and category.
• data/raw/uswest_peaks_features.csv
• data/interim/yellowstone_poi_score.csv
• logs/yellowstone_poi_explain.csv

---

Weighted Linear Model (WLM)

This suite provides a WeightedLinearModel, a sci-kit compatible rule-based model. The final score is calculated as: score = intercept + Σ(contribution_of_each_feature).

The contribution from each feature is determined by its configured mode:

• presence: If the feature is present, add the coefficient value.
• value: Multiply the feature's value by the coefficient.
• base_multiplier: If the feature is present, multiply the base_score_column's value by the coefficient.

For a detailed guide, see the Weighted Linear Model Readme.

---

Advanced Workflow: Bootstrapping a Model

The suite is uniquely designed to solve the "cold start" problem where no labeled data exists. You can bootstrap a powerful supervised model from your own expertise.

1. Encode Expertise: Manually define your heuristic rules in the configuration for the Weighted Linear Model (WLM).
2. Generate Weak Labels: Run the WLM to produce an initial ranked list.
3. Curate a Training Set: Hand-pick a small, diverse subset of these scored items and adjust their scores to create a high-quality "gold" training set.
4. Switch to Supervised Learning: Change a single line in the model configuration (model: WLM -> model: RFR) and run the train and tune steps to create a RandomForestRegressor that learns the nuanced patterns from your curated labels. All data extraction and cleanup for the WLM model will continue to be used for RFR.

This process combines the best of both worlds: it starts with your domain knowledge and uses machine learning to scale and refine it.