GIS Tools - De-duplication Tool

What Is Deduplication in GIS?

Deduplication is the operation of finding and removing duplicate records from a dataset. In a tabular context, duplicates are rows where every field has the same value as another row. In a spatial context, duplicates can also be features whose geometries are identical (or nearly identical) even if their attributes differ slightly. The goal is the same in both cases: a clean dataset where each entity appears exactly once.

Duplicates creep into GIS datasets through many channels. A field crew submits the same observation twice. A merge of two source files brings the same feature in from each. A geocoding pipeline assigns the same coordinate to multiple addresses. A quality control script rejects a row that gets re-imported manually. Whatever the source, the consequences are the same: counts inflate, statistics distort, and downstream analysis produces wrong answers.

The gis.tools deduplication tool finds duplicate features in a loaded layer using a configurable matching strategy. You pick which fields to compare (or all fields, or just geometry, or both), and the tool surfaces matching groups for review. You can then keep one representative from each group and drop the rest, or merge attributes across the group, or export the duplicates separately for inspection. Everything runs in the browser against the in-memory FeatureCollection.

Deduplication is one of the highest-leverage cleanup operations because the cost of running it is low and the consequences of skipping it can be severe.

How Deduplication Works

Hash-based exact matching

The simplest deduplication strategy hashes each row's selected fields into a string and groups rows with identical hashes. This is O(N) and catches exact duplicates instantly. The hash function is typically a concatenation of the field values with a separator.

Geometry comparison

For spatial duplicates, the tool can compare geometries by exact equality (every coordinate matches) or by approximate equality (a tolerance distance). Exact equality works for features that came from the same source. Approximate equality catches features with minor coordinate noise from different geocoding rounds.

Fuzzy attribute matching

For near-duplicates with minor text differences ("Main Street" vs "Main St."), the tool can apply fuzzy string matching using Levenshtein distance, Jaro-Winkler, or token-set similarity. These find rows that are probably the same despite spelling variations.

Group selection

Once duplicate groups are identified, the tool offers strategies for picking which row to keep:

First: keep the first row in each group, drop the rest.
Last: keep the last row.
Newest: keep the row with the latest date in a specified field.
Most complete: keep the row with the fewest null fields.
Manual: review each group and pick by hand.

Key Parameters and Options

Match fields

Pick which fields to compare. "All fields" is exhaustive but strict. Picking just id or name is more permissive.

Geometry matching mode

Ignore, exact, or approximate (with tolerance distance).

Fuzzy threshold

For fuzzy matching, set the maximum edit distance or minimum similarity score that counts as a match.

Keep strategy

First, last, newest, most complete, or manual selection.

Output options

Replace the layer with deduplicated data, save duplicates to a separate file, or both.

Practical Applications

Address list cleanup

A marketing team merges addresses from multiple list providers and ends up with the same household appearing 4-5 times across the combined dataset. Deduplication on address and zip collapses each household to one row, reducing mailing costs and avoiding annoyed recipients.

Wildlife sighting consolidation

A conservation biologist consolidates wildlife sightings from multiple field crews who all reported the same nest. Deduplication on geometry (with a 5-meter tolerance) merges the duplicates into one observation per nest.

Permit application QA

A building department finds that some applications were submitted both online and on paper, ending up in the database twice. Deduplication on applicant_name and parcel_id flags the duplicates for manual review.

Asset inventory cleanup

A utility company merges asset data from two legacy systems and finds that some hydrants are listed in both with slightly different attributes. Fuzzy matching on asset_id (which has a different format in each system) catches the duplicates for reconciliation.

Real estate listing deduplication

A real estate aggregator pulls listings from multiple MLS sources and the same property may appear multiple times. Deduplication on parcel_id (or address + price) consolidates them into a single listing per property.

Citizen complaint consolidation

A 311 system receives multiple reports of the same pothole from different citizens. Deduplication on geometry (with a 10-meter tolerance) groups them into a single work order.

Membership database merge

A nonprofit merges donor lists from a CRM and a fundraising platform. Fuzzy matching on email and last name catches duplicate donors with slightly different name capitalization.

Step-by-Step Workflow

Load your dataset into the GeoJSON, KML, Shapefile & GIS File Viewer.
Run a Data Profiling Report first to see if duplicates are likely.
Open the deduplication tool.
Pick the match fields — start with the most distinctive identifier.
Choose the geometry mode if relevant.
Set fuzzy matching options if attribute spelling varies.
Run the match — the tool reports the number of duplicate groups found.
Review the duplicate groups in the preview panel.
Pick a keep strategy (first, newest, most complete, or manual).
Apply the deduplication — the layer updates in place.
Optionally export the removed duplicates separately for audit.

Worked Example

A voter registration team merges voter rolls from a state database and a county database. The combined dataset has 215,000 records but the team suspects significant overlap. They load the GeoJSON of registered voters and run the deduplication tool with match fields last_name, first_name, dob, and zip.

The tool identifies 18,400 duplicate groups containing 43,200 records — meaning roughly 24,800 records are duplicates. They pick the "most complete" keep strategy so that the kept row in each group has the most non-null fields. After deduplication, the dataset has 190,200 records. They export the removed 24,800 duplicates as a separate file and send it to the state for reconciliation. Without this cleanup, voter mailings would have been sent twice to thousands of households.

Common Pitfalls and Gotchas

Whitespace differences. "Smith " and "Smith" are different unless you trim. Use the trim option.
Case sensitivity. "SMITH" and "Smith" are different unless you lowercase. Use the case-insensitive toggle.
Encoding mismatches. Composed and decomposed Unicode (café vs cafe\u0301) compare unequal.
Nulls treated as equal. Two rows with null in the match field will group together, which may or may not be what you want.
Order sensitivity. The "keep first" strategy depends on the input order, which may not be meaningful.
Aggressive fuzzy matching. Low Levenshtein thresholds can group rows that are not actually duplicates.
Approximate geometry tolerance. Too tight and you miss real duplicates; too loose and you collapse distinct features.
Loss of useful attribute variation. Sometimes the differences across duplicates contain important information; merging strategies can preserve them.

Tips for Best Results

Always run a Data Profiling Report first to estimate the duplicate rate.
Start with strict exact matching and loosen only if needed.
Use the "most complete" keep strategy to preserve the highest-quality row.
Export removed duplicates separately for audit and recovery.
Trim and lowercase string fields before matching.
For fuzzy matching, test the threshold on a sample first.
Document the deduplication criteria in metadata for reproducibility.
Run deduplication after merging sources, not before, to catch cross-source duplicates.

Comparison with pandas, OpenRefine, and SQL

Pandas implements deduplication via df.drop_duplicates() with subset and keep options. OpenRefine offers cluster-and-edit for fuzzy matching. SQL uses GROUP BY with aggregate functions to collapse duplicates.

The gis.tools deduplication tool combines elements of all three: pandas-style exact matching, OpenRefine-style fuzzy clustering, and SQL-style group selection — with the addition of geometry-based matching for spatial duplicates. It runs in the browser without setup.

Performance Considerations

Exact hash-based deduplication is O(N) and runs in milliseconds even for million-row datasets. Fuzzy matching is O(N²) in the worst case (every row compared to every other) and can be slow for large datasets — limit fuzzy matching to subsets blocked by an exact key first. Geometry comparison is O(N) for exact matching and O(N log N) for tolerance-based matching with a spatial index.

Data Privacy and Browser-Based Processing

Deduplication runs entirely in the browser. Sensitive data — voter rolls, patient lists, donor records — never leaves your machine. This is essential for compliance with HIPAA, FERPA, and other data protection regulations.

Related GIS Concepts

Record linkage. The broader discipline of identifying matching records across datasets. Deduplication is intra-dataset linkage.

Entity resolution. The process of figuring out which records refer to the same real-world entity.

Levenshtein distance. The minimum number of single-character edits (insertions, deletions, substitutions) to transform one string into another.

Blocking. A performance optimization that limits fuzzy matching to candidate pairs sharing an exact key.

Master data management. The enterprise practice of maintaining authoritative versions of key data entities, often involving deduplication.

Frequently Asked Questions

How do I find duplicates with slightly different spellings?

Enable fuzzy matching and set the Levenshtein threshold or similarity score.

Can I deduplicate by geometry alone?

Yes. Pick the geometry-only match mode with exact or tolerance-based equality.

What happens to the removed duplicates?

By default they are discarded. You can also export them to a separate file for audit.

Can I undo a deduplication?

Reload the original file. The tool does not modify your source data — it operates on the in-memory copy.

How do I pick which duplicate to keep?

The keep strategy options are first, last, newest, most complete, or manual selection.

About Deduplication