Deduplication (record linkage)
Deduplication, or record linkage, is the process of identifying records that refer to the same real-world entity, such as a farmer or plot, across or within datasets, and merging or linking them so each entity is counted once.
Also known as: record linkage · entity resolution · duplicate detection
Why does it matter?
Duplicates are common in first-mile data: farmers registered by several buyers, names spelled differently, re-registrations each season. They inflate farmer counts and plot areas, hide side-selling and let volumes be declared twice. Deduplication is also a prerequisite for fair premium payments and programme targeting.
How does it work?
- Standardise: clean names, dates, phone formats and place names.
- Block: limit comparisons to plausible candidates, for example same district.
- Compare: compute similarity on several attributes, including plot overlap.
- Score and decide: automatic merge for clear matches, human review for uncertain ones.
- Link: keep a record of merged IDs so history is not lost.
Common pitfalls
- Over-merging different people with common names in the same village, such as relatives.
- Deleting records instead of linking them, which breaks the audit trail.
- Deduplicating within one dataset only, when duplicates span buyers.
- Relying on phone numbers, which are shared and change.
Cross-dataset deduplication is one of the pre-competitive checks in Farmer1st’s fraud detection service.
Frequently asked questions
How are duplicate farmers detected?
By comparing attributes such as names (with phonetic and spelling variants), dates of birth, phone numbers, ID numbers, villages, photos and plot geometries, then scoring candidate pairs and reviewing uncertain matches.
Is deduplication across companies compatible with data protection?
It can be, when there is a lawful basis such as consent, the processing is limited to what is necessary, and companies do not see each other's commercial data. Governance should be documented.
Tell us about your supply chain.
Ten minutes with our scoping assistant. We come back with a first sketch of your service.
