The duplicate you can see and the one you can't
Two rows in a contact export, about forty lines apart. Row 312: Jon Smyth, j.smyth@company.com, (555) 867-5309. Row 353: Jonathan Smith, jonathan.smith@company.org, 555.867.5309. Same phone number, different punctuation. Same person, almost certainly. Same company, different email domain because somebody updated it in one system and not the other.
Your dedupe tool looked at those two rows and said: not a match. It wasn't broken. It did exactly what it was built to do. It counted characters, found too many differences, and moved on.
There are two kinds of duplicates in every database. The first kind looks like a duplicate. "John Smith" and "John Smith " with a trailing space. "Acme Corp" and "Acme Corp." Those get caught on the first pass by almost anything, and they are not the reason your marketing team keeps getting complaints about people receiving the same email twice.
The second kind doesn't look like anything. It survives every cleanup, every annual CRM tidy, every manual review, because a human scanning 12,000 rows will never put row 312 next to row 353. This article is about that second kind: how character-based matching finds the first group, where it stops, and what semantic matching does about the rest. The duplicate detection conversation, fuzzy vs semantic, usually gets framed as a choice between two tools. It isn't. It's a stack, and each layer handles what the one below it can't.
Exact and rule-based matching: the floor
Exact matching is the baseline. Two records match if the compared fields are identical, character for character. "john@acme.com" equals "john@acme.com". Change the case, add a space, and you have nothing.
It's fast, it's cheap, and it catches real duplicates: the double form submission, the same CSV imported twice, the contact created by two reps in the same hour. On typical e-commerce or CRM data, exact matching on email alone usually accounts for somewhere between 10 and 20 percent of the duplicates actually present.
Rule-based matching is the next step up, and it's where most teams spend years. You write rules. Strip punctuation before comparing. Lowercase everything. Treat "St." as "Street". Treat "Inc." and "Incorporated" as the same suffix. Normalize phone numbers to digits only so that (555) 867-5309 and 555.867.5309 collapse into the same string.
Those rules work. The problem is that they never stop. You write one for "St." and "Saint", then you need "Ft." and "Fort", "Mt." and "Mount", "Corp" and "Corporation", "Ltd" and "Limited", "&" and "and". Every industry has its own abbreviations. Every country has its own address conventions. Eventually the rule library is larger and more fragile than the matching logic it supports, and one person understands it.
Normalization is still worth doing. It just isn't a matching strategy. Standardize first (AutoFormat handles the casing, punctuation, phone formats, and address abbreviations), then match on clean inputs. You'll get better results from every layer above.
Fuzzy matching: Levenshtein, Jaro-Winkler, and the ceiling
Fuzzy matching stops asking whether two strings are identical and starts asking how far apart they are.
Levenshtein distance counts the minimum number of single-character edits (insert, delete, substitute) needed to turn one string into the other. "John Smith" to "John Smyth" is one substitution. "John Smith" to "Jon Smith" is one deletion. Divide the distance by the length of the longer string, subtract from one, and you get a similarity score. Both of those land around 0.90. Clear matches.
Jaro-Winkler works differently. It measures matching characters and transpositions, then adds a bonus for strings that share a prefix. That bonus is deliberate, because people get the beginnings of names right more often than the ends. "Acme Inc" and "Acme Incorporated" score well above what raw edit distance would give them (Levenshtein puts them near 0.50, Jaro-Winkler closer to 0.90) because the first eight characters are identical.
Phonetic algorithms are the third common family. Soundex and Metaphone convert words to a sound-based code, so "Smith" and "Smyth" collapse to the same key regardless of spelling. Useful for names collected over the phone. Useless for company names and addresses.
All three ask the same question in different accents: how similar do these two strings look? That question has a ceiling. Everything above the ceiling is a spelling variation. Everything below it is meaning, and character comparison has no access to meaning at all.
Where fuzzy matching runs out
Back to our two rows. "Jon Smyth" to "Jonathan Smith" is six edits: five insertions to get from "Jon" to "Jonathan", one substitution to get from "Smyth" to "Smith". Normalized, that's roughly 0.57. Jaro-Winkler is kinder because of the shared "Jon" prefix and lands in the high 0.70s.
Neither number helps you. At a 0.85 threshold, you miss the match. At a 0.75 threshold, you catch it along with several hundred pairs of people who merely share a first name. The score isn't wrong, it's just measuring the wrong thing.
Here are the cases that show up in nearly every customer file we see, with the scores fuzzy matching typically produces:
- Nicknames. "William Chen" and "Bill Chen". Around 0.70. Flagged as uncertain at best, and there are too many of these to review by hand.
- Legal suffixes and trade names. "Acme Inc", "Acme Incorporated", and "ACME Technologies". The first two score high on Jaro-Winkler. The third scores low against both, even though it's the same vendor.
- Name changes. "Sarah Johnson, sarah.j@email.com" and "Sarah Miller, sarah.j@email.com". Name similarity is low. The email is the only evidence, and name-only matching never sees it.
- Accents and international formats. "François Müller, +33 6 12 34 56 78" and "Francois Mueller, 0033612345678". Diacritics drag the score down, and the two phone strings share almost no characters until you normalize them.
- Title and honorific noise. "Dr. Robert James Thompson III, MD" and "Bob Thompson". Different lengths, different tokens, nothing to work with.
- Initialisms. "International Business Machines" and "IBM". "Johnson & Johnson", "Johnson and Johnson", and "J&J" across three systems.
Embeddings without the maths
Semantic matching compares meaning instead of characters, and the mechanism behind it is simpler than the name suggests.
An embedding turns a piece of text into a list of numbers. Not arbitrary numbers: numbers positioned so that text with similar meaning produces similar values. Think of them as coordinates. Latitude and longitude put two places near each other on a map. An embedding does the same thing with meaning, using a few hundred dimensions instead of two so it can capture more nuance than "north" and "east".
"Acme Technologies LLC" becomes a point. "ACME TECH" becomes a different point, sitting very close by. "Zebra Logistics" sits a long way off. Measuring the distance between two points (usually with cosine similarity, which returns 1.0 for identical meaning and 0.0 for unrelated) gives you a score that behaves like a fuzzy match score but is built on completely different evidence.
The model producing those coordinates was trained on an enormous amount of text. In the process it learned that Bob is short for Robert, that LLC and Inc both signal a business entity, that "St." before a name usually means Saint and after a number usually means Street. Every hand-written rule you were going to maintain forever, the model absorbed from examples. That is the whole trick, and it's why semantic matching handles the nickname, suffix, and initialism cases that character comparison can't reach.
It's also why semantic matching isn't a replacement for the layers below it. Embeddings are slower than string comparison and less certain about exact identifiers. SmartMatch runs exact matching first (fast, free, catches the obvious), then character similarity for near misses, then semantic similarity for the hard cases, and weighs corroborating fields like normalized phone and email at every stage. Each layer reduces the work for the next one.
Why nobody compares every record to every other record
There's a practical problem hiding underneath all of this. Comparing every record to every other record means roughly n²/2 comparisons. Ten thousand records is about 50 million comparisons. A hundred thousand records is about 5 billion. No threshold choice saves you from that.
Blocking is the standard answer. You group records into buckets on a cheap attribute first, then run the expensive comparison only inside each bucket. Block on the first three characters of a company name, or a postcode, or a normalized email domain. "Acme Technologies" and "ACME TECH" both land in the ACM bucket and get compared. "Zebra Logistics" never enters the conversation.
Crude blocking has a failure mode you should know about: anything that disagrees on the blocking key is invisible. Block on surname and you lose Sarah Johnson and Sarah Miller forever. That's why the better approach uses the embeddings themselves. Approximate nearest neighbour search indexes all the points and retrieves the closest handful to any given record without scanning the whole set. Same idea as a search engine, applied to your contact list.
In practice you generate embeddings once, index them, pull the top five or ten nearest candidates per record, and evaluate only those. The comparison count stays linear instead of quadratic, which is why a semantic pass on a 10,000-row file finishes in minutes rather than hours.
Thresholds, precision vs. recall, and the merge decision
Every matching approach eventually hands you one question: how similar is similar enough?
Set the cutoff at 0.98 and you catch near-identical records only. Very few false positives, and you've missed every Bob Johnson / Robert Johnson pair, which was the reason you went semantic in the first place. Set it at 0.70 and you drown. "Johnson Controls" and "Johnson & Johnson" will score somewhere around 0.75 because they share a prominent name. They are not the same company. Review enough pairs like that and your team stops trusting the queue, which is worse than having no queue.
Two numbers describe the tradeoff. Precision is the share of flagged pairs that turn out to be real duplicates: flag 100, get 85 real ones, precision is 85 percent. Recall is the share of actual duplicates you found: 160 out of 200 existing pairs is 80 percent recall. Raising the threshold raises precision and lowers recall. Lowering it does the reverse. F1 combines them into one number when you want a single dial to watch.
Don't guess the starting point. Take a batch of pairs your team already merged by hand, run them through matching, and look at the scores they produce. If your known duplicates cluster above 0.86, start at 0.86. As rough anchors: vendor and company records, where abbreviations run wild, often settle around 0.82 to 0.88. Contact records with name plus company plus email give the model more to work with, so 0.90 to 0.95 is common. For most business data, precision above 80 percent and recall above 75 percent is a healthy place to land. Chasing 99 percent produces a threshold tuned to last quarter's data and nothing else.
Then re-check. Naming conventions drift, new systems feed in new conventions, and acquisitions bring whole new patterns. A quarterly pass against a fresh sample of known duplicates keeps the cutoff honest.
What to automate, what to review, and what to leave alone
The two halves of this argument used to end in different places: never auto-merge anything, or let the model decide. Both are wrong in the same way. The right answer depends on whether a unique identifier corroborates the match.
Use three bands instead of one threshold:
- Auto-merge: high similarity plus an exact match on a genuinely unique field. Same normalized email, or same normalized phone and same postcode. The merge is confirmed by something other than the score. This is safe, and it's usually a surprisingly large share of your pairs.
- Review: high similarity with no corroborating identifier, or a corroborating identifier with lower similarity. Jon Smyth and Jonathan Smith sit here if the phone numbers hadn't matched. A human spends ten seconds and moves on.
- Leave alone: below your cutoff. Don't put these in the queue "just in case". A review queue nobody finishes is the same as no review queue.
Running it on your own records
The review step is where this becomes real work, so it's worth making it fast. SmartMatch surfaces each pair with its similarity score and a field-by-field breakdown showing which fields agreed, which disagreed, and which were empty. You approve, reject, or edit. No configuring embedding models, no choosing a blocking key, no standing up search infrastructure.
The shape of the job is the same whatever tool you use: import, clean, review, export. DataBridge reads your records out of Mailchimp, Shopify, Klaviyo, HubSpot, or Salesforce (or you upload a CSV). AutoFormat standardizes the inputs so matching has clean text to work with. SmartMatch finds the pairs. You review. Then you get a CSV or JSON export plus a change log listing every merge, which you import back into your platform yourself. Nothing gets changed in your live system without you doing it.
After each cleaning run, which takes a few minutes, the Clarity Score is recalculated. Its four sub-scores are Completeness, Consistency, Uniqueness, and Validity, and dedupe work shows up most visibly in Uniqueness. It's a useful way to see whether the thresholds you picked actually moved anything.
One customer ran a 12,000-row CRM export through this after years of annual cleanups with their CRM's built-in matching. Their existing tool had found 847 duplicate pairs. SmartMatch found 2,341, including 1,494 that character-level matching had never been able to see. Their marketing team had spent months blaming a segmentation problem.
You don't have to take the numbers on faith. Sign up for a 7-day free trial—no credit card required—and see SmartMatch run against your own records, with the scores and field breakdowns in front of you, before you decide what to merge.
Frequently Asked Questions
What is the difference between fuzzy and semantic duplicate detection?
Fuzzy matching compares the characters in two strings and scores how similar they look, which catches typos, spacing, and spelling variants. Semantic matching compares meaning using embeddings, so it recognizes that Bob and Robert, or Acme Inc and ACME Technologies, point at the same entity even when the characters barely overlap.
Will semantic matching create more false positives than fuzzy matching?
It surfaces more candidate pairs, and some of those will be wrong ("Johnson Controls" and "Johnson & Johnson" is the classic). That's why every pair comes with a similarity score and a field-by-field breakdown, and why you should only auto-merge when a unique identifier like a normalized email or phone confirms the match.
How much slower is semantic matching?
Less than you'd expect, because blocking and nearest-neighbour search keep the comparison count manageable. A 10,000-row file that takes under a minute with plain character matching typically finishes a full multi-pass cleaning run in a few minutes, after which the Clarity Score is recalculated.