Data quality
The record matcher finds the records that stand for the same customer, supplier or product - in a list merged from several systems, or in one system that has been filled in by hand for years. Each field is compared the way that suits it: people's names with their initials and typing slips, organisation names with the legal form left out and the words in any order, e-mail addresses, telephone numbers however they are written, postcodes and other codes, numbers, dates and names that sound alike. Each agreement counts for as much as it tells, by the Fellegi-Sunter method, with weights set by hand or learned from the data, and only records that share a blocking key are compared, on a background thread, so a list of thousands takes moments.
Pairs that score between the two thresholds wait in a review queue, side by side with each field's similarity and weight, for a person to call the same or not the same from the keyboard. Matches join into clusters, and each cluster gets a golden record: every column taken by its own survivorship rule - the most recent, the most trusted source, the most complete record, the largest value and others - or from a record chosen by hand. Every run, decision and choice goes into an audit trail. Records come in as CSV or JSON; golden records, scored pairs and the records with their cluster go out as CSV, and the work saves as a JSON file to carry on with later or to run again on a server.
Try it with your own records in confidence: they are opened and matched by your browser, on your device. Nothing is uploaded, no copy reaches us or anyone else, and nothing is kept once you close this tab.
Duplicate customers in a tile wholesaler's list
Brackenfold Tiles, a made-up trade wholesaler, has merged the customer lists of its trade counter, its web shop and its accounts system: 314 invented customers, generated in this page. Some are in it twice or three times - with a typing slip in a name, an initial for a first name, a firm's trades the other way round or its legal form written out, a new address, a new e-mail address, or a telephone number written another way - and a few are partners who share a surname and an address but are different customers. Find the matches, see why a pair scored as it did, move the thresholds, work the review queue and look at the clusters.
Golden records for a supplier list
Larchmont Facilities, an invented facilities management firm, keeps its suppliers in three systems - finance, purchasing and accounts payable - and these 170 made-up suppliers come from all three: the same firm entered again with a misspelt name, its legal form written another way, a move, a new telephone number or no VAT number, and a few sister firms that share a building, a telephone and an e-mail domain but not a VAT number. Build each supplier's golden record by its rules, correct one by hand, take a sister firm out of a cluster it does not belong to, read the audit trail, then run the same settings over 6,000 suppliers.
// Node 20 or later, from the Gold package: match last night's supplier extract with the settings and decisions saved from the page.
import { readFileSync, writeFileSync } from 'node:fs';
import { readMatchFile, readMatchSession, matchRecords, goldenCsv, pairsCsv } from './dist/node/bmx-record-matcher.mjs';
const { columns, records } = readMatchFile(readFileSync('suppliers.csv', 'utf8'), 'suppliers.csv');
const work = readMatchSession(readFileSync('larchmont-suppliers.match.json', 'utf8'));
const run = matchRecords(records, work.config, { decisions: work.decisions, golden: work.golden });
const possible = (records.length * (records.length - 1)) / 2;
console.log(`${run.result.compared} of ${possible} pairs compared in ${run.result.ms} ms: ${run.clusters.length} suppliers with duplicates`);
for (const b of run.result.bigBlocks) console.warn(`Left out: ${b.size} records share the key "${b.key}"`);
writeFileSync('suppliers-golden.csv', goldenCsv(run.golden, columns));
const review = run.pairs.filter(p => p.status === 'review');
writeFileSync('suppliers-to-review.csv', pairsCsv(review, run.ids, run.config.fields.map(f => f.name), run.result.sims));