Setu · an archived project

Five minutes against five months

I built a system to find duplicate companies in Singapore's registry. It worked. There were no duplicates to find.

On the last day, I typed an old company name into Singapore's official registry search. United Overseas Land. The registrar came back with UOL Group Limited and printed the old name underneath it: formerly known as.

That was the thing my system did that the registrar supposedly could not do. I had believed otherwise for five months, on the strength of a documentation page that mentioned previous identifiers under one search filter and said nothing either way about the others. I read the silence as an answer. Checking it took about five minutes, and I did it in the same week I decided to shut the project down.

The assumption

Every entity-resolution system rests on one belief about its data: that the same real thing shows up more than once, under names that do not quite match. Acme Pte Ltd and ACME PTE. LTD. and Acme Private Limited. Resolving that mess is a genuinely hard problem and a genuinely valuable one, which is why so much machinery exists to do it.

I assumed Singapore's company registry was that kind of data. I never checked, because it did not occur to me that it was a thing you would check. Messy data is the water you swim in.

A registry is not messy. A registrar issues exactly one identifier per company and refuses to register a second company under a name already taken. It is not a pile of records that happens to be clean. It is a system whose entire job is to guarantee that each entity appears once.

I had built a duplicate detector for a dataset that cannot contain duplicates of itself.

What got built

The work itself was not sloppy, which is part of why it took so long to see. There was a resolver that matched company names five different ways, from exact identifier down to fuzzy similarity. There was a scoring layer that weighted how much to trust each source. When two records looked like the same company, the pair went to a council of language models that argued about it: one built the case to merge, a second from a different vendor argued against, a third read both and decided. A merge required both vendors to agree independently, so if either fell silent, nothing merged.

Behind that sat the part I was proudest of. A separate measurement layer scored the detector against the registry's own identifiers, and a gate held every result back from publication until the measured precision cleared a statistical bar. Nothing could be published on the strength of looking right. It had to be measured right first.

It ran unattended on a small server for thirty-one days. Every scheduled job succeeded. The measurement layer reported honestly the entire time. Nothing crashed and nothing lied. The decisions were written down as they were made, rejected options alongside chosen ones, and four of them are worth reading against what happened next.

The machine told me early

The detector flagged 58 candidate pairs. For 53 of them, both companies carried a registry identifier, so the question could be settled outright rather than argued about.

Flagged pairs confirmed as genuine duplicates. All 53 were different companies. See the measurements.

Precision was zero. Not low, not disappointing — zero, over every case where the answer could be checked. My public page reported that number as unmeasured and refused to publish anything, which was the gate doing exactly what it had been built to do.

I read it as a calibration problem. I spent weeks looking for the missing signal that would let a true match prove itself, and wrote up why the measurement was hard.

It was not hard. It was finished. A detector aimed at a registry has nothing true left to find, and mine had been telling me so, in a number, every night, for a month.

There was a second thing I had wrong, and I only found it in the last week by reading the transcripts instead of the summary. The council merged nothing across 58 pairs, and I had been quoting that as evidence the adversarial design worked — two vendors, genuine disagreement, no unearned merges. In fact it argued for merging on four pairs, and on all four the dissenting vendor returned output the parser could not read, so it abstained and the merge was blocked. The gate held. It held on a crash rather than on an argument, and the summary I had been reading could not tell the two apart. The transcripts are here.

Then I went looking for the mess somewhere else

If the registry had no duplicates, perhaps the join between two government datasets would. Government procurement records list supplier names as free text with no identifier attached, so matching an award to a registered company is exactly the fuzzy problem I had tooling for.

I sampled sixty supplier names, found that forty-six of forty-eight matched exactly, and concluded the join was nearly solved. Then someone pointed out that I had taken the first sixty rows of the table, which are ordered by insertion and stacked with large-agency technology contracts. Accenture. KPMG. NEC. Clean corporate names, every one.

A proper random sample of 500 gave a different answer.

Suppliers matching a registered company exactly. 393 of 500, 95% confidence interval 74.8 to 82.0. See the method.

So a fifth of them did not match, and that gap looked like the fuzzy tail I had built for. I wrote that it was, and then measured it rather than leaving the claim standing. Normalising punctuation and legal suffixes recovered none of the 107 misses. Loosening to similarity produced candidates, and the candidates were these:

SP CONSULTANCY against SP HR CONSULTANCY. BLACKPEAK SINGAPORE against BLACKBERRY SINGAPORE. TAN BOON KEE against TAN BOON KENG — two different people.

Turning the threshold down did not recover true matches. It manufactured false ones, which is the same failure the measurement layer had already found on the registry, arrived at from a completely different direction. The missing fifth were sole traders, foreign firms, individuals and companies since struck off. They had no counterpart to match against. No better model would have found them.

What I actually got wrong

It would be tidy to say the lesson is that registries are clean, so do not build entity resolution against one. That is true and almost useless — it applies to about four people.

The real mistake was structural. I checked everything that was expensive to check and nothing that was cheap. I measured precision to a confidence interval, built a gate to stop myself publishing an unearned number, and argued pairs through two independent model vendors. Meanwhile the load-bearing belief underneath all of it — that there was something there to find — went unexamined for five months, and the test that settled it took five minutes.

Rigour applied downstream of an unchecked assumption produces very well-measured nonsense.

It kept happening while I wrote this

I would like to report that having seen the pattern, I stopped doing it. On the day I wrote this page I did it four more times.

The forty-six-of-forty-eight figure was mine, from a sample I had not randomised, and I put it in front of someone as a finding. When I first wrote up the misses, I said they matched at no similarity threshold — which I had not measured, having measured exact matching and assumed the rest.

Then it got funnier. Correcting that second error turned up a phantom row in my own results: 108 records where there should have been 107. I diagnosed it as company names containing line breaks, fixed it, and got a byte-identical result, because the fix was a no-op. I diagnosed it again as supplier names containing line breaks. There were none. The actual cause was a database client printing a status line I had counted as data, and I found it only by looking at the raw output instead of reasoning about it.

Two unchecked assumptions about the data, then two unchecked assumptions about the bug in my correction of them. Same error each time, made by someone who had spent the week writing this page.

That is the part worth keeping. This is not a mistake you make once and learn from. It is a standing hazard, and the defence is not vigilance, which does not scale. It is asking one boring question early enough to hurt: what would I have to see to know this is wrong, and how long would it take to look?

If the answer is five minutes, go and look now.