How we know our trust scores aren't noise

A note on authorship: this article was written by an AI, working from our benchmark runs and notes, and reviewed by the UpTrust team.

We took a public record of who trusts whom, kept it entirely away from our engine, and asked the engine to predict it. Run first with every topic melted into a single graph, reading only positive evidence, it came out at chance level on the pooled comparison. Run separately inside each topic, from each observer's own point of view, the same computation carried a very real signal. Letting it read distrust as well as trust lifted both.

The pooled scores: 0.503 with topics mixed, 0.649 per topic, and 0.735 per topic once distrust is taken into account. Pooled is the strict reading. Ranked within each observer's own list — that is, the way a product interface actually reads a score — that same run with distrust gives 0.787 with topics mixed and 0.811 per topic. This article explains both readings, how the test works, and what it does not settle.

What a trust score has to prove

Any system that scores people can pass off noise as meaning. Give a number to a thousand users and sort them: the ranking can look serious even when it says nothing. The right question is simple: what external fact must this score predict, and what could prove it wrong?

A trust score is only real if there is a ground truth it has to predict — one you did not manufacture yourself. If no outcome can prove it wrong, the score is decorative. So we went looking for a public record of who actually trusts whom, and asked whether our engine could predict it from evidence that never contained it.

The test

Our ground truth is Epinions, a consumer-review site where members publicly declared which other members they trusted, and whom they distrusted. This test rests on one particular collection of that record: Extended Epinions, assembled by Paolo Massa and Paolo Avesani. It is the only source that contains both the explicit distrust declarations and the topic of each rating, and every number in this article depends on it. No one at UpTrust produced any of its data.

Stanford's SNAP project publishes a separate graph from the same site: 75,879 people and 508,837 directed trust edges. That one serves us for measuring how the engine scales. It has neither topic labels nor distrust declarations, so no number in this article comes from it.

The absence of leakage in this test is stronger than a simple hold-out. The answer key is a file of explicit trust and distrust declarations. Those declarations are not in the graph the engine reads — and never were. What the engine reads is a different record: the helpfulness ratings members gave one another's reviews, each carrying the topic the review was about. Those ratings become topic-labeled evidence. There was nothing to remove from a training graph, because the answers were never in it.

The sample is sized to fit in memory: reviews on the topics ranked 18th through 45th among the 60 with the most edges, then a deterministic sample of 70% of members, an edge being kept only if both people survive. In the run that reads only positive evidence, that leaves 13,459 people, about 167,000 evidence edges, and 226,195 declarations to predict, of which about 10% are distrust.

The run that also reads distrust follows the same recipe, but counting low ratings as negative evidence brings in 4,215 edges the first run never saw. It is therefore a different graph, and the set of pairs it can score is not the same. When a number below belongs to only one of the two runs, we say so.

We measure all of this with AUC (Area Under the Curve — the area under the ROC curve). AUC is the probability that, given one true trust declaration and one true distrust declaration, the score ranks the trusted person higher. 0.5 is a coin flip. 1.0 is perfect. Values well above 0.5 are signal that the score bears on a fact it was never told. We do not report confidence intervals here, so read the small gaps with caution.

The result

We ran the same engine two ways on the same held-out pairs. The first melts every topic into a single graph and computes trust over the whole. The second runs the engine separately inside each topic and reads each pair at its strongest topic: the same person can thus be trustworthy on one topic and unknown on another. Both readings are observer-relative. Each asks how much one given member trusts another given member, and neither produces a site-wide reputation score. The only difference between them is whether the topics stay separate.

Take the positive-evidence-only run first. Its two readings are scored on one and the same population: the 49,371 declarations for which the per-topic run produced a belief for the pair in at least one topic. Producing a belief takes more than two people merely appearing together in the same topic. The observer must have rated someone within that topic, the ratings must chain from the observer all the way to the target, and the resulting belief must survive pruning. Co-presence without a path from observer to target produces nothing at all. We used the per-topic run's coverage because it is the set where a topical belief exists at all. We did not run the mirror comparison on the mixed-topics run's own coverage.

On those 49,371 pairs, with topics mixed, the engine scored 0.503. That is chance. Trust computed over all topics at once told us nothing about whether a given member had declared trust or distrust. The same algorithm run inside each topic scored 0.649. That is real signal, from the same computation. The only change was to stop flattening the topics into a single graph.

We then let the engine read distrust as well as trust. The negative evidence is dense: it accounts for only 2.5% of the evidence edges in our sample, and adding it lifts every score. This run covers its own set of pairs, so its head-to-head is a fresh comparison, not a rerun of the 49,371. Within it, the mixed-topics run gives 0.627 and the per-topic run 0.735 — so per-topic keeps its edge on identical pairs. That is consistent with topic partitioning and negative evidence each helping in their own right, but the two runs do not cover the same population, so we do not add them together.

The control is direct evidence with no propagation at all. It is scored on the pairs where the observer rated the target — an even smaller population, and each run covers its own version of it. It gives 0.500 without negative evidence and 0.705 with it. Read those two numbers on their own terms, not against the head-to-head numbers. The 0.500 says that the sheer quantity of evidence between two people predicts nothing by itself. It is how that evidence is distributed across topics that carries the signal.

Two readings of the same numbers

Pooled AUC lines up every pair from every observer in a single ranking. It is the strict test, and it penalizes a score whose calibration differs from one observer to the next. But a product interface never asks that question. It ranks a single observer's own list. Read that way, both runs fare markedly better.

Every number in this table comes from the run that reads distrust — the one whose pooled scores were 0.627 with topics mixed and 0.735 per topic. In this reading, the mixed-topics run is no longer a coin flip. It scores 0.787, and per-topic leads by only 0.024, versus 0.108 pooled. Most of the pooled gap appears to come down to calibration across observers, which ranking within a single observer's list cancels out.

The only cell in this article that reads at chance level, the 0.503, belongs to the positive-evidence-only run. We have since computed the per-observer metrics for that run as well: the same correction takes its mixed-topics reading from 0.503 to 0.654, and per-topic keeps the lead at 0.688. Per-topic wins on every metric we computed, in both runs. The top-ranked numbers are brushing up against a ceiling, so read them as error rates: random puts the wrong person on top 3.0% of the time, the mixed-topics run 2.3%, per-topic 1.4%.

The last row is the one to watch. When an observer has explicitly declared distrust of someone who also appears among their topical acquaintances, the per-topic ranking still places that person in the top three about 21% of the time. That is better than chance, at 40%, and it is not good enough to rely on. A production interface along the lines of "most trustworthy on this topic" must exclude explicitly negative relationships outright, rather than counting on the ranking to bury them.

Why mixing topics costs signal

One graph for every topic means one belief for every context a person appears in. What the numbers point to as the price paid is calibration. Propagated belief piles up on the members the whole site rates heavily, and the mass a given observer's row carries depends on how active they are. A number that means "highly trustworthy" in one observer's row can mean "barely known" in another's.

Pooled AUC lines all those rows up together, so that difference becomes the entire test. Ranking within a single observer's list removes it, and most of the gap leaves with it: the mixed run gives 0.627 pooled and 0.787 per observer, on the same run. The deficit we measured with topics mixed therefore points to cross-observer calibration as the main contributor.

Running the engine inside a topic keeps contexts separate, and asks a narrower question the evidence can answer: on this topic, from where you stand, who has earned trust. That is the premise UpTrust is built on, and this is the measurement that supports it. Topic partitioning recovers a pooled signal the mixed-topics run does not carry, and it stays ahead within each observer's own list as well — by a much thinner margin.

Read this result narrowly. It says that our engine, with topics mixed and positive evidence only, is at chance level on this pooled task, and that topic partitioning recovers the signal. It is not a measurement of crowd reputation scores in general. We tested none here, and nothing in this article settles how a reputation aggregator would fare on the same data.

What this leaves open

• Declared trust is socially motivated. People return trust to those who extend it to them, and no score fully predicts a web of reciprocity: even a perfect model would land well below an AUC of 1.0. The ceiling here is real, and both scores sit under it.
• A good share of the signal is one relationship recorded twice. Direct evidence alone, with no inference at all, reaches 0.705 once negative evidence is included, and inference on the pairs with no direct edge stays close to chance. A good share of the 0.735 therefore comes down to noticing that an observer found a member's reviews unhelpful and concluding that they probably went on to declare distrust. The two files record the same relationship — a weaker result than predicting a fact from independent evidence.
• Multi-hop inference carries little on its own. Restricted to pairs where the observer never rated the target — and a rating in the reverse direction still counts as never — the mixed run gives 0.463 without negative evidence, below chance, and 0.534 with it. Per-topic gives 0.507 and 0.530, but on this subset it produces a belief for only a small fraction of the pairs, 4.4% in the positive-evidence run, so treat those two numbers as thin readings. Propagation extends reach and fills out a ranking. On this data, it does not predict declared trust on its own.
• This is one dataset, one sample, one pruning setting (tau 0.01). We did no random-seed sweep, reported no confidence intervals, and did not measure how much the numbers move from one sample to the next. About 10% of the declarations are distrust: the classes are imbalanced, and that is why we report AUC and no accuracy figure.
• The benchmark runs a reimplementation of the production algorithm. It is not the live service. On small graphs, we verify it entry by entry against the production code, but that verification covers the unpruned version, while every number shown here is pruned at tau 0.01. This test therefore does not validate every UpTrust interface.
• Epinions is another site, with other people. That is exactly why we use it. It gives us outside proof. A score that only ever agreed with our own data would prove nothing.

Checking our work

Here is the honest state of reproducibility. Every number shown here requires Extended Epinions, and its original host is gone. The files survive in a code mirror and in the Internet Archive. We are not going to print links we have not tested, so that part remains pending.

Landing on the same numbers takes more than the data. The converter counts ratings of 4 and 5 as positive evidence and, for the signed runs, counts 1s and 2s as negative evidence and discards the 3s. The sample takes the 60 topics with the most edges, keeps ranks 18 through 45, then keeps an edge only if both people pass a deterministic 70% filter. The pruning setting is tau 0.01. And it all depends on the version of our propagation formula.

One step in there is easy to get wrong. The 70% filter multiplies a member ID by 2654435761 and takes the remainder modulo 10, and that computation must run in floating point, the way awk does it. Epinions member IDs are long enough that the same arithmetic in exact integers selects a different sample: 13,370 people instead of 13,459. Ask us and we will send you the harness with the full recipe. We will publish a single command here as soon as the download path is reliable.

The propagation formula matters, because we changed it in July 2026, after these numbers were first measured. We replayed the entire test on the new formula in August 2026, same sample and same pruning setting. Every per-topic number came back identical. Six mixed-topics numbers moved, five pooled and one per-observer, each by 0.005 or less, and the two gaps we cite move with them: 0.024 becomes 0.023 and 0.108 becomes 0.104. Nothing moved enough to change what any number means. The values on this page are July's.

We are making this test a lock on our own work. The baseline is on record: 20.9% per topic against 26.2% with topics mixed and 39.8% at random, alongside a macro-AUC of 0.811 and a top-ranked rate of 98.6%. Ranking the feed by topical trust will not ship beyond the team until the blend it uses does at least as well, on these per-observer measures, as each of the two signals taken alone, and until it leaves the distrust-contamination row no worse. That baseline was published in July without having been replayed against our next propagation change. We have replayed it since: it held.

Why this matters

UpTrust builds trust graphs. Our engine reads records of who recommends whom and computes personal, explainable rankings, along with structure: communities, bridges. We should be judged on tests like this one, which had a real way to fail. With topics mixed and positive evidence only, our own engine came out at chance level on the pooled comparison. Within each observer's own list, negative evidence included, that same configuration reached 0.787. Topic partitioning recovered a pooled signal the mixed-topics run does not carry, and led on every per-observer measure as well — on data we did not build, against answers it never read.

A version of this test can run anywhere you have one record to build evidence from and a separate, held-out record of positive and negative judgments: reviews, code approvals, citations, recommendations. If you are considering a trust or reputation layer and want to know whether it carries signal or only looks like it does, talk to us.