1 minute readReference note
How are reputation-analysis tools checked before their findings are used?
In brief
Test an analytical tool against the task it is meant to support, using examples whose interpretation can be independently checked. Software running successfully is only the first check.
A sentiment or identity test set should include easy and difficult cases: namesakes, quotations about another entity, historical events, mixed sentiment, and uncertain claims. Reviewers compare results with the sources and record error types instead of relying solely on an overall accuracy figure.
Check missing data, duplicates, coverage changes, and sensitivity to wording. Evaluate performance on consequential reputation errors as well as straightforward examples. Concentrate human review on mistakes affecting identity, allegations, status, and important audience decisions.
Updates need the same scrutiny. A changed model or collection method can alter results without any change in public information. Versioned methods and retained examples help separate measurement changes from actual reputation findings.