CHECK THE FACTS (CTF)
Turning Neshan's unresolved grey points into a crowdsourced verification system
A case study — reframing a verification problem, testing the crowd, and proving it can be trusted.
IN SHORT
How do you verify places your editors can't?
Neshan had a growing pool of grey points — places users had already added but editors couldn't confidently verify. The real problem wasn't contribution; it was a verification gap: many places needed local knowledge editors couldn't access remotely. I helped design CTF, a crowdsourced verification system that asks nearby users the simple question “does this place exist?” and resolves it through trust-weighted aggregation. I ran three rounds of usability testing and designed a ground-truth accuracy system to prove CTF's answers were correct. After launch, CTF's share of daily new points resolved grew from near zero to 20–30% by month six — and kept rising.
THE PROBLEM STARTED WITH A COFFEE SHOP
A place that existed but couldn't be found
A few of us were looking for newly opened places in our city — and couldn't find some of them in Neshan as trusted, published places. They had already been added by users, but sat unpublished as grey points: shown on the map in grey, and missing from search.
01 · DISCOVERY
A newly-opened place exists in the real world.
02 · SEARCH FAILS
Searching Neshan, it doesn't show up — grey points aren't in search.
03 · ALREADY THERE
A user already added it — but it's stuck, unpublished, as a grey point.
04 · CTF RESOLVES
The crowd confirms it exists; it publishes to the map.
RESEARCH CHANGED THE PROBLEM DEFINITION
Three angles on one stalled queue
I investigated from three sides: the editors who process points, the users who contribute them, and the closest product doing something similar.
EDITOR INTERVIEWS · FINDING
The queue fills faster than it empties
Volume: inflow far exceeds what editors can process. Capability: many grey points carry only a name + location, so editors cross-check external maps by hand — and often still can't decide.
IN-APP FEEDBACK · FINDING
Contributors felt ignored
Users repeatedly asked why the places they added were never published. Verbatim: “Why don't the places I add ever get approved?”
BENCHMARK · GOOGLE MAPS
Same insight, different context
Google's “Check the facts” routes place-questions to nearby users — the same core idea. CTF differs with purpose-built eligibility + an explicit trust system.
GOOGLE MAPS — “Check the facts” (the closest analog)
PROBLEM DEFINITION
The real problem was verification, not contribution
Asking editors “does Neshan have this place?” flipped the question. Users had already added many of these places — they sat in the grey-point pool, unverified. So the problem was never “how do we get more contributions?” It was “why do the places users already contribute get stuck before verification?”
BEFORE — how it first looked
“Users can't find new places” → a CONTRIBUTION problem
AFTER — what it actually was
“Users already added them” → a VERIFICATION problem → a LOCAL-KNOWLEDGE problem
The missing fact lives near the place — not at a desk.
PRODUCT HYPOTHESIS
Route to the crowd — but never trust it blindly
The answer had two deliberate halves. First, crowd over headcount: only people near a place reliably know it exists, so scaling editors could never close the capability gap. Second, explicit trust over raw consensus — because an unchecked crowd invites both honest error and gaming.
- Grey pointunverified place
- Nearby userslikely local knowledge
- Question“does this exist?”
- Yes / No / Don't knowsimple answer
- Trust-weighted aggregationnot raw votes
- Decisionpublish · grey · reject
Editors run in parallel as a high-authority override — CTF resolves what they can't, it doesn't replace them.
EXPLORATION
Many directions — and a decision on each
Before settling the CTF question, I explored the format widely. What mattered wasn't the number of options — it was making a clear call on each, and knowing why.
Map vs. no map
Kept the map — users rely on streets and nearby places as evidence for whether a place is real.
Slider vs. simple answers
A quick guerrilla test showed sliders confused people. Dropped for plain Yes / No / Don't know.
POI layer on vs. off
Nearby POIs help orient users but can also become false ‘proof’ a place exists.
Question wording
Iterated copy repeatedly — ‘ask me again later’ → ‘next’, ‘not sure’ → ‘don't know’.
Some of the question & card variants explored
TESTING
Three rounds, three things you couldn't guess from a desk
Three rounds of testing each challenged a different assumption. What mattered wasn't the number of tests — it was that each one changed a specific design decision.
ROUND 1
The map answered for them
- ASSUMPTION
- Users draw on their real knowledge of the place.
- EVIDENCE
- Many read the map as the answer key (name on map = “exists”). A real place I deliberately renamed fooled many on autopilot.
- CHANGE
- Replaced the standard red pin with a question-mark marker — the pin asks, it doesn't assert.
ROUND 2
Wording failed
- ASSUMPTION
- Users read the question and answer from it.
- EVIDENCE
- Confident on published points, hesitant on grey ones — and “ask me again later” / “not sure” confused people.
- CHANGE
- Simplified the wording to “next” and “don't know.” Needed a cleaner re-run.
ROUND 3
Uncertainty was a signal
- ASSUMPTION
- More votes = a better answer.
- EVIDENCE
- With a ≥3-positive-vote rule, the crowd got only some right — and 68% of all answers were “don't know.”
- CHANGE (MINDSET)
- Not a failure. It proved the crowd is cautious and accuracy must be measured — leading straight to the accuracy system.
HOW DO WE KNOW THE CROWD IS RIGHT?
We tested the testers
Trusting a crowd raises a fair worry: what if people just answer wrong? So before relying on CTF's answers, we checked them against places where we already knew the truth. We fed CTF two kinds of places on purpose:
KNOWN-REAL PLACES
Recently verified places we knew were real. The crowd should confirm these — if they rejected one, that's a visible mistake.
FAKES WE MADE OURSELVES
Real places we deliberately altered — moved the location, removed the photos — so they looked believable but were definitely wrong. The crowd should reject these.
Because we knew the real answer for every test place, we could measure exactly how often the crowd got it right — and we measured the human editors the same way, holding both to the same bar. It runs continuously, with a weekly accuracy check.
Recently verified places we know exist. Correct outcome = CTF confirms them.
Real places we moved & stripped of photos — believable but definitely wrong. Correct outcome = CTF rejects them.
↓ fed in, mixed with normal traffic
↓ compare every decision to the answer we already knew
Weekly accuracy score — for CTF and editors, on the same bar
Ongoing measurement
CHOOSING A WEB MVP
Reduce the cost of learning before paying for native
Native Android development would have been expensive if the hypothesis failed. The PM and I chose to validate the flow on web first — an approach Neshan had used before. Native = better integrated UX, higher cost and risk. Web = faster validation, lower risk if the idea fails, at the cost of a context switch. The decision: optimize for learning speed.
Native
- + Better, fully-integrated in-app UX
- − High build cost
- − High risk if the hypothesis fails
Web MVP
CHOSEN- + Faster to validate the core idea
- + Lower cost
- + Low risk if the hypothesis fails
- − A context switch out of the app
Decision: optimize for learning speed — validate on web first, then build native.
HOW THE SYSTEM WORKS
End-to-end architecture
User contribution → grey point → eligibility → relevant-user selection → question → answer → trust → aggregation → threshold decision → resolution, with editorial fallback running in parallel as a high-authority override. CTF doesn't replace editors; it resolves what they can't.
The full flow across four lanes — left to right
- Add place
- Grey point
- Answer question
- Eligibility
- Pick nearby users
- Ask question
- Trust + aggregate
- Threshold → decision
- Parallel review
- High-authority override
- Publish / grey / reject
- Final map state
Editorial runs in parallel to CTF, not after it.
LAUNCH & RESULTS
A ramp from near-zero
Share of daily new-point additions CTF reaches a decision on. It launched near zero and climbed as contributor trust accumulated — the ramp is the story, not a single number. This is a resolution rate, not a publish rate: rejecting a fake place counts as much as publishing a real one.
Share of daily new points CTF resolves
↗ Launched near zero and climbed as contributor trust accumulated — still rising. The ramp is the story, not any single number.
MY CONTRIBUTION
Clear ownership boundaries
A senior project means being honest about what was mine and what wasn't.
- Problem reframing
- Research
- Interaction design
- Testing & iteration
- Accuracy harness
- Product direction
- Trust UX
- Launch
- Web MVP
- Trust algorithm
- Engineering
- Editorial operations
REFLECTION & NEXT STEPS
What I'd carry forward
User behavior isn't always intuitive — the map-as-truth assumption and the renamed attention-check surfaced only through testing. The crowd's 68% caution was honest signal, not a bug. Designing the accuracy harness alongside the feature was the most senior decision in the project.
Next: metadata CTF (Phase 2, rolling out); improvements to how points enter CTF (designed, in development); a standing accuracy dashboard; and contributor satisfaction (CSAT).
WHAT I LEARNED
- Contribution ≠ verification.
- The map can become an authority.
- Uncertainty is information.
- Crowdsourcing needs quality control.
- The most valuable design work happened beyond the final UI.
CLOSING
A place that couldn’t be found — now found at scale
CTF started with a small, human frustration: a real place we couldn’t find. It ended as a system that turns the knowledge of nearby people into a map others can trust — resolving the undecidable points no editor could reach, and measuring itself honestly along the way.