Skip to content

fix: stabilize analytics sample cohort #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
silentgeckoaudit3801 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: stabilize analytics sample cohort #5

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion scripts/collect-metrics.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,10 @@ async function fetchAllAgents() {

async function sampleInboxMetrics(agents, sampleSize = 20) {
console.log(`Sampling ${sampleSize} agent inboxes...`);
const sample = agents.slice(0, sampleSize);
// Keep the cohort stable when the API changes its result ordering.
const sample = [...agents]
.sort((a, b) => a.verifiedAt.localeCompare(b.verifiedAt) || a.btcAddress.localeCompare(b.btcAddress))
.slice(0, sampleSize);
Comment on lines +37 to +40

@gemini-code-assist gemini-code-assist Bot Jun 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Methodological Issue: Selection Bias in Cohort Sampling

Sorting the agents by verifiedAt ascending means the sample cohort is permanently frozen to the 20 oldest registered agents in the network. As the network grows, new agents are completely excluded from the sample.

This introduces a severe selection bias because:

  1. The behavior of the oldest 20 agents is extrapolated to the entire network using scaleFactor = agents.length / sample.length.
  2. If these 20 oldest agents become inactive or have unusually high activity compared to the rest of the network, the extrapolated metrics (total messages, sats received/sent) will be highly inaccurate.

Recommended Solution: Deterministic, Unbiased Hashed Sampling

To keep the cohort stable and deterministic without biasing it to the oldest agents, we should sort by a property that is randomly distributed across all agents, such as a hash of their btcAddress.

Using a hash (like SHA-256) is crucial because:

  • It ensures that new agents have an equal chance of being sampled.
  • It avoids address format bias (e.g., legacy addresses starting with 1 sorting before Bech32 addresses starting with bc1 lexicographically).
Suggested change
// Keep the cohort stable when the API changes its result ordering.
const sample = [...agents]
.sort((a, b) => a.verifiedAt.localeCompare(b.verifiedAt) || a.btcAddress.localeCompare(b.btcAddress))
.slice(0, sampleSize);
// Keep the cohort stable and unbiased by sorting by the hash of the BTC address.
const crypto = require('crypto');
const getHash = (addr) => crypto.createHash('sha256').update(addr || '').digest('hex');
const sample = [...agents]
.sort((a, b) => getHash(a.btcAddress).localeCompare(getHash(b.btcAddress)))
.slice(0, sampleSize);


let totalMessages = 0;
let totalSatsReceived = 0;
Expand Down