Backtest

We compare simulated threads with real ones on the same metrics, and a blind judge tries to tell them apart. These are the current numbers, including the ones that look bad. See about for how comments are made.

statusok
runAt2026-11-10T18:05:35.955Z
seriesStartedAt2026-11-10T17:23:30.453Z

heldOut

posts60
realComments1,750
simComments1,497

metrics

namerealsimdistanceunitnote
words per comment, top-level22280.110KSmedian shown; n 826 real / 875 sim
words per comment, reply31300.073KSmedian shown; n 924 real / 622 sim
top-level share per thread0.7820.6670.207KSmedian shown; n 58 real / 58 sim; 2 capped posts excluded
max depth per thread220.121KSmedian shown; n 58 real / 58 sim; 2 capped posts excluded
share of comments <= 10 words0.1950.1960.000abs
depth histogram (0..6+){"0":0.472,"1":0.236,"2":0.1423,"3":0.0697,"4":0.0389,"5":0.024,"6":0.0171}{"0":0.5845,"1":0.2231,"2":0.1062,"3":0.0474,"4":0.0261,"5":0.0094,"6":0.0033}0.113TV
length bucket, top-level{"xs":0.2276,"s":0.3378,"m":0.2712,"l":0.138,"xl":0.0254}{"xs":0.216,"s":0.2537,"m":0.3749,"l":0.1303,"xl":0.0251}0.104TV
length bucket, reply{"xs":0.1667,"s":0.2608,"m":0.3323,"l":0.1991,"xl":0.0411}{"xs":0.1672,"s":0.2605,"m":0.4035,"l":0.1334,"xl":0.0354}0.072TV
children per top-level comment (0..6+){"0":0.7542,"1":0.1404,"2":0.0472,"3":0.0242,"4":0.0182,"5":0.0048,"6":0.0109}{"0":0.7726,"1":0.1394,"2":0.048,"3":0.0251,"4":0.008,"5":0.0034,"6":0.0034}0.020TV
archetype, top{"alternative_pointer":0.0978,"domain_veteran":0.0217,"pedant":0.0109,"enthusiast":0.2228,"pricing_sceptic":0.0109,"security_worrier":0.0326,"site_nitpicker":0.1033,"feature_requester":0.0761,"curious_asker":0.1413,"cynic":0.0815,"helper":0.0109,"topic_debater":0.0435,"tangent":0.0652,"joker":0.0815}{"alternative_pointer":0.088,"domain_veteran":0.072,"pedant":0.016,"enthusiast":0.196,"pricing_sceptic":0.076,"security_worrier":0.068,"site_nitpicker":0.004,"feature_requester":0.112,"curious_asker":0.244,"cynic":0.036,"helper":0.016,"topic_debater":0.024,"tangent":0.012,"joker":0.036}0.120JSJev labels, 184 real / 250 sim
archetype, reply{"alternative_pointer":0.037,"domain_veteran":0.0417,"pedant":0.1019,"enthusiast":0.0694,"pricing_sceptic":0.0093,"security_worrier":0.0185,"site_nitpicker":0.0093,"feature_requester":0.0185,"curious_asker":0.0694,"cynic":0.0787,"helper":0.1065,"topic_debater":0.3009,"tangent":0.0787,"joker":0.0602}{"alternative_pointer":0.08,"domain_veteran":0.1267,"pedant":0.06,"enthusiast":0.0467,"pricing_sceptic":0.0067,"security_worrier":0.0667,"site_nitpicker":0.04,"feature_requester":0.08,"curious_asker":0.14,"cynic":0.0333,"helper":0.14,"topic_debater":0.14,"tangent":0.02,"joker":0.02}0.120JSJev labels, 216 real / 150 sim
stance, top{"positive":0.3587,"neutral":0.2391,"mixed":0.125,"negative":0.2772}{"positive":0.288,"neutral":0.28,"mixed":0.156,"negative":0.276}0.005JSJev labels, 184 real / 250 sim
relation, reply{"agree_extend":0.3981,"disagree":0.3194,"correct":0.0556,"answer":0.1065,"question":0.0509,"joke":0.0417,"tangent":0.0278}{"agree_extend":0.5667,"disagree":0.2333,"correct":0.0467,"answer":0.0867,"question":0.06,"joke":0.0067,"tangent":0}0.041JSJev labels, 216 real / 150 sim
anecdote rate0.2200.3330.113absJev label, score >= 1.5
informal rate0.2120.1400.072absJev label, score >= 1.5
question rate0.2430.3770.133abscontains '?'
quote rate0.0500.0490.000absa line starting with '>'
code rate0.0070.0050.003absreal: <pre><code> block; sim: backticks or 4-space indent
tone 0..3{"0":0.2275,"1":0.465,"2":0.2375,"3":0.07}{"0":0.22,"1":0.5325,"2":0.24,"3":0.0075}0.070TVJev abrasive score
em dash0.00600.006absconditionals.style definition
exclamation0.1290.1140.015absconditionals.style definition
bullet list0.01400.014absconditionals.style definition
starts great/love/nice/cool/awesome/amazing0.0300.0160.014absconditionals.style definition
multi-paragraph0.3490.1860.163absconditionals.style definition
addresses-parent score, reply1.7501.9400.406KSmedian shown; n 216 real / 150 sim; Jev score 0..2

judge

judgenunavailableaccuracyci95note
muse-wonder-1.023810.832[0.7792,0.8741]top 88/119, reply 110/119; picked A 128/238; 1 invalid answers excluded; judge separates real from sim
jev23910.259[0.208,0.3185]top 30/120, reply 32/119; picked A 87/239; 0 invalid answers excluded; below chance: judge systematically favors simulated text; do not interpret this as indistinguishability
muse-wonder-1.0 vs naive23550.889[0.8428,0.9234]top 98/118, reply 111/117; picked A 117/235; 0 invalid answers excluded
jev vs naive23550.106[0.0731,0.1523]top 11/118, reply 14/117; picked A 103/235; 0 invalid answers excluded

cost

perFreePostUsd0.013
perCommentUsd0.001
chatOnlyEstimatePerFreePostUsd0.025
projection1000PostsUsd13.230
evalSpendUsd5.930
noteThe primary cost and 1,000-post projection use the complete free-job ledger measurement below. The chat-only estimate excludes Jev and interest and is shown separately. From the backtest engine's own per-call costs (cached per post, so resumed runs keep earlier posts): comments $2.5055 over 1497 accepted comments (retries included); digest+interest+plan $0.7624 over 60 posts. chatOnlyEstimatePerFreePostUsd = fixed per post + 7.5 x perComment (mean free size). evalSpendUsd is the ledger's judge+eval rows since the evaluation series began, including resumed work (jev/eval 1947 calls $0.0779, jev/moderation 2 calls $0.0001, meta/eval 1812 calls $3.5243, meta/judge 601 calls $2.3281); the persisted series start keeps earlier cached work in this total. Script totals: non-Jev $0.0000 this run (naive baseline $0.2299, judge replies $0.2475 cumulative), Jev 0 calls this invocation, series ledger $0.0779.

runtimeMeasurement

measuredAt2026-11-10T17:52:06.310Z
posts2
perFreePostUsd0.013
perCommentUsd0.001
projection1000PostsUsd13.229
noteMeasured free jobs against real providers and the preview database: one fetched page and one dead URL fallback. Includes digest, interest, plan, comments, Jev gates, and retries. Excludes submission moderation, payment fees, database, and hosting. Small sample; the projection assumes the same mix.

notes