Scoring MAHORAGA's `WAIT` — how to find out whether confidence 0.88 was right without touching your account

MAHORAGA attaches a confidence number to every decision not to buy, and nobody has ever scored one. Here is the procedure for actually placing the opposite orde

Done when you can take one "do not buy" call from your own tool, place the opposite order for real, and walk away with a single number: the minimum move that call needed in order to be wrong.

1. The blank this tool kept leaving

MAHORAGA (github.com/ygwyg/MAHORAGA · MIT · 868 stars) is a 24/7 agent that runs on Cloudflare. It scrapes community posts and SEC filings into per-symbol sentiment scores, hands those to a model, and gets back BUY/SELL/HOLD plus a confidence.

Here is what it actually recorded when I ran it locally.

There is not one "buy" in that table. Every recorded call is a decision not to order. And every one of them carries a confidence number.

One correction first. In an earlier transfer check I wrote this tool's three WAITs as 0.88 · 0.89 · 0.95. Those three are from the equity path (GME, BB, BONK.X). The crypto path's three-run consistency table is 0.88 · 0.90 · 0.89, and the SKIP 0.20 is also crypto. This article keeps the two apart.

Here is the tool's own reasoning on GME (the tool's output):

"At the supplied $23.49 price, there is insufficient evidence of an attractive entry. … Confidence reflects the case for waiting, not a prediction of a price decline."

Hold on to that last sentence. It is the reason I placed no shorts in section 3.

MAHORAGA's dashboard with real verdicts in it

This is the tool at work. Top left is the live Alpaca paper account, below it the 29 signals collected this cycle, and bottom right is what the model wrote about each symbol. Look at the place where WAIT sits next to a confidence number — that number is the thing this article is trying to score.

How far you get without an account — local boot and where the wall is

Green is what runs without a Cloudflare account, red is what does not. Everything up to Ready on http://127.0.0.1:8787 is free — and the whole procedure below lives inside that green stretch. You do not have to deploy anything to get verdicts.

This tool issues no levels. The 5% stop and 10% take-profit are config constants (stop_loss_pct, take_profit_pct), not judgments. So the only quantitative thing it produces is the confidence — and nobody has ever checked whether that confidence was any good.

2. Why this has been empty until now

Scorecards have only ever existed for things you bought. A purchase leaves a fill, a fee, a P&L. Not buying leaves nothing at all.

But WAIT is not a blank. It is a testable claim: right now, staying out beats going in. With a 0.88 attached, more so.

Here is why it stayed empty. To measure a decision not to buy, you have to actually buy — and the moment you do, your account moves. Capital gets tied up, the loss limit gets eaten into, the performance rollup gets contaminated. The act of measuring destroys the thing being measured. So nobody measured it.

A trial order steps around that. It carries the judgment through to the close in a separate ledger that touches none of your balance, P&L, limits, or performance totals.

The decision not to buy gets a scorecard too. Without touching your account.

This is not a claim that nobody else could do it. A backtest could approximate it in principle. But a backtest is a replay of the past, so it cannot catch the moment and the quote the call was actually made at — and a live account brings back the destruction above. There was a gap between those two, and this article is a record of actually working in it.

3. The recipe — the thing that actually runs

3.1 First, get a verdict out of MAHORAGA

The worker and the screen are two separate apps. wrangler dev starts the API worker; the dashboard lives in its own directory.

If you only want the verdict, two lines will do it:

curl -H "Authorization: Bearer $TOKEN" localhost:8787/agent/trigger
curl -H "Authorization: Bearer $TOKEN" localhost:8787/agent/status

signalResearch carries verdict, confidence, entry_quality and reasoning. That is where the tool's job ends and the scoring begins.

3.2 Place the opposite, for real

The full recipe is in 03-trial-order-mahoraga-wait.sh. Here is its heart.

curl -sS -X POST "$WELTIQ_BASE_URL/v1/trial-buys" \
  -H "Authorization: Bearer $WELTIQ_API_KEY" -H 'Content-Type: application/json' \
  -d '{
    "venue": "binance-spot", "symbol": "BTCUSDT",
    "client_order_id": "mahoraga-wait-1-wait-088",
    "side": "long",
    "exit_plan": {"close_after_seconds": 600},
    "client_tracking_id": "mahoraga-wait-1",
    "tags": ["counterfactual", "part-29A"],
    "metadata": {"verdict": "WAIT", "confidence": "0.88",
                 "source": "reports/a/29-account-walls/RUN-LOG.md:148"}
  }'

Three things are decided before you send anything.

① The side is long. WAIT and SKIP both mean "do not buy", and the opposite of that is buying. I placed no shorts — because of that sentence in section 1. Attaching a short to a call that never implied a decline is not a counterfactual, it is invention.

② The exit is a deadline and nothing else. This tool issues no levels, so I invented no price conditions. close_after_seconds: 600, full stop.

③ Confidence goes in as a string. Send "confidence": 0.88 as a number and you get a 500. The request body is hashed, and the determinism rule rejects floats. Eight of my first attempts died that way; strings went through.

And note that the verdict and the confidence are never interpreted by the server. The intent belongs to the tool; we hold the execution record. The only thing joining them is client_tracking_id.

3.3 Pull it back — two calls

curl -sS "$WELTIQ_BASE_URL/v1/forward-tests?tracking=mahoraga-wait-1&limit=200" \
  -H "Authorization: Bearer $WELTIQ_API_KEY"

curl -sS "$WELTIQ_BASE_URL/v1/forward-tests/orders?tracking=mahoraga-wait-1&format=csv" \
  -H "Authorization: Bearer $WELTIQ_API_KEY"

The second line is the account-wide endpoint added on 2026-09-28. You no longer pick a test first — the old path was "one list call plus one export per test", which in a month where you used the whole quota meant 1,001 calls. Now it is one.

Server-generated close orders inherit the tracking ID and the tags, so seeding only the entry still brings the exit row back in that CSV. That is where the round-trip cost comes from.

When you join, do not use awk. The metadata and exit_plan columns contain quoted commas, and cutting on the field separator silently shifts your columns. The recipe joins with an inline python3 csv module and says why in a comment.

Environment notice: the numbers below came off an isolated server with synthetic quotes. This is not production, and the P&L is not a market fact — what is being measured is the wiring and the scoring frame, not whether the tool was right.

4. What came back

All four BTC/USD cases went 201 → filled → closed on the 600-second deadline.

Entry notional 6.8379237160 USDT, quantity 0.00008 BTC. exit_trigger_px is null — a trade closed on a deadline has no trigger price, and nothing is invented to fill that column.

The equity path (GME, BB) was accepted with 201 and then came back from placement as rejected:market_closed. No fill means no P&L, so "was the wait right" is unresolved for those this cycle. I did not fake the clock to get around it.

4.1 The one number to read out of that table

Break-even 0.2223%. Meaning: had you gone in at that moment, BTC would have had to rise at least 0.2223% within ten minutes just to get your money back. It did not, so sitting out was right. Conditionally: "provided the price did not move more than 0.2223% during those ten minutes."

Turned around, that is the tool's threshold. For this call to have been wrong, BTC needed to rise at least 0.2223% in ten minutes.

4.2 The surprise — confidence made no difference at all

All four net P&Ls are identical to the last decimal: −0.0152017520. Confidence 0.88 and confidence 0.20 produced the same number.

The reason is plain. All four were placed at the same moment against the same quote. In a frozen market, P&L is decided by the round-trip cost, not by confidence.

This is not a failure. It is a condition the frame revealed as it was being built, and it settles the design of the next cycle.

To measure any correlation between confidence and outcome, the counterfactual has to be placed at the moment and the price the call was made. Batch it up and replay it later, and the confidence disappears while the cost remains.

Which means the feature only pays off if the counterfactual order goes out automatically the instant the tool issues the call. Replaying it after the fact yields nothing.

Trial buy detail — one counterfactual's plan and what triggered it

This is our screen. Expanding a row shows the side, the exit plan, the effective close time and what decided it, and the trigger condition with its quote. Look at the row where exit_trigger is close_after and exit_trigger_px is empty — a deadline close has no trigger price, and the screen says so rather than filling it with a zero.

5. What this combination cannot tell you

① One sample says nothing. A wait that ends in a loss is not a wait that was right — you need a distribution. The crypto sample here is four cases, and those four turned out to be indistinguishable from each other (section 4.2). Statistics would need the monthly 1,000 spread across the actual decision times.

② The counterfactual measures one side only. "What if I had bought" is measured. "What did I save by not buying" is still a judgment call. This table gives you a threshold, not a value.

③ We do not say the tool was right or wrong. We measure and record. Even the "right" above is a verdict against a stated criterion, and a conditional one.

④ The most confident wait dropped out of the sample. BONK.X carried 0.95, the highest in the cycle, and I could not run it. The tool's own text says "A displayed price of $0 is likely rounded or invalid and cannot establish an entry valuation" — there is no price in the record to anchor to. Substituting one would make it invention rather than a counterfactual.

⑤ The confidence is not a property of the tool. Same market, same prompt, same conclusion ("do not buy"), and confidence split 0.88–0.90 against 0.20. That difference belongs to the model, not the tool. The counterfactual inherits the contamination intact — pin and record the model every cycle.

⑥ The exchange decides what sitting out is worth. This is the most practical finding of the cycle.

In crypto, a ten-minute wait is wrong if the price moves 0.22%. A 0.22% move in ten minutes is not rare in Bitcoin — this tool's waits are defended by a very thin band of cost. US equities are the reverse: round-trip cost is near zero, so waiting there is not justified by cost at all, only by whether the pullback actually came.

⑦ Synthetic quotes. The P&L above is a model result off fixed quotes on an isolated server. What was confirmed is the wiring and the scoring frame, not exchange performance or real fee levels.

6. When to use this and when not to

Use it if you are gating on confidence. MAHORAGA's default gate is min_analyst_confidence 0.6. If you are filtering on that number, you have almost certainly never checked whether it correlates with anything. This procedure produces the first sample. Just obey section 4.2 — place it at the moment of the call.

Use it for tools that issue no levels. A tool like this one, which produces only a direction and a confidence, looks like a tool with empty mandatory fields from a transfer point of view. From a counterfactual point of view it is the opposite — supplying the side is not invention, it is the definition of the counterfactual — which makes it a good sample.

Do not use it if position weight is the output. The server decides the quantity, so "10% of capital" collapses to the minimum tradeable unit. Tools whose metrics are scale-free lose nothing; tools whose answer is the weight lose half of it.

Do not use it to score a long horizon through a short window. A trial order lives at most 168 hours. Cut a three-month judgment down to seven days and what you are measuring is no longer that tool's judgment. This tool issues no horizon so it escapes that, but the next few parts do not.

Do not attach a short to a call that never implied a decline. That is invention, not a counterfactual. What the original text claimed is what fixes the side.

Evidence for this article: the counterfactual measurement record · per-case request and response text · this part's recipe · the tool's own run log. Every number above came out of those files, and so did the things I could not measure (GME and BB closed, BONK.X with no price).