Scoring the BUY that orallexa's confidence gate blocked — and the WAIT rows nobody could grade

orallexa's gate stopped a BUY with p_up 0.55. I placed that never-placed order for real, without touching the account, and measured what it would have taken to

You've finished this article successfully if you can take one WAIT, SKIP, or gate-blocked BUY from your own tool, get a one-line scorecard for it without your account moving at all, and feed that line into your own scoring script.

The human-side lines are real commands rewritten as dialogue; the outputs are measured and the numbers are untouched.

1. The blank this tool kept leaving

I ran orallexa in part 34. It carries the heaviest measurement apparatus of any tool in this series — Brier scores, reliability diagrams, ECE, Platt calibration, all in the code. It doesn't just print a confidence number; it was built to grade that number itself.

Here is one basket run over 16 symbols (raw/probe-basket.json). The verdict split was BUY 5 · SELL 1 · WAIT 10.

The raw and on-screen confidence values for all 16 symbols cluster into a handful of numbers. The on-screen values are 42.6%, 47.6% and 50.8%. Meanwhile the p_up (upside probability) that ships with the same judgment spreads much wider — **look at the p_up column**. That, not the on-screen confidence, is what gets graded

Look at the p_up column rather than the on-screen confidence. That distinction becomes the whole article in a moment.

orallexa's own UI, MSFT on SWING/1D. Compare the CONFIDENCE panel at the bottom with the UPSIDE PROBABILITY above it — two numbers on one screen that both read as "how sure am I", and they disagree. The one that enters scoring is **the probability above**, not the confidence below

For anyone meeting this tool here: that's what one judgment looks like. Part 34's conclusion was that the number on screen and the number being graded are not the same number, and this article starts on top of that.

The tool also has a confidence gate. The pilot defaults it to 60%.

DECISION MSFT decision=BUY confidence=50.8% risk=MEDIUM elapsed=4.1s
SKIP MSFT — confidence 50.8% below gate 60.0%
PILOT_DONE trades=0 decisions=1

The judgment is BUY. The gate blocked it, so no order went out. trades=0.

That's the blank. Nobody knows whether that gate was right. No order means no outcome, and no outcome means nothing to grade.

And a bigger blank — eleven WAIT rows

The tool's Brier audit (brier_audit.py) reads the decision log and grades it. What it grades matters: the value it uses for forecast_p is not the on-screen confidence but probabilities.up (brier_audit.py:301).

Run it as shipped and it reads zero rows, because the decision-log path doesn't match. Point it at the real log and it reads 20 rows — **of which very few are gradeable**. Look at the Platt line near the bottom: "Need at least 30 observations". That's the sample size the tool asks for itself

Of the 20 rows, 11 WAIT rows drop out as "no directional claim, not a scoring candidate" (RUN-LOG.md:214). Of the 9 BUY/SELL rows that remain, 8 are "scored": null — and the reason is not the gate, it's that less than a day has passed.

But those 11 WAIT rows do carry probabilities.up (NVDA 0.40 · AAPL 0.45 · QQQ 0.35 …). What's missing isn't the probability. It's the direction.

2. Why this stayed empty

The scorecard for "decided not to buy" has been missing for a structural reason, not a lazy one.

To measure it you have to actually buy, and buying moves the account. Capital gets tied up, the loss budget gets eaten, and the result lands in your performance numbers. The act of measuring damages the thing being measured. So nobody measured it.

A gate-blocked BUY is the worst case of this. Lower the gate and place it for real, and that judgment is now inside the account — you've removed the gate in order to test the gate.

The trades you decided not to take get a scorecard too. Without your account moving.

A trial order carries that judgment to completion on a separate ledger that never touches your balance, P&L, loss limits or performance totals. The gate stays exactly where it is; only the order it blocked goes out separately.

One contract line first

If you want to carry a number like p_up on the order, carry it as a string.

"metadata": {"p_up": "0.55"}     // send it as a number and you get a 500

canonical_json rejects floats under its determinism rule (weltiq/core/canonical.py:18 — "float 금지 — Decimal 또는 str을 사용 (결정론)"). On my first attempt I sent confidence as a number, all eight cases came back 500, and switching to strings put them through.

3. The recipe — the thing that actually runs

The recipe is here: 07-trial-order-orallexa-calibration.sh. Four steps.

The intent stays with the tool. All that goes to the server is client_order_id, client_tracking_id and string metadata; the gate value and the reasoning live only in tool_log.csv. What joins them later is a single tracking id.

Step ③ is what changed. Recovery used to be "one list call plus one export per test". Now the whole filtered set comes back in a single call, and the columns match a single-test export character for character, so concatenating them doesn't quietly break. The server's automatic exit order also inherits the entry's tracking id — realised P&L lands on that exit row, so without the inheritance a ?tracking= recovery would bring back entries and leave the P&L behind.

Where the tick grid bites — send the raw value first

The BTC-USD judgment came with levels (raw/btc-decision.json): yf_entry_price 84138.4609375 · stop_loss_price 82034.9994 · take_profit_price 88345.384 · approved: true.

Send them raw and you get this.

Nothing gets silently rounded. A condition price has to sit on the same grid as a real order, and a server quietly altering a price you supplied is the more dangerous behaviour.

Writing down the direction of the rounding matters. Stop rounded up (82034.9994 → 82035.00), target rounded down (88345.384 → 88345.38). For a long, raising the stop brings it closer and lowering the target brings it closer — both conservative. If you don't record which way you rolled, nobody can later tell whether the rounding produced the result.

How direction was chosen

All three went out long, because all three have p_up > p_down. There are WAIT rows that do imply downside (META 0.25/0.45 · AMD 0.25/0.45 · PLTR 0.30/0.42). But nowhere in orallexa's records is there a price for those three symbols. The rule is "stay near the price band the tool recorded", and that rule can't be kept here, so I didn't run them. Inventing a price turns a counterfactual into fiction.

Environment for the numbers below: an isolated temporary server · synthetic quotes (each value taken from the price band in that tool's own records) · not production · the P&L is not a market fact.

4. What came back

Three cases. All three were accepted with 201.

Read the last column, not the P&L. Pin the quote at the price the tool recorded, look at net P&L 600 seconds later, and in a frozen market that net is exactly the round-trip cost. So |net| ÷ entry notional is "how far would this have had to move in ten minutes for that judgment to be worth anything".

0.2224%. Put the other way round — for the BUY that gate blocked to have been worth taking, BTC had to rise at least 0.2224% inside ten minutes.

The tool's own stop at 82,035 and target at 88,345.38 were never touched in that window (−2.51% / +4.99% from the 84,147.91 entry). The spread the tool set with risk_reward_ratio: 2.0 assumes a daily horizon, and that is a different time scale from the ten-minute window that measures the "right now" of a wait.

Our trial detail — one counterfactual accepted, filled and closed on its deadline. Look at close_reason and the exit_trigger_px beside it: **a deadline close has no trigger quote, so it's null**. Nothing gets filled in with a fake zero

The unmeasured case on the same screen. Accepted with 201, then returned from placement as rejected:market_closed. Look at the **quota display** — with no fill at all, this one's single unit is refunded

The two US equity cases are unmeasured

us-equity was closed at run time.

venue=us-equity  local_time=2026-09-28T00:27-04:00  is_open=False  state=closed
next_open=2026-09-28T09:30-04:00   (= 22:30 KST)

I did not fake the clock to get around it. A fill obtained that way isn't a fill that passed the live rules. Run the same recipe between 22:30 and 05:00 KST and it lands on the same criterion.

So what happens to the Brier sample

The value of this piece isn't the 0.2224% above. It's here, and it splits three ways.

(a) The gate-blocked BUY is already a scoring candidate. orallexa writes each judgment to decision_log and the Brier audit reads that log — it never checks whether an order went out. Of the 9 BUY rows in raw/brier-scored.json, 8 are "scored": null, and the reason is the one-day wait, not the gate. So MSFT's and BTC's p_up 0.55 are already in scope contractually. What the counterfactual trial order adds here is actual, sooner and cheaper — a ten-minute window with real costs inside it.

(b) WAIT rows are out of scope today. The counterfactual opens that door. Eleven of the 20 rows drop out as "no directional claim", yet those rows carry probabilities.up. What's missing is direction, not probability, and a counterfactual order supplies exactly that. Leave forecast_p = probabilities.up alone and fill only actual. The sample goes from 20 rows to 31.

(c) Two places in the contract block it. Both were measured this round.

1. On our side: a float in metadata is a 500 (section 2). To carry a probability as a first-class number and compute Brier server-side you need a dedicated Decimal field, not metadata. metadata is hashed contract body, so putting numbers there collides with the design itself.
2. On orallexa's side: the lowest reliability bucket is 50-60% (brier_audit.py:390). The WAIT rows at p_up 0.40, 0.45, 0.35 have no bucket to land in. And Platt calibration demands 30 or more observations ("Need at least 30 observations, got 1"). Growing the sample with counterfactuals is precisely the road to that 30 — but lowering the bucket floor to zero has to come first.

It is contractually possible. It needs a dedicated numeric field on our side and a lower bucket floor on theirs. Until then, string metadata plus scoring outside works.

5. What this combination can't tell you

① One sample says nothing. The 0.2224% above is one case. A wait that ends in a loss was not thereby correct — you need a distribution, and building one is what this path is for. The 30 observations orallexa asked for are still a long way off.

② A counterfactual measures one side only. It measures "what if I had bought". "What did not buying save me" is still a judgment. Only the first is in that table.

③ We don't say the tool was right or wrong. We measure and record. What the table says is not "orallexa's gate was correct" but "for that gate to have been wrong, 0.2224% was needed inside ten minutes". Those are different sentences.

④ The P&L is not a market fact. It's arithmetic on synthetic quotes. What was confirmed is the wiring and the scoring frame, not how BTC actually moved.

⑤ Batch them late and the confidence signal disappears. This was the hardest constraint this round. Fire the counterfactuals in one later batch instead of at the moment of the judgment and, in a frozen market, what determines P&L is not confidence but round-trip cost. In another piece, four cases at confidence 0.88, 0.90, 0.89 and 0.20 came back with identical P&L to the last decimal. To measure confidence against outcome, the counterfactual order has to go out at the instant the tool emits the wait.

⑥ The time scales don't match. orallexa's levels assume a daily horizon; this window was ten minutes. A second set matched to the tool's own horizon is the next job.

⑦ The downside WAITs could not be run. Their prices aren't in orallexa's records. The most valuable samples are still missing.

6. When to use this, and when not to

Use it when:

• your tool emits a probability or confidence as a number and that number has no scorecard. orallexa is exactly this case
• a gate or filter blocks orders. Whether the blocking side was right is precisely what never gets recorded
• your scoring tool states its own sample requirement. Thirty observations or 384, you can only reach that number on a path that doesn't use the account

Don't use it when:

• you want a verdict from one case. This path builds distributions; a single case mostly measures round-trip cost
• the tool's main output is allocation percentage. The server sets quantity, so allocation collapses
• the tool's records contain no price. The moment you invent one it stops being a counterfactual. Writing "not run" is the better outcome
• you want to measure US equities while the market is closed. It will be accepted and come back rejected:market_closed, unmeasured. Don't fake the clock

Evidence for this article: axis A counterfactual log · per-case request/response originals · recipe. The original tool run is in part 34, orallexa.