The trades you decided not to take now get a scorecard — two uses of trial orders, checked against ten tools

A large share of what these tools output is \"don't buy\", and that half of the scorecard was simply blank. Here's a measured account of why it was blank and wh

You've finished this article successfully if you can take one "decided not to buy" call from your own tool and get a scorecard for it without your account moving.

0. What's measured and what isn't

Measured: whether a verdict fits the fields of a trial order · whether it then runs to completion · whether the account really doesn't move · whether you can retrieve it later and score it.

Not measured: whether the tool is right. Returns. Which tool is better.

That distinction is the whole series. I'm on the measuring side, not the claiming side.

1. At a glance — the missing half

Run thirty-odd judgment tools and you notice the scorecard only ever covers what you bought.

When a tool says WAIT, HOLD, or AVOID, that is a testable claim: "not entering now beats entering now." Nobody scores it.

How much is missing — of the ten tools here, six had a no-trade verdict in the record. One is extreme: all four recorded verdicts were no-trade (WAIT 0.88 · 0.90 · 0.89 plus SKIP 0.20) — confidence numbers attached, and nobody ever checked whether that confidence was justified.

Why it was blank — the one line this article turns on

"You could backtest it" comes straight back. In principle, true. What actually blocked it is something else.

To score a decision not to buy, you have to actually buy it. And the moment you do, your account moves — capital is tied up, the loss limit is eaten into, performance aggregates get polluted. The act of measuring destroys the thing being measured.

So it went unmeasured. A trial buy carries that judgment to completion on a separate ledger that touches no balance, no P&L, no limit, no performance aggregate.

I didn't take the "account doesn't move" claim on faith — I counted. Across four verification runs, nine account-side tables compared in full, byte for byte identical, before and after.

The trial buy screen — this month's usage and the entry form Usage sits at the top (8 / 1000). Note what the form below it doesn't have: a quantity field — size comes from the instrument's rules and you can't change it here. That's exactly why tools become comparable.

2. Setup cost — none

Unlike the rest of this series, there's nothing new to install. Your tool is already there; on the weltiq side you need a pro account and one API key carrying the run-forward-tests scope.

What you do have is a budget: 1,000 a month on pro, 33 a day — enough to cycle thirty-one tools once daily. One approved new entry costs one; exits, queries, exports and identical resends cost nothing.

3. The two uses — why this feature exists

Use ① Score the decision not taken

Issue a counterfactual on the symbol and the moment the tool passed on: "what if we had gone in then?"

Supplying a direction here is not inventing data — it is defining the counterfactual. That distinction matters. In an earlier transfer check I lacked this framing and failed tools that only produce "wait". That verdict is reversed.

Use ② Change one field, same instant

Alongside the real order (or a baseline), issue a variant with exactly one field changed, against the same quotes. Same book, so the difference can only come from that field.

Since nothing touches the balance you can run as many arms as you like. Measured: six fired at once, 6/6 accepted, 0.02 seconds, nothing blocked. But N arms cost N against the quota.

4. Which tool for which use

5. Where it meshes — what actually happened

Environment note: everything below ran on an isolated server with synthetic quotes. What's confirmed is the wiring and the decision framework; the P&L is not a fact about the market.

5.1 The waits were right — conditionally

Four went to fill and exit, and all four said the same thing: break-even at 0.2223–0.2224%.

Entering at that moment needed a move of more than 0.22% just to break even. It didn't move that far, so waiting was right.

Counterfactual detail — entry through auto-exit Rounds issued as counterfactuals against 29A's WAIT 0.88 and 12's Hold. Note "condition fired: relative time reached" and, on the exit row below, tracking ID: tool-a-run-42 — the auto-exit inherits the entry's tracking ID.

5.2 But something more important came out

Confidence and outcome did not diverge — and could not have.

Fire four at the same instant against the same quotes and the P&L is set by round-trip cost, not by confidence. The numbers matched to the last decimal.

Which surfaces a design requirement:

The counterfactual has to go out automatically, at the moment and the price of the verdict.

Replay them in a batch later and nothing shows. The trial order has to leave at the instant the tool says WAIT. That's why every individual recipe in this series has that shape.

5.3 What a wait is worth is set by the venue

In crypto, a 0.22% move is enough to make the wait wrong. In US equities the round trip costs almost nothing, so "waiting" is never justified on cost grounds at all.

The same verdict is worth different amounts in different markets. That's market structure, not a property of the tool.

5.4 What one field changes

04 freqtrade was the cleanest. Three stop widths filled at the same price against the same book, and the recorded session low broke only the −0.05% line → 2.9 seconds stop_loss versus 46 seconds close_after.

That one field decided why and when the trade closed. It did not change the P&L (the quotes were fixed) — which tells you what an A/B here actually measures: the path, not the return.

15 Moss produced a paradox. The target multiple that was raised because of fees then fails to close the very trade it was derived from at its target.

23 arena flipped signs. Same fill, same exit, total P&L fixed — net P&L −7.03 / +2.98 / +1.98 / −7.03bp. But switching venue also changes lot size and minimum notional, so quantity differs by 13× — isolating "fees only" is not possible on our side.

14 QuantDinger couldn't be compared at all, because the two grid rows carry no per-trade levels. Why it failed, and what would have to exist, is the body of part 11.

Two arms of a comparison Two arms differing in one field, side by side in the same list. The tracking ID is the same on both and the memo says which arm is which — that's what ties them into one experiment later.

5.5 Getting it back — the round trip

Planting only pays off if you can retrieve it. I planted twelve and measured.

The tool keeps the intent; we keep the execution record. One tracking ID ties them together.

The four rows that failed to join were all server-created auto-exits — and that's where realized P&L lives. Retrieval was returning entries and leaving the P&L behind. Fixed; the capture in 5.1 shows the result.

Export The CSV and JSON export buttons, with the order and fill table beneath. Look at the "my record" column — tags, memo, tracking ID come back exactly as sent.

6. What this series showed — and what doesn't work

6.1 One of ten went in untouched

In the earlier transfer check I pushed the recorded output of thirty-one tools in as-is. Exactly one went in with nothing changed, and the thing that snagged was always the exit deadline. Even the eight that recorded one are all unusable — absolute timestamps are in the past, durations exceed the 7-day cap.

6.2 The 7-day cap truncates the judgment

When a tool says "3–6 months" and you put 7 days in that slot, what you measure is not that tool's judgment. Rounding adjusts a value; truncation changes a decision. You must not read a truncated round's scorecard as that tool's scorecard — and right now nothing in the record marks it as truncated.

6.3 Three caveats that travel with everything above

1. One sample tells you nothing. A wait that would have lost money wasn't thereby correct. You need a distribution — which is what the 1,000-a-month budget is for.
2. A counterfactual measures one side only. "What if I'd bought" is measurable; "what did I save by not buying" is still a judgment.
3. We don't say whether the tool was right. We measure and record.

6.4 Still not working

• Synthetic quotes. Wiring and contract are confirmed; exchange behaviour is not.
• US equity cases are unresolved — accepted, then returned as market_closed at placement.
• An unfilled arm leaves no record of the book it saw. If one side of an A/B doesn't fill, there's no field proving both saw the same quotes.
• Dust remainders — the reference size is small, so a partial fill can leave a remainder below the minimum notional; the exit then waits and the P&L never settles. Exclude or count those separately.
• The borrow fee on US equity shorts is charged as zero — the longer a short is held, the more favourable our record is versus the market.

Registering without a deadline Fill in a target but leave the time condition empty and this is what you get. Note that your inputs are still there — you don't retype anything.

7. Squares nobody touched

Things this series did not do. A map of blanks for the next one.

• A counterfactual that fires automatically at verdict time — what 5.2 demands. Today a human issues it afterwards
• Distribution by confidence band — what appears once 384 samples exist
• Feeding an exported record into a tool's own review step — this series stopped at transfer
• US equities during a session — the cases the closed market blocked

8. Where to start

If your tool waits a lot, start with ①. Better still if it attaches a confidence number, as 29A does — being able to line confidence up against outcome is rare.

If you tune parameters, start with ②. Part 9 is the easiest to follow: change one stop width and fire two arms at the same instant.

If neither applies, it isn't time yet. A tool that can't produce a deadline doesn't fit this path, and the moment you invent one you've created a judgment the tool never made.

Evidence for this article: counterfactual runs · A/B runs · round-trip run · transfer check across 31 tools — every number above comes from that record, and so does everything that failed.