Scoring TradingAgents' `Hold` — finding out whether \"do not add here\" was right

Hold is the one rating nobody ever grades. Here is the procedure for doing the exact thing the tool forbade, in a ledger that never touches your account, and re

Done when you can split a "stay in the position" rating into two counterfactuals — add now, exit now — place one of them for real, and read that rating's threshold off the result as a number.

1. The blank this tool kept leaving

TradingAgents (github.com/TauricResearch/TradingAgents · Apache-2.0 · 107,945 stars, observed 2026-09-22) runs eleven agents over a single symbol in sequence: four analysts, three researchers, a trader, three risk analysts, then the portfolio manager. The rating has five steps — Buy · Overweight · Hold · Underweight · Sell.

Here is one BTC-USD call I got out of it (2026-09-21, the tool's output).

**Rating**: Hold
**Price Target**: not provided
**Time Horizon**: 3-6 months

And the summary underneath names exactly what not to do.

"Maintain BTC-USD exposure only at a controlled, non-leveraged size; do not add at the current 81,526.82 level beneath the 82,253.84–83,000 resistance zone. … Treat a sustained close below 78,473.91 as a reason to reduce risk, with materially greater concern below the 73,216.37 50 SMA."

Its medium-term read was bullish — "BTC-USD has a constructive medium-term trend but an unattractive immediate entry point."

That is the blank. A Buy leaves a record when you act on it, a Sell likewise, but Hold is a call to do nothing, so it never gets graded. Except this Hold is not a blank. It is a testable claim: adding at 81,526.82 right now is a bad trade.

The risk team arguing with each other

This is the tool's signature screen. Inside Current Report, the neutral analyst is mediating between the aggressive and the conservative one. Look at Progress on the left — the trader is already completed and the risk team is still running. That debate has to finish before the portfolio manager's one-line Rating appears. That line is what this article scores.

One thing pinned down in advance. This Hold is from the 2026-09-21 run. When I re-ran the same configuration thirteen times the next day, Hold came up 0 out of 13 (Sell 7, Underweight 6). The same tool swings that much under the same settings — section 5 comes back to it.

2. Why this has been empty until now

Scorecards have only ever existed for things you bought. Hold places no order, so there is no fill, so there is no P&L, so there is nothing to grade.

Here is why it stayed empty. To test whether "do not add here" was right, you have to actually add — and the moment you do, your account moves. Capital gets tied up, the loss limit gets eaten into, the performance rollup gets contaminated. The act of measuring destroys the thing being measured. So nobody measured it.

A trial order steps around that. It carries the judgment through to the close in a separate ledger that touches none of your balance, P&L, limits, or performance totals.

The decision not to buy gets a scorecard too. Without touching your account.

This is not a claim that nobody else could do it. A backtest could approximate it in principle. But a backtest is a replay of the past, so it cannot catch the moment and the quote the call was actually made at — and a live account brings back the destruction above.

Hold has one extra property: it splits into two counterfactuals — add now, and exit now. Place them separately and a rating that is currently never evaluated becomes measurable. This cycle only placed the first one; the second needs an existing position to be meaningful.

3. The recipe — the thing that actually runs

3.1 First, get a rating out of TradingAgents

There is no dashboard. Search the repository and you will not find one line of streamlit, gradio, fastapi or flask. What exists is a terminal TUI. Do not go looking for a URL.

git clone https://github.com/TauricResearch/TradingAgents.git && cd TradingAgents
uv venv -p 3.12 .venv && uv pip install --python .venv/bin/python .
python -m cli.main

Install took 3.288 seconds for 85 packages. It asks for the ticker, the date and the analysts interactively, and then the screen streams for about four minutes.

The finished install — TradingAgents CLI on first launch

The green welcome box and the Workflow Steps line under it summarise the whole tool — analyst team → research team → trader → risk management → portfolio management. If those five stages appear, the install worked. Step 1: Ticker Symbol at the bottom is the first question.

Eleven agents working through in order

Progress at the top left is the live view. Look at the status bar at the very bottom — Agents: 1/11 | LLM: 4 | Tools: 10 | Tokens: 70.8k↑ 4.0k↓ | ⏱ 00:40 tells you how many agents have finished and how many tokens you have spent. On a tool that costs money per run, having that pinned to the screen matters.

Output lands in a dated folder. Calling graph.save_reports() from Python splits it into a thirteen-file tree, and the one file we need is 5_portfolio/decision.md — the **Rating** line and the summary.

3.2 Do the forbidden thing, for real

The full recipe is in 04-trial-order-tradingagents-hold.sh. Here is its heart.

curl -sS -X POST "$WELTIQ_BASE_URL/v1/trial-buys" \
  -H "Authorization: Bearer $WELTIQ_API_KEY" -H 'Content-Type: application/json' \
  -d '{
    "venue": "binance-spot", "symbol": "BTCUSDT",
    "client_order_id": "tradingagents-hold-1-hold-add",
    "side": "long",
    "exit_plan": {"close_after_seconds": 600,
                  "stop_loss": "78473.91", "take_profit": "83000"},
    "client_tracking_id": "tradingagents-hold-1",
    "tags": ["counterfactual", "part-12"],
    "metadata": {"verdict": "Hold", "rating": "Hold",
                 "level_roles": "assigned_by_harness"}
  }'

Three things are decided before you send anything.

① The counterfactual is "add now". That is the action the tool forbade, so that is the action you take.

② No short. This tool's medium-term read is "constructive medium-term trend". Below 78,473.91 is a risk-reduction signal, not a claim that price falls.

③ Be honest about the levels. The tool never called these two numbers a stop and a target — its record literally says Price Target: not provided. What I did was move the two levels it did name ("reason to reduce risk" at 78,473.91, the top of the resistance zone at 83,000) into exit_plan, and the role assignment is mine. The numbers themselves are verbatim; not one digit was changed.

A loose end: in the same file, the top of the resistance zone reads 82,253.84 in the summary and 82,252.84 in the body. That is an inconsistency inside the tool's own output, so I quoted it as it is rather than quietly normalising it.

Numbers in metadata go in as strings. Send a float like 0.88 and you get a 500 — the request body is hashed and the determinism rule rejects floats.

3.3 Pull it back — two calls

curl -sS "$WELTIQ_BASE_URL/v1/forward-tests?tracking=tradingagents-hold-1&limit=200" \
  -H "Authorization: Bearer $WELTIQ_API_KEY"

curl -sS "$WELTIQ_BASE_URL/v1/forward-tests/orders?tracking=tradingagents-hold-1&format=csv" \
  -H "Authorization: Bearer $WELTIQ_API_KEY"

The second line is the account-wide endpoint added on 2026-09-28. You no longer pick a test first. Server-generated close orders inherit the tracking ID and the tags, so seeding one entry still brings the exit row back — which is where the round-trip cost comes from.

When you join, do not use awk. The metadata and exit_plan columns contain quoted commas, and cutting on the field separator silently shifts your columns. The recipe joins with an inline python3 csv module and says why in a comment.

And the rating never reaches the server. The intent belongs to the tool; we hold the execution record. The only thing joining them is client_tracking_id.

Environment notice: the numbers below came off an isolated server with synthetic quotes. This is not production, and the P&L is not a market fact — what is being measured is the wiring and the scoring frame, not whether the tool was right.

4. What came back

201 → filled → closed on the 600-second deadline.

Neither of the two levels the tool named (78,473.91 and 83,000) was touched in the ten-minute window. That is what close_reason: close_after means.

4.1 The one number to read out of that table

Break-even 0.2224%. Had you added at that moment, BTC would have had to rise at least 0.2224% within ten minutes just to get your money back. It did not, so "do not add here" was right. Conditionally — "provided the price did not rise more than 0.2224% during those ten minutes."

But the value of this part is not the number. It is that a "stay in the position" rating turns out to be translatable into a counterfactual at all. Split a Hold into "add now" and "exit now" and a rating that is currently never evaluated becomes something you can measure.

4.2 The surprise — confidence could not have made a difference

In the same cycle I placed four waits from another tool the same way. Confidence 0.88, 0.90, 0.89 and 0.20 produced net P&Ls identical to the last decimal.

The reason is plain. They were placed at the same moment against the same quote. In a frozen market, P&L is decided by the round-trip cost, not by confidence.

This is not a failure. It is a condition the frame revealed as it was being built.

To measure any correlation between confidence and outcome, the counterfactual has to be placed at the moment and the price the call was made. Batch it up and replay it later, and the confidence disappears while the cost remains.

Which means the feature only pays off if the counterfactual order goes out automatically the instant the tool issues the call. Replaying it after the fact yields nothing.

Trial buy list — the counterfactual rounds on one screen

This is our screen. Each row is one counterfactual, and look at the status column on the right — a round that reached a fill and a round that bounced at placement are separated by a single word. That is how you decide which rows go into the scoring and which stay "unresolved".

5. What this combination cannot tell you

① One sample says nothing. This part's sample is one case. A wait that ends in a loss is not a wait that was right — you need a distribution.

② The counterfactual measures one side only. "What if I had bought" is measured. "What did I save by not buying" is still a judgment call. And this cycle only placed one of the two halves of the Hold; "exit now" requires an existing position.

③ We do not say the tool was right or wrong. We measure and record. Even the "right" above is a verdict against a stated criterion, and a conditional one.

④ This Hold did not reproduce. The next day, thirteen runs of the same configuration produced Hold 0/13 (Sell 7, Underweight 6). So what this counterfactual scored is not "this tool's Hold" but "that one run's Hold". Unless the model, the date and the inputs are pinned and recorded per cycle, what you are measuring gets blurry fast.

⑤ The role assignment is mine. As stated in 3.2, I put a tool's two named levels into stop and target slots on a record that says Price Target: not provided. The numbers are verbatim but the slots are my choice. A different assignment could produce a different result.

⑥ The horizon does not fit in the window. This tool's Time Horizon is 3–6 months; a trial order lives at most 168 hours. A ten-minute window measures "right now", not that horizon. Keeping those two apart in writing matters.

⑦ The exchange decides what sitting out is worth. This is the most practical finding of the cycle.

In crypto, a ten-minute wait is wrong if the price moves 0.22%. A 0.22% move in ten minutes is not rare in Bitcoin. US equities are the reverse: round-trip cost is near zero, so waiting there is not justified by cost at all, only by whether the pullback actually came.

⑧ Synthetic quotes. The P&L above is a model result off fixed quotes on an isolated server. What was confirmed is the wiring and the scoring frame.

6. When to use this and when not to

Use it for tools that issue a lot of Holds. This is the only part in the series that re-grades "stay in the position". Splitting a Hold into two counterfactuals turns an unevaluated rating into a measurable one, and the more Holds your tool issues, the more that is worth.

Use it when the call contains a specific prohibition. "Do not add at 81,526.82" names an action, so the counterfactual defines itself. "Proceed cautiously" cannot be translated — there is no action that was skipped.

Do not judge the tool on one run. Thirteen runs of the same configuration produced zero Holds. This tool's run-to-run variance is large, and the counterfactual inherits all of it.

Do not use it to score the tool's horizon. Cut a 3–6 month judgment down to 168 hours and what you are measuring is no longer that tool's judgment. Rounding adjusts a value; truncation changes the judgment.

Do not use it if position weight is the output. This tool mostly issues allocation instructions ("50%–75% of a normal allocation"). The server decides the quantity here, so that weight disappears. Direction and timing are what survive.

Evidence for this article: the counterfactual measurement record · per-case request and response text · this part's recipe · the tool's own run log. Every number above came out of those files, including the inconvenient Hold 0/13.