Scoring AlpacaTradingAgent's `HOLD` — the call that places no order but still issues every level

This tool writes order_type: none and hands you an absolute stop and target anyway. Here is the procedure for building a counterfactual out of it with zero conv

Done when, faced with a call that says "no order" while handing you a full set of levels, you can build a counterfactual out of it without altering a single digit — filling in only the side.

1. The blank this tool kept leaving

AlpacaTradingAgent (github.com/huygiatrng/AlpacaTradingAgent · Apache-2.0 · 271 stars) runs thirteen agents over a symbol, produces BUY/HOLD/SELL along with stop and target prices, and sends the result to an Alpaca paper account.

Here is one call I got (NVDA, the tool's output, final_trade_intent).

action: HOLD    target_position: NEUTRAL    order_intent.order_type: none
advisory_rating: Hold    confidence: medium
risk_controls.stop_loss_price: 220.92    risk_controls.take_profit_price: 234.75

BTC/USD has the same shape.

action: HOLD    target_position: NEUTRAL    order_type: none
risk_controls.stop_loss_price: 84800.0    take_profit_price: 88500.0

And on entry it wrote (the tool's output):

"No immediate entry. Consider a small staged entry only after a confirmed 4-hour reclaim and retest above $86,000, or a validated pivot hold above $84,940."

This is the cleanest-shaped call in the series. The tool says "no order" (order_type: none) and still issues a stop and a target as absolute prices. But since no order goes out, nothing is ever scored — nobody finds out whether that level arrangement survives real costs.

Market analyst report — entry, stop and target laid out in a table

This is the tool's signature screen. Look at the Summary Table at the end of the report — entry, target, stop, R:R and current action are all numbers or explicit conditions. If those cells are full of numbers, it worked — this is where this tool separates from tools that hand you one word. And Invalidation just above it states the condition under which it would be wrong.

I ran three independent passes and the levels were remarkably stable — the stop on all three NVDA runs was 220.92, identical.

Its refusals are numeric, too. nvda-3 did not say "this feels wrong"; it said (the tool's output):

"the proposed pullback setup offers only about 1.7:1 reward-to-risk, below the required minimum."

2. Why this has been empty until now

Scorecards have only ever existed for things you bought. order_type: none means no order, no fill, no P&L, nothing to grade.

But this HOLD is not a blank. With a stop at 84,800 and a target at 88,500 it is a complete trade plan, and a testable claim: right now, staying out beats going in.

Here is why it stayed empty. To test that claim, you have to actually buy — and the moment you do, your account moves. Capital gets tied up, the loss limit gets eaten into, the performance rollup gets contaminated. The act of measuring destroys the thing being measured. So nobody measured it.

A trial order steps around that. It carries the judgment through to the close in a separate ledger that touches none of your balance, P&L, limits, or performance totals.

The decision not to buy gets a scorecard too. Without touching your account.

This is not a claim that nobody else could do it. A backtest could approximate it in principle. But a backtest is a replay of the past, so it cannot catch the moment and the quote the call was actually made at — and a live account brings back the destruction above.

3. The recipe — the thing that actually runs

3.1 First, get a call out of the tool

git clone https://github.com/huygiatrng/AlpacaTradingAgent.git && cd AlpacaTradingAgent
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt          # 204 packages, about 770MB
cp env.sample .env

Five lines in .env are enough — two Alpaca paper keys, ALPACA_USE_PAPER=True, an LLM provider and an LLM key. Finnhub, FRED, CoinDesk and Reddit keys are optional.

This one does have a web UI.

Thirteen agents finishing in order

The Analysis Status table is the live view. When all thirteen rows go green, one pass is done. Look at Tool Calls: 26 · LLM Calls: 13 · Generated Reports: 13 under the table — if tool calls read 0, the judgment was produced without data, and you can tell at a glance.

The dashboard just after a fill — guardrails and the Alpaca paper account

Safety Guardrails at the top is the non-LLM layer — kill switch, order size cap, concentration limit, daily loss, drawdown, rejection streak. Look at the second row of Recent Orders: NVDA · canceled — that is what an equity order does when it is placed while the US market is shut. We hit the same wall in section 4.

Auto-execution ships switched off (trade_enabled: False). If you only want the call, leave it that way.

3.2 Take the levels verbatim and fill in the side

The full recipe is in 06-trial-order-alpaca-agent-hold.sh. Here is its heart.

curl -sS -X POST "$WELTIQ_BASE_URL/v1/trial-buys" \
  -H "Authorization: Bearer $WELTIQ_API_KEY" -H 'Content-Type: application/json' \
  -d '{
    "venue": "binance-spot", "symbol": "BTCUSDT",
    "client_order_id": "alpaca-agent-hold-1-btc",
    "side": "long",
    "exit_plan": {"close_after_seconds": 600,
                  "stop_loss": "84800", "take_profit": "88500"},
    "client_tracking_id": "alpaca-agent-hold-1",
    "tags": ["counterfactual", "part-31"],
    "metadata": {"action": "HOLD", "target_position": "NEUTRAL",
                 "order_type": "none", "levels": "verbatim_from_tool"}
  }'

There is not one conversion in there. 84800 and 88500 are the numbers in risk_controls; the only field I supplied is side.

Why long: the level arrangement is already a long — stop below the current price, target above (84,800 < 84,940 < 88,500 / 220.92 < 227.30 < 234.75). Flip it to a short and the tool's own arrangement breaks. There is no sentence implying a decline either — "only about 1.7:1 reward-to-risk" says the reward is small, not that price falls. So I placed no short.

A note on ticks. 84800, 88500, 220.92 and 234.75 are all multiples of the 0.01 tick and pass as they are. That is because this tool issues integers and two decimals, not because I tidied anything. Tools that issue floats are different — elsewhere in the same cycle stop 82034.9994 came back 422, and only went through as 201 once rounded to 82035.00. The server does not round for you. When you must round, round conservatively — on a long, round the stop up and the target down. Both move the condition closer.

Numbers in metadata go in as strings. Floats return 500 — the request body is hashed and the determinism rule rejects them.

3.3 Pull it back — two calls

curl -sS "$WELTIQ_BASE_URL/v1/forward-tests?tracking=alpaca-agent-hold-1&limit=200" \
  -H "Authorization: Bearer $WELTIQ_API_KEY"

curl -sS "$WELTIQ_BASE_URL/v1/forward-tests/orders?tracking=alpaca-agent-hold-1&format=csv" \
  -H "Authorization: Bearer $WELTIQ_API_KEY"

Two venues, still one call. The account-wide endpoint added on 2026-09-28 does not make you pick a test first — the old path was "one list call plus one export per test", 1,001 calls in a month where you used the whole quota.

Server-generated close orders inherit the tracking ID and the tags, so seeding only entries still brings the exit rows back. That is where the round-trip cost comes from.

When you join, do not use awk. The metadata and exit_plan columns contain quoted commas, and cutting on the field separator silently shifts your columns. The recipe joins with an inline python3 csv module and says why in a comment.

The exported trade record — where the tool's log joins on

This is our screen. Look at the two columns client_tracking_id and client_order_id — the verdict and the confidence are not here, they are in the tool's own log. Those two columns are the only join, which is exactly why the intent stays with the tool and we hold only the execution record.

Environment notice: the numbers below came off an isolated server with synthetic quotes. This is not production, and the P&L is not a market fact — what is being measured is the wiring and the scoring frame, not whether the tool was right.

4. What came back

The tool's stop at 84,800 and target at 88,500 were never touched in the ten-minute window (against the fixed 84,940 quote that is −0.165% to the stop and +4.2% to the target). That is what close_reason: close_after means.

The NVDA side was accepted and bounced at placement because the US market was shut. Unresolved. I did not fake the clock — to measure it again, run the same recipe between 22:30 and 05:00 KST.

4.1 The one number to read out of that table

Break-even 0.2223%. Had you gone in at that moment, BTC would have had to rise at least 0.2223% within ten minutes just to get your money back. It did not, so that HOLD was right. Conditionally — "provided the price did not rise more than 0.2223% during those ten minutes."

Turned around, that is the call's threshold. For this HOLD to have been wrong, BTC needed to rise at least 0.2223% in ten minutes.

4.2 The surprise — confidence could not have made a difference

In the same cycle I placed four waits from another tool the same way. Confidence 0.88, 0.90, 0.89 and 0.20 produced net P&Ls identical to the last decimal, and this part's BTC figure lines up with those four digit for digit.

The reason is plain. They were placed at the same moment against the same quote. In a frozen market, P&L is decided by the round-trip cost, not by confidence. And this tool's confidence is only high/medium/low — all six runs came back medium, so there was no variation to correlate against in the first place.

This is not a failure. It is a condition the frame revealed as it was being built.

To measure any correlation between confidence and outcome, the counterfactual has to be placed at the moment and the price the call was made. Batch it up and replay it later, and the confidence disappears while the cost remains.

Which means the feature only pays off if the counterfactual order goes out automatically the instant the tool issues the call. Replaying it after the fact yields nothing.

5. What this combination cannot tell you

① One sample says nothing. What got measured here is one case. A wait that ends in a loss is not a wait that was right — you need a distribution.

② The counterfactual measures one side only. "What if I had bought" is measured. "What did I save by not buying" is still a judgment call.

③ We do not say the tool was right or wrong. We measure and record. Even the "right" above is a verdict against a stated criterion, and a conditional one.

④ Half of it is unresolved. NVDA never filled because the market was closed. This part's conclusion rests on a single crypto case.

⑤ The entry condition is not in the execution contract. This tool writes "only enter conditionally" as free text — "only after a confirmed 4-hour reclaim and retest above $86,000" lives in a sentence, not in a structured field. The counterfactual ignores that condition and measures going in now. That is what a counterfactual is, but it has to be said plainly: what is being measured is not "that trade once the condition was met".

⑥ The horizon does not fit in the window. This tool's Time Horizon is 2–10 trading days. A ten-minute window measures "right now", not that horizon.

⑦ The exchange decides what sitting out is worth. This is the most practical finding of the cycle, and it is sharpest here — the same HOLD from the same tool is worth different amounts in two markets.

In crypto, a ten-minute wait is wrong if the price moves 0.22%. A 0.22% move in ten minutes is not rare in Bitcoin — this HOLD is defended by a very thin band of cost. US equities are the reverse: round-trip cost is near zero, so waiting there is not justified by cost at all, only by whether the pullback actually came.

⑧ Synthetic quotes. The P&L above is a model result off fixed quotes on an isolated server. What was confirmed is the wiring and the scoring frame.

6. When to use this and when not to

Use it for tools that issue levels as absolute prices. That is why this part was the cleanest of the six. When "no order" and "a complete set of levels" sit in the same object, filling in the side finishes the counterfactual. No conversion means you can say afterwards, truthfully, that you never touched the numbers.

Use the level arrangement as your evidence for the side. Stop below the current price and target above it is already a long. action: HOLD does not mean there is no direction in the record.

Do not attach a short to a call that never implied a decline. "The reward is small" and "it goes down" are different statements. What the original text claimed is what fixes the side.

Do not run it while the market is closed. That is why the NVDA case here is unresolved. Crypto is 24/7, but place US equity counterfactuals between 22:30 and 05:00 KST.

Do not use it to draw a calibration curve. This tool's confidence has three values and all six runs came back medium. An axis with no variation cannot carry a correlation. If confidence is what you want to score, pick a tool that emits a continuous value — and obey section 4.2 and place it at the moment of the call.

Evidence for this article: the counterfactual measurement record · per-case request and response text · this part's recipe · the tool's own run log. Every number above came out of those files, including the NVDA case that stayed unresolved.