orallexa — what the 50.8% confidence on its dashboard actually means, traced to the bottom

One of the very few trading repos that ships Platt calibration, a Brier audit and bias tracking. Run 16 tickers through it and confidence takes exactly three va

Done when: you can see for yourself how the confidence number on this tool's screen is built, and carry that number into a weltiq slot until it actually fills.

Raw logs and source data: reports/a/34-orallexa/

The human lines below are real commands rewritten as conversation. The outputs are measured and the numbers are untouched.

1. What this is

alex-jb/orallexa-ai-trading-agent scores one ticker on technical indicators and layers a Claude judgement over that score.

There is a reason this tool gets its own part. Of everything covered in this series so far, it carries the most machinery for measuring confidence: Platt calibration (engine/platt_calibration.py), a Brier audit with a reliability diagram (markets/auto/brier_audit.py), bias tracking (engine/bias_tracker.py), per-source accuracy (engine/source_accuracy.py), a decision log (engine/decision_log.py). Papers are cited in the docstrings.

So this part asks one question. What does the "50.8% confidence" on the screen mean?

orallexa's own UI, running MSFT on SWING · 1D with CLAUDE AI OVERLAY switched on. Look at the three cells across the bottom — SIGNAL Strong 79/100, **CONFIDENCE Low 48%**, RISK Low. Directly above them sits **73% UPSIDE PROBABILITY** in large type. Two numbers that both read as "how sure is it", on one screen, **25 points apart.** The BREAKING line at the top reports that the figure moved +18% from the previous run (55%)

How the tool is wired

scripts/run_daily_pilot.py          ← autonomous entry point (launchd, 09:35 ET daily)
    ↓
core/brain.run_prediction           ← use_debate defaults to False
    ↓
skills/prediction.py                technical score → one Claude call
    ↓  confidence = scale_confidence(raw LLM value)
engine/decision_log.py              appends to memory_data/decision_log.json
    ↓
bot/alpaca_executor.py              confidence gate → Alpaca order

There are three more entry points beside it — app.py (Streamlit), api_server.py (FastAPI, which the UI above talks to) and core/loop.py. The same ticker gets a different path and a different answer. This piece pins everything to run_daily_pilot.py.

2. Installing it

Python 3.11 or later. I ran 3.11.16.

git clone https://github.com/alex-jb/orallexa-ai-trading-agent.git
cd orallexa-ai-trading-agent
python3.11 -m venv .venv
.venv/bin/pip install -r requirements.txt     # 163 seconds

Fill in .env like this. One Anthropic key is enough for judgements; the Alpaca keys are only needed to place orders.

ANTHROPIC_API_KEY=...
ALPACA_API_KEY=...          # paper (free)
ALPACA_SECRET_KEY=...

I used OpenRouter instead of calling Anthropic directly. Leave the SDK alone and add ANTHROPIC_BASE_URL=https://openrouter.ai/api; it goes through.

2.1 Where it stops — the autonomous entry point never reads .env

The README tells you to put your key in .env. But the only files that call load_dotenv are api_server.py and desktop_agent/main.py. Run scripts/run_daily_pilot.py directly and the key is never picked up.

And the failure is silent. I ran my first pass without knowing this, got a 401 on every call, and the output still looked entirely normal — WAIT · confidence 32.8% · "Claude overlay applied". That "Claude overlay applied" line gets stamped on even when the model returned a 401. On failure, llm/ui_analysis.py:229 quietly returns a default (WAIT·40%), and the layer above treats it as a normal response. Nothing gives it away except that the run finishes in half a second.

The workaround is one line.

set -a; source ./.env; set +a          # export first
PYTHONPATH=. .venv/bin/python scripts/run_daily_pilot.py --tickers NVDA --dry-run

How to tell whether you are stuck: a judgement that takes about 4 seconds means the model ran; 0.5 seconds means it did not. The clock is the honest indicator here.

2.2 The buried one — install it fresh today and two thirds of the LLM paths are dead

requirements.txt:2 is the bare line anthropic, with no version. Install today and you get 1.8.0, whose Messages.create() has no temperature argument.

Two calls, one with CLAUDE AI OVERLAY on and one with it off, come back **byte for byte identical.** The server log carries Claude overlay failed: Messages.create() got an unexpected keyword argument 'temperature' every time, while the response is a 200 OK with normal-looking values. 22 of the 33 call sites pass temperature

Which 22 matters — llm/debate.py (Bull/Bear/Judge), llm/debate_graph.py, engine/multi_agent_analysis.py, engine/daily_intel.py, engine/scenario_sim.py, llm/perspective_panel.py. The repository's own description — "8-source signal fusion, Bull/Bear/Judge debate on Claude Opus 4.7" — is entirely inside that list.

The 11 that still work include prediction_decision_report, which the autonomous entry point uses. That is why the pilot runs. The reason 83 test files and 18,112 lines of tests miss this is that every one of them swaps the client for a mock — nobody checks the real SDK's signature. The fix is a single line (pin an upper bound on the dependency), but as shipped it does not run.

3. Running it once

3.1 First judgement

You say this

Take a look at NVDA on the swing timeframe. Don't place anything.

And this comes back

PILOT_START tickers=['NVDA'] confidence_gate=60.0% dry_run=True
HTTP Request: POST https://openrouter.ai/api/v1/messages "HTTP/1.1 200 OK"
DECISION NVDA decision=WAIT confidence=42.6% risk=MEDIUM elapsed=5.0s
PILOT_DONE trades=0 decisions=1

set -a; source ./.env; set +a
PYTHONPATH=. .venv/bin/python scripts/run_daily_pilot.py --tickers NVDA --dry-run

That output means it worked. A 200 OK together with 4–5 seconds means the model really answered.

3.2 Bringing up the dashboard

This tool's dashboard is two processes, backend and frontend.

# Terminal 1 — API
set -a; source ./.env; set +a
.venv/bin/python api_server.py            # → http://localhost:8002

# Terminal 2 — UI
cd orallexa-ui && npm install && npm run dev   # → http://localhost:3000

1. Addresses: the UI is http://localhost:3000, the API http://localhost:8002. The README does not give the port — it is at api_server.py:1578. Edit that line to change it
2. There is no login. It opens without an account
3. On first open you get a ticker box on the left and STANDBY in the middle. Enter a ticker and press RUN SIGNAL to fill the screen. There is also a demo mode you can click through with no keys
4. Where it trips: the README's copy-paste block has a disclaimer wedged into the middle of it, so pasting it verbatim breaks. npm install took 6 seconds and Next booted in 129ms

3.3 Four places beginners get stuck

4. Getting serious — the scenarios

4.1 Scenario D. Is "80% confident" really 80%? (= S4)

The code says confidence is capped at 82% (models/confidence.py:23). So I ran 16 tickers once each and captured both the number that reaches the screen and the raw value the model actually wrote.

Across all 16 tickers the model wrote exactly **three raw confidence values — 52, 58 and 62.** The displayed values are three as well: 42.6%, 47.6%, 50.8%. Highest minus lowest is **8.2 points**. Nothing came anywhere near the 82% ceiling. The p_up figure that ships with the same judgement, meanwhile, spreads from **0.15 to 0.57**

Three lines

1. Confidence took three values across 16 tickers and spanned 8.2 points. Running the same ticker (NVDA) five times gave a raw value of 52.0 on 5 of 5
2. Displayed = raw × 0.82. Despite its name and its docstring ("capped at 82%"), scale_confidence() multiplies every value by 0.82 rather than capping it. 52 → 42.6, 62 → 50.8
3. 80% is structurally unreachable. The highest raw value the model wrote, 62, reaches the screen as 50.8%

So the 50.8% on the screen is not a probability. It is exactly the pathology this tool's own docstrings cite — Prophet Arena (arxiv 2510.17638) on LLMs compressing into the middle band — with one more multiplication by 0.82 laid on top.

It goes one step further. On paths where no LLM call is attached, the confidence is a formula outright: |technical score − 50| × 2 × 0.82. The 48% in the section 1 screenshot is precisely that (technical score 79 → |79−50|×2 = 58 → 58 × 0.82 = 47.6). That screen had CLAUDE AI OVERLAY switched on (§2.2). The 73% UPSIDE on the same card is a separate number from the probability model.

4.2 Does the Brier log fill up? (Scenario D-2)

This tool ships an audit that grades its own confidence. markets/auto/brier_audit.py reads the decision log and produces a Brier score, a reliability diagram and an ECE, and it computes how much better things would have been under Platt calibration. I ran it.

As shipped it reports loaded 0 historical decisions — the decision-log path is **hardcoded to the author's own Desktop** (brier_audit.py:189-190). The report output path is too, so a ~/Desktop/Interview-Prep/Projects/alex-brain/… folder appeared on my Mac. Point the two constants at the real log and it reads 20 rows, of which exactly **one** can be scored

Three lines

1. The log does fill. The pilot writes one row per judgement into decision_log.json. It reached 20 rows today
2. But only one row was scoreable. The audit skips WAIT (correctly — there is no directional claim), and 11 of the 20 rows are WAIT. Of the remaining nine, one was more than a day old, and its Brier was 0.2704 (forecast_p 0.52, actual −2.26% → actual 0)
3. Platt never ran. It needs at least 30 observations and there was 1 — _Skipped: Need at least 30 observations, got 1._

And the part that matters most: the number the audit grades is not confidence but probabilities.up (brier_audit.py:301). The 50.8% confidence on the screen never enters this audit at all. The measuring instrument is there; it does not measure the number people look at.

Platt is not wired into the stock path. The only places that import engine/platt_calibration.py are its own tests and this audit, and inside the audit it is a single "what would it have been" paragraph. The author wrote it into that function's docstring himself — "before flipping the live wire-up flag on polymarket_daily.py / stock decision path". So: not yet.

The log also misses its own purpose. The docstring of run_daily_pilot.py says the script exists to accumulate "100 eligible production debate rows". But the pilot calls with use_debate at its default False. Zero of the 20 rows carry a debate, and entry_price is 0.0 on all 20.

4.3 Scenarios S5 and S7. The autonomous path never passes a stop

The bracket-order code (entry plus stop plus target as one unit) is genuinely there. alpaca_executor.py:176-186 builds an OrderClass.BRACKET. One condition guards it — if stop_loss > 0 and take_profit > 0.

So I intercepted the arguments that actually go to Alpaca and put the two paths side by side.

The same execute_signal() on the same account, and **the autonomous path produces order_class: null · stop_loss: null** while the path where a human fills in levels produces order_class: "bracket". Two arguments are the only difference. run_daily_pilot.py:134-139 does not pass them

I confirmed it with a real order too. I dropped the gate to 45 and sent one MSFT order.

Order submitted: BUY 9 MSFT @ $500.40 (conf=51%, stop=$0.00, target=$0.00)

The tool's own log says stop=$0.00. Alpaca's record of the order agreed: order_class: "simple", legs: 0 (order 69e14508-…, cancelled once checked).

Three lines

1. Brackets are for the human path only. They never switch on in the path you would leave running around the clock
2. Nothing watches the position after entry. There is no stop trailing, no partial exit and no re-evaluation loop anywhere in the code; close_position is an HTTP endpoint a person presses
3. On default settings no order goes out at all. The pilot's default gate is 60% and the highest confidence this tool produced was 50.8% — SKIP MSFT — confidence 50.8% below gate 60.0%. Multiplying by 0.82 has quietly changed what the gate means (a displayed 60% is a raw 73.2)

A correction to the part 33 teaser. At the end of the previous part I called this tool "the one with the best-equipped machinery for feeding confidence into size". The machinery part is right; the feeding-into-size part is wrong. Confidence is used only as a gate (skip below 40, skip below --confidence). Size comes from the constant DEFAULT_POSITION_PCT = 5.0 on the autonomous path and from risk_level (LOW/MEDIUM/HIGH) on the human path. There is no place in any path where confidence shrinks the quantity.

4.4 Scenario S5-b. A failed order is counted as a success

I dropped the gate to 45 and sent a BTC-USD order.

Order failed for BTC-USD: {"message":"code=400, message=invalid symbol: BTC-USD"}
TRADE BTC-USD decision=BUY result=?
PILOT_DONE trades=1 decisions=1

The broker rejected it with a 400, and the daily summary says trades: 1 with status "traded" (run_daily_pilot.py:141 writes traded without looking at what execute_signal returned). Anyone running this under launchd and checking with grep TRADE will count orders that never left as orders that did.

4.5 The dashboard says "not connected" while the same keys place orders

/api/alpaca/account returns null while /api/alpaca/positions reads the real position correctly. One line causes it — alpaca_executor.py:86 calls int(account.daytrade_count) and Alpaca hands back None. The dashboard then shows "NOT CONNECTED — Set ALPACA_API_KEY … in .env". **The keys are already there, and the same process places orders**

4.6 Scenarios I did not run

• S1 and S2 (fetching data): data is yfinance and nothing else, so it is not the point here
• S3 (reviewing my own record): §4.2 covered it using this tool's decision log
• S8 (disconnect and reconnect): not applicable — there is no resident loop, just a script that wakes once a day
• S9 and S10: not done in this part

5. Connecting it to weltiq

5-1. What moved across, and what I had to fill in

This is everything the tool produced (BTC-USD, autonomous path, measured).

Two blanks. But the code that fills them is inside this repository — skills/risk_management.py. It is only the autonomous entry point that never calls it. So I put the tool's own judgement into the tool's own module and took the levels and the size from there. I invented no numbers.

Something else shows up here. What sets the levels and the size is risk_level (MEDIUM), and confidence never enters the calculation. 50.8% or 82%, the size comes out the same.

The only thing I took from our side is the price reference. The tool prices against yfinance BTC-USD (84138.4609375) and we fill on binance-spot BTCUSDT (84120.005). Carrying the absolute numbers across would carry the basis between two markets with them, so I applied the tool's percentages to our quote → stop 82015.05, target 88323.91.

POST /v1/strategy-rules   → doc_hash 128ed65d…
POST /v1/ai/thesis        → {doc_hash, text}   # rules without a rationale are refused
POST /v1/containers       → {source:"ai", doc_hash, venue:"binance-spot",
                             symbols:["BTCUSDT"], tf:"1m", balance:"10000",
                             stop_max_loss:"150", stop_after_hours:2}

5-2. It filled, and a stop went on with it (= S5 · S6)

Slot detail. Look at the ORDERS block — buy market 0.02 is filled, and **above it sits sell (close) stop_market 0.02 in state acked**. That is exactly what the tool failed to attach in §4.3. The FILLS block shows the fill at 84094.41860050 and a fee of 1.68188837 USDT

The fill came 36 seconds after the slot was created, and I stopped the slot and closed the position two minutes later.

Realised P&L −0.7665, total fees 3.36301024, balance 10,000 → 9995.87048874 (−0.0413%). Ledger reconciliation came back matched: true with zero orphan intents.

The P&L of one round trip means nothing. What to look at is not the number but the fact that the blanks got filled.

5-3. What this combination does and does not do

The best way to use it is this. Run the tool with --dry-run only, treat it as a judgement generator, and hand the levels, the sizing, the stop watching and the crypto symbol mapping to the slot. There is no reason to enable the Alpaca path until §4.3 and §4.4 are fixed.

What this combination does not do: neither side feeds confidence into size. And it does not make the judgement any more likely to be right.

The thread that runs through the execution stages held again. The rule document's doc_hash carries the same body through backtest and paper. The live stage is not open yet — "the same document goes straight through" is true up to that point and the rest is planned, with the promotion ladder as the gate.

Cleanup record: the slot was stopped and the position liquidated (status: stopped, positions: null). Deleting it did not work — DELETE returns a 409 positions_open even with the position flat. That is a known defect on our side, so I left it rather than forcing it. The tool's Alpaca order was cancelled and the account holds no position created by this piece.

6. Having used it

What was good

• The measurement machinery is the thickest in the field. Brier, reliability diagram, ECE, Platt calibration and bias tracking are all real code, with the papers cited. This part pulled them apart because they exist, not because they don't — most tools have nothing to pull apart
• The audit report says unflattering things about itself. An ECE above 0.20 is described in its own code as "system is telling you it doesn't know what it's doing"
• Every judgement carries a sentence of rationale. The one quoted in section 5 is it
• The UI is good. The upside-probability bar, the change-since-last-run alert and the backtest table all sit on one screen
• It is MIT, and Alpaca paper is hardcoded, so no live-account URL exists in the code

What was not

• Confidence is effectively three values. 8.2 points of spread across 16 tickers. And since the displayed value is raw × 0.82, 80% is structurally unreachable
• The calibration machinery is not wired into the stock path. Platt is a "what if" paragraph, and the Brier audit grades probabilities.up, not confidence
• The audit paths are hardcoded to the author's Desktop, so on anyone else's machine it reads zero rows and instead creates folders on their desktop
• The autonomous path never passes a stop (§4.3). And it counts failed orders as successes (§4.4)
• The README's headline features do not run on a fresh install today — one unpinned dependency kills 22 of the 33 LLM call sites (§2.2)
• There are four entry points and they disagree. The same MSFT came out at 50.8% on the CLI and 48% in the UI
• No push in two and a half months

Who it fits and who it doesn't

Fits — anyone who wants to see how instruments like confidence and calibration get written in code. platt_calibration.py and brier_audit.py are well-written; they are just not connected. And anyone who takes the judgement and wires their own execution under it.

Doesn't — anyone who wants the README's 8-source fusion, debate or multi-agent paths (§2.2). Anyone planning to trust the tool with their stop (§4.3). Anyone trading crypto (there is no order path).

7. Next

Next is achaljhawar/1rok. If this part was about "two confidence numbers on one screen, 25 points apart", the next one is about a claim that is half true — "the same tool surface" for every model — where the prompt forks three ways depending on how well the model does structured output. We saw four entry points give different answers for one ticker here, and that is the next question along.