🟢 piker/brokers/ib/_util.py - New utility script added
🛠️ vnc_click_hack -> Refactored to initialize VNC client outside context
🛠️ client connection -> Simplified by removing crash handler context
🛠️ moved client movement -> Now inside async with block for proper cleanup
🛠️ added comments -> Clarified screen position and hotkey actions
Implement workaround for macOS POSIX shm 31-character name limit by
hashing long keys. Add comprehensive documentation for macOS-specific
compatibility fixes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
For the new github image, a high-level look at its basic
features/usage/docs and prosing around our expected default usage with
the `piker.brokers.ib` backend.
Lol, a cheeky unforeseen bug due to TOML's lack of a null type and
thinking i can render an `Optional` field on a `msgspec.Struct`
(defaulted to `None`) the `binance.symcache.toml` cache file..
I didn't catch this when i first updated to the 3.1 API in f7caa75228
because i never did a cache-files flush.. lesson learned and we **really
need tests for this**!!
To start this is just a shell for the test, there's no checking logic
yet.. put it as `test_accounting.test_ib_account_with_duplicated_mktids()`.
The test is composed for now to be completely runtime-free using only
the offline txn-ledger / symcache / account loading APIs, ideally we
fill in the activated symbology-data-runtime cases once we figure a sane
way to handle incremental symcache updates for backends like IB..
To actually fill the test out with real checks we still need to,
- extract the problem account file from my ib.algopape into the test
harness data.
- pick some contracts with multiple fqmes despite a single bs_mktid and
ensure they're aggregated as a single `Position` as well as,
* ideally de-duplicating txns from the account file section for the
mkt..
* warning appropriately about greater-then-one fqme for the bs_mktid
and providing a way for the ledger re-writing to choose the
appropriate `<venue>` as the "primary" when the
data-symbology-runtime is up and possibly use it to incrementally
update the IB symcache and store offline for next use?
Since under `trio`-cancellation the `.remove()` is a checkpoint and will
be masked by a taskc AND we **always want to remove the rect** despite
the surrounding teardown conditions.
In the `translate_and_relay_brokerd_events()` loop task that is, such
that we never crash on a `status_msg = book._active.pop(oid)` in the
'closed' status handler whenever a double removal happens.
Turns out there were unforeseen races here when a benign backend error
would cause an order-mode dialog to be cancelled (incorrectly) and then
a UI side `.on_cancel()` would trigger too-early removal from the
`book._active` table despite the backend sending an actual 'closed'
event (much) later, this would crash on the now missing entry..
So instead we now,
- obviously use `book._active.pop(oid, None)`
- emit a `log.warning()` (not info lol) on a null-read and with a less
"one-line-y" message explaining the double removal and maybe *why*.
For backends which opt to set the new `BrokerdPosition.bs_mktid` field,
give (matching logic) priority to it such that even if the `.symbol`
field doesn't match the mkt currently focussed on chart, it will
always match on a provider's own internal asset-mapping-id. The original
fallback logic for `.fqme` matching is left as is.
As an example with IB, a qqq.nasdaq.ib txn may have been filled on
a non-primary venue as qqq.directedea.ib, in this case if the mkt is
displayed and focused on chart we want the **entire position info** to
be overlayed by the `OrderMode` UX without discrepancy.
Other refinements,
- improve logging and add a detailed edge-case-comment around the
`.on_fill()` handler to clarify where if a benign 'error' msg is
relayed from a backend it will cause the UI to operate as though the
order **was not-cleared/cancelled** since the `.on_cancel()` handler
will have likely been called just before, popping the `.dialogs`
entry. Return `bool` to indicate whether the UI removed-lines
/ added-fill-arrows.
- inverse the `return` branching logic in `.on_cancel()` to reduce
indent.
- add a very loud `log.error()` in `Status(resp='error')` case-block
ensuring the console yells about the order being cancelled, also
a todo for the weird msg-field recursion nonsense..
Such that backends can deliver their own internal unique
`MktPair.bs_mktid` when they can't seem to get it right via the
`.fqme: str` export.. (COUGH ib, you piece of sh#$).
Also add todo for possibly replacing the msg with a `Position.summary()`
"snapshot" as a better and more rigorously generated wire-ready msg.
Despite a `.bs_mktid` ideally being a bijection with `MktPair.fqme`
values, apparently some backends (cough IB) will switch the .<venue>`
part in txn records resulting in multiple account-conf-file sections for
the same dst asset. Obviously that means we can't allocate new
`Position` entries keyed by that `bs_mktid`, instead be sure to **update
them instead**!
Deats,
- add case logic to avoid pp overwrites using a `pp_objs.get()` check.
- warn on duplicated pos entries whenever the current account-file
entry's `mkt` doesn't match the pre-existing position's.
- mk `Position.add_clear()` return a `bool` indicating if the record was
newly added, warn when it was already existing/added prior.
Also,
- drop the already deprecated `open_pps()`, also from sub-pkg exports.
- draft TODO for `Position.summary()` idea as a replacement for
`BrokerdPosition`-msgs.
It actually works for vncAuth(2) (thank god!) which the previous
`asyncvnc` **did not**, and seems to be mostly based on the work
from the `asyncvnc` author anyway (so all my past efforts don't seem to
have been in vain XD).
Deats,
- switch to `pyvnc` async API (using `asyncio` again obvi) in
`.ib._util._vnc_click_hack()`.
- add `pyvnc` as src installed dep from GH.
- drop `asyncvnc` as dep.
Other,
- update `pytest` version range to avoid weird auto-load plugin exposed
by `xonsh`?
- add a `tool.pytest.ini_options` to project file with vars to,
- disable that^ `xonsh` plug using `addopts = '-p no:xonsh'`.
- set a `testpaths` to avoid running anything but that subdir.
- try out the `'progress'` style console output (does it work?).
Buncha updates and improvements,
- adjust sub-namespace imports according to console warnings.
- iterate all detected screens in a loop and instead report which is the
primary and the current.
- type annotate all vars where non-obvious, particularly the`Qt` refs.
Such that if/when the `push()` ticker callback (closure) errors
internally, we actually eventually bubble the error out-and-up from the
`asyncio.Task` and from there out the `.to_asyncio.open_channel_from()` to
the parent `trio.Task`..
It ended up being much more subtle to solve then i would have liked
thanks to,
- whatever `Ticker.updateEvent.connect()` does behind the scenes in
terms of (clearly) swallowing with only log reporting any exc raised
in the registered callback (in our case `push()`),
- `asyncio.Task.set_excepion()` never working and instead needing to
resort to `Task.cancel()`, catching `CancelledError` and re-raising
the stashed `maybe_exc` from `push()` when set..
Further this ports `.to_asyncio.open_channel_from()` usage to use
the new `chan: tractor.to_asyncio.LinkedTaskChannel` fn-sig API, namely
for `_setup_quote_stream()` task. Requires the latest `tractor` updates
to the inter-eventloop-chan iface providing a `.set_nowait()` and
`.get()` for the `asyncio`-side.
Impl deats within `_setup_quote_stream()`,
- implement `push()` error-bubbling by adding a `maybe_exc` which can be
set by that callback itself or by its registering task; when set it is
both,
* reported on by the `teardown()` cb,
* re-raised by the terminated (via `.cancel()`) `asyncio.Task` after
woken from its sleep, aka "cancelled" (since that's apparently one
of the only options.. see big rant further todo comments).
- add explicit error-tolerance-tuning via a `handler_tries: int` counter
and `tries_before_raise: int` limit such that we only bubble
a `push()` raised exc once enough tries have consecutively failed.
- as mentioned, use the new `chan` fn-sig support and thus the new
method API for `asyncio` -> `trio` comms.
- a big TODO XXX around the need to use a better sys for terminating
`asyncio.Task`s whether it's by delegating to some `.to_asyncio`
internals after a factor-out OR by potentially going full bore `anyio`
throughout `.to_asyncio`'s impl in general..
- mk `teardown()` use appropriate `log.<level>()`s based on outcome.
Surroundingly,
- add a ton of doc-strings to mod fns previously missing them.
- improved / added-new comments to `wait_on_data_reset()` internals and
anything changed per ^above.
Using `tractor.trionics.collapse_eg()` as needed and doing
some renames, in similar style as elsewhere:
- `pcs` -> `rent_cs`,
- `n` -> `tn` for nursery handles,
Also,
- tweak the `._reconnect_forever()` while loop to use the
(also) `trio`-internal
`mc_state: trio._channel.MemoryChannelState = snd._state` instead
of `snd._close` to poll for open send/receive consumer task counts
since,
1. it seems more reliable then using the `snd._closed`,
2. there's no other way to access the info.. afaik?
- handle `ConnectionRejected` explicitly alongside handshake-errs as
a retry case.
- add a base-exc handler which `.exception()` reports the reconnect
attempt failure explicitly.
- drop some lingering `Optional` usage.
Instead of the insignificantly named dev branch from recent `trio`
/ py3.13 updates work; it makes more sense to keep a dedicated pin (as
we have prior) for the moment. Also re-org the masked @goodboy dev-env
lines + comments to bottom of file.
Namely changes for the `registry_addrs: list`, enable_transports: list`
and related `tractor._addr` primitive requirements.
Other updates include,
- passing `maybe_enable_greenback=True`,
- additional exc logging around `pikerd` syncing/booting,
- changing to newer `Context.wait_for_result()`,
- dropping (unnecessary?) `maybe_open_crash_handler()` around `pikerd` ep.
Such that we can avoid other (pretty unreliable) "alternative" checks to
determine whether a real-time quote should be waited on or (when venue
is closed) we should just signal that historical backfilling can
commence immediately.
This has been a todo for a very long time and it turned out to be much
easier to accomplish than anticipated..
Deats,
- add a new `is_current_time_in_range()` dt range checker to predicate
whether an input range contains `datetime.now(start_dt.tzinfo)`.
- in `.ib.feed.stream_quotes()` add a `venue_is_open: bool` which uses
all of the new ^^ to determine whether to branch for the
short-circuit-and-do-history-now-case or the std real-time-quotes
should-be-awaited-since-venue-is-open, case; drop all the old hacks
trying to workaround not figuring that venue state stuff..
Other,
- also add a gpt5 composed parser to `._util` for the
`ib_insync.ContractDetails.tradingHours: str` for before i realized
there was a `.tradingSessions` property XD
- in `.ib_feed`,
* add various EG-collapsings per recent tractor/trio updates.
* better logging / exc-handling around ticker quote pushes.
* stop clearing `Ticker.ticks` each quote iteration; not sure if this
is needed/correct tho?
* add masked `Ticker.ticks` poll loop that logs.
- fix some `str.format()` usage in `._util.try_xdo_manual()`
You'd think they could be bothered to make either a "log" or "warning"
msg type instead of a `type='error'`.. but alas, this attempts to detect
all such "warning"-errors and never proxy them to the clearing engine
thus avoiding the cancellation of any associated (by `reqid`)
pre-existing orders (control dialogs).
Also update all surrounding log messages to a more multiline style.
Topically, throughout various (seemingly) console-UX-affecting or benign
spots in the code base; nothing that required more intervention beyond
things superficial. A few spots also include `trio.Nursery` ref renames
(always to something with a `tn` in it) and log-level reductions to
quiet (benign) console noise oriented around issues meant to be solved
long..
Note there's still a couple spots i left with the loose-ify flag because
i haven't fully tested them without using the latest version of
`tractor.trionics.collapse_eg()`, but more then likely they should flip
over fine.
That is inside embedded `.accounting.calc.dyn_parse_to_dt()` closure add
an optional `_invalid: list` param to where we can report
bad-timestamped records which we instead override and return as
`from_timestamp(0.)` (when the parser loop falls through) and report
later (in summary ) from the `.accounting.calc.iter_by_dt()` caller
. Add some logging and an optional debug block for future tracing.
Since apparently porting to the new docker container enforces using
a vnc password and `asyncvnc` seems to have a bug/mis-config whenever
i've tried a pw over a wg tunnel..?
Soo, this tries out the old `i3ipc`-win-focus + `xdo` click hack when
the above fails.
Deats,
- add a mod-level `try_xdo_manual()` to wrap calling
`i3ipc_xdotool_manual_click_hack()` with an oserr handler, ensure we
don't bother trying if `i3ipc` import fails beforehand tho.
- call ^ from both the orig case block and the failover from the
vnc-client case.
- factor the `+no_setup_msg: str` out to mod level and expect it to be
`.format()`-ed.
- refresh todo around `asyncvnc` pw ish..
- add a new `i3ipc_fin_wins_titled()` window-title scanner which
predicates input `titles` and delivers any matches alongside the orig
focused win at call time.
- tweak `i3ipc_xdotool_manual_click_hack()` to call ^ and remove prior
unfactored window scanning logic.
The root daemon, pikerd, needs to be adjusted to use diff default
registry addrs to also utilize non-TCP, but for now this gets us started
testing; so far so good B)
That is to use the new `tractor.msg.types.Aid` struct to pull the
`brokerd` info from the `tractor.Channel.aid: Aid` attr as well as more
generally handling the new `Channel.raddr.proto_key: str` and no longer
assuming a TCP IPC transport; this per the recent `tractor.ipc`
subsys which adds multi-IPC-transports!
Downstream tweaks to match,
- use an "opt-in" field set to display in the `brokerd` info pane in
`.ui._feedstatus.mk_feed_label()`.
|_ also add some todos and drop some seemingly unneeded form sizing
calcs?
- tweak `.ui._label` to allow not using markdown, though ended up not
doing that since it looked too plain..
Using `tractor.trionics.collapse_eg()` as needed to avoid, at the least,
crash-worthy (in debug-mode REPL-ing terms) nested cancellation egs that
exhibit on SIGINT/ctl-c of each "app" (chart & daemon).
Also a bit of renaming of all `trio.Nursery`s to `tn`, the new "task
nursery" shorthand-var-name being used in all our other `tractor`
related projects.
wygud
Tyler Goodlet