Commit Graph

3421 Commits (52ac1053aaeb523ffe944b1703abfe68d16098f7)

Author SHA1 Message Date
Tyler Goodlet 8195fae289 Add a `trigger_all` arg to update cycle func; allows hard history updates 2022-04-16 13:25:14 -04:00
Tyler Goodlet 30656eda39 Use a `DisplayState` in the graphics update loop
The graphics update loop is much easier to grok when all the UI
components which potentially need to be updated on a cycle are arranged
together in a high-level composite namespace, thus this new
`DisplayState` addition. Create and set this state on each
`LinkedSplits` chart set and add a new method `.graphics_cycle()` which
let's a caller trigger a graphics loop update manually. Use this method
in the fsp graphics manager such that a chain can update new history
output even if there is no real-time feed driving the display loop (eg.
when a market is "closed").
2022-04-16 13:25:14 -04:00
Tyler Goodlet 2564acea1b Facepalm**2: only update on special "update" msg 2022-04-16 13:25:14 -04:00
Tyler Goodlet b3efa2874b Facepalm: display state must be linked charts specific 2022-04-16 13:25:14 -04:00
Tyler Goodlet ad1bbe74ad Manually trigger graphics loops updates on msgs from the fsp chain 2022-04-16 13:25:14 -04:00
Tyler Goodlet 761b823939 Always fire a "step/update message" on every fsp history update 2022-04-16 13:25:14 -04:00
Tyler Goodlet b75a3310fe Factor sync part of graphics update into func, add `trigger_update()`` 2022-04-16 13:25:14 -04:00
Tyler Goodlet ed8cfcf66d Drop `arrow` from install deps 2022-04-16 13:23:42 -04:00
Tyler Goodlet 72ec34ffd2 Port to `pendulum` equivalent apis throughout 2022-04-16 13:23:42 -04:00
Tyler Goodlet d334e61b1f Drop 22s timeout on reset hack 2022-04-16 13:23:38 -04:00
goodboy fbabfb78e0
Merge pull request #294 from pikers/broker_bumpz
Broker bumpz
2022-04-13 08:10:44 -04:00
Tyler Goodlet 4d23f6e4d7 Drop need for `ib_insync.IB.qualifyContractsAsync()' mod
As per https://github.com/erdewit/ib_insync/pull/454 the more correct
way to do this is with `.reqContractDetailsAsync()` which we wrap with
`Client.con_deats()` and which works just as well. Further drop all the
`dict`-ifying that was being done in that method and instead always
return `ContractDetails` object in an fqsn-like explicitly keyed `dict`.
2022-04-13 00:39:15 -04:00
Tyler Goodlet 8b1c521ae9 Ignore symbol-not-found errors 2022-04-13 00:39:15 -04:00
Tyler Goodlet 7586e20ab4 Use new unpacker helper name 2022-04-13 00:39:15 -04:00
Tyler Goodlet 80d70216f7 Drop back down ohlc bars request count to not trigger feed hack 2022-04-13 00:39:15 -04:00
Tyler Goodlet d1f45b0883 Add `ShmArray.last()` docstr 2022-04-13 00:39:15 -04:00
Tyler Goodlet 00a7f20292 Up the shm size to 10d of 1s ohlc 2022-04-13 00:39:15 -04:00
Tyler Goodlet 0178fcd26f Increase shm size to days of 1s steps 2022-04-13 00:39:15 -04:00
Tyler Goodlet 24fa1b8ff7 Support an array field map to `ShmArray.push()`, start index 3days in 2022-04-13 00:39:15 -04:00
Tyler Goodlet 66ea74c6d5 Put back more bars iters in loop to handle no-data in range cases 2022-04-13 00:39:15 -04:00
Tyler Goodlet b579d4b1f5 Get ib data feed hackzorz workin
ib has a throttle limit for "hft" bars but contained in here is some
hackery using ``xdotool`` to reset data farms auto-magically B)

This copies the working script into the ib backend mod as a routine and
now uses `trio.run_process()` and calls into it from the `get_bars()`
history retriever and then waits for "data re-established" events to be
received from the client before making more history queries.

TL;DR summary of changes:
- relay ib's "system status" events (like for data farm statuses)
  as a new "event" msg that can be processed by registers of
  `Client.inline_errors()` (though we should probably make a new
  method for this).
- add `MethodProxy.status_event()` which allows a proxy user to register
  for a particular "system event" (as mentioned above), which puts
  a `trio.Event` entry in a small table can be set by an relay task if
  there are any detected waiters.
- start a "msg relay task" when opening the method proxy which does
  the event setting mentioned above in the background.
- drop the request error handling around the proxy creation, doesn't
  seem necessary any more now that we have better error propagation from
  `asyncio`.
- add event waiting logic around the data feed reset hackzorin.
- change the order relay task to only log system events for now (though
  we need to do some better parsing/logic to get tws-external order
  updates to work again..
2022-04-13 00:39:15 -04:00
Tyler Goodlet 874374af06 Drop `pandas` use in ib backend for history
Found an issue (that was predictably brushed aside XD) where the
`ib_insync.util.df()` helper was changing the timestamps on bars data to
be way off (probably a `pandas.Timestamp` timezone thing?).

Anyway, dropped all that (which will hopefully let us drop `pandas` as
a hard dep) and added a buncha timestamp checking as well as start/end
datetime return values using `pendulum` so that consumer code can know
which "slice" is output.

Also added some WIP code to work around "no history found" request
errors where instead now we try to increment backward another 200
seconds - not sure if this actually correct yet.
2022-04-13 00:39:15 -04:00
Tyler Goodlet 62d073dc18 More IB repairs..
Make the throttle error propagate through to `trio` again by adding
`dict`-msg support between the two loops such that errors can be
re-raised on the `trio` side. This is all integrated into the
`MethoProxy` and accompanying result relay task.

Further fix a longer standing issue where sometimes the `ib_insync`
order entry method will raise a weird assertion error because it detects
some internal order-id state issue.. Just ignore those and make relay
back an error to the ems in such cases.

Add a bunch of notes for todos surrounding data feed reset hackery.
2022-04-13 00:39:15 -04:00
Tyler Goodlet 3e125625b1 Attempt to better handle history throttles using flag 2022-04-13 00:39:15 -04:00
Tyler Goodlet 8395a1fcfe IB: Comment on lowercase for the fqsn key 2022-04-13 00:39:15 -04:00
Tyler Goodlet 957686a9fe Comment exception debug in ib request error block 2022-04-13 00:39:15 -04:00
Tyler Goodlet 1e433ca4f4 Support "expiry" suffixes for derivatives with ib
To start we only have futes working but this allows both searching
and loading multiple expiries of the same instrument by specifying
different expiries with a `.<expiry>` suffix in the symbol key (eg.
`mnq.globex.20220617`). This also paves the way for options contracts
which will need something similar plus a strike property. This change
set also required a patch to `ib_insync` to allow retrieving multiple
"ambiguous" contracts from the `IB.reqContractDetailsAcync()` method,
see https://github.com/erdewit/ib_insync/pull/454 for further discussion
since the approach here might change.

This patch also includes a lot of serious reworking of some `trio`-`asyncio`
integration to use the newer `tractor.to_asyncio.open_channel_from()`
api and use it (with a relay task) to open a persistent connection with
an in-actor `ib_insync` `Client` mostly for history requests.

Deats,
- annot the module with a `_infect_asyncio: bool` for `tractor` spawning
- add a futes venu list
- support ambiguous futes contracts lookups so that all expiries will
  show in search
- support both continuous and specific expiry fute contract
  qualification
- allow searching with "fqsn" keys
- don't crash on "data not found" errors in history requests
- move all quotes msg "topic-key" generation (which should now be
  a broker-specific fqsn) and per-contract quote processing into
  `normalize()`
- set the fqsn key in the symbol info init msg
- use `open_client_proxy()` in bars backfiller endpoint
- include expiry suffix in position update keys
2022-04-13 00:39:15 -04:00
Tyler Goodlet 937406534c Maybe spawn `brokerd` in `asyncio` mode if declared in backend mod 2022-04-13 00:39:15 -04:00
Tyler Goodlet b26b66cc66 Add context-styled `asyncio` client proxy for ib
This adds a new client manager-factory: `open_client_proxy()` which uses
the newer `tractor.to_asyncio.open_channel_from()` (and thus the
inter-loop-task-channel style) a `aio_client_method_relay()` and
a re-implemented `MethodProxy` wrapper to allow transparently calling
`asyncio` client methods from `trio` tasks. Use this proxy in the
history backfiller task and add a new (prototype)
`open_history_client()` which will be used in the new storage management
layer. Drop `get_client()` which was the portal wrapping equivalent of
the same proxy but with a one-task-per-call approach. Oh, and
`Client.bars()` can take `datetime`, so let's use it B)
2022-04-13 00:39:15 -04:00
Tyler Goodlet 7936dcafbf Make linux timeout the same 2022-04-13 00:39:15 -04:00
Tyler Goodlet d32c26c5d7 Add flag to avoid logging json to console 2022-04-13 00:39:15 -04:00
Tyler Goodlet d2d3286fb8 Use `asyncio` in `Client.get_quote()` 2022-04-13 00:39:15 -04:00
goodboy 310a17e93b
Merge pull request #301 from pikers/no_git_prot_w_pip
No git prot w pip, v3 actions.
2022-04-13 00:38:18 -04:00
Tyler Goodlet a45156cbb7 Use checkout and setup-python v3 actions and drop dev install 2022-04-12 22:14:22 -04:00
Tyler Goodlet 6324624811 Try https? 2022-04-12 17:29:25 -04:00
Tyler Goodlet 3762466a58 Try running CI on 3.10 and drop eager install 2022-04-12 17:29:25 -04:00
Tyler Goodlet 289a69bf41 Stop using unecrypted git prot for edit deps 2022-04-12 17:29:25 -04:00
goodboy 253cbf901c
Merge pull request #295 from pikers/fqsns
Fqsns for cross-broker ticker naming
2022-04-11 09:20:36 -04:00
Tyler Goodlet 4b0ca40b17 Document "fqsn" on `Symbol` method 2022-04-11 08:48:17 -04:00
Tyler Goodlet ebe2680355 Change `uncons_fqsn()` -> `unpack_fqsn()` 2022-04-11 01:01:36 -04:00
Tyler Goodlet e92632bd34 Remove old commented nan checking lines 2022-04-10 21:51:22 -04:00
Tyler Goodlet 32e316ebff Drop nl 2022-04-10 17:33:02 -04:00
Tyler Goodlet f604437897 Remove symbol key from first quote from ib feed 2022-04-10 17:33:02 -04:00
Tyler Goodlet c9e6c81459 Expect fqsn input to paper clearing engine 2022-04-10 17:33:02 -04:00
Tyler Goodlet ce7d630676 Pass in fqsn from fsp admin apis 2022-04-10 17:33:02 -04:00
Tyler Goodlet 6ac60fbe22 Expect fqsns through fsp machinery 2022-04-10 17:33:02 -04:00
Tyler Goodlet 998a5acd92 Crypto$ backend updates
- move to 3.9+ type annots
- add initial draft `open_history_client()` endpoints
- deliver `'fqsn'` keys in quote-stream init msgs
2022-04-10 17:33:00 -04:00
Tyler Goodlet 493e45e70a Strip broker name from symbol on pp msg updates 2022-04-10 17:30:02 -04:00
Tyler Goodlet c7f3e59105 Expect fqsn in ems and order mode
Use fqsn as input to the client-side EMS apis but strip broker-name
stuff before generating and sending `Brokerd*` msgs to each backend for
live order requests (since it's weird for a backend to expect it's own
name, though maybe that could be a sanity check?).

Summary of fqsn use vs. broker native keys:
- client side pps, order requests and general UX for order management
  use an fqsn for tracking
- brokerd side order dialogs use the broker-specific symbol which is
  usually nearly the same key minus the broker name
- internal dark book and quote feed lookups use the fqsn where possible
2022-04-10 17:30:02 -04:00
Tyler Goodlet d62a636bcc Pass concatted pre-fqsn directly to feed api 2022-04-10 17:30:02 -04:00