There were some imports missing or unused as well as a variety of spots
that had grokability issues due to missing type hints.
Other tweaks as part some more thorough manual testing:
- always raise when not `brokers.toml` section since the API can never
work (no free data without keys).
- inline the `Asset.atype='crypto_currency` field despite it maybe not
being the best value for `OptionPair` instruments..
- tossed in a now-masked pause block for debugging history queries in
`Client.bars()`.
- commented out all the live order ctl (internal) endpoints for now
since they're unused.
Since currently we're only using this IPC subsys for `deribit`, and
generally speaking we're primarly supporting options markets (which are
fairly "slow moving"), flip to a default of NOT resetting the `NoBsWs`
on timeout since doing so normally breaks the jsron-rpc IPC session.
Without a proper `fixture` passed to `open_autorecon_ws()` (which we
should eventually implement!!) relying on a timeout-to-reset more or
less will just cause breakage issues - a proper reconnect sequence must
be implemented before using that feature.
Deats,
- expose and proxy through the `msg_recv_timeout` from
`open_jsonrpc_session()` into the underlying `open_autorecon_ws()`
call.
Namely handling backends which do not provide a default "frame
size-duration" in their init-config by making the backfiller guess the
value based on the first frame received.
Deats,
- adjust `start_backfill()` to take a more explicit
`def_frame_duration: Duration` expected to be unpacked from any
backend hist init-config by the `tsdb_backfill()` caller which now
also computes a value from the first received frame when the config
section isn't provided.
- in `start_backfill()` we now always expect the `def_frame_duration`
input and always decrement the query range by this value whenever
a `NoData` is raised by the provider-backend paired with an explicit
`log.warning()` about the handling.
- also relay any `DataUnavailable.args[0]` message from the provider
in the handler.
- repair "gap reporting" which checks for expected frame duration vs.
that received with much better humanized logging on the missing
segment using `pendulum.Interval/Duration.in_words()` output.
The `open_history_client()` provider endpoint can *optionally*
deliver a `frame_types: dict[int, pendulum.Duration]` subsection in its
`config: dict[str, dict]` (as was implemented with the `ib` backend).
This allows the `tsp` backfilling machinery to use this "recommended
frame duration" to subtract from the `last_start_dt` any time a `NoData`
gap is signalled by the `get_hist()` call allowing gaps to be ignored
safely without missing history by knowing the next earliest dt we can
query from using the `end_dt`. However, currently all crypto$ providers
haven't implemented this feat yet..
As such only try to use the `frame_types` feature if provided when
handling `NoData` conditions inside `tsp.start_backfill()` and otherwise
raise as normal.
For example in the paper-eng, if you have a backend that doesn't fully
support a symcache (yet) it's handy to be able to ignore processing
other paper-eng txns when all you care about at the moment is the
simulated symbol.
NOTE, that currently this will still result in a key-error when you load
more then one mkt with the paper engine (for which the backend does not
have the symcache implemented) since no fqme ad-hoc query was made for
the 2nd symbol (and i'm not sure we should support that kinda hackery
over just encouraging the sym-cache being added?). Def needs a little
more thought depending on how many backends are never going to be able
to (easily) support caching..
Instead just check for the field (which i'm not huge on the key-name for
anyway) and if not found get the "last price" from the real-time shm
buffer's latest 'close' sample.
Unrelatedly, use a `subs.copy()` in the `Router.client_broadcast()` loop
such that if a `client_stream` is popped on connection failure, we don't
RTE for the "size changed on iteration".
This must have broke at some point during the new `MktPair` and thus
`.fqme: str` updates; mas-o-menos the symbol key in the quote-msg-`dict`
was NOT set to the `MktPair.bs_fqme: str` value and thus wasn't being
processed by the downstream sampling and feed subsys.
So fix that as well as a few other refinements,
- set the `topic: mkt.bs_fqme` in quote msgs obvi.
- drop the "wait for first clearing vlm" quote poll loop; going to fix
the sampler to handle a `first_quote` without a `'last'` key.
- add some typing around calls to `get_mkt_info()`.
- rename `stream_messages()` -> `iter_normed_quotes()`.
Since `tractor`'s new and improved inter-actor cancellation semantics
are much more pedantic, AND bc we use the `ServiceMngr` for spawning
service actors on-demand, the caller of `maybe_spawn_daemon()` should
NEVER conduct a so called "out of band" `Actor`-runtime cancel request
since this is precisely the job of our `ServiceMngr` XD
Add a super in depth note explaining the underlying issue and adding
a todo list of how we should prolly augment `tractor` to make such cases
easier to grok and fix in the future!
Given it's a fairly simple yet useful abstraction, it makes sense to
offer this sub-sys alongside the core `tractor` runtime lib.
Without going into extreme detail on the impl changes (it'll come in
the commit that moves to the other repo) here is the high level summary:
------ - ------
- rename `Services` -> `ServiceMngr` and use an factory `@acm`
to guarantee a single-instance-per-actor using a niche approach for a
singleton object using a default keyword-arg B)
- the mod level `open_service_mngr()` and `get_service_mngr()` are the
new allocation/access API.
- add a `ServiceMngr.start_service()` method which does the work of both
spawning a new subactor (for the daemon) and uses its portal to start
the mngr side supervision task.
- open actor/task nurseries inside the `@acm` allocator.
Adjust other dependent subsystems to match:
------ - ------
- use `open_service_mngr()` when first allocated in `open_pikerd()`.
- use `get_service_mngr()` instead of importing the class ref inside
`.service.maybe_spawn_daemon()`, `.brokers._daemon.spawn_brokerd()`
and `.data._sampling.spawn_samplerd()` using a `partial` to pack in
the endpoint ctx kwargs (unpacked inside `.start_service()` XD).
Tyler Goodlet
Nelson Torres