Since the new FSP system will require time aligned data amongst actors,
it makes sense to share broker data feeds as much as possible on a local
system. There doesn't seem to be downside to this approach either since
if not fanning-out in our code, the broker (server) has to do it anyway
(and who knows how junk their implementation is) though with more
clients, sockets etc. in memory on our end. It also preps the code for
introducing a more "serious" pub-sub systems like zeromq/nanomessage.
Start a draft normalization format for (sampled) tick data.
Ideally we move toward the dense tick format (DFT) enforced by
techtonicDB, but for now let's just get a dict of something simple
going: `{'type': 'trade', 'price': <price}` kind of thing. This
gets us started being able to real-time chart from all data feed
back-ends. Oh, and hack in support for XAUUSD..and get subactor
logging workin.
Add a `Client.find_contract()` which internally takes
a <symbol>.<exchange> str as input and uses `IB.qualifyContractsAsync()`
internally to try and validate the most likely contract. Make the module
script call this using `asyncio.run()` for console testing.
Infected `asyncio` support is being added to `tractor` in
goodboy/tractor#121 so delegate to all that new machinery.
Start building out an "actor-aware" api which takes care of all the
`trio`-`asyncio` interaction for data streaming and request handling.
Add a little (shudder) method proxy system which can be used to invoke
client methods from another actor. Start on a streaming api in
preparation for real-time charting.
Start working towards meeting the backend client api.
Infect `asyncio` using `trio`'s new guest mode and demonstrate
real-time ticker streaming to console.
Since the new FSP system will require time aligned data amongst actors,
it makes sense to share broker data feeds as much as possible on a local
system. There doesn't seem to be downside to this approach either since
if not fanning-out in our code, the broker (server) has to do it anyway
(and who knows how junk their implementation is) though with more
clients, sockets etc. in memory on our end. It also preps the code for
introducing a more "serious" pub-sub systems like zeromq/nanomessage.
This is something I've been meaning to try for a while and will likely
make writing tick data to a db more straight forward (filling in NaN
values is more matter of fact) plus it should minimize bandwidth usage.
Note, it'll require stream consumers to be considerate of non-full
quotes arriving and thus using the first "full" quote message to fill
out dynamically formatted systems or displays.
For easy testing of questrade historical data from cli.
Re-org the common cli components into a new package to avoid having all
commands defined in a top-level module.
There's some expected limitations with the number of sticks allowed in
a single query (they say 2k but I've been able to pull 20k). Also note
without a paid data sub there's a 15m delay on 1m sticks (we'll hack
around that shortly, don't worry).
Gets us better throughput when polling multiple endpoints (eg. option
and stock quotes simultaneously) since slower round trip request won't
block faster ones when using multiple connections.
- stop displaying search bar widget on <ctrl-c>
- if there's existing search bar content highlight it automatically
to allow user to start typing new content right away
- when activated allow search bar to insert its own set of keybinding
controls; restore prior bindings on exit
Fixes to `tractor` that resolve issues with async generators being
non-task safe make the need for the mutex lock in
`DataFeed.open_stream()` unnecessary. Also, don't bother pushing empty
quotes from the publisher; avoids hitting the network when possible.
Questrade's API is half baked and can't handle concurrency.
It allows multiple concurrent requests to most endpoints *except*
for the auth endpoint used to refresh tokens:
https://www.questrade.com/api/documentation/security
I've gone through extensive dialogue with their API team and despite
making what I think are very good arguments for doing the request
serialization on the server side, they decided that I should instead
do the "locking" on the client side. Frankly it doesn't seem like they
have that competent an engineering department as it took me a long time
to explain the issue even though it's rather trivial and probably not
that hard to fix; maybe it's better this way.
This adds a few things to ensure more reliable token refreshes on
expiry:
- add a `@refresh_token_on_err` decorator which can be used on `_API`
methods that should refresh tokens on failure
- decorate most endpoints with this *except* for the auth ep
- add locking logic for the troublesome scenario as follows:
* every time a request is sent out set a "request in progress" event
variable that can be used to determine when no requests are currently
outstanding
* every time the auth end point is hit in order to refresh tokens set
an event that locks out other tasks from making requests
* only allow hitting the auth endpoint when there are no "requests in
progress" using the first event
* mutex all auth endpoint requests; there can only be one outstanding
- don't hit the accounts endpoint at client startup; we want to
eventually support keys from multiple accounts and you can disable
account info per key and just share the market data function
Adjust feed locking around internal manager `yields` to make this work.
Also, change quote publisher to deliver a list of quotes for each
retrieved batch. This was actually broken for option streaming since
each quote was being overwritten due to a common `key` value for all
expiries. Asjust the `packetizer` function accordingly to work for
both options and stocks.
The pub-sub data feed system was factored into `tractor` as an
experimental api / subsystem. Move to using that which greatly
simplifies the data feed architecture.
Start working toward a more general (on-demand) pub-sub system which
can be brought into ``tractor``. Right now this just means making
the code in the `fan_out_to_ctxs()` less specific but, eventually
I think this function should be coupled with a decorator and shipped
as a standard "message pattern".
Additionally,
- try out making `BrokerFeed` a `@dataclass`
- strip out all the `trio.Event` / uneeded nursery / extra task crap
from `start_quote_stream()`
If quotes are pushed using the adjusted contract symbol (i.e. with
trailing '-1' suffix) the subscriber won't receive them under the
normal symbol. The logic was wrong for determining whether to add
a suffix (was failing for any symbol with an exchange suffix)
which was causing normal data feed subscriptions to fail to match
in every case.
I did some testing of the `optionsIds` parameter to the option quote
endpoint and found that it limits you to 100 symbols so it's not
practical for real-time "all-strike"" chain updating; we have to stick
to filters for now. The only real downside of this is that it seems
multiple filters across multiple symbols is quite latent. I need to
toy with it more to be sure it's not something slow on the client side.
Oh, and store option contract to ids in a `dict` for now as we may want
to try the `optionsIds` thing again down the road as I coordinate with
the QT tech team.
Add some extra fields to each quote that QT should already be
providing (instead of hiding them in the symbol and request contract
info); namely, the expiry and contact type (i.e. put or call).
Define the base set of fields to be displayed in an option chain
UI and add a quote formatter.
Tyler Goodlet
Tyler Goodlet