For easy testing of questrade historical data from cli.
Re-org the common cli components into a new package to avoid having all
commands defined in a top-level module.
There's some expected limitations with the number of sticks allowed in
a single query (they say 2k but I've been able to pull 20k). Also note
without a paid data sub there's a 15m delay on 1m sticks (we'll hack
around that shortly, don't worry).
Gets us better throughput when polling multiple endpoints (eg. option
and stock quotes simultaneously) since slower round trip request won't
block faster ones when using multiple connections.
This required some copy-paste of code from @matham's branch:
https://github.com/kivy/kivy/pull/5241
namely, the stuff in the `utils_async.py` module. I've added all that as
a standalone file for now.
Update the pipfile to use `kivy`'s master branch (since there seems to
be some lingering cython issues in the current release wheels).
- stop displaying search bar widget on <ctrl-c>
- if there's existing search bar content highlight it automatically
to allow user to start typing new content right away
- when activated allow search bar to insert its own set of keybinding
controls; restore prior bindings on exit
Look up the broker module and set up the loglevel locally.
Ask the arbiter for a portal to the data daemon since we can't
pass one to a subactor by reference.
Fixes to `tractor` that resolve issues with async generators being
non-task safe make the need for the mutex lock in
`DataFeed.open_stream()` unnecessary. Also, don't bother pushing empty
quotes from the publisher; avoids hitting the network when possible.
Questrade's API is half baked and can't handle concurrency.
It allows multiple concurrent requests to most endpoints *except*
for the auth endpoint used to refresh tokens:
https://www.questrade.com/api/documentation/security
I've gone through extensive dialogue with their API team and despite
making what I think are very good arguments for doing the request
serialization on the server side, they decided that I should instead
do the "locking" on the client side. Frankly it doesn't seem like they
have that competent an engineering department as it took me a long time
to explain the issue even though it's rather trivial and probably not
that hard to fix; maybe it's better this way.
This adds a few things to ensure more reliable token refreshes on
expiry:
- add a `@refresh_token_on_err` decorator which can be used on `_API`
methods that should refresh tokens on failure
- decorate most endpoints with this *except* for the auth ep
- add locking logic for the troublesome scenario as follows:
* every time a request is sent out set a "request in progress" event
variable that can be used to determine when no requests are currently
outstanding
* every time the auth end point is hit in order to refresh tokens set
an event that locks out other tasks from making requests
* only allow hitting the auth endpoint when there are no "requests in
progress" using the first event
* mutex all auth endpoint requests; there can only be one outstanding
- don't hit the accounts endpoint at client startup; we want to
eventually support keys from multiple accounts and you can disable
account info per key and just share the market data function
Adjust feed locking around internal manager `yields` to make this work.
Also, change quote publisher to deliver a list of quotes for each
retrieved batch. This was actually broken for option streaming since
each quote was being overwritten due to a common `key` value for all
expiries. Asjust the `packetizer` function accordingly to work for
both options and stocks.
The pub-sub data feed system was factored into `tractor` as an
experimental api / subsystem. Move to using that which greatly
simplifies the data feed architecture.
Start working toward a more general (on-demand) pub-sub system which
can be brought into ``tractor``. Right now this just means making
the code in the `fan_out_to_ctxs()` less specific but, eventually
I think this function should be coupled with a decorator and shipped
as a standard "message pattern".
Additionally,
- try out making `BrokerFeed` a `@dataclass`
- strip out all the `trio.Event` / uneeded nursery / extra task crap
from `start_quote_stream()`
This allows for using a monitor to select the current option chain
symbol!
The deats:
- start a bg task which streams the monitor selected symbol
- dynamically repopulate expiry buttons on a newly published symbol
- move static widget creation into a chain method to avoid multiple
quotes requests at startup
- rename a bunch of methods
If quotes are pushed using the adjusted contract symbol (i.e. with
trailing '-1' suffix) the subscriber won't receive them under the
normal symbol. The logic was wrong for determining whether to add
a suffix (was failing for any symbol with an exchange suffix)
which was causing normal data feed subscriptions to fail to match
in every case.
I did some testing of the `optionsIds` parameter to the option quote
endpoint and found that it limits you to 100 symbols so it's not
practical for real-time "all-strike"" chain updating; we have to stick
to filters for now. The only real downside of this is that it seems
multiple filters across multiple symbols is quite latent. I need to
toy with it more to be sure it's not something slow on the client side.
Oh, and store option contract to ids in a `dict` for now as we may want
to try the `optionsIds` thing again down the road as I coordinate with
the QT tech team.
This is an optimization to improve performance when the UI is fed real
time data. Instead of resorting all rows on every quote update, only
re-render when the sort key appears in the quote data, and further, only
resort rows which are changed using bisection based widget insertion to
avoid having `kivy` re-add widgets (and thus re-render graphics) more
often than absolutely necessary.
There's still a ton to polish (and some bugs to fix) but this is a first
working draft of a real-time option chain!
Insights and todos:
- `kivy` widgets need to be cached and reused (eg. rows, cells, etc.)
for speed since it seems creating new ones constantly is quite taxing
on the CPU
- the chain will tear down and re-setup the option data feed stream each
time a different contract expiry button set is clicked
- there's still some weird bug with row highlighting where it seems rows
added from a new expiry set (which weren't previously rendered) aren't
being highlighted reliably
`Row`:
- `no_cell`: support a list of keys for which no cells will be created
- allow passing in a `cell_type` at instantiation
`TickerTable`:
- keep track of rendered rows via a private `_rendered` set
- don't create rows inside `append_row()` expect caller to do it
- never render already rendered widgets in `render_rows()`
Miscellaneous:
- generalize `update_quotes()` to not be tied to specific quote fields
and allow passing in a quote `formatter()` func
- don't bother creating a nursery block until necessary in main
- more commenting
Add some extra fields to each quote that QT should already be
providing (instead of hiding them in the symbol and request contract
info); namely, the expiry and contact type (i.e. put or call).
Define the base set of fields to be displayed in an option chain
UI and add a quote formatter.
Copy out `kivy.clock.triggered` from version 1.10.1 since it isn't yet
available in the `trio`/async branch and use it to throttle the callback
rate. Use a `collections.deque` to LIFO iterate widgets each call
using the heuristic that it's more likely the mouse is still within the
currently highlighted (or it's adjacent neighbors) widget as opposed
to some far away widget (the case when the mouse is moved very
drastically across the window).
Thanks yet again to @tshirtman for suggesting this.
Instead of defining a `on_mouse_pos()` on every widget simply
register and track each widget and loop through them all once (or as much
as is necessary) in a single callback. The assumption here is that we
get a performance boost by looping widgets instead of having `kivy` loop
and call back each widget thus avoiding costly python function calls.
Well that was a doozy; had to rejig pretty much all of it.
The deats:
- Track broker components in a new `DataFeed` namedtuple
- port to new list based batch quotes (not dicts any more)
- lock access to cached broker-client / data-feed instantiation
- respawn tasks that fail due to the network
Tyler Goodlet
Tyler Goodlet