pikers
/
piker
3
0
Fork 0
Code Issues 5 Pull Requests 7 Packages Projects Releases Wiki Activity

Deribit broker fix #8

Open
ntorres wants to merge 39 commits from deribit_fix into nix-qt6-fix
pull from: deribit_fix
merge into: pikers:nix-qt6-fix
Conversation 3 Commits 39 Files Changed 11
ntorres commented 2024-08-28 00:02:52 +00:00
Collaborator

In the brokers init file, at this point the feed is working, to check the tables use vd tool.

In the brokers init file, at this point the feed is working, to check the tables use vd tool.
ntorres added 1 commit 2024-08-28 00:02:53 +00:00
fcf68b3fe9 Added cryptofeed and pyarrow necessary for the feed, enable deribit 
in the brokers init file, at this point the feed is working, to check
the tables use vd tool.
ntorres force-pushed deribit_fix from fcf68b3fe9 to 53148b453d 2024-08-28 16:57:14 +00:00 Compare
ntorres force-pushed deribit_fix from 53148b453d to 00406028ea 2024-08-28 18:04:53 +00:00 Compare
ntorres force-pushed deribit_fix from 00406028ea to cd92ef8b3a 2024-08-28 19:01:35 +00:00 Compare
ntorres force-pushed deribit_fix from cd92ef8b3a to b2cfa3444f 2024-08-28 23:58:58 +00:00 Compare
ntorres changed title from Deribit broker fix. to Deribit broker fix 2024-08-29 22:47:52 +00:00
ntorres added 2 commits 2024-11-05 12:03:52 +00:00
499b2d0090 fix syms for venues. 
little refactor in get_config, and created get_fh_config for cryptofeed.
ntorres added 15 commits 2024-11-08 21:59:17 +00:00
ntorres added 2 commits 2024-11-11 14:23:10 +00:00
ntorres force-pushed deribit_fix from 278cb87f43 to a3ab8dd8fe 2024-11-12 15:20:22 +00:00 Compare
ntorres added 1 commit 2024-11-12 15:22:41 +00:00
ntorres added 3 commits 2024-11-12 19:52:04 +00:00
ntorres added 2 commits 2024-11-13 13:47:21 +00:00
5304a36b87 get_assets now uses public endpoint 
It's better if the data is available through a public endpoint.
ntorres added 4 commits 2024-11-13 13:52:24 +00:00
e9fa422916 Added missing fields for kucoin. 
feeCategory, makerFeeCoefficient, takerFeeCoefficient and st.
38a1f0b9ee `kucoin`: repair live quotes streaming.. 
This must have broke at some point during the new `MktPair` and thus
`.fqme: str` updates; mas-o-menos the symbol key in the quote-msg-`dict`
was NOT set to the `MktPair.bs_fqme: str` value and thus wasn't being
processed by the downstream sampling and feed subsys.

So fix that as well as a few other refinements,
- set the `topic: mkt.bs_fqme` in quote msgs obvi.
- drop the "wait for first clearing vlm" quote poll loop; going to fix
  the sampler to handle a `first_quote` without a `'last'` key.
- add some typing around calls to `get_mkt_info()`.
- rename `stream_messages()` -> `iter_normed_quotes()`.
45788b0b53 .clearing._ems: Don't require `first_quote['last']` 
Instead just check for the field (which i'm not huge on the key-name for
anyway) and if not found get the "last price" from the real-time shm
buffer's latest 'close' sample.

Unrelatedly, use a `subs.copy()` in the `Router.client_broadcast()` loop
such that if a `client_stream` is popped on connection failure, we don't
RTE for the "size changed on iteration".
ntorres added 6 commits 2024-11-15 14:24:43 +00:00
ntorres added 1 commit 2024-11-15 17:27:02 +00:00
ntorres added 2 commits 2024-11-15 18:24:19 +00:00
6555ccfbba config refactor 
only one get_config method for api class and cryptofeed feed handler
guille approved these changes 2024-11-16 14:07:43 +00:00
piker/brokers/deribit/api.py
@ -231,1 +226,3 @@
async def get_balances(self, kind: str = 'option') -> dict[str, float]:
async def _json_rpc_auth_wrapper(self, *args, **kwargs) -> JSONRPCResult:
"""Background task that adquires a first access token and then will
guille commented 2024-11-16 13:56:45 +00:00
Collaborator

need to update this comment

need to update this comment
goodboy commented 2024-11-17 00:18:25 +00:00
Owner

Ok so yeah, as per a nice in depth paired bug scan today i think we made some good progress on gettin this bby almost fully workin again B)

Things we solved and i need to commit:

  • fix to history endpoint which ensures we only raise NoData for actual frame gaps and instead raise DataUnavailable when either there is zero history avail OR we’re already past the start_dt of the contract’s creation (as per OptionPair.creation_timestamp: int)
  • an import error in .venues due to a missing typing.Optional.
  • an adjustment to the .data.open_jsonrpc_session() to pass and proxy through msg_recv_timeout: float to the underlying open_autorecon_ws() call |_ since option markets generally are going to be wayy less active (vlm wise) and using a naive timeout on ws-msg rates is likely going to cause overdone connection resets thus killing the rpc session repeatedly - see notes in patch).

still TODO:

  • i’m pretty sure the live opts feeds aren’t quite working due to (as similarly solved recently for kucoin) the live quote msg 'fqme' field not matching that expected by the sampler actor? |_ needs some investigation but likely solving the fqme “superfluous ISO token” issue will mostly resolve this?

  • write up an issue on dealing with an approach to a piker-style FQME for generic options mkts.. |_ likely tie in with current discussion in our GH issue: https://github.com/pikers/piker/issues/467

Ok so yeah, as per a nice in depth paired bug scan today i think we made some good progress on gettin this bby almost fully workin again B) Things we solved and i need to commit: - fix to history endpoint which ensures we only raise `NoData` for actual frame gaps and instead raise `DataUnavailable` when either there is zero history avail OR we're already past the `start_dt` of the contract's creation (as per `OptionPair.creation_timestamp: int`) - an import error in `.venues` due to a missing `typing.Optional`. - an adjustment to the `.data.open_jsonrpc_session()` to pass and proxy through `msg_recv_timeout: float` to the underlying `open_autorecon_ws()` call |_ since option markets generally are going to be wayy less active (vlm wise) and using a naive timeout on ws-msg rates is likely going to cause overdone connection resets thus killing the rpc session repeatedly - see notes in patch). still TODO: - i'm pretty sure the live opts feeds aren't quite working due to (as similarly solved recently for `kucoin`) the live quote msg `'fqme'` field not matching that expected by the sampler actor? |_ needs some investigation but likely solving the fqme "superfluous ISO token" issue will mostly resolve this? - write up an issue on dealing with an approach to a piker-style FQME for generic options mkts.. |_ likely tie in with current discussion in our GH issue: https://github.com/pikers/piker/issues/467
goodboy commented 2024-11-19 22:54:54 +00:00
Owner

@ntorres ok so pushing up a sub-dev-branch for resolving a the history backfilling which is summarized in the sub-pr here ^^

@ntorres ok so pushing up a sub-dev-branch for resolving a the history backfilling which is summarized in the sub-pr here ^^
🚀 1
goodboy commented 2024-11-20 02:24:29 +00:00
Owner

Also added live feed support fully workin to that pr eh!

STILL TODO:

  • trying to use the mkt search seems to still be borked on eth?

    screenshot-2024-11-19_21-21-04.png

  • we need to rework the 'trade' msgs from crypto feed which are currently not formatted correctly for our “modern spec”:

    • FOLLOW UP todo is to make a msgspec contract for these msgs and use the new typed msg-spec features in tractor to enforce it Bo
Also added live feed support fully workin to that pr eh! #### STILL TODO: - [ ] trying to use the mkt search seems to still be borked on `eth`? ![screenshot-2024-11-19_21-21-04.png](/attachments/f6b58e95-9393-45c2-8a6b-ba7715f4cd91) - [ ] we need to rework the `'trade'` msgs from crypto feed which are currently not formatted correctly for our "modern spec": - [ ] FOLLOW UP todo is to make a `msgspec` contract for these msgs and use the new typed msg-spec features in `tractor` to enforce it Bo
screenshot-2024-11-19_21-21-04.png
73 KiB
This pull request can be merged automatically.
You are not authorized to merge this pull request.
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b deribit_fix nix-qt6-fix
git pull origin deribit_fix

Step 2:

Merge the changes and update on Gitea.
git checkout nix-qt6-fix
git merge --no-ff deribit_fix
git push origin nix-qt6-fix
Sign in to join this conversation.
Reviewers
No reviewers
guille
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: pikers/piker#8
There is no content yet.
Delete Branch "deribit_fix"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?