Commit Graph

78 Commits (a4bd51a01b4488143e325cf8d81930f198f6c6a1)

Author SHA1 Message Date
Guillermo Rodriguez 47bf45f30e
Merge pull request #464 from pikers/elasticsearch_integration
Elasticsearch integration
2023-02-24 16:38:37 -03:00
Esmeralda Gallardo b96e2c314a
Minor style changes and removed unnecesary comments 2023-02-24 15:11:15 -03:00
Esmeralda Gallardo f96d6a04b6
Fixed UnboundLocalError on _ahab. Added test for marketstore's initialization 2023-02-22 13:28:07 -03:00
Guillermo Rodriguez acc6249d88
Remove unnesesary arguments to some pikerd functions, fix container init error
by switching from log reading to quering es health endpoint, fix install on ci
and add more logging.
2023-02-21 20:45:10 -03:00
Tyler Goodlet 0b678c97f4 Pass `loglevel: str` cli value through to service tests 2023-02-21 12:02:26 -05:00
Esmeralda Gallardo 4122c482ba
Added new tests for elasticsearch's and marketstore's initialization and stop 2023-02-21 13:34:29 -03:00
Esmeralda Gallardo b5cdf14036
Modified elasticsearch file name to 'elastic' to avoid name errors. Applied changes suggested in the pr. 2023-02-21 13:34:29 -03:00
Tyler Goodlet b3400f0d9c Add `loglevel: str` fixture, passthrough to `open_test_pikerd()` 2023-02-21 10:54:18 -05:00
Tyler Goodlet cd3e9b1b2a Move quest fixtures to test mod, clean out old travis fixture 2023-02-21 10:54:18 -05:00
Tyler Goodlet 69074f4fa5 Bump up service tree spawn timeout a couple secs 2023-01-26 17:59:25 -05:00
Tyler Goodlet fe4fb37b58 Add service tree tests for data-feeds and the EMS 2023-01-24 15:15:27 -05:00
Tyler Goodlet 7cfd431a2b Yield `Services` in `open_test_pikerd()` fixture 2023-01-24 15:15:27 -05:00
Tyler Goodlet 37e0ec7b7d Assert fixture caller is `pikerd` 2023-01-13 13:21:49 -05:00
Tyler Goodlet 6a1bb13feb Add base `pikerd` service tree custom check test 2023-01-13 13:21:49 -05:00
Tyler Goodlet 593db0ed0d Only run `kraken` feed tests in CI, use `open_test_pikerd()` 2023-01-10 11:09:19 -05:00
Tyler Goodlet 06622105cd Add a `open_test_pikerd()` acm fixture for easy booting of the service stack 2023-01-10 11:09:19 -05:00
Tyler Goodlet f232d6d4ee Add `ci_env` detector fixture 2023-01-10 11:09:19 -05:00
Tyler Goodlet b7e1443618 Use ETH on kraken to ensure enough quotes 2023-01-10 11:09:19 -05:00
Tyler Goodlet 5d021ffb85 Bump up timeout on multi-feed test for CI 2023-01-10 11:09:19 -05:00
Tyler Goodlet e348968113 Add multi-broker streaming test using both `binance` and `kraken` 2023-01-10 11:09:19 -05:00
Tyler Goodlet 36868bb86e Add `kraken` test, ensure single broker-provider for now 2023-01-10 11:09:19 -05:00
Tyler Goodlet bb6452b969 Further feed syncing fixes wrt to `Flumes`
Sync per-symbol sampler loop start to subscription registers such that
the loop can't start until the consumer's stream subscription is added;
the task-sync uses a `trio.Event`. This patch also drops a ton of
commented cruft.

Further adjustments needed to get parity with prior functionality:
- pass init msg 'symbol_info' field to the `Symbol.broker_info: dict`.
- ensure the `_FeedsBus._subscriptions` table uses the broker specific
  (without brokername suffix) as keys for lookup so that the sampler
  loop doesn't have to append in the brokername as a suffix.
- ensure the `open_feed_bus()` flumes-table-msg returned sent by
  `tractor.Context.started()` uses the `.to_msg()` form of all flume
  structs.
- ensure `maybe_open_feed()` uses `tractor.MsgStream.subscribe()` on all
  `Flume.stream`s on cache hits using the
  `tractor.trionics.gather_contexts()` helper.
2023-01-10 11:09:19 -05:00
Tyler Goodlet 32b36aa042 Expect init startup quotes from each symbol 2023-01-10 11:09:19 -05:00
Tyler Goodlet 18dc8b08e4 First draft aggregate feedz support
Orient shm-flow-arrays around the new idea of a `Flume` which provides
access, mgmt and basic measure of real-time data flow sets (see water
flow management semantics).

- We discard the previous idea of a "init message" which contained all
  the shm attachment info and instead send a startup message full of
  `Flume.to_msg()`s which are symmetrically loaded on the caller actor
  side.

- Create data-flows "entries" for every passed in fqsn such that the consumer gets back
  streams and shm for each, now all wrapped in `Flume` types. For now we
  allocate `brokermod.stream_quotes()` tasks 1-to-1 for each fqsn
  (instead of expecting each backend to do multi-plexing, though we
  might want that eventually) as well a `_FeedsBus._subscriber` entry
  for each. The pause/resume management loop is adjusted to match.
  Previously `Feed`s were  allocated 1-to-1 with each fqsn.

- Make `Feed` a `Struct` subtype instead of a `@dataclass` and move all
  flow specific attrs to the new `Flume`:
  - move `.index_stream()`, `.get_ds_info()` to `Flume`.
  - drop `.receive()`: each fqsn entry will now require knowledge of
    separate streams by feed users.
  - add multi-fqsn tables: `.flumes`, `.streams` which point to the
    appropriate per-symbol entries.

- Async load all `Flume`s from all contexts and all quote streams using
  `tractor.trionics.gather_contexts()` on the client `open_feed()` side.

- Update feeds test to include streaming 2 symbols on the same (binance)
  backend.
2023-01-10 11:09:18 -05:00
Tyler Goodlet c7d5db5f90 Start data feed layer test suite
Initial test that starts a `binance` feed and reads the quote messages
alongside shm buffers for 1s and 1m OHLC; just prints to console for
now.

Template out parametrization for multi-symbol quote-multiplexed feeds
which coming soon B)
2023-01-10 11:09:18 -05:00
Tyler Goodlet 1bf1965a8b Drop `tractor.log` level override fixture 2023-01-10 11:09:18 -05:00
Tyler Goodlet 63e9af002d Drop `tractor.testing` import in qt tests 2022-10-28 16:09:55 -04:00
Tyler Goodlet 2312b6aeb2 Fix conftest config mod import 2021-09-11 18:15:42 -04:00
Tyler Goodlet 088ea8c3ec Skip CLI tests due to questrade dependency for now 2021-05-22 15:12:12 -04:00
Tyler Goodlet 0228db7811 Skip questrade tests for now
They need to be run with a local private API key and haven't been ported
to latest data apis anyway. It's also a broker most peeps aren't going
to be using any time soon.
2021-05-20 16:11:49 -04:00
Tyler Goodlet 44eb9aa9ff Create a blank config if none exists 2019-03-18 00:04:34 -04:00
Tyler Goodlet 88bb5b9f05 Handle file not found 2019-03-17 23:22:00 -04:00
Tyler Goodlet b99c5d2abe Use confdir if provided even when in CI 2019-03-17 23:04:04 -04:00
Tyler Goodlet d0b1d0954f Catch multi-errors too? 2019-03-03 11:01:34 -05:00
Tyler Goodlet 6cbea10b83 Forgot an error 2019-02-26 19:15:53 -05:00
Tyler Goodlet 2306799110 Speed bump quoter a slight bit 2019-02-25 22:40:19 -05:00
Tyler Goodlet 1fd033d351 Token should always be from a practice account 2019-02-25 22:12:33 -05:00
Tyler Goodlet 414734f803 Add travisCI specific test suite integration
Questrade is the default broker backend (for now) so the CI
can run using a practice account token handed down through an
env variable. If we add a cached directory to the build then the token
should remain persistent in the brokers config and will only need to be
updated if something goes wrong.

Also, add a `--confdir` flag for pytest much in the same way as for
the `piker` cli.
2019-02-25 20:23:20 -05:00
Tyler Goodlet 1a0427db08 Use `DataFeed` api in streaming tests 2019-02-21 23:10:24 -05:00
Tyler Goodlet 57bef52438 Capture the right logger 2019-02-10 17:29:08 -05:00
Tyler Goodlet 308ceb1772 Use search method for `piker api` test 2019-02-10 17:28:43 -05:00
Tyler Goodlet ea289540b3 Add a test to verify auth endpoint "locking" 2019-02-09 21:58:49 -05:00
Tyler Goodlet 5339f754a1 Add a token refresh test that exhibits an API race issue 2019-02-04 00:16:16 -05:00
Tyler Goodlet 6cc8b4cc2f Test duplicate feed type quoting 2018-12-23 21:27:47 -05:00
Tyler Goodlet 1d1be9dd77 Include option stream subscription change in test 2018-12-11 17:10:36 -05:00
Tyler Goodlet 9c7ca84fef Include strike and expiry in option quotes 2018-12-09 13:40:26 -05:00
Tyler Goodlet 12d5627860 Aggregate streaming tests and test stocks + options together 2018-12-01 16:14:33 -05:00
Tyler Goodlet 2df5c76828 Adjust cli tests for new quotes list output 2018-12-01 16:13:15 -05:00
Tyler Goodlet 15dec65ba1 Add an options streaming test 2018-11-30 08:18:54 -05:00
Tyler Goodlet 48a9c389c5 Add loglevel support to tests 2018-11-30 08:18:13 -05:00