Commit Graph

13 Commits (6555ccfbba46c62d6c9dd72ee984161da52a29c7)

Author SHA1 Message Date
Tyler Goodlet cc5b21a7e6 Prep service mngr for move to `tractor.hilevel`
Given it's a fairly simple yet useful abstraction, it makes sense to
offer this sub-sys alongside the core `tractor` runtime lib.

Without going into extreme detail on the impl changes (it'll come in
the commit that moves to the other repo) here is the high level summary:
------ - ------
- rename `Services` -> `ServiceMngr` and use an factory `@acm`
  to guarantee a single-instance-per-actor using a niche approach for a
  singleton object using a default keyword-arg B)
  - the mod level `open_service_mngr()` and `get_service_mngr()` are the
    new allocation/access API.
- add a `ServiceMngr.start_service()` method which does the work of both
  spawning a new subactor (for the daemon) and uses its portal to start
  the mngr side supervision task.
- open actor/task nurseries inside the `@acm` allocator.

Adjust other dependent subsystems to match:
------ - ------
- use `open_service_mngr()` when first allocated in `open_pikerd()`.
- use `get_service_mngr()` instead of importing the class ref inside
  `.service.maybe_spawn_daemon()`, `.brokers._daemon.spawn_brokerd()`
  and `.data._sampling.spawn_samplerd()` using a `partial` to pack in
  the endpoint ctx kwargs (unpacked inside `.start_service()` XD).
2024-08-20 23:01:51 -03:00
Tyler Goodlet 7ab97fb21d Add marketstore client as storage-backend module
To kick off our (tsdb) storage backends this adds our first implementing
a new `Storage(Protocol)` client interface. Going foward, the top level
`.storage` pkg-module will now expose backend agnostic APIs and helpers
whilst specific backend implementations will adhere to that middle-ware
layer.

Deats:
- add `.storage.marketstore.Storage` as the first client implementation,
  moving all needed (import) dependencies out from
  `.service.marketstore` as well as `.ohlc_key_map` and `get_client()`.
- move root `conf.toml` loading from `.data.history` into
  `.storage.__init__.open_storage_client()` which now takes in a `name:
  str` and does all the work of loading the correct backend module, its
  config, and determining if a service-instance can be contacted and
  a client loaded; in the case where this fails we raise a new
  `StorageConnectionError`.
- add a new `.storage.get_storagemod()` just like we have for brokers.
- make `open_storage_client()` also return the backend module such that
  the history-data layer can make backend specific calls as needed (eg.
  ohlc_key_map).
- fall back to a basic non-tsdb backfill when `open_storage_client()`
  raises the new connection error.
2023-06-27 13:41:47 -04:00
Tyler Goodlet 29211b200d Start `piker.storage` subsys: cross-(ts)db middlewares
The plan is to offer multiple tsdb and other storage backends (for
a variety of use cases) and expose them similarly to how we do for
broker and data providers B)
2023-06-27 13:41:47 -04:00
Tyler Goodlet f0a346dcc3 Some linting fixes after trying out `ruff` 2023-05-24 17:25:23 -04:00
Tyler Goodlet bd8e4760d5 Port everything strictly to `Position.mkt` and `Flume.mkt` 2023-05-24 12:16:28 -04:00
Tyler Goodlet e7a172b656 Reimplement marketstore and elasticsearch daemons
Using the new `._ahab.start_ahab_service()` mngr of course, and now
support user config overrides (such that our defaults can be modified by
a keen user, say using a config file, or for testing). This is where the
functionality moved out of the `pikerd` init has been moved - instead of
being triggered by bool flag inputs to that factory.

For marketstore actually support overriding the entire yaml config via
runtime `_yaml_config_str: str` formatting with any passed user dict,
primarily focussing on supporting override of the sockaddrs for testing.
2023-05-23 14:16:02 -04:00
Tyler Goodlet 3294defee1 `fqme` adjustments to marketstore module
Mostly renaming from the old acronym. This also contains necessary
conf.toml loading in order to call `open_storage_client()` which now
does not have default network contact info.
2023-05-17 10:46:32 -04:00
Tyler Goodlet f8c8f63e87 Drop `Optional` usage from marketstore module 2023-05-09 14:49:46 -04:00
Tyler Goodlet 0a8dd7b6da Try to disable `snappy` compression on variables; it breaks everything XD 2023-05-09 14:49:46 -04:00
Tyler Goodlet a462de6f2d Use a single log for entire `.service` subsys 2023-05-09 14:49:26 -04:00
Tyler Goodlet 75b7a8b56e `marketstore`: Pull default socket from server config 2023-03-09 15:37:43 -05:00
Tyler Goodlet dd87d1142e Bump mkts timeout to 2s 2023-03-09 15:37:42 -05:00
Tyler Goodlet afac553ea2 Move all docker and external db code to `piker.service` 2023-03-09 15:37:42 -05:00