505 lines
15 KiB
Markdown
505 lines
15 KiB
Markdown
# macOS Compatibility Fixes for Piker/Tractor
|
|
|
|
This guide documents macOS-specific issues encountered when running `piker` on macOS and their solutions. These fixes address platform differences between Linux and macOS in areas like socket credentials, shared memory naming, and async runtime coordination.
|
|
|
|
## Table of Contents
|
|
|
|
1. [Socket Credential Passing](#1-socket-credential-passing)
|
|
2. [Shared Memory Name Length Limits](#2-shared-memory-name-length-limits)
|
|
3. [Shared Memory Cleanup Race Conditions](#3-shared-memory-cleanup-race-conditions)
|
|
4. [Async Runtime (Trio/AsyncIO) Coordination](#4-async-runtime-trioasyncio-coordination)
|
|
|
|
---
|
|
|
|
## 1. Socket Credential Passing
|
|
|
|
### Problem
|
|
|
|
On Linux, `tractor` uses `SO_PASSCRED` and `SO_PEERCRED` socket options for Unix domain socket credential passing. macOS doesn't support these constants, causing `AttributeError` when importing.
|
|
|
|
```python
|
|
# Linux code that fails on macOS
|
|
from socket import SO_PASSCRED, SO_PEERCRED # AttributeError on macOS
|
|
```
|
|
|
|
### Error Message
|
|
|
|
```
|
|
AttributeError: module 'socket' has no attribute 'SO_PASSCRED'
|
|
```
|
|
|
|
### Root Cause
|
|
|
|
- **Linux**: Uses `SO_PASSCRED` (to enable credential passing) and `SO_PEERCRED` (to retrieve peer credentials)
|
|
- **macOS**: Uses `LOCAL_PEERCRED` (value `0x0001`) instead, and doesn't require enabling credential passing
|
|
|
|
### Solution
|
|
|
|
Make the socket credential imports platform-conditional:
|
|
|
|
**File**: `tractor/ipc/_uds.py` (or equivalent in `piker` if duplicated)
|
|
|
|
```python
|
|
import sys
|
|
from socket import (
|
|
socket,
|
|
AF_UNIX,
|
|
SOCK_STREAM,
|
|
)
|
|
|
|
# Platform-specific credential passing constants
|
|
if sys.platform == 'linux':
|
|
from socket import SO_PASSCRED, SO_PEERCRED
|
|
elif sys.platform == 'darwin': # macOS
|
|
# macOS uses LOCAL_PEERCRED instead of SO_PEERCRED
|
|
# and doesn't need SO_PASSCRED
|
|
LOCAL_PEERCRED = 0x0001
|
|
SO_PEERCRED = LOCAL_PEERCRED # Alias for compatibility
|
|
SO_PASSCRED = None # Not needed on macOS
|
|
else:
|
|
# Other platforms - may need additional handling
|
|
SO_PASSCRED = None
|
|
SO_PEERCRED = None
|
|
|
|
# When creating a socket
|
|
if SO_PASSCRED is not None:
|
|
sock.setsockopt(SOL_SOCKET, SO_PASSCRED, 1)
|
|
|
|
# When getting peer credentials
|
|
if SO_PEERCRED is not None:
|
|
creds = sock.getsockopt(SOL_SOCKET, SO_PEERCRED, struct.calcsize('3i'))
|
|
```
|
|
|
|
### Implementation Notes
|
|
|
|
- The `LOCAL_PEERCRED` value `0x0001` is specific to macOS (from `<sys/un.h>`)
|
|
- macOS doesn't require explicitly enabling credential passing like Linux does
|
|
- Consider using `ctypes` or `cffi` for a more robust solution if available
|
|
|
|
---
|
|
|
|
## 2. Shared Memory Name Length Limits
|
|
|
|
### Problem
|
|
|
|
macOS limits POSIX shared memory names to **31 characters** (defined as `PSHMNAMLEN` in `<sys/posix_shm_internal.h>`). Piker generates long descriptive names that exceed this limit, causing `OSError`.
|
|
|
|
```python
|
|
# Long name that works on Linux but fails on macOS
|
|
shm_name = "piker_quoter_tsla.nasdaq.ib_hist_1m" # 39 chars - too long!
|
|
```
|
|
|
|
### Error Message
|
|
|
|
```
|
|
OSError: [Errno 63] File name too long: '/piker_quoter_tsla.nasdaq.ib_hist_1m'
|
|
```
|
|
|
|
### Root Cause
|
|
|
|
- **Linux**: Supports shared memory names up to 255 characters
|
|
- **macOS**: Limits to 31 characters (including leading `/`)
|
|
|
|
### Solution
|
|
|
|
Implement automatic name shortening for macOS while preserving the original key for lookups:
|
|
|
|
**File**: `piker/data/_sharedmem.py`
|
|
|
|
```python
|
|
import hashlib
|
|
import sys
|
|
|
|
def _shorten_key_for_macos(key: str) -> str:
|
|
'''
|
|
macOS has a 31 character limit for POSIX shared memory names.
|
|
Hash long keys to fit within this limit while maintaining uniqueness.
|
|
'''
|
|
# macOS shm_open() has a 31 char limit (PSHMNAMLEN)
|
|
# Use format: /p_<hash16> where hash is first 16 hex chars of sha256
|
|
# This gives us: / + p_ + 16 hex chars = 19 chars, well under limit
|
|
# We keep the 'p' prefix to indicate it's from piker
|
|
if len(key) <= 31:
|
|
return key
|
|
|
|
# Create a hash of the full key
|
|
key_hash = hashlib.sha256(key.encode()).hexdigest()[:16]
|
|
short_key = f'p_{key_hash}'
|
|
return short_key
|
|
|
|
|
|
class _Token(Struct, frozen=True):
|
|
'''
|
|
Internal representation of a shared memory "token"
|
|
which can be used to key a system wide post shm entry.
|
|
'''
|
|
shm_name: str # actual OS-level name (may be shortened on macOS)
|
|
shm_first_index_name: str
|
|
shm_last_index_name: str
|
|
dtype_descr: tuple
|
|
size: int # in struct-array index / row terms
|
|
key: str | None = None # original descriptive key (for lookup)
|
|
|
|
def __eq__(self, other) -> bool:
|
|
'''
|
|
Compare tokens based on shm names and dtype, ignoring the key field.
|
|
The key field is only used for lookups, not for token identity.
|
|
'''
|
|
if not isinstance(other, _Token):
|
|
return False
|
|
return (
|
|
self.shm_name == other.shm_name
|
|
and self.shm_first_index_name == other.shm_first_index_name
|
|
and self.shm_last_index_name == other.shm_last_index_name
|
|
and self.dtype_descr == other.dtype_descr
|
|
and self.size == other.size
|
|
)
|
|
|
|
def __hash__(self) -> int:
|
|
'''Hash based on the same fields used in __eq__'''
|
|
return hash((
|
|
self.shm_name,
|
|
self.shm_first_index_name,
|
|
self.shm_last_index_name,
|
|
self.dtype_descr,
|
|
self.size,
|
|
))
|
|
|
|
|
|
def _make_token(
|
|
key: str,
|
|
size: int,
|
|
dtype: np.dtype | None = None,
|
|
) -> _Token:
|
|
'''
|
|
Create a serializable token that uniquely identifies a shared memory segment.
|
|
'''
|
|
if dtype is None:
|
|
dtype = def_iohlcv_fields
|
|
|
|
# On macOS, shorten long keys to fit the 31-char limit
|
|
if sys.platform == 'darwin':
|
|
shm_name = _shorten_key_for_macos(key)
|
|
shm_first = _shorten_key_for_macos(key + "_first")
|
|
shm_last = _shorten_key_for_macos(key + "_last")
|
|
else:
|
|
shm_name = key
|
|
shm_first = key + "_first"
|
|
shm_last = key + "_last"
|
|
|
|
return _Token(
|
|
shm_name=shm_name,
|
|
shm_first_index_name=shm_first,
|
|
shm_last_index_name=shm_last,
|
|
dtype_descr=tuple(np.dtype(dtype).descr),
|
|
size=size,
|
|
key=key, # Store original key for lookup
|
|
)
|
|
```
|
|
|
|
### Key Design Decisions
|
|
|
|
1. **Hash-based shortening**: Uses SHA256 to ensure uniqueness and avoid collisions
|
|
2. **Preserve original key**: Store the original descriptive key in the `_Token` for debugging and lookups
|
|
3. **Custom equality**: The `__eq__` and `__hash__` methods ignore the `key` field to ensure tokens are compared by their actual shm properties
|
|
4. **Platform detection**: Only applies shortening on macOS (`sys.platform == 'darwin'`)
|
|
|
|
### Edge Cases to Consider
|
|
|
|
- Token serialization across processes (the `key` field must survive IPC)
|
|
- Token lookup in dictionaries and caches
|
|
- Debugging output (use `key` field for human-readable names)
|
|
|
|
---
|
|
|
|
## 3. Shared Memory Cleanup Race Conditions
|
|
|
|
### Problem
|
|
|
|
During teardown, shared memory segments may be unlinked by one process while another is still trying to clean them up, causing `FileNotFoundError` to crash the application.
|
|
|
|
### Error Message
|
|
|
|
```
|
|
FileNotFoundError: [Errno 2] No such file or directory: '/p_74c86c7228dd773b'
|
|
```
|
|
|
|
### Root Cause
|
|
|
|
In multi-process architectures like `tractor`, multiple processes may attempt to clean up shared resources simultaneously. Race conditions during shutdown can cause:
|
|
|
|
1. Process A unlinks the shared memory
|
|
2. Process B tries to unlink the same memory → `FileNotFoundError`
|
|
3. Uncaught exception crashes Process B
|
|
|
|
### Solution
|
|
|
|
Add defensive error handling to catch and log cleanup races:
|
|
|
|
**File**: `piker/data/_sharedmem.py`
|
|
|
|
```python
|
|
class ShmArray:
|
|
# ... existing code ...
|
|
|
|
def destroy(self) -> None:
|
|
'''
|
|
Destroy the shared memory segment and cleanup OS resources.
|
|
'''
|
|
if _USE_POSIX:
|
|
# We manually unlink to bypass all the "resource tracker"
|
|
# nonsense meant for non-SC systems.
|
|
shm = self._shm
|
|
name = shm.name
|
|
try:
|
|
shm_unlink(name)
|
|
except FileNotFoundError:
|
|
# Might be a teardown race where another process
|
|
# already unlinked it - this is fine, just log it
|
|
log.warning(f'Shm for {name} already unlinked?')
|
|
|
|
# Also cleanup the index counters
|
|
if hasattr(self, '_first'):
|
|
try:
|
|
self._first.destroy()
|
|
except FileNotFoundError:
|
|
log.warning(f'First index shm already unlinked?')
|
|
|
|
if hasattr(self, '_last'):
|
|
try:
|
|
self._last.destroy()
|
|
except FileNotFoundError:
|
|
log.warning(f'Last index shm already unlinked?')
|
|
|
|
|
|
class SharedInt:
|
|
# ... existing code ...
|
|
|
|
def destroy(self) -> None:
|
|
if _USE_POSIX:
|
|
# We manually unlink to bypass all the "resource tracker"
|
|
# nonsense meant for non-SC systems.
|
|
name = self._shm.name
|
|
try:
|
|
shm_unlink(name)
|
|
except FileNotFoundError:
|
|
# might be a teardown race here?
|
|
log.warning(f'Shm for {name} already unlinked?')
|
|
```
|
|
|
|
### Implementation Notes
|
|
|
|
- This fix is platform-agnostic but particularly important on macOS where the shortened names make debugging harder
|
|
- The warnings help identify cleanup races during development
|
|
- Consider adding metrics/counters if cleanup races become frequent
|
|
|
|
---
|
|
|
|
## 4. Async Runtime (Trio/AsyncIO) Coordination
|
|
|
|
### Problem
|
|
|
|
The `TrioTaskExited` error occurs when trio tasks are cancelled while asyncio tasks are still running, indicating improper coordination between the two async runtimes.
|
|
|
|
### Error Message
|
|
|
|
```
|
|
tractor._exceptions.TrioTaskExited: but the child `asyncio` task is still running?
|
|
>>
|
|
|_<Task pending name='Task-2' coro=<wait_on_coro_final_result()> ...>
|
|
```
|
|
|
|
### Root Cause
|
|
|
|
`tractor` uses "guest mode" to run trio as a guest in asyncio's event loop (or vice versa). The error occurs when:
|
|
|
|
1. A trio task is cancelled (e.g., user closes the UI)
|
|
2. The cancellation propagates to cleanup handlers
|
|
3. Cleanup tries to exit while asyncio tasks are still running
|
|
4. The `translate_aio_errors` context manager detects this inconsistent state
|
|
|
|
### Current State
|
|
|
|
This issue is **partially resolved** by the other fixes (socket credentials and shared memory), which eliminate the underlying errors that trigger premature cancellation. However, it may still occur in edge cases.
|
|
|
|
### Potential Solutions
|
|
|
|
#### Option 1: Improve Cancellation Propagation (Tractor-level)
|
|
|
|
**File**: `tractor/to_asyncio.py`
|
|
|
|
```python
|
|
async def translate_aio_errors(
|
|
chan,
|
|
wait_on_aio_task: bool = False,
|
|
suppress_graceful_exits: bool = False,
|
|
):
|
|
'''
|
|
Context manager to translate asyncio errors to trio equivalents.
|
|
'''
|
|
try:
|
|
yield
|
|
except trio.Cancelled:
|
|
# When trio is cancelled, ensure asyncio tasks are also cancelled
|
|
if wait_on_aio_task:
|
|
# Give asyncio tasks a chance to cleanup
|
|
await trio.lowlevel.checkpoint()
|
|
|
|
# Check if asyncio task is still running
|
|
if aio_task and not aio_task.done():
|
|
# Cancel it gracefully
|
|
aio_task.cancel()
|
|
|
|
# Wait briefly for cancellation
|
|
with trio.move_on_after(0.5): # 500ms timeout
|
|
await wait_for_aio_task_completion(aio_task)
|
|
|
|
raise # Re-raise the cancellation
|
|
```
|
|
|
|
#### Option 2: Proper Shutdown Sequence (Application-level)
|
|
|
|
**File**: `piker/brokers/ib/api.py` (or similar broker modules)
|
|
|
|
```python
|
|
async def load_clients_for_trio(
|
|
client: Client,
|
|
...
|
|
) -> None:
|
|
'''
|
|
Load asyncio client and keep it running for trio.
|
|
'''
|
|
try:
|
|
# Setup client
|
|
await client.connect()
|
|
|
|
# Keep alive - but make it cancellable
|
|
await trio.sleep_forever()
|
|
|
|
except trio.Cancelled:
|
|
# Explicit cleanup before propagating cancellation
|
|
log.info("Shutting down asyncio client gracefully")
|
|
|
|
# Disconnect client
|
|
if client.isConnected():
|
|
await client.disconnect()
|
|
|
|
# Small delay to let asyncio cleanup
|
|
await trio.sleep(0.1)
|
|
|
|
raise # Now safe to propagate
|
|
```
|
|
|
|
#### Option 3: Detection and Warning (Current Approach)
|
|
|
|
The current code detects the issue and raises a clear error. This is acceptable if:
|
|
1. The error is rare (only during abnormal shutdown)
|
|
2. It doesn't cause data loss
|
|
3. Logs provide enough info for debugging
|
|
|
|
### Recommended Approach
|
|
|
|
For **piker**: Implement Option 2 (proper shutdown sequence) in broker modules where asyncio is used.
|
|
|
|
For **tractor**: Consider Option 1 (improved cancellation propagation) as a library-level enhancement.
|
|
|
|
### Testing
|
|
|
|
Test the fix by:
|
|
|
|
```python
|
|
# Test graceful shutdown
|
|
async def test_asyncio_trio_shutdown():
|
|
async with open_channel_from(...) as (first, chan):
|
|
# Do some work
|
|
await chan.send(msg)
|
|
|
|
# Trigger cancellation
|
|
raise KeyboardInterrupt
|
|
|
|
# Should cleanup without TrioTaskExited error
|
|
```
|
|
|
|
---
|
|
|
|
## Summary of Changes
|
|
|
|
### Files Modified in Piker
|
|
|
|
1. **`piker/data/_sharedmem.py`**
|
|
- Added `_shorten_key_for_macos()` function
|
|
- Modified `_Token` class to store original `key`
|
|
- Modified `_make_token()` to use shortened names on macOS
|
|
- Added `FileNotFoundError` handling in `destroy()` methods
|
|
|
|
2. **`piker/ui/_display.py`**
|
|
- Removed assertion that checked for 'hist' in shm name (incompatible with shortened names)
|
|
|
|
### Files to Modify in Tractor (Recommended)
|
|
|
|
1. **`tractor/ipc/_uds.py`**
|
|
- Make socket credential imports platform-conditional
|
|
- Handle macOS-specific `LOCAL_PEERCRED`
|
|
|
|
2. **`tractor/to_asyncio.py`** (Optional)
|
|
- Improve cancellation propagation between trio and asyncio
|
|
- Add graceful shutdown timeout for asyncio tasks
|
|
|
|
### Platform Detection Pattern
|
|
|
|
Use this pattern consistently:
|
|
|
|
```python
|
|
import sys
|
|
|
|
if sys.platform == 'darwin': # macOS
|
|
# macOS-specific code
|
|
pass
|
|
elif sys.platform == 'linux': # Linux
|
|
# Linux-specific code
|
|
pass
|
|
else:
|
|
# Other platforms / fallback
|
|
pass
|
|
```
|
|
|
|
### Testing Checklist
|
|
|
|
- [ ] Test on macOS (Darwin)
|
|
- [ ] Test on Linux
|
|
- [ ] Test shared memory with names > 31 chars
|
|
- [ ] Test multi-process cleanup race conditions
|
|
- [ ] Test graceful shutdown (Ctrl+C)
|
|
- [ ] Test abnormal shutdown (kill signal)
|
|
- [ ] Verify no memory leaks (check `/dev/shm` on Linux, `ipcs -m` on macOS)
|
|
|
|
---
|
|
|
|
## Additional Resources
|
|
|
|
- **macOS System Headers**:
|
|
- `/usr/include/sys/un.h` - Unix domain socket constants
|
|
- `/usr/include/sys/posix_shm_internal.h` - Shared memory limits
|
|
|
|
- **Python Documentation**:
|
|
- [`socket` module](https://docs.python.org/3/library/socket.html)
|
|
- [`multiprocessing.shared_memory`](https://docs.python.org/3/library/multiprocessing.shared_memory.html)
|
|
|
|
- **Trio/AsyncIO**:
|
|
- [Trio Guest Mode](https://trio.readthedocs.io/en/stable/reference-lowlevel.html#using-guest-mode-to-run-trio-on-top-of-other-event-loops)
|
|
- [Tractor Documentation](https://github.com/goodboy/tractor)
|
|
|
|
---
|
|
|
|
## Contributing
|
|
|
|
When implementing these fixes in your own project:
|
|
|
|
1. **Test thoroughly** on both macOS and Linux
|
|
2. **Add platform guards** to prevent cross-platform breakage
|
|
3. **Document platform-specific behavior** in code comments
|
|
4. **Consider CI/CD** testing on multiple platforms
|
|
5. **Handle edge cases** gracefully with proper logging
|
|
|
|
If you find additional macOS-specific issues, please contribute to this guide!
|