goodboy
/
tractor
1
0
Fork 1
Code Issues 1 Pull Requests 4 Packages Projects Releases Wiki Activity
tractor/tests/test_docs_examples.py

136 lines
3.9 KiB
Python
Raw Permalink Normal View History

Plan asyncio test set
2021-11-05 14:42:43 +00:00
'''
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
Let's make sure them docs work yah?
Plan asyncio test set
2021-11-05 14:42:43 +00:00
'''
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
from contextlib import contextmanager
Search for guard and strip instead of hardcoding
2020-02-10 17:59:44 +00:00
import itertools
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
import os
import sys
import subprocess
import platform
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
import shutil
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
import pytest
Adjust other examples tests to expect `pathlib` objects
2023-01-27 21:41:16 +00:00
from conftest import (
examples_dir,
)
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
@pytest.fixture
Adjust other examples tests to expect `pathlib` objects
2023-01-27 21:41:16 +00:00
def run_example_in_subproc(
loglevel: str,
testdir,
arb_addr: tuple[str, int],
):
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
@contextmanager
def run(script_code):
kwargs = dict()
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
if platform.system() == 'Windows':
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
# on windows we need to create a special __main__.py which will
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
# be executed with ``python -m <modulename>`` on windows..
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
shutil.copyfile(
Adjust other examples tests to expect `pathlib` objects
2023-01-27 21:41:16 +00:00
examples_dir() / '__main__.py',
str(testdir / '__main__.py'),
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
)
Search for guard and strip instead of hardcoding
2020-02-10 17:59:44 +00:00
# drop the ``if __name__ == '__main__'`` guard onwards from
# the *NIX version of each script
windows_script_lines = itertools.takewhile(
lambda line: "if __name__ ==" not in line,
script_code.splitlines()
)
script_code = '\n'.join(windows_script_lines)
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
script_file = testdir.makefile('.py', script_code)
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
# without this, tests hang on windows forever
kwargs['creationflags'] = subprocess.CREATE_NEW_PROCESS_GROUP
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
# run the testdir "libary module" as a script
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
cmdargs = [
sys.executable,
'-m',
# use the "module name" of this "package"
'test_example'
]
else:
script_file = testdir.makefile('.py', script_code)
cmdargs = [
sys.executable,
str(script_file),
]
Fix some borked tests
2021-01-09 01:40:49 +00:00
# XXX: BE FOREVER WARNED: if you enable lots of tractor logging
# in the subprocess it may cause infinite blocking on the pipes
# due to backpressure!!!
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
proc = testdir.popen(
cmdargs,
**kwargs,
)
assert not proc.returncode
yield proc
proc.wait()
assert proc.returncode == 0
yield run
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
@pytest.mark.parametrize(
'example_script',
Run parallel examples
2021-01-18 02:24:56 +00:00
# walk yields: (dirpath, dirnames, filenames)
Move pydantic-click hang example to new dir, skip in test suite
2022-06-26 19:26:18 +00:00
[
(p[0], f) for p in os.walk(examples_dir()) for f in p[2]
Run parallel examples
2021-01-18 02:24:56 +00:00
if '__' not in f
Don't run stdlib example as part of test set
2021-02-21 20:41:21 +00:00
and f[0] != '_'
Move pydantic-click hang example to new dir, skip in test suite
2022-06-26 19:26:18 +00:00
and 'debugging' not in p[0]
and 'integration' not in p[0]
Skip `advanced_faults/` subset in docs examples tests
2023-01-27 22:17:17 +00:00
and 'advanced_faults' not in p[0]
Move pydantic-click hang example to new dir, skip in test suite
2022-06-26 19:26:18 +00:00
],
Add 2-way streaming example to readme and scripts
2021-07-31 16:10:25 +00:00
Don't run stdlib example as part of test set
2021-02-21 20:41:21 +00:00
ids=lambda t: t[1],
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
)
def test_example(run_example_in_subproc, example_script):
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
"""Load and run scripts from this repo's ``examples/`` dir as a user
would copy and pasing them into their editor.
Add initial root actor debugger tests
2020-10-03 23:35:18 +00:00
On windows a little more "finessing" is done to make
``multiprocessing`` play nice: we copy the ``__main__.py`` into the
test directory and invoke the script as a module with ``python -m
test_example``.
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
"""
Run parallel examples
2021-01-18 02:24:56 +00:00
ex_file = os.path.join(*example_script)
Add 2-way streaming example to readme and scripts
2021-07-31 16:10:25 +00:00
if 'rpc_bidir_streaming' in ex_file and sys.version_info < (3, 9):
pytest.skip("2-way streaming example requires py3.9 async with syntax")
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
with open(ex_file, 'r') as ex:
code = ex.read()
with run_example_in_subproc(code) as proc:
proc.wait()
err, _ = proc.stderr.read(), proc.stdout.read()
Fix some borked tests
2021-01-09 01:40:49 +00:00
# print(f'STDERR: {err}')
# print(f'STDOUT: {out}')
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
# if we get some gnarly output let's aggregate and raise
Guard against `asyncio` cancelled logged to console
2022-07-11 01:06:21 +00:00
if err:
errmsg = err.decode()
errlines = errmsg.splitlines()
last_error = errlines[-1]
if (
'Error' in last_error
# XXX: currently we print this to console, but maybe
# shouldn't eventually once we figure out what's
# a better way to be explicit about aio side
# cancels?
and 'asyncio.exceptions.CancelledError' not in last_error
):
raise Exception(errmsg)
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
assert proc.returncode == 0