goodboy
/
tractor
1
0
Fork 1
Code Issues 1 Pull Requests 4 Packages Projects Releases Wiki Activity
tractor/tests/test_docs_examples.py

144 lines
4.3 KiB
Python
Raw Normal View History

Plan asyncio test set
2021-11-05 14:42:43 +00:00
'''
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
Let's make sure them docs work yah?
Plan asyncio test set
2021-11-05 14:42:43 +00:00
'''
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
from contextlib import contextmanager
Search for guard and strip instead of hardcoding
2020-02-10 17:59:44 +00:00
import itertools
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
import os
import sys
import subprocess
import platform
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
import shutil
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
import pytest
Factor `repodir()` helper into conftest.py
2020-10-13 18:49:31 +00:00
from conftest import repodir
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
def examples_dir():
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
"""Return the abspath to the examples directory.
"""
return os.path.join(repodir(), 'examples')
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
@pytest.fixture
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
def run_example_in_subproc(loglevel, testdir, arb_addr):
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
@contextmanager
def run(script_code):
kwargs = dict()
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
if platform.system() == 'Windows':
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
# on windows we need to create a special __main__.py which will
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
# be executed with ``python -m <modulename>`` on windows..
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
shutil.copyfile(
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
os.path.join(examples_dir(), '__main__.py'),
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
os.path.join(str(testdir), '__main__.py')
)
Search for guard and strip instead of hardcoding
2020-02-10 17:59:44 +00:00
# drop the ``if __name__ == '__main__'`` guard onwards from
# the *NIX version of each script
windows_script_lines = itertools.takewhile(
lambda line: "if __name__ ==" not in line,
script_code.splitlines()
)
script_code = '\n'.join(windows_script_lines)
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
script_file = testdir.makefile('.py', script_code)
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
# without this, tests hang on windows forever
kwargs['creationflags'] = subprocess.CREATE_NEW_PROCESS_GROUP
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
# run the testdir "libary module" as a script
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
cmdargs = [
sys.executable,
'-m',
# use the "module name" of this "package"
'test_example'
]
else:
script_file = testdir.makefile('.py', script_code)
cmdargs = [
sys.executable,
str(script_file),
]
Fix some borked tests
2021-01-09 01:40:49 +00:00
# XXX: BE FOREVER WARNED: if you enable lots of tractor logging
# in the subprocess it may cause infinite blocking on the pipes
# due to backpressure!!!
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
proc = testdir.popen(
cmdargs,
**kwargs,
)
assert not proc.returncode
yield proc
proc.wait()
assert proc.returncode == 0
yield run
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
@pytest.mark.parametrize(
'example_script',
Run parallel examples
2021-01-18 02:24:56 +00:00
# walk yields: (dirpath, dirnames, filenames)
Move pydantic-click hang example to new dir, skip in test suite
2022-06-26 19:26:18 +00:00
[
(p[0], f) for p in os.walk(examples_dir()) for f in p[2]
Run parallel examples
2021-01-18 02:24:56 +00:00
if '__' not in f
Don't run stdlib example as part of test set
2021-02-21 20:41:21 +00:00
and f[0] != '_'
Move pydantic-click hang example to new dir, skip in test suite
2022-06-26 19:26:18 +00:00
and 'debugging' not in p[0]
and 'integration' not in p[0]
],
Add 2-way streaming example to readme and scripts
2021-07-31 16:10:25 +00:00
Don't run stdlib example as part of test set
2021-02-21 20:41:21 +00:00
ids=lambda t: t[1],
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
)
def test_example(run_example_in_subproc, example_script):
Always print any std streams to console in docs examples tests
2022-11-06 21:42:13 +00:00
'''
Load and run scripts from this repo's ``examples/`` dir as a user
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
would copy and pasing them into their editor.
Add initial root actor debugger tests
2020-10-03 23:35:18 +00:00
On windows a little more "finessing" is done to make
``multiprocessing`` play nice: we copy the ``__main__.py`` into the
test directory and invoke the script as a module with ``python -m
test_example``.
Always print any std streams to console in docs examples tests
2022-11-06 21:42:13 +00:00
'''
Run parallel examples
2021-01-18 02:24:56 +00:00
ex_file = os.path.join(*example_script)
Add 2-way streaming example to readme and scripts
2021-07-31 16:10:25 +00:00
if 'rpc_bidir_streaming' in ex_file and sys.version_info < (3, 9):
pytest.skip("2-way streaming example requires py3.9 async with syntax")
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
with open(ex_file, 'r') as ex:
code = ex.read()
with run_example_in_subproc(code) as proc:
Always print any std streams to console in docs examples tests
2022-11-06 21:42:13 +00:00
try:
proc.wait(timeout=5)
finally:
err = proc.stderr.read()
Guard against `asyncio` cancelled logged to console
2022-07-11 01:06:21 +00:00
errmsg = err.decode()
Always print any std streams to console in docs examples tests
2022-11-06 21:42:13 +00:00
out = proc.stdout.read()
outmsg = out.decode()
if out:
print(f'STDOUT: {out.decode()}')
# if we get some gnarly output let's aggregate and raise
if err:
print(f'STDERR:\n{errmsg}')
errmsg = err.decode()
errlines = errmsg.splitlines()
last_error = errlines[-1]
if (
'Error' in last_error
# XXX: currently we print this to console, but maybe
# shouldn't eventually once we figure out what's
# a better way to be explicit about aio side
# cancels?
and 'asyncio.exceptions.CancelledError' not in last_error
):
raise Exception(errmsg)
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
assert proc.returncode == 0