jc211
/
tractor
forked from goodboy/tractor
1
0
Fork 0
Code Pull Requests Activity
tractor/tests/test_docs_examples.py

115 lines
3.3 KiB
Python
Raw Permalink Normal View History

Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
"""
Let's make sure them docs work yah?
"""
from contextlib import contextmanager
Search for guard and strip instead of hardcoding
2020-02-10 17:59:44 +00:00
import itertools
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
import os
import sys
import subprocess
import platform
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
import shutil
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
import pytest
Factor `repodir()` helper into conftest.py
2020-10-13 18:49:31 +00:00
from conftest import repodir
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
def examples_dir():
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
"""Return the abspath to the examples directory.
"""
return os.path.join(repodir(), 'examples')
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
@pytest.fixture
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
def run_example_in_subproc(loglevel, testdir, arb_addr):
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
@contextmanager
def run(script_code):
kwargs = dict()
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
if platform.system() == 'Windows':
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
# on windows we need to create a special __main__.py which will
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
# be executed with ``python -m <modulename>`` on windows..
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
shutil.copyfile(
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
os.path.join(examples_dir(), '__main__.py'),
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
os.path.join(str(testdir), '__main__.py')
)
Search for guard and strip instead of hardcoding
2020-02-10 17:59:44 +00:00
# drop the ``if __name__ == '__main__'`` guard onwards from
# the *NIX version of each script
windows_script_lines = itertools.takewhile(
lambda line: "if __name__ ==" not in line,
script_code.splitlines()
)
script_code = '\n'.join(windows_script_lines)
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
script_file = testdir.makefile('.py', script_code)
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
# without this, tests hang on windows forever
kwargs['creationflags'] = subprocess.CREATE_NEW_PROCESS_GROUP
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
# run the testdir "libary module" as a script
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
cmdargs = [
sys.executable,
'-m',
# use the "module name" of this "package"
'test_example'
]
else:
script_file = testdir.makefile('.py', script_code)
cmdargs = [
sys.executable,
str(script_file),
]
Fix some borked tests
2021-01-09 01:40:49 +00:00
# XXX: BE FOREVER WARNED: if you enable lots of tractor logging
# in the subprocess it may cause infinite blocking on the pipes
# due to backpressure!!!
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
proc = testdir.popen(
cmdargs,
**kwargs,
)
assert not proc.returncode
yield proc
proc.wait()
assert proc.returncode == 0
yield run
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
@pytest.mark.parametrize(
'example_script',
Add initial root actor debugger tests
2020-10-03 23:35:18 +00:00
[
f for f in os.listdir(examples_dir())
if (
('__' not in f) and
('debugging' not in f)
)
],
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
)
def test_example(run_example_in_subproc, example_script):
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
"""Load and run scripts from this repo's ``examples/`` dir as a user
would copy and pasing them into their editor.
Add initial root actor debugger tests
2020-10-03 23:35:18 +00:00
On windows a little more "finessing" is done to make
``multiprocessing`` play nice: we copy the ``__main__.py`` into the
test directory and invoke the script as a module with ``python -m
test_example``.
Better document the window's gotcha solution in test code
2020-02-09 19:59:22 +00:00
"""
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
ex_file = os.path.join(examples_dir(), example_script)
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
with open(ex_file, 'r') as ex:
code = ex.read()
with run_example_in_subproc(code) as proc:
proc.wait()
err, _ = proc.stderr.read(), proc.stdout.read()
Fix some borked tests
2021-01-09 01:40:49 +00:00
# print(f'STDERR: {err}')
# print(f'STDOUT: {out}')
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
# if we get some gnarly output let's aggregate and raise
Only error the exs test when "Error" in last line of output
2020-02-10 17:07:32 +00:00
errmsg = err.decode()
errlines = errmsg.splitlines()
if err and 'Error' in errlines[-1]:
raise Exception(errmsg)
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
assert proc.returncode == 0