jc211
/
tractor
forked from goodboy/tractor
1
0
Fork 0
Code Pull Requests Activity
tractor/tests/test_docs_examples.py

94 lines
2.5 KiB
Python
Raw Normal View History

Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
"""
Let's make sure them docs work yah?
"""
from contextlib import contextmanager
import os
import sys
import subprocess
import platform
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
import shutil
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
import pytest
def confdir():
dirname = os.path.dirname
dirpath = os.path.abspath(
dirname(dirname(os.path.realpath(__file__)))
)
return dirpath
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
def examples_dir():
return os.path.join(confdir(), 'examples')
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
@pytest.fixture
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
def run_example_in_subproc(loglevel, testdir, arb_addr):
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
@contextmanager
def run(script_code):
kwargs = dict()
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
if platform.system() == 'Windows':
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
# on windows we need to create a special __main__.py which will
# be executed with ``python -m __main__.py`` on windows..
shutil.copyfile(
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
os.path.join(examples_dir(), '__main__.py'),
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
os.path.join(str(testdir), '__main__.py')
)
# drop the ``if __name__ == '__main__'`` guard
script_code = '\n'.join(script_code.splitlines()[:-4])
script_file = testdir.makefile('.py', script_code)
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
# without this, tests hang on windows forever
kwargs['creationflags'] = subprocess.CREATE_NEW_PROCESS_GROUP
Use the windows "gotchcas" fix for example tests Apply the fix from @chrizzFTD where we invoke the entry point using module exec mode on a ``__main__.py`` and import the ``test_example::`main()` from within that entry point script.
2020-02-08 01:20:46 +00:00
# run the "libary module" as a script
cmdargs = [
sys.executable,
'-m',
# use the "module name" of this "package"
'test_example'
]
else:
script_file = testdir.makefile('.py', script_code)
cmdargs = [
sys.executable,
str(script_file),
]
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
proc = testdir.popen(
cmdargs,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
**kwargs,
)
assert not proc.returncode
yield proc
proc.wait()
assert proc.returncode == 0
yield run
Add tests for all docs examples Parametrize our docs example test to include all (now fixed) examples from the `READ.rst`. The examples themselves have been fixed/corrected to run but they haven't yet been updated in the actual docs. Once #99 lands these example scripts will be directly included in our documentation so there will be no possibility of presenting incorrect examples to our users! This technically fixes #108 even though the new example aren't going to be included directly in our docs until #99 lands.
2020-02-09 07:01:39 +00:00
@pytest.mark.parametrize(
'example_script',
[f for f in os.listdir(examples_dir()) if '__' not in f],
)
def test_example(run_example_in_subproc, example_script):
ex_file = os.path.join(examples_dir(), example_script)
Initial examples testing attempt A per #98 we need tests for examples from the docs as they would be run by a user copy and pasting the code. This adds a small system for loading examples from an "examples/" directory and executing them in a subprocess while checking the output. We can use this to also verify end-to-end expected logging output on std streams (ex. logging on stderr). To expand this further we can parameterize the test list using the contents of the examples directory instead of hardcoding the script names as I've done here initially. Also, fix up the current readme examples to have the required/proper `if __name__ == '__main__'` script guard.
2020-01-31 22:06:26 +00:00
with open(ex_file, 'r') as ex:
code = ex.read()
with run_example_in_subproc(code) as proc:
proc.wait()
err, _ = proc.stderr.read(), proc.stdout.read()
# if we get some gnarly output let's aggregate and raise
if err and b'Error' in err:
raise Exception(err.decode())
assert proc.returncode == 0