Demonstrations of select features of the TestScribe tool.

Table of Contents

Here is the demo project. It’s intentionally made as simple as possible to make it easier to understand.
Feel free to download it and play with it. Recommend setting up the IDE quick launch to make it easier to try the examples.

For more realistic examples, see how the tool uses itself to test here.

Mock a class instance

The function search_name takes a Service object as a parameter. The service may involve a database or a network call. It’s easier to test by mocking it.

An example test run output:

...
Please provide the value for the parameter (service) of type: (tsdemo.service.Service) []: m
Created a mock: Mock: name (m_service) spec (<class 'tsdemo.service.Service'>)
Please provide the value for the parameter (keyword) of type: (str) []: Bob
Calling search_name(service=Mock: name (m_service) spec (<class 'tsdemo.service.Service'>), keyword='Bob')
m_service's search_a_name method is called
with: keyword='key: Bob'.
Call stack:
  File "/home/ray/code/testscribe-demo/tsdemo/simple_mock.py", line 11, in search_name
    name = service.search_a_name("key: " + keyword)

Please provide the value for the return value of type: (str) []: real Bob
Mock call return value: 'real Bob'
***** Result:
type: <class 'str'>
value:
{"name": "real Bob"}
***** Result end
...

Here is the generated unit test code.

Here is a demo video

Create a class instance

The function get_person_age takes a Person object as a parameter.

An example test run output:

...
Please provide the value for the parameter (p) of type: (tsdemo.person.Person) []: c("Bob", 10)
Calling get_person_age(p=tsdemo.person.Person("Bob", 10))
***** Result:
type: <class 'int'>
value:
10
***** Result end...

Here is the generated unit test code.

A real-world example of an end-to-end test scenario and how to update it

The tool uses itself to test. See this FAQ for more details.

test_retry_invalid_input_output is a generated test targeting a wrapper function retry_invalid_input_output. The wrapper function encapsulates the testing logic of running an end-to-end testing session with an invalid input and return the command line output.

The following demo video demonstrates how this test can be easily updated when the error message is changed.

Here is the commit diff of the change.

Updating the test only involves a few routine key presses and visual inspection.

Here is the updated test.

Exception result

When a test run results in an exception, the tool can display the exception information and generate the appropriate test code.

This target function always_raise_exception will always raise an exception.

An example test run output:

...
Calling always_raise_exception()
The function always_raise_exception throws an exception.
Traceback (most recent call last):
  File "/home/ray/code/testscribe/src/testscribe/execution.py", line 86, in get_args_and_call
    ret_value = call_target_function(func, args)
  File "/home/ray/code/testscribe/src/testscribe/execution_util.py", line 87, in call_target_function
    return func(*binded_args.args, **binded_args.kwargs)
  File "/home/ray/code/testscribe-demo/tsdemo/raise_exception.py", line 7, in always_raise_exception
    raise Exception("test exception")
Exception: test exception
...

Here is the generated test.

Test a method

When the target function is a method of a class, the tool will automatically call the constructor to create an instance first and invoke the method on that instance.

This method greet is the target.

An example test run output:

...
Prepare to create an instance of the class: Greeter
Getting parameters for the function (Greeter)
Please provide the value for the parameter (my_name) of type: (str) []: Bob
Calling Greeter(my_name='Bob')
Prepare to call the target function.
Getting parameters for the function (Greeter.greet)
Please provide the value for the parameter (to) of type: (str) []: Alice
Calling Greeter.greet(to='Alice')
***** Result:
type: <class 'str'>
value:
Hello Alice. My name is Bob
***** Result end
...

Here is the generated unit test code.

Test multiple input sets

See this FAQ for the background information.

The eval_expression function takes an expression as a string and returns its evaluation result. We will test different expressions with multiple test runs.

Here is a demo video comparing testing with TestScribe with the traditional testing approach. TestScribe makes you about 5 times more productive in this test scenario ( 30 seconds vs 140 seconds ).
This assumes perfect execution of the traditional test method. In practice, looking up documentation and fixing errors usually takes additional time. More importantly, the saved steps are the boring parts.

Here are the test runs:

...
Please provide the value for the parameter (expr) of type: (str) []: 3 + 5
Calling eval_expression(expr='3 + 5')
***** Result:
type: <class 'int'>
value:
8
***** Result end
Test name: [_]: _addition
...

...
Please provide the value for the parameter (expr) of type: (str) [3 + 5]: 2 + 4
Calling eval_expression(expr='2 + 4')
***** Result:
type: <class 'int'>
value:
6
***** Result end
Test name: [_]: _addition
...

...
Please provide the value for the parameter (expr) of type: (str) [2 + 4]: 6 * 9
Calling eval_expression(expr='6 * 9')
***** Result:
type: <class 'int'>
value:
54
***** Result end
Test name help: 'test_' prefix will be added automatically. Use a leading '_' to include the target function name as part of the prefix.
Test name: [_]: _multiplication
...

You only need to type in the expressions (3+5, 2+4, 6 * 9) and the test names (_addition, _multiplication).

Here is the generated tests.

Here is the manually written test for comparison.

Try for yourself to see which method (manual vs using the tool) makes you and people who read your tests more productive for this test.

Multiple class instances in a list

The function get_average_age takes a list of Person object as a parameter.

Create multiple real instances in a list

An example test run:

...
Please provide the value for the parameter (person_list) of type: (typing.List[tsdemo.person.Person]) []: [tsdemo.person.Person("a", 2), tsdemo.person.Person("b", 3)]
Calling get_average_age(person_list=[tsdemo.person.Person("a", 2), tsdemo.person.Person("b", 3)])
***** Result:
type: <class 'int'>
value:
2
***** Result end
...

Create multiple mocks in a list

An example test run:

Please provide the value for the parameter (person_list) of type: (typing.List[tsdemo.person.Person]) []: [m, m]
Created a mock: Mock: name (m_person) spec (<class 'tsdemo.person.Person'>)
Created a mock: Mock: name (m_person_1) spec (<class 'tsdemo.person.Person'>)
Calling get_average_age(person_list=[<testscribe.mock_proxy.MockProxy object at 0x7f43dd744af0>, <testscribe.mock_proxy.MockProxy object at 0x7f43dd744e50>])
Mock object m_person's ( age ) attribute is accessed for the first time.
Call stack:
  File "/home/ray/code/testscribe-demo/tsdemo/objects_in_list.py", line 12, in get_average_age
    total += p.age

Please provide the value for the age attribute of type: (int) []: 2
Mock attribute value: 2
Mock object m_person_1's ( age ) attribute is accessed for the first time.
Call stack:
  File "/home/ray/code/testscribe-demo/tsdemo/objects_in_list.py", line 12, in get_average_age
    total += p.age

Please provide the value for the age attribute of type: (int) []: 3
Mock attribute value: 3
***** Result:
type: <class 'int'>
value:
2
***** Result end

Here is the generated unit test code.

Raise an exception in a mock call

The function search_name takes a Service object as a parameter. The search_a_name method on that Service object is called. An example test run output to test the behavior when the call throws an exception:

...
Please provide the value for the parameter (service) of type: (tsdemo.service.Service) [m(tsdemo.service.Service, 'm_service')]: 
Created a mock: Mock: name (m_service) spec (<class 'tsdemo.service.Service'>)
Please provide the value for the parameter (keyword) of type: (str) [Bob]: 
Calling search_name(service=Mock: name (m_service) spec (<class 'tsdemo.service.Service'>), keyword='Bob')
m_service's search_a_name method is called
with: keyword='key: Bob'.
Call stack:
  File "/home/ray/code/testscribe-demo/tsdemo/simple_mock.py", line 11, in search_name
    name = service.search_a_name("key: " + keyword)

Please provide the value for the return value of type: (str) [real Bob]: throw(Exception("search failed"))
Mock call return value: InputValue(expression='throw(Exception("search failed"))', value=Exception('search failed'))
The function search_name throws an exception.
...
Exception: search failed

Test name help: 'test_' prefix will be added automatically. Use a leading '_' to include the target function name as part of the prefix.
Test name: [_]: _exception is propagated
...

Notice that this test run use an earlier test run’s input as default input.

Here is the generated unit test code.

Ignore the return value of a mock call

The function show takes a Printer object as a parameter. The display method on that Printer object is called. The returned string is not used by this function. It’s easier to ignore this return value when the object is mocked during a test. And the generated test is simpler.

An example test run output to ignore the mocked display call:

...
Please provide the value for the parameter (text) of type: (str) []: a
Please provide the value for the parameter (printer) of type: (tsdemo.ignore_mock_return.Printer) []: m
Created a mock: Mock: name (m_printer) spec (<class 'tsdemo.ignore_mock_return.Printer'>)
Calling show(text='a', printer=Mock: name (m_printer) spec (<class 'tsdemo.ignore_mock_return.Printer'>))
m_printer's display method is called
with: text='a'.
Call stack:
  File "/home/ray/code/testscribe-demo/tsdemo/ignore_mock_return.py", line 17, in show
    printer.display(text)

Please provide the value for the return value of type: (str) []: ignore
Mock call return value: InputValue(expression='ignore', value='Ignored')
...

Here is the generated unit test code.

Patch a function

The function call_fixed_func has a fixed dependency on the calculate function.

To replace the calculate function with a mock object for a test, add the following code to the setup function

patch_with_mock(calculate)

The setup function definition is here. Uncomment the line before starting a test run which specifies this setup function. The configuration file is here

The setup function is called before the target function is called. The patch_with_mock function replaces the target function with a mock object of the same type as the target function.

...
Calling the setup function setup.setup.
setup function in tests is called.
Created a mock: Mock: name (m_calculate) spec (<function calculate at 0x7fe62436eb80>)
Patch tsdemo.patch_function.calculate with Mock: name (m_calculate) spec (<function calculate at 0x7fe62436eb80>)
Prepare to call the target function.
Getting parameters for the function (call_fixed_func)
Please provide the value for the parameter (num) of type: (int) []: 2
Calling call_fixed_func(num=2)
m_calculate is called
with: seed=2.
Call stack:
  File "/home/ray/code/testscribe-demo/tsdemo/patch_function.py", line 12, in call_fixed_func
    return calculate(num)

Please provide the value for the return value of type: (int) []: 1
Mock call return value: 1
***** Result:
type: <class 'int'>
value:
1
***** Result end
...

Here is the generated unit test code.

Patch a string

The function get_db_name has a fixed dependency on the DB_NAME variable.

To replace the string for a test, add the following code to the setup function

    patch_with_expression(target_str="tsdemo.patch_string.DB_NAME", expression="'test'")

Notice that the string value has to be quoted to be a valid Python expression.

The setup function definition is here. Uncomment the line before starting a test run which specifies this setup function.

Test run output:

...
setup function in tests is called.
Patch tsdemo.patch_string.DB_NAME with 'test'
Prepare to call the target function.
Getting parameters for the function (get_db_name)
Calling get_db_name()
***** Result:
type: <class 'str'>
value:
test
***** Result end
...

Here is the generated unit test code.

Input alias

To make inputs easier, you can define aliases.

In the example create real instances in a list, instead of typing the fully qualified class name, you can create an alias for the class name by adding the following to the setup function.

define_alias(alias=”ps”, full_str=”tsdemo.person.Person”)

Here is a test run which produces the same result as the example referenced above.

Please provide the value for the parameter (person_list) of type: (typing.List[tsdemo.person.Person]) []: [ps("a", 2), ps("b", 3)]
Expanded alias: ps 2 times.
Result after the expansion: [tsdemo.person.Person("a", 2), tsdemo.person.Person("b", 3)]
Calling get_average_age(person_list=[tsdemo.person.Person("a", 2), tsdemo.person.Person("b", 3)])

Wrapper function

To test a magic string method here, a wrapper function is used.

Here is the generated test code.

Annotate a class instance member variable with type information

Proper type information makes input easier in some cases.

In this class the model field is annotated with type information. For comparison, the owner field is not.

As the result, the model field input doesn’t need to be quoted. The owner field input does.

Here is an example test run:

...
Please provide the value for the parameter (car) of type: (tsdemo.annotate_field_type.Car) []: m
Created a mock: Mock: name (m_car) spec (<class 'tsdemo.annotate_field_type.Car'>)
Calling get_car_info(car=Mock: name (m_car) spec (<class 'tsdemo.annotate_field_type.Car'>))
Mock object m_car's ( model ) attribute is accessed for the first time.
Call stack:
  File "/home/ray/code/testscribe-demo/tsdemo/annotate_field_type.py", line 15, in get_car_info
    return f"Car model: {car.model}, owner: {car.owner}"

Please provide the value for the model attribute of type: (str) []: camery
Mock attribute value: 'camery'
Mock object m_car's ( owner ) attribute is accessed for the first time.
Call stack:
  File "/home/ray/code/testscribe-demo/tsdemo/annotate_field_type.py", line 15, in get_car_info
    return f"Car model: {car.model}, owner: {car.owner}"

Please provide the value for the owner attribute of type: (any) []: "Bob"
Mock attribute value: 'Bob'
***** Result:
type: <class 'str'>
value:
Car model: camery, owner: Bob
***** Result end
...

Results that contain class instances

The class doesn’t implement a custom __repr__ method

The result display and generated assertions will be based on member fields.

The target function create_simple_class_instance returns an instance of the class SimpleClass, which doesn’t define a custom __repr__ method.

Here is an example test run:

...
***** Result:
type: tsdemo.object_result.SimpleClass
value:
Object(type (tsdemo.object_result.SimpleClass), members ({'str_field': 'a', 'int_field': 1}))
***** Result end
...

Here is the generated test code.

The class implements a custom __repr__ method

The result display and generated assertions will be based on the result of repr(object).

The target function create_class_with_repr_instance returns an instance of the class ClassWithRepr, which defines a custom __repr__ method.

Here is an example test run:

...
***** Result:
type: tsdemo.object_with_repr_result.ClassWithRepr
value:
Object(type (tsdemo.object_with_repr_result.ClassWithRepr), repr (ClassWithRepr(str_field='a', int_field=1)))
***** Result end
...

Here is the generated test code.

Scribe files

The tool generates machine-readable YAML formatted files with the tscribe extension alongside the unit test files.

Here is a simple example that tests an add function.

You can find more examples generated for other demos on this page here.

A real world example of how the tool makes it easy to make some batch updates to tests

This change fixed the order of a generated assertion after an exception is thrown. The expected value should be on the right hand side of the equation.

After making this change to the production code, running the sync-all command once updated all the generated tests.

Assert class instances in mock call parameters

The tool automatically generates assertion for complex mock call parameters such as class instances. This would be cumbersome to do manually.

The target function call_b_method calls a method on its parameter with an instance of the class A.

Here is an example test run:

...
Please provide the value for the parameter (b) of type: (tsdemo.object_in_mock_call_param.B) []: m
Created a mock: Mock: name (m_b) spec (<class 'tsdemo.object_in_mock_call_param.B'>)
Calling call_b_method(b=Mock: name (m_b) spec (<class 'tsdemo.object_in_mock_call_param.B'>))
m_b's do method is called
with: a=tsdemo.object_in_mock_call_param.A(i=1).
Call stack:
  File "/home/ray/code/testscribe-demo/tsdemo/object_in_mock_call_param.py", line 15, in call_b_method
    b.do(A(1))
...

Here is the generated test code.