hitchstory 0.21.0

Type-safe YAML-based example specification driven development framework for python.

HitchStory

HitchStory is a StrictYAML based python integration testing library that runs in pytest.

With it, you can write integration tests that rewrite themselves and tests that write your docs:

It can be used to quickly and easily integration test and generate docs for any kind of app. Examples:

• A website
• An interactive command line app
• A REST API
• A Python API

Example

example.story:

Log in as James:
  given:
    browser: firefox  # preconditions
  steps:
  - Enter text:
      username: james
      password: password
  - Click: log in
  
See James analytics:
  based on: log in as james  # inheritance
  following steps:
  - Click: analytics

test_hitchstory.py:

from hitchstory import BaseEngine, GivenDefinition, GivenProperty
from hitchstory import Failure, strings_match
from hitchstory import StoryCollection
from strictyaml import Str
from pathlib import Path
from os import getenv

class Engine(BaseEngine):
    """Interprets and validates the hitchstory stories."""

    given_definition = GivenDefinition(
        browser=GivenProperty(
            # Available validators: https://hitchdev.com/strictyaml/using/
            Str()
        ),
    )
    
    def __init__(self, rewrite=False):
        self._rewrite = rewrite

    def set_up(self):
        print(f"Using browser {self.given['browser']}")

    def click(self, name):
        print(f"Click on {name}")
        
        if name == "analytics":
            raise Failure(f"button {name} not found")
    
    def enter_text(self, **textboxes):
        for name, text in textboxes.items():
            print(f"Enter {text} in {name}")
    
    def tear_down(self):
        pass


collection = StoryCollection(
    # All .story files in this file's directory.
    Path(__file__).parent.glob("*.story"),

    Engine(
        # If REWRITE environment variable is set to yes -> rewrite mode.
        rewrite=getenv("REWRITE", "no") == "yes"
    )
)

#Manually run stories within pytest tests
#def test_log_in_as_james():
#    collection.named("Log in as james").play()

#def test_see_james_analytics():
#    collection.named("See James analytics").play()

# Automagically add all stories as tests.
# E.g. "Log in as James" -> "def test_login_in_as_james"
collection.with_external_test_runner().ordered_by_name().add_pytests_to(
    module=__import__(__name__) # This module
)

Run log in as James test

This runs "test_log_in_as_james", a pytestified version of "Log in as James" while -s (don't capture output) lets you see the results of running the steps.

pytest -s test_hitchstory.py -k test_log_in_as_james

Outputs:

============================= test session starts ==============================
platform linux -- Python n.n.n, pytest-n.n.n, pluggy-n.n.n
rootdir: /path/to
collected 2 items / 1 deselected / 1 selected

test_hitchstory.py Using browser firefox
Enter james in username
Enter password in password
Click on log in
.

======================= 1 passed, 1 deselected in 0.1s ========================

Run failing test

Failing tests have highlighting and colors when run for real.

pytest -k test_see_james_analytics test_hitchstory.py

Outputs:

============================= test session starts ==============================
platform linux -- Python n.n.n, pytest-n.n.n, pluggy-n.n.n
rootdir: /path/to
collected 2 items / 1 deselected / 1 selected

test_hitchstory.py F                                                     [100%]

=================================== FAILURES ===================================
___________________________ test_see_james_analytics ___________________________

story = Story('see-james-analytics')

    def hitchstory(story=story):
>       story.play()
E       hitchstory.exceptions.StoryFailure: RUNNING See James analytics in /path/to/example.story ... FAILED in 0.1 seconds.
E
E             based on: log in as james  # inheritance
E             following steps:
E             - Click: analytics
E
E
E       hitchstory.exceptions.Failure
E
E           Test failed.
E
E       button analytics not found

/src/hitchstory/story_list.py:50: StoryFailure
----------------------------- Captured stdout call -----------------------------
Using browser firefox
Enter james in username
Enter password in password
Click on log in
Click on analytics
=========================== short test summary info ============================
FAILED test_hitchstory.py::test_see_james_analytics - hitchstory.exceptions.StoryFailure: RUNNING See James analytics in /path/to/example.story ... FAILED in 0.1 seconds.

      based on: log in as james  # inheritance
      following steps:
      - Click: analytics


hitchstory.exceptions.Failure

    Test failed.

button analytics not found
======================= 1 failed, 1 deselected in 0.1s ========================

Install

$ pip install hitchstory

Using HitchStory: With Pytest

If you already have pytest set up and running integration tests, you can use it with hitchstory:

• Self rewriting tests with pytest and hitchstory

Using HitchStory: Engine

How to use the different features of the story engine:

• Hiding stacktraces for expected exceptions
• Given preconditions
• Gradual typing of story steps
• Match two JSON snippets
• Match two strings and show diff on failure
• Extra story metadata - e.g. adding JIRA ticket numbers to stories
• Story with parameters
• Story that rewrites itself
• Raising a Failure exception to conceal the stacktrace
• Arguments to steps
• Strong typing

Using HitchStory: Documentation Generation

How to autogenerate documentation from your tests:

• Generate documentation with extra variables and functions
• Generate documentation from story

Using HitchStory: Inheritance

Inheriting stories from each other:

• Inherit one story from another simply
• Story inheritance - given mapping preconditions overridden
• Story inheritance - override given scalar preconditions
• Story inheritance - parameters
• Story inheritance - steps
• Variations

Using HitchStory: Runner

Running the stories in different ways:

• Continue on failure when playing multiple stories
• Flaky story detection
• Play multiple stories in sequence
• Run one story in collection
• Shortcut lookup for story names

Approach to using HitchStory

Best practices, how the tool was meant to be used, etc.

• Can I do BDD with hitchstory? How do I do BDD with hitchstory?
• Complementary tools
• Executable specifications
• Flaky Tests
• Does hitchstory let "the business" write stories while you just write the code?
• Separation of Test Concerns
• Test Artefact Environment Isolation
• Test concern leakage
• Tests as an investment
• What is the difference betweeen a test and a story?
• The importance of test realism
• Testing non-deterministic code
• Specification Documentation Test Triality

Design decisions and principles

Design decisions are justified here:

• Declarative User Stories
• Why does hitchstory mandate the use of given but not when and then?
• Why is inheritance a feature of hitchstory stories?
• Why does hitchstory not have an opinion on what counts as interesting to "the business"?
• Why does hitchstory not have a command line interface?
• Principles
• Why does HitchStory have no CLI runner - only a pure python API?
• Why Rewritable Test Driven Development (RTDD)?
• Why does HitchStory use StrictYAML?

Why not X instead?

HitchStory is not the only integration testing framework. This is how it compares with the others:

• Why use Hitchstory instead of Behave, Lettuce or Cucumber (Gherkin)?
• Why not use the Robot Framework?
• Why use hitchstory instead of a unit testing framework?

Using HitchStory: Setup on its own

If you want to use HitchStory without pytest:

• Creating a basic command line test runner

Using HitchStory: Behavior

Miscellaneous docs about behavior of the framework:

• Abort a story with ctrl-C
• Upgrade breaking changes between v0.14 and v0.15
• Handling failing tests
• Running a single named story successfully