From 1f99714844abdb364c4a718d76fbacc62241d9fc Mon Sep 17 00:00:00 2001 From: Ross McDonald Date: Thu, 1 Mar 2018 15:55:16 -0600 Subject: [PATCH 1/5] Proposed updates to contract testing article. --- articles/testing/contract-testing.md | 81 +++++++++++++++++++++------- 1 file changed, 63 insertions(+), 18 deletions(-) diff --git a/articles/testing/contract-testing.md b/articles/testing/contract-testing.md index 80ebe55..4f08211 100644 --- a/articles/testing/contract-testing.md +++ b/articles/testing/contract-testing.md @@ -1,37 +1,82 @@ # Contract Testing -Scenarios makes it easy to incorporate your OAS / Swagger API specification into your testing process. A few benefits to doing this include: +Scenarios makes it easier than ever to integrate your OpenAPI specification into +your testing process. One of the easiest ways to do this is through a testing +method called __contract testing__. Contract testing is a simple verification +process where Stoplight verifies that the API responses match the "contract" +within a connected OpenAPI specification. -- **DRY**: Don't re-create test assertions that check for what is already described in your API contract. -- **Governance**: Quickly figure out if the API that was created actually conforms to the API design that was initially agreed upon. -- **Sync Manager**: Your API spec is the single point of truth that describes your API. From it, you might generate documentation, sdks, mock servers, etc. Incorporating your spec into your tests makes sure that your API spec accurately represents your API over time. +Benefits of contract testing include: + +- **Don't Repeat Yourself**: Don't re-create test assertions that check for what + is already described in your API specification. Let your scenario do the heavy + lifting, validating that an API implementation matches the OpenAPI + specification. + +- **Governance**: Quickly figure out if an API implementation conforms to the + OpenAPI specification that was initially agreed upon. Run a contract test + scenario on a schedule to ensure the API never violates the specification. + +- **Single Source of Truth**: Your API specification is the single source of + truth that describes your API. From it, you can generate documentation, SDKs, + mock servers, and more. Incorporating the specification into your testing + pipeline ensures that the it accurately represents your API implementation + over time. -> If you don't have an API specification yet, you can create one using the Stoplight modeling tool! +> If you don't have an API specification yet, you can create one using the +> [Stoplight modeling tool](../modeling/modeling-introduction.md)! -## Connecting The Spec +## Connecting a Spec -The first thing you need to do to get started with contract testing is connect your API spec to the Scenarios Collection. + -1. Create a new (or open an existing) **Scenario file** in the Stoplight editor +To get started with contract testing, the first thing you will need to do is +generate a scenario from an OpenAPI specification. To get started: + +1. Create a new (or open an existing) **scenario** in the Stoplight editor 2. Select **Swagger/OAS Coverage** in the Scenarios menu to the left -3. Open **Contract Test Settings** -4. Click **+ Add Spec** -5. Select a file from either **This Project** or an **External URL** -6. You are all set! You can now test against an API spec. +3. Open the **Contract Test Settings** menu +4. Click **Add Spec**, and select an OpenAPI specification to connect + +You are all set! Once the specification has been connected, you can +automatically generate a contract testing scenario for your spec using the +Coverge Report, described below. ## Using the Coverage Report -The coverage report gives you a quick overview of which parts of the connected specs are covered by test assertions in the current Scenario Collection. +The coverage report gives you a quick overview of which parts of the connected +specs are covered by test assertions in the current Scenario Collection. -You can use the coverage report to quickly stub out a new scenario. Just click the status codes in the table matrix for the steps you want to add to your scenario (in order). Once you've added all the steps, click the "Create Scenario" button in the top right. This will create a scenario with as much setup as possible, using the connected spec for data. It will set your request body, set variables in a sensible way, automatically setup contract tests, and more. +You can use the coverage report to quickly stub out a new scenario. To start: -You will likely need to tweak the resulting scenario a little bit, but this process will usually get you most of the way to a complete scenario, with contract test assertions in place! +1. Click the status codes in the table matrix for the steps you want to add to + your scenario. Note that the order in which the endpoints are clicked + determines the order in which they will appear in the scenario. For example, + if an API object needs to be created before it can be removed, then you will + want to choose the 'create object' endpoint before the 'delete object' + endpoint. + +2. Once all of the desired endpoints have been selected, click the __Create + Scenario__ button in the top right to generate the scenario. + +3. You will automatically be navigated to the new scenario, complete with + contract test assertions for each selected endpoint. + + +> You will likely need to modify the resulting scenario to fit your use case ## Automatic Contract Test Assertion -After linking your spec to the Scenario Collection, contract test assertions will be automatically added for step assertions. + -Stoplight will look through your API specification for a operation that matches the step's HTTP method + URL, and use the response status code returned from the API to look up the JSON schema. In the example below, we are testing the 200 response schema in our API spec for the GET /todos/{todoId} endpoint. +After linking your spec to the Scenario Collection, contract test assertions will be automatically added for each step assertion. -When this step is run, the HTTP response structure will be validated against the matched JSON schema from our API spec, and any errors will be added to the test results. +When the scenario is generated, Stoplight will look through the API +specification for an operation that matches the step's HTTP method and URL. The +response code returned from the API is then used to look up the corresponding +JSON schema. + +When a contract assertion step is run, the HTTP response structure will be +validated against the matched JSON schema from the connected API specification. +Any validation errors will automatically be added to the test results. From 823f0cfa48532f7104a79c82782ba6b3fdff6ab6 Mon Sep 17 00:00:00 2001 From: Robert Wallach Date: Tue, 6 Mar 2018 15:10:21 -0600 Subject: [PATCH 2/5] Update contract-testing.md --- articles/testing/contract-testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/articles/testing/contract-testing.md b/articles/testing/contract-testing.md index 4f08211..b8f0521 100644 --- a/articles/testing/contract-testing.md +++ b/articles/testing/contract-testing.md @@ -20,7 +20,7 @@ Benefits of contract testing include: - **Single Source of Truth**: Your API specification is the single source of truth that describes your API. From it, you can generate documentation, SDKs, mock servers, and more. Incorporating the specification into your testing - pipeline ensures that the it accurately represents your API implementation + pipeline ensures that it accurately represents your API implementation over time. From fb465246cda4c1c344926125be985b5f9983f2f9 Mon Sep 17 00:00:00 2001 From: Robert Wallach Date: Tue, 6 Mar 2018 15:16:28 -0600 Subject: [PATCH 3/5] Update contract-testing.md --- articles/testing/contract-testing.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/articles/testing/contract-testing.md b/articles/testing/contract-testing.md index b8f0521..9723eb0 100644 --- a/articles/testing/contract-testing.md +++ b/articles/testing/contract-testing.md @@ -41,7 +41,7 @@ generate a scenario from an OpenAPI specification. To get started: You are all set! Once the specification has been connected, you can automatically generate a contract testing scenario for your spec using the -Coverge Report, described below. +Coverge Report, as described below. ## Using the Coverage Report @@ -50,12 +50,15 @@ specs are covered by test assertions in the current Scenario Collection. You can use the coverage report to quickly stub out a new scenario. To start: -1. Click the status codes in the table matrix for the steps you want to add to - your scenario. Note that the order in which the endpoints are clicked - determines the order in which they will appear in the scenario. For example, - if an API object needs to be created before it can be removed, then you will - want to choose the 'create object' endpoint before the 'delete object' - endpoint. +1. Click the **status codes** in the table matrix for the steps you want to add to + your scenario. + +<-- theme:info --> +> Note that the order in which the endpoints are clicked +determines the order in which they will appear in the scenario. For example, +if an API object needs to be created before it can be removed, then you will +want to choose the 'create object' endpoint before the 'delete object' +endpoint. 2. Once all of the desired endpoints have been selected, click the __Create Scenario__ button in the top right to generate the scenario. From fb54460cbac1836bdb243d6935bb83a5f7c04e80 Mon Sep 17 00:00:00 2001 From: Robert Wallach Date: Tue, 6 Mar 2018 15:54:15 -0600 Subject: [PATCH 4/5] Update contract-testing.md --- articles/testing/contract-testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/articles/testing/contract-testing.md b/articles/testing/contract-testing.md index 9723eb0..486bd4d 100644 --- a/articles/testing/contract-testing.md +++ b/articles/testing/contract-testing.md @@ -53,7 +53,7 @@ You can use the coverage report to quickly stub out a new scenario. To start: 1. Click the **status codes** in the table matrix for the steps you want to add to your scenario. -<-- theme:info --> + > Note that the order in which the endpoints are clicked determines the order in which they will appear in the scenario. For example, if an API object needs to be created before it can be removed, then you will From e6e993c1f6c905c358fbf49d2c5cf978fcc921d3 Mon Sep 17 00:00:00 2001 From: Ross McDonald Date: Wed, 7 Mar 2018 16:42:56 -0600 Subject: [PATCH 5/5] Updates --- articles/testing/contract-testing.md | 45 +++++++++++++++------------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/articles/testing/contract-testing.md b/articles/testing/contract-testing.md index 4f08211..7f70c61 100644 --- a/articles/testing/contract-testing.md +++ b/articles/testing/contract-testing.md @@ -2,28 +2,29 @@ Scenarios makes it easier than ever to integrate your OpenAPI specification into your testing process. One of the easiest ways to do this is through a testing -method called __contract testing__. Contract testing is a simple verification +method called **contract testing**. Contract testing is a simple verification process where Stoplight verifies that the API responses match the "contract" within a connected OpenAPI specification. Benefits of contract testing include: -- **Don't Repeat Yourself**: Don't re-create test assertions that check for what +* **Don't Repeat Yourself**: Don't re-create test assertions that check for what is already described in your API specification. Let your scenario do the heavy lifting, validating that an API implementation matches the OpenAPI specification. -- **Governance**: Quickly figure out if an API implementation conforms to the +* **Governance**: Quickly figure out if an API implementation conforms to the OpenAPI specification that was initially agreed upon. Run a contract test scenario on a schedule to ensure the API never violates the specification. -- **Single Source of Truth**: Your API specification is the single source of +* **Single Source of Truth**: Your API specification is the single source of truth that describes your API. From it, you can generate documentation, SDKs, mock servers, and more. Incorporating the specification into your testing - pipeline ensures that the it accurately represents your API implementation + pipeline ensures that the spec accurately represents your API implementation over time. + > If you don't have an API specification yet, you can create one using the > [Stoplight modeling tool](../modeling/modeling-introduction.md)! @@ -34,10 +35,10 @@ Benefits of contract testing include: To get started with contract testing, the first thing you will need to do is generate a scenario from an OpenAPI specification. To get started: -1. Create a new (or open an existing) **scenario** in the Stoplight editor -2. Select **Swagger/OAS Coverage** in the Scenarios menu to the left -3. Open the **Contract Test Settings** menu -4. Click **Add Spec**, and select an OpenAPI specification to connect +1. Create a new (or open an existing) **scenario** in the Stoplight editor +2. Select **Swagger/OAS Coverage** in the Scenarios menu to the left +3. Open the **Contract Test Settings** menu +4. Click **Add Spec**, and select an OpenAPI specification to connect You are all set! Once the specification has been connected, you can automatically generate a contract testing scenario for your spec using the @@ -46,31 +47,33 @@ Coverge Report, described below. ## Using the Coverage Report The coverage report gives you a quick overview of which parts of the connected -specs are covered by test assertions in the current Scenario Collection. +specs are covered by test assertions in the current Scenario Collection. You can use the coverage report to quickly stub out a new scenario. To start: -1. Click the status codes in the table matrix for the steps you want to add to - your scenario. Note that the order in which the endpoints are clicked - determines the order in which they will appear in the scenario. For example, - if an API object needs to be created before it can be removed, then you will - want to choose the 'create object' endpoint before the 'delete object' - endpoint. +1. Click the status codes in the table matrix for the steps you want to add to + your scenario. Note that the order in which the endpoints are clicked + determines the order in which they will appear in the scenario. For example, + if an API object needs to be created before it can be removed, then you will + want to choose the 'create object' endpoint before the 'delete object' + endpoint. -2. Once all of the desired endpoints have been selected, click the __Create - Scenario__ button in the top right to generate the scenario. +2. Once all of the desired endpoints have been selected, click the **Create + Scenario** button in the top right to generate the scenario. -3. You will automatically be navigated to the new scenario, complete with - contract test assertions for each selected endpoint. +3. You will automatically be navigated to the new scenario, complete with + contract test assertions for each selected endpoint. + > You will likely need to modify the resulting scenario to fit your use case ## Automatic Contract Test Assertion -After linking your spec to the Scenario Collection, contract test assertions will be automatically added for each step assertion. +After linking your spec to the Scenario Collection, contract test assertions +will be automatically added as steps within your scenario. When the scenario is generated, Stoplight will look through the API specification for an operation that matches the step's HTTP method and URL. The