BDD, the bigger sibling of Test Driven Development (TDD) is aimed at testing the behaviour of an application or integration from end-user perspectives versus testing smaller pieces (units) of functionality.
SpecFlow, a BDD framework for the .NET ecosystem, tests features and scenarios written in the Gherkin format. This is a human readable format written in spoken language, allowing your team to create a shared understanding of the behaviour of the feature you’re building and testing.
Feature: Membership
Scenario: Creating a new member
Given there are no members
And the register page exists
When I am on the register page
And I create the following members
| Username | Name | Email | Password |
| anne | Anne | anne@example.org | I_Love_Bob_23! |
| bob | Bob | bob@example.com | Umbraco4Life$ |
Then there should be 2 members
An example Gherkin definition - Membership.feature
This format outlines the desired outcome based on the expected behaviour of an application given a known initial state:
You may also be able to relate the Given, When and Then format to Arrange, Act and Assert discussed elsewhere in automated testing.
Gherkin is not limited to defining specifications in English, so there's no need to break out the Babel-fish to describe your behaviours, just write in your team’s supported language of choice!
Let's walk through setting up a SpecFlow project to test an example Umbraco site.
In the following example, we'll be testing the Membership feature developed using Umbraco's built in UmbRegisterController with the Gherkin feature file above.
A dotnet new template package exists for SpecFlow, that can be installed and run to bootstrap an example testing project: SpecFlow.Templates.DotNet.
The specflowproject template can be run with additional arguments to specify the .NET framework and testing provider you'd like to use, and also whether to include the useful FluentAssertions library.
For our use case, we'll include a few additional dependencies, Microsoft.AspNetCore.Mvc.Testing to host the site in-memory and AngleSharp to parse and interact with the returned HTML documents.
Be sure to match the versions of .NET for the SpecFlow project and the Microsoft.AspNetCore.Mvc.Testing package to the .NET version of your Umbraco web project.
# Install the template and create a new SpecFlow project
dotnet new --install SpecFlow.Templates.DotNet
dotnet new specflowproject -o MyProject.Specs -f net6.0 -t xunit -fa true
dotnet sln add MyProject.Specs
# Reference the Umbraco web project
dotnet add MyProject.Specs reference MyProject.Web
# Install additional dependencies
dotnet add MyProject.Specs package Microsoft.AspNetCore.Mvc.Testing
dotnet add MyProject.Specs package AngleSharp
dotnet add MyProject.Specs package AngleSharp.Io
Command line steps for installing and creating a SpecFlow project
After following the steps above, your solution should have a new SpecFlow testing project with the following folder structure:
You can rename these folders or add more to group your code if you wish. All code in our example project written for this article is available on GitHub for you to explore.
A basic SpecFlow project structure
As the developer implementing the tests, you will need to create step definition methods for each of the lines within the Gherkin feature file. Performing the actions they describe within the context of the application feature under test.
These are the methods decorated with [Given("")], [When("")] and [Then("")] attributes in MembershipStepDefinitions below.
The [Binding] attribute informs SpecFlow to bind the Gherkin specification to your classes and methods containing your step definitions.
[Binding]
public sealed class MembershipStepDefinitions
{
private readonly AngleSharpWebDriver _webDriver;
private readonly UmbracoDriver _umbracoDriver;
public MembershipStepDefinitions(
AngleSharpWebDriver webDriver,
UmbracoDriver umbracoDriver)
{
_webDriver = webDriver;
_umbracoDriver = umbracoDriver;
}
[Given(@"the register page exists")]
public void GivenTheRegisterPageExists() => _umbracoDriver.EnsureRegisterPageExists();
[Given(@"there are no members")]
public void GivenThereAreNoMembers() => _umbracoDriver.EnsureNumberOfMembers(0);
[When(@"I am on the register page")]
public async Task WhenIAmOnTheRegisterPage() => await _webDriver.VisitPageAsync("/register/");
[When(@"I create the following members")]
public async Task WhenICreateTheFollowingMembers(Table table)
{
foreach (var row in table.Rows)
{
_webDriver.InputValue("RegisterModel.Name", row["Name"]);
_webDriver.InputValue("RegisterModel.Email", row["Email"]);
_webDriver.InputValue("RegisterModel.Password", row["Password"]);
_webDriver.InputValue("RegisterModel.ConfirmPassword", row["Password"]);
await _webDriver.SubmitFormAsync();
}
}
[Then(@"there should be (.*) members")]
public void ThenThereShouldBeMembers(int numberOfMembers) => _umbracoDriver.EnsureNumberOfMembers(numberOfMembers);
}
Step definitions for the Membership feature - MembershipStepDefinitions.cs
It's also possible to take in values to these methods through regular expression matching and defining data tables in the feature file.
In our example, we do this to create a list of members and check the expected membership count.
A common pattern used within BDD is defining Drivers, allowing you to encapsulate common operations to alter the state of the application under test.
To simplify interaction with Umbraco from our step definitions, helper methods can be created to ensure content nodes (the registration page) have been created and check the current number of members in our site.
public class UmbracoDriver
{
private readonly IContentService _contentService;
private readonly IMemberService _memberService;
public UmbracoDriver(TestWebContext webContext)
{
_contentService = webContext.Services.GetRequiredService<IContentService>();
_memberService = webContext.Services.GetRequiredService<IMemberService>();
}
public IContent EnsureHomePageExists()
{
var homePage = _contentService.Create("Home", Constants.System.Root, Home.ModelTypeAlias);
var result = _contentService.SaveAndPublish(homePage);
result.Success.Should().BeTrue();
return homePage;
}
public IContent EnsureRegisterPageExists()
{
var homePage = EnsureHomePageCreated();
var registerPage = _contentService.Create("Register", homePage, Register.ModelTypeAlias);
var result = _contentService.SaveAndPublish(registerPage);
result.Success.Should().BeTrue();
return registerPage;
}
public void EnsureNumberOfMembers(int numberOfMembers) => _memberService.Count().Should().Be(numberOfMembers);
}
Driver for interacting with Umbraco services - UmbracoDriver.cs
To download and interact with web pages, a Driver can be created wrapping around our web application and AngleSharp's browsing context tracking the current document we're browsing.
Common interactions with a HTML document have been defined in our AngleSharpWebDriver to visit a URL, enter a value into an <input /> on a <form> and submit said form.
public class AngleSharpWebDriver
{
private readonly IBrowsingContext _browsingContext;
private readonly Url _baseUrl;
private IDocument? _document;
public AngleSharpWebDriver(TestWebContext webContext)
{
var httpClient = webContext.Client;
_baseUrl = new Url(httpClient.BaseAddress!.ToString());
var requester = new HttpClientRequester(httpClient);
var configuration = Configuration.Default
.With(requester)
.WithDefaultLoader();
_browsingContext = BrowsingContext.New(configuration);
}
public async Task VisitPageAsync(string path)
{
var url = new Url(_baseUrl, path);
_document = await _browsingContext.OpenAsync(url);
}
public void InputValue(string field, string value)
{
_document.Should().NotBeNull();
var form = _document.Forms.FirstOrDefault();
form.Should().NotBeNull();
var input = form!
.QuerySelectorAll<IHtmlInputElement>("input")
.FirstOrDefault(i => i.Name == field);
input.Should().NotBeNull();
input!.Value = value;
}
public async Task SubmitFormAsync()
{
_document.Should().NotBeNull();
var form = _document.Forms.FirstOrDefault();
form.Should().NotBeNull();
_document = await form!.SubmitAsync();
}
}
Driver for interacting with webpages - AngleSharpWebDriver.cs
SpecFlow has a unique dependency injection capability allowing you to inject context and share application state between steps in a scenario.
This context has been consumed from our Drivers but can also be directly accessed from within Step Definitions (and Hooks, more on that later).
The lifecycle of a context is limited to a scenario and shared between steps in the scenario. It cannot be shared between scenarios or features.
public class TestWebContext : IDisposable
{
private readonly TestWebApplicationFactory _applicationFactory = new();
public HttpClient Client => _applicationFactory.CreateClient();
public IServiceProvider Services => _applicationFactory.Services;
public void Dispose() => _applicationFactory.Dispose();
}
Our shared context, storing application state - TestWebContext.cs
In our example, the web application can be initialised using TestWebApplicationFactory a custom WebApplicationFactory<>, which exposes a custom HttpClient for interfacing via HTTP requests to the in-memory web server and accessing services via the ServiceProvider for our Umbraco site.
Remember to cleanup your resources! SpecFlow ensures they're disposed of correctly between scenario executions when contexts implement the IDisposable interface.
All of this testing is possible to be run in-memory with the power of the Microsoft.AspNetCore.Mvc.Testing package and the Microsoft.Data.Sqlite provider for Umbraco.
This removes the dependency on an external database or spinning up a full Kestrel or IIS server to run your tests against. Allowing you to run the tests independently within a CI/CD pipeline.
public class TestWebApplicationFactory : WebApplicationFactory<Startup>
{
private readonly string _connectionString = "Data Source=InMemory;Mode=Memory;Cache=Shared;Pooling=True";
private readonly SqliteConnection _sharedConnection;
public TestWebApplicationFactory()
{
_sharedConnection = new SqliteConnection(_connectionString);
_sharedConnection.Open();
}
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
var currentDirectory = Directory.GetCurrentDirectory();
var path = Path.Combine(currentDirectory, "appsettings.Testing.json");
builder.ConfigureAppConfiguration(config =>
{
config.AddJsonFile(path);
});
builder.UseEnvironment("Development");
}
protected override void Dispose(bool disposing)
{
base.Dispose(disposing);
_sharedConnection.Dispose();
}
}
The factory to configure the application - TestWebApplicationFactory.cs
The custom WebApplicationFactory<> wraps around and adapts the Startup commonly found in Umbraco web projects, configuring the App Settings and Connections Strings.
To ensure a SQLite connection is shared and maintained throughout the tests, we create and open a connection at the factory level.
To aide in initialisation, you can use a custom App Settings configuration file to setup Umbraco in unattended mode, allowing for a fresh install to be created on every run.
It's recommended to use a utility like uSync to create, persist and recreate the required Settings for your site. Running an import of Content Types, Data Types and Member Types the on first boot.
{
"Umbraco": {
"CMS": {
"Unattended": {
"UpgradeUnattended": true,
"InstallUnattended": true,
"UnattendedUserName": "admin",
"UnattendedUserEmail": "admin@example.org",
"UnattendedUserPassword": "I_Am_R00T!"
}
}
},
"uSync": {
"Settings": {
"ImportOnFirstBoot": true
}
},
"ConnectionStrings": {
"umbracoDbDSN": "Data Source=InMemory;Mode=Memory;Cache=Shared;Pooling=True",
"umbracoDbDSN_ProviderName": "Microsoft.Data.Sqlite"
}
}
Testing specific configuration - appsettings.Testing.json
There are actually two kinds of bindings in SpecFlow; firstly Step Definitions as previously discussed and secondly Hooks for defining operations to occur before or after your test suite, features or scenarios run.
Hooks can be useful to construct and tear down a shared application state that you expect as part of your test suite across features, scenarios or steps.
Additionally bindings can be restricted and scoped to run only for specific features, scenarios or tags. This can be done using the [Scope] attribute as seen below in ExampleHooks.
Hooks that bind to before and after events for the test run and features must be static methods.
[Binding]
public class ExampleHooks
{
[BeforeTestRun]
public static void BeforeTestRun()
{
}
[AfterTestRun]
public static void AfterTestRun()
{
}
[BeforeFeature]
[Scope(Feature = "Membership")]
public static void BeforeMembershipFeature()
{
}
[AfterFeature]
[Scope(Feature = "Membership")]
public static void AfterMembershipFeature()
{
}
[BeforeScenario]
[Scope(Feature = "Membership", Scenario = "Creating a new member")]
public void BeforeCreatingANewMemberScenario()
{
}
[AfterScenario]
[Scope(Feature = "Membership", Scenario = "Creating a new member")]
public void AfterCreatingANewMemberScenario()
{
}
}
Example hook methods for binding to scenario, feature or test events - ExampleHooks.cs
Here is an non-exhaustive list of potential ways you could extend the example above:
The opportunity to test the behaviour of your application over individual units is endless.
Maybe you have built a package you'd like to perform automated testing on but don't want to write a Mock / Fake for every dependent Umbraco service?
Try integration testing with an example site like above.
Potentially take your testing one step further and use Playwright to carry out end-to-end behaviour testing of your site, all from within .NET!
