Testing if your Umbraco website is behaving naughty or nice

tagged with .NET 6 .NET 7 .NET Core Backend Developer Members Testing v10 v11 v12 v13

You may have never heard of Behaviour Driven Development (BDD) but consider it an option for your Christmas wishlist this year, you too can elevate your automated testing of Umbraco.

BDD, the bigger sibling of Test Driven Development (TDD) is aimed at testing the behaviour of an application or integration from end-user perspectives versus testing smaller pieces (units) of functionality.

SpecFlow, a BDD framework for the .NET ecosystem, tests features and scenarios written in the Gherkin format. This is a human readable format written in spoken language, allowing your team to create a shared understanding of the behaviour of the feature you’re building and testing.

Feature: Membership

    Scenario: Creating a new member
        Given there are no members
        And the register page exists
        When I am on the register page
        And I create the following members
          | Username | Name | Email            | Password       |
          | anne     | Anne | anne@example.org | I_Love_Bob_23! |
          | bob      | Bob  | bob@example.com  | Umbraco4Life$  |
        Then there should be 2 members

An example Gherkin definition - Membership.feature

This format outlines the desired outcome based on the expected behaviour of an application given a known initial state:

  • Given the initial state of the application;
  • When I interact with the application;
  • Then the state of the application should now be.

You may also be able to relate the Given, When and Then format to Arrange, Act and Assert discussed elsewhere in automated testing.


Gherkin is not limited to defining specifications in English, so there's no need to break out the Babel-fish to describe your behaviours, just write in your team’s supported language of choice!

Setting up SpecFlow

Let's walk through setting up a SpecFlow project to test an example Umbraco site.  

In the following example, we'll be testing the Membership feature developed using Umbraco's built in UmbRegisterController with the Gherkin feature file above.

Creating a new Project

A dotnet new template package exists for SpecFlow, that can be installed and run to bootstrap an example testing project: SpecFlow.Templates.DotNet.

The specflowproject template can be run with additional arguments to specify the .NET framework and testing provider you'd like to use, and also whether to include the useful FluentAssertions library.

For our use case, we'll include a few additional dependencies, Microsoft.AspNetCore.Mvc.Testing to host the site in-memory and AngleSharp to parse and interact with the returned HTML documents.


Be sure to match the versions of .NET for the SpecFlow project and the Microsoft.AspNetCore.Mvc.Testing package to the .NET version of your Umbraco web project.

# Install the template and create a new SpecFlow project
dotnet new --install SpecFlow.Templates.DotNet
dotnet new specflowproject -o MyProject.Specs -f net6.0 -t xunit -fa true
dotnet sln add MyProject.Specs

# Reference the Umbraco web project
dotnet add MyProject.Specs reference MyProject.Web

# Install additional dependencies
dotnet add MyProject.Specs package Microsoft.AspNetCore.Mvc.Testing
dotnet add MyProject.Specs package AngleSharp
dotnet add MyProject.Specs package AngleSharp.Io

Command line steps for installing and creating a SpecFlow project

After following the steps above, your solution should have a new SpecFlow testing project with the following folder structure:

  • Drivers - Where we define common operations;
  • Features - Where your Gherkin feature files live;
  • Hooks - Where we can hook into events during the testing lifecycle;
  • Steps - Where we will wire up the behaviours defined in our feature file.

You can rename these folders or add more to group your code if you wish. All code in our example project written for this article is available on GitHub for you to explore.

Screenshot of the solution explorer showing a typical SpecFlow project structure: Drivers, Features, Hooks and Steps folders with feature files and appropriate classes in each

A basic SpecFlow project structure

Defining Steps

As the developer implementing the tests, you will need to create step definition methods for each of the lines within the Gherkin feature file. Performing the actions they describe within the context of the application feature under test.

These are the methods decorated with [Given("")], [When("")] and [Then("")] attributes in MembershipStepDefinitions below.

The [Binding] attribute informs SpecFlow to bind the Gherkin specification to your classes and methods containing your step definitions.

public sealed class MembershipStepDefinitions
    private readonly AngleSharpWebDriver _webDriver;
    private readonly UmbracoDriver _umbracoDriver;

    public MembershipStepDefinitions(
        AngleSharpWebDriver webDriver,
        UmbracoDriver umbracoDriver)
        _webDriver = webDriver;
        _umbracoDriver = umbracoDriver;

    [Given(@"the register page exists")]
    public void GivenTheRegisterPageExists() => _umbracoDriver.EnsureRegisterPageExists();

    [Given(@"there are no members")]
    public void GivenThereAreNoMembers() => _umbracoDriver.EnsureNumberOfMembers(0);

    [When(@"I am on the register page")]
    public async Task WhenIAmOnTheRegisterPage() => await _webDriver.VisitPageAsync("/register/");

    [When(@"I create the following members")]
    public async Task WhenICreateTheFollowingMembers(Table table)
        foreach (var row in table.Rows)
            _webDriver.InputValue("RegisterModel.Name", row["Name"]);
            _webDriver.InputValue("RegisterModel.Email", row["Email"]);
            _webDriver.InputValue("RegisterModel.Password", row["Password"]);
            _webDriver.InputValue("RegisterModel.ConfirmPassword", row["Password"]);

            await _webDriver.SubmitFormAsync();

    [Then(@"there should be (.*) members")]
    public void ThenThereShouldBeMembers(int numberOfMembers) => _umbracoDriver.EnsureNumberOfMembers(numberOfMembers);

Step definitions for the Membership feature - MembershipStepDefinitions.cs

It's also possible to take in values to these methods through regular expression matching and defining data tables in the feature file.

In our example, we do this to create a list of members and check the expected membership count.

Drivers Group Common Operations

A common pattern used within BDD is defining Drivers, allowing you to encapsulate common operations to alter the state of the application under test.

To simplify interaction with Umbraco from our step definitions, helper methods can be created to ensure content nodes (the registration page) have been created and check the current number of members in our site.

public class UmbracoDriver
    private readonly IContentService _contentService;
    private readonly IMemberService _memberService;

    public UmbracoDriver(TestWebContext webContext)
        _contentService = webContext.Services.GetRequiredService<IContentService>();
        _memberService = webContext.Services.GetRequiredService<IMemberService>();

    public IContent EnsureHomePageExists()
        var homePage = _contentService.Create("Home", Constants.System.Root, Home.ModelTypeAlias);

        var result = _contentService.SaveAndPublish(homePage);

        return homePage;

    public IContent EnsureRegisterPageExists()
        var homePage = EnsureHomePageCreated();

        var registerPage = _contentService.Create("Register", homePage, Register.ModelTypeAlias);

        var result = _contentService.SaveAndPublish(registerPage);

        return registerPage;

    public void EnsureNumberOfMembers(int numberOfMembers) => _memberService.Count().Should().Be(numberOfMembers);

Driver for interacting with Umbraco services - UmbracoDriver.cs

To download and interact with web pages, a Driver can be created wrapping around our web application and AngleSharp's browsing context tracking the current document we're browsing.

Common interactions with a HTML document have been defined in our AngleSharpWebDriver to visit a URL, enter a value into an <input /> on a <form> and submit said form.

public class AngleSharpWebDriver
    private readonly IBrowsingContext _browsingContext;
    private readonly Url _baseUrl;

    private IDocument? _document;

    public AngleSharpWebDriver(TestWebContext webContext)
        var httpClient = webContext.Client;

        _baseUrl = new Url(httpClient.BaseAddress!.ToString());

        var requester = new HttpClientRequester(httpClient);

        var configuration = Configuration.Default

        _browsingContext = BrowsingContext.New(configuration);

    public async Task VisitPageAsync(string path)
        var url = new Url(_baseUrl, path);
        _document = await _browsingContext.OpenAsync(url);

    public void InputValue(string field, string value)

        var form = _document.Forms.FirstOrDefault();


        var input = form!
            .FirstOrDefault(i => i.Name == field);


        input!.Value = value;

    public async Task SubmitFormAsync()

        var form = _document.Forms.FirstOrDefault();


        _document = await form!.SubmitAsync();

Driver for interacting with webpages - AngleSharpWebDriver.cs

Sharing State with Contexts

SpecFlow has a unique dependency injection capability allowing you to inject context and share application state between steps in a scenario.

This context has been consumed from our Drivers but can also be directly accessed from within Step Definitions (and Hooks, more on that later).


The lifecycle of a context is limited to a scenario and shared between steps in the scenario. It cannot be shared between scenarios or features.

public class TestWebContext : IDisposable
    private readonly TestWebApplicationFactory _applicationFactory = new();

    public HttpClient Client => _applicationFactory.CreateClient();

    public IServiceProvider Services => _applicationFactory.Services;

    public void Dispose() => _applicationFactory.Dispose();

Our shared context, storing application state - TestWebContext.cs

In our example, the web application can be initialised using TestWebApplicationFactory a custom WebApplicationFactory<>, which exposes a custom HttpClient for interfacing via HTTP requests to the in-memory web server and accessing services via the ServiceProvider for our Umbraco site.


Remember to cleanup your resources! SpecFlow ensures they're disposed of correctly between scenario executions when contexts implement the IDisposable interface.

Working with WebApplicationFactory

All of this testing is possible to be run in-memory with the power of the Microsoft.AspNetCore.Mvc.Testing package and the Microsoft.Data.Sqlite provider for Umbraco.

This removes the dependency on an external database or spinning up a full Kestrel or IIS server to run your tests against. Allowing you to run the tests independently within a CI/CD pipeline.

public class TestWebApplicationFactory : WebApplicationFactory<Startup>
    private readonly string _connectionString = "Data Source=InMemory;Mode=Memory;Cache=Shared;Pooling=True";
    private readonly SqliteConnection _sharedConnection;

    public TestWebApplicationFactory()
        _sharedConnection = new SqliteConnection(_connectionString);

    protected override void ConfigureWebHost(IWebHostBuilder builder)
        var currentDirectory = Directory.GetCurrentDirectory();
        var path = Path.Combine(currentDirectory, "appsettings.Testing.json");

        builder.ConfigureAppConfiguration(config =>


    protected override void Dispose(bool disposing)

The factory to configure the application - TestWebApplicationFactory.cs

The custom WebApplicationFactory<> wraps around and adapts the Startup commonly found in Umbraco web projects, configuring the App Settings and Connections Strings.

To ensure a SQLite connection is shared and maintained throughout the tests, we create and open a connection at the factory level.

Custom App Settings File

To aide in initialisation, you can use a custom App Settings configuration file to setup Umbraco in unattended mode, allowing for a fresh install to be created on every run.

It's recommended to use a utility like uSync to create, persist and recreate the required Settings for your site. Running an import of Content Types, Data Types and Member Types the on first boot.

  "Umbraco": {
    "CMS": {
      "Unattended": {
        "UpgradeUnattended": true,
        "InstallUnattended": true,
        "UnattendedUserName": "admin",
        "UnattendedUserEmail": "admin@example.org",
        "UnattendedUserPassword": "I_Am_R00T!"
  "uSync": {
    "Settings": {
      "ImportOnFirstBoot": true
  "ConnectionStrings": {
    "umbracoDbDSN": "Data Source=InMemory;Mode=Memory;Cache=Shared;Pooling=True",
    "umbracoDbDSN_ProviderName": "Microsoft.Data.Sqlite"

Testing specific configuration - appsettings.Testing.json

Hook Into Events

There are actually two kinds of bindings in SpecFlow; firstly Step Definitions as previously discussed and secondly Hooks for defining operations to occur before or after your test suite, features or scenarios run.

Hooks can be useful to construct and tear down a shared application state that you expect as part of your test suite across features, scenarios or steps.

Additionally bindings can be restricted and scoped to run only for specific features, scenarios or tags. This can be done using the [Scope] attribute as seen below in ExampleHooks.


Hooks that bind to before and after events for the test run and features must be static methods.

public class ExampleHooks
    public static void BeforeTestRun()

    public static void AfterTestRun()

    [Scope(Feature = "Membership")]
    public static void BeforeMembershipFeature()

    [Scope(Feature = "Membership")]
    public static void AfterMembershipFeature()

    [Scope(Feature = "Membership", Scenario = "Creating a new member")]
    public void BeforeCreatingANewMemberScenario()

    [Scope(Feature = "Membership", Scenario = "Creating a new member")]
    public void AfterCreatingANewMemberScenario()

Example hook methods for binding to scenario, feature or test events - ExampleHooks.cs

What Next?

Here is an non-exhaustive list of potential ways you could extend the example above:

  • Match the members data to an expected list of members;
  • Validate error messages displaying on the form when invalid information is submitted;
  • Ensure labels, placeholders and ARIA definitions are on the inputs to meet accessibility requirements;
  • Check the page redirects correctly after submission.

The opportunity to test the behaviour of your application over individual units is endless.

Maybe you have built a package you'd like to perform automated testing on but don't want to write a Mock / Fake for every dependent Umbraco service?

Try integration testing with an example site like above.

Potentially take your testing one step further and use Playwright to carry out end-to-end behaviour testing of your site, all from within .NET!