Learn how to import external data with EF Core and then work with it in the Backoffice as if it was content nodes using Umbraco UI Builder.
Setup
The examples throughout this blogpost will be running Umbraco 12.3.1.
I have a simple setup to begin with where I have a website project running Umbraco, and a referenced Persistence project for all the EF Core data handling. It looks like this:
Initial project structure
For the data example I had ChatGPT help in generating a list of christmas movies with some additional meta data. The dataset looks like this and is an Excel sheet:
Firstly, to import the movie data we need to set up the database tables to hold this data.
To do this we will use EF Core as it makes it pretty easy for us to manage related data.
The Genre column is a comma separated list of genres, we want to split those off into their own table and then have a many to many relation between the genres and the movies.
First of all we need to install the Umbraco.Cms.Persistence.EFCorepackage in the persistence project, so install the package through NuGet or via the dotnet CLI.
Set up the EF Core database models
Next we add a Models folder with a Movie.cs file:
using System.ComponentModel.DataAnnotations.Schema;
namespace TwentyFourDays.Persistence.Models;
[Table("movie")]
public class Movie
{
public int Id { get; set; }
public required string Name { get; set; }
public int ReleaseYear { get; set; }
public string? MainActor { get; set; }
public List<MovieGenre> Genres { get; set; }
}
[Table("movieGenre")]
public class MovieGenre
{
public int Id { get; set; }
public required string Name { get; set; }
public List<Movie> Movies { get; set; }
}
Tablename will automatically become the class name, but if you want to overwrite it to e.g. keep with lowercased table names you can add a data annotation.
The important thing to note here is that a movie has a list of MovieGenre and a MovieGenre has a list of Movie. This is what tells EF Core that they have a many to many relationship.
Set up the EF Core database context
To set up the models as EF Core models that relate to database tables we need to set up the EF Core DbContext.
This is what we can use to perform CRUD operations on the db with strongly typed models.
There are many ways to configure the DbContext, but in this example I will show how to set it up to auto generate ids when inserting data, as well as how to automatically generate the many to many relation table.
In a new DbContexts/MovieContext.cs file I've added the following:
using Microsoft.EntityFrameworkCore;
using TwentyFourDays.Persistence.Models;
namespace TwentyFourDays.Persistence.DbContexts;
public class MovieContext : DbContext
{
public MovieContext(DbContextOptions<MovieContext> options) : base(options)
{
}
public required DbSet<Movie> Movies { get; set; }
public required DbSet<MovieGenre> MovieGenres { get; set; }
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<Movie>(entity =>
{
entity.HasKey(e => e.Id);
entity.Property(e => e.Id).ValueGeneratedOnAdd();
});
modelBuilder.Entity<Movie>()
.HasMany(e => e.Genres)
.WithMany(e => e.Movies)
.UsingEntity(e => e.ToTable("movieToMovieGenre"))
.HasKey(e => e.Id);
}
}
The bottom part saying the Movie entity has many Genres which again has many Movies is how EF Core knows that it's a many to many relation. We can then specify the tablename (movieToMovieGenre) to generate for the relations table - meaning we don't need an explicit model for it.
Once this is done we need to register the MovieContext in DI:
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using TwentyFourDays.Persistence.DbContexts;
using Umbraco.Cms.Core.Composing;
using Umbraco.Cms.Core.DependencyInjection;
using Umbraco.Extensions;
namespace TwentyFourDays.Persistence;
public class Composer : IComposer
{
public void Compose(IUmbracoBuilder builder)
{
builder.Services.AddUmbracoDbContext<MovieContext>((serviceProvider, options) =>
{
options.UseUmbracoDatabaseProvider(serviceProvider);
});
}
}
Generating migrations
Next we generate the EF Core migrations code. First of all we need to install another package in the website project:
This is what allows us to generate migrations code via the command line.
If you do not have it installed already you also need the EF Core CLI tool. It can be installed by running dotnet tool install --global dotnet-ef in the terminal.
Next we can generate the migrations, for a place to put them I've added a Generated folder in the root of the Persistence project.
Assuming you are in the root of the persistence project with your terminal you can call this command:
So we give it a name (CreateTables), a reference to the Context, an output path and a source project path pointing to the website project.
Now we have the generated migration:
Seeing the generated migration
The final thing to do, is hook it up so the website runs any pending migrations on startup (otherwise you'd need to do another CLI tool command each time you want to apply migrations).
For this we will use an Umbraco INotificationHandler targeting the UmbracoApplicationStartedNotification.
In a new Migrations/MovieMigration.cs file I've added:
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Logging;
using TwentyFourDays.Persistence.DbContexts;
using Umbraco.Cms.Core.Events;
using Umbraco.Cms.Core.Notifications;
namespace TwentyFourDays.Persistence.Migrations;
public class MovieMigration : INotificationAsyncHandler<UmbracoApplicationStartedNotification>
{
private readonly MovieContext _movieContext;
private readonly ILogger<MovieMigration> _logger;
public MovieMigration(MovieContext movieContext, ILogger<MovieMigration> logger)
{
_movieContext = movieContext;
_logger = logger;
}
public async Task HandleAsync(UmbracoApplicationStartedNotification notification, CancellationToken cancellationToken)
{
_logger.LogInformation("Starting movie migrations");
var pendingMigrations = await _movieContext.Database.GetPendingMigrationsAsync(cancellationToken);
if (pendingMigrations.Any())
{
_logger.LogInformation($"Found pending movie migrations: {string.Join(',', pendingMigrations)}");
await _movieContext.Database.MigrateAsync(cancellationToken);
}
_logger.LogInformation("Completed movie migrations");
}
}
Next we set up a MoviesImportService that can handle all the parsing logic:
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using OfficeOpenXml;
using Umbraco.Cms.Core.Extensions;
namespace TwentyFourDays.Persistence.Services;
public class MoviesImportService
{
private readonly ILogger<MoviesImportService> _logger;
private readonly IHostEnvironment _hostEnvironment;
public MoviesImportService(ILogger<MoviesImportService> logger, IHostEnvironment hostEnvironment)
{
_logger = logger;
_hostEnvironment = hostEnvironment;
}
public async Task Import()
{
_logger.LogInformation("Starting movies import");
var excelPath = Path.Combine(_hostEnvironment.MapPathContentRoot(Umbraco.Cms.Core.Constants.SystemDirectories.Data), "chatgpt-25-christmas-movies.xlsx");
if (!File.Exists(excelPath))
{
_logger.LogError($"Couldn't find movies file on path: {excelPath}");
return;
}
// Adapted from https://www.c-sharpcorner.com/article/using-epplus-to-import-and-export-data-in-asp-net-core/
await using var fileStream = File.Open(excelPath, FileMode.Open);
using var package = new ExcelPackage(fileStream);
var worksheet = package.Workbook.Worksheets.FirstOrDefault();
if (worksheet is null)
{
_logger.LogError("Couldn't find worksheet");
return;
}
// TODO: Map movies from excel to Movie model
// TODO: Insert movies to db
_logger.LogInformation("Import complete");
}
}
In my case the Excel sheet is in the websites /umbraco/Data folder so I can easily target its path via some Umbraco extension methods and constants as shown above:
Location of the Excel file
Next we add some methods that help parse through this data:
private static IEnumerable<Movie> MapMoviesFromExcel(ExcelWorksheet worksheet)
{
var movies = new List<Movie>();
var rowCount = worksheet.Dimension.Rows;
// Starts on row 2 as row 1 are headers
for (var row = 2; row <= rowCount; row++)
{
var name = worksheet.Cells[row, 1].Value?.ToString()?.Trim() ?? "";
int.TryParse(worksheet.Cells[row, 2].Value?.ToString()?.Trim(), out var releaseYear);
var genres = GetGenres(worksheet.Cells[row, 3].Value?.ToString()?.Trim());
var mainActor = worksheet.Cells[row, 4].Value?.ToString()?.Trim() ?? "";
movies.Add(new Movie
{
Name = name,
ReleaseYear = releaseYear,
Genres = genres,
MainActor = mainActor
});
}
return movies;
}
private static List<MovieGenre> GetGenres(string? value)
{
var movieGenres = new List<MovieGenre>();
if (string.IsNullOrWhiteSpace(value))
{
return movieGenres;
}
var genres = value.Split(',');
foreach (var genre in genres)
{
var trimmedGenre = genre.Trim();
if (string.IsNullOrWhiteSpace(trimmedGenre))
{
continue;
}
movieGenres.Add(new MovieGenre
{
Name = trimmedGenre
});
}
return movieGenres;
}
So we just get the content of each field, and map it into the Movie model. The only special thing is we take the content of the genre field - split on comma and create a new list of MovieGenre that gets added to the Movie.
The final step is to insert the movies into the database, for that we will create a new repository at Repositories/MovieRepository.cs:
using TwentyFourDays.Persistence.DbContexts;
using TwentyFourDays.Persistence.Models;
using Umbraco.Cms.Persistence.EFCore.Scoping;
namespace TwentyFourDays.Persistence.Repositories;
public class MovieRepository
{
private readonly IEFCoreScopeProvider<MovieContext> _scopeProvider;
public MovieRepository(IEFCoreScopeProvider<MovieContext> scopeProvider)
{
_scopeProvider = scopeProvider;
}
public async Task Insert(Movie movie)
{
using var scope = _scopeProvider.CreateScope();
await scope.ExecuteWithContextAsync<Task>(async db =>
{
var moviesFromDb = db.Movies;
var movieExists = moviesFromDb.FirstOrDefault(x => x.Id == movie.Id);
if (movieExists is not null)
{
return;
}
var movieGenresFromDb = db.MovieGenres;
var movieGenreList = new List<MovieGenre>();
foreach (var movieGenre in movie.Genres)
{
// If a genre of the same name already exists then we just use that as to not have duplicates
var exists = movieGenresFromDb.FirstOrDefault(x => x.Name == movieGenre.Name);
movieGenreList.Add(exists ?? movieGenre);
}
movie.Genres = movieGenreList;
db.Movies.Add(movie);
await db.SaveChangesAsync();
});
scope.Complete();
}
}
For each movie we check if one with the same ID exists and then return it right away. In the case of calling it with the Excel sheet data that will never happen as they don't have an ID, but this repo could be called from other code.
We also need to check if a movieGenre with the same name exists, as otherwise we will have lots of duplicates.
It would technically be better to bulk insert performance wise, but then we would need to manage the ids of all movie genres ourselves.
Now we can finish up the import service:
var movies = MapMoviesFromExcel(worksheet);
foreach (var movie in movies)
{
await _movieRepository.Insert(movie);
}
Finally we can call the importservice from somewhere - for quick testing I added an API controller, but down the line it would probably be something like a Hangfire job or some other button that can be pushed in the backoffice.
After triggering the import, data is populated in the database tables:
To be able to see our movies and edit them in the backoffice we can use the Umbraco UI Builder package.
First of all we need to install the NuGet package Umbraco.UIBuilder.
Next we need to set up a UI Builder repository - which is what the package needs to know how to get 1 movie, get a list of movies, save a movie, etc.
So in the website within a new UiBuilder folder I'll add a MovieUiBuilderRepository. It inherits from Umbraco.UIBuilder.Persistence.Repository<Movie, int> where the types are the entity type and the primary id type. In my case the entity is my Movie class, and its Id property is an int.
You can use your IDE to help generate the missing members to get a starting point like this:
using System.Linq.Expressions;
using TwentyFourDays.Persistence.Models;
using Umbraco.Cms.Core.Models;
using Umbraco.UIBuilder;
using Umbraco.UIBuilder.Persistence;
namespace TwentyFourDays.UiBuilder;
public class MovieUiBuilderRepository : Repository<Movie, int>
{
public MovieUiBuilderRepository(RepositoryContext context) : base(context)
{
}
protected override int GetIdImpl(Movie entity)
{
throw new NotImplementedException();
}
protected override Movie GetImpl(int id)
{
throw new NotImplementedException();
}
protected override Movie SaveImpl(Movie entity)
{
throw new NotImplementedException();
}
protected override void DeleteImpl(int id)
{
throw new NotImplementedException();
}
protected override IEnumerable<Movie> GetAllImpl(Expression<Func<Movie, bool>> whereClause, Expression<Func<Movie, object>> orderBy, SortDirection orderByDirection)
{
throw new NotImplementedException();
}
protected override PagedResult<Movie> GetPagedImpl(int pageNumber, int pageSize, Expression<Func<Movie, bool>> whereClause, Expression<Func<Movie, object>> orderBy,
SortDirection orderByDirection)
{
throw new NotImplementedException();
}
protected override long GetCountImpl(Expression<Func<Movie, bool>> whereClause)
{
throw new NotImplementedException();
}
}
The first method is easy enough - it is how to get the unique Id from the entity:
protected override int GetIdImpl(Movie entity)
{
return entity.Id;
}
However, all of the other methods require some additional work, we need a method for the following things:
Get a Movie from an Id
Save a Movie entity
Delete a Movie by its Id
GetAll Movies with a where expression, order expression and order direction
Get a page of Movies with paging and a where expression, order expression and order direction
Get a count of Movies based on a where clause
So let's head back to the MovieRepository we used before to insert data from the excel sheet and create some corresponding methods using EF Core to do that!
Get all movies
We are starting at the bottom, because the list of movies is the first thing we can test.
For the different needs we have for getting a list of movies we can do one method in our MovieRepository that gets passed the Expressions from the Ui Builder override methods.
We can also add optional paging as some of the overrided methods need it and others do not:
public async Task<(int TotalResults, IEnumerable<Movie>? Movies)> GetAll(
Expression<Func<Movie, bool>>? whereClause,
Expression<Func<Movie, object>>? orderBy,
bool ascending,
int? skip = null,
int? take = null)
{
using var scope = _scopeProvider.CreateScope();
var totalResults = 0;
var items = await scope.ExecuteWithContextAsync(async db =>
{
var movies = db.Movies.AsQueryable();
if (whereClause is not null)
{
movies = movies.Where(whereClause);
}
if (orderBy is not null)
{
movies = ascending ? movies.OrderBy(orderBy) : movies.OrderByDescending(orderBy);
}
totalResults = movies.Count();
if (skip is not null && take is not null)
{
movies = movies.Skip(skip.Value).Take(take.Value);
}
return movies;
});
scope.Complete();
return (totalResults, items.ToList());
}
You may notice that we can get all Movies from the database as IQueryable by calling db.Movies.AsQueryable() and then we can do the optional where filtering and ordering to build our final Movie collection.
At this point we can finish a few of the UI Builder repository methods:
You may notice that we have to do async calls in sync code here. Unfortunately Umbraco UI Builder does not support async repositories at the time of writing this - there is anopen discussion that you can voice your support for.
At this point we can add our UI Builder config to get a section in the backoffice that loads our movies by adding a UiBuilderConfigurator file:
using TwentyFourDays.Persistence.Models;
using Umbraco.UIBuilder.Configuration;
using Umbraco.UIBuilder.Configuration.Builders;
namespace TwentyFourDays.UiBuilder;
public class UiBuilderConfigurator : IConfigurator
{
public void Configure(UIBuilderConfigBuilder builder)
{
builder.AddSection("Movies", sectionConfig => sectionConfig
.Tree(treeConfig => treeConfig
.AddCollection<Movie>(
x => x.Id,
"Movie",
"Movies",
"List of movies",
"icon-movie",
"icon-movie",
collectionConfig => collectionConfig
.SetRepositoryType<MovieUiBuilderRepository>())));
}
}
Before my user can see the new section I need to add it under one of my user groups, then I can start my site up, and load the collection which uses the MovieUiBuilderRepository for the CRUD operations.
For now we haven't configured it much and can just see that there is some sort of data:
Obviously we dont want the listview to display the model name. We can extend the collectionConfig in the UiBuilderConfigurator to add some extra listview config:
With this bit of config we set the name property for our "nodes", and say we want a column in the listview for releaseyear and mainactor.
At this point due to already implementing the whereClause and orderBy in our fetch of the movies it is possible to click columns to sort by that property. We can also search on the name out of the box.
I will add a bit of extra configuration for the default sort to be by year and to enable searching by actor name as well:
This is already quite nice, however if we click a movie an error is thrown. So let's move on to the next override method in the UI Builder repository.
Get a movie by its Id
First in the MovieRepository we create our new GetById method:
public async Task<Movie?> GetById(int id)
{
using var scope = _scopeProvider.CreateScope();
var movie = await scope.ExecuteWithContextAsync(async db =>
{
return db.Movies.Include(x => x.Genres).FirstOrDefault(x => x.Id == id);
});
scope.Complete();
return movie;
}
Only thing to note here, if you are not that familiar with EF Core, is that if we had just called the db.Movies.FirstOrDefault without a .Include(x => x.Genres) then we would get the movies but without their Genre relations from the MovieGenre table. But since EF Core knows about their relationship and the Movie class has a Genres property with a list of Genres then we can include it in our query really easily.
Now we can call this new method from our MovieUiBuilderRepository:
protected override Movie GetImpl(int id)
{
return _movieRepository.GetById(id).GetAwaiter().GetResult();
}
Now if we boot up the site and click one of the movies in the listview it will open the movie - however we still haven't configured the different fields to show in the editor, so for now it will only have its name.
Setting up editor fields
Let's add our fields to our UiBuilderConfigurator:
If data is of a simple type then the UI Builder package can automatically pick a suitable data type - however, it is possible to set your own.
If I boot up the site I can see that the Release Year is a standard numeric type where I cannot type letters, and Main Actor is is a standard textstring. However Genres is in our case of the type List<MovieGenre> which it understandably cannot figure out:
Initial editor fields
Using custom dataeditors in UI Builder
For the movie genres I'd like to show a list with a checkbox per genre that we have in the database, and then allow the user to select multiple of them.
One of the great strengths of UI Builder is the fact that we can use any property editor. To make it easy I will install the Our.Umbraco.Community.Contentment package and use that.
First of all we will make a new method in our MovieRepository that allows us to pull a list of Movie Genres:
public async Task<IEnumerable<MovieGenre>> GetAll()
{
using var scope = _scopeProvider.CreateScope();
var items = await scope.ExecuteWithContextAsync(async db => db.MovieGenres);
scope.Complete();
return items.ToList();
}
Note
Make sure to ToList() the items as they will otherwise we disposed by the time you need them.
And then we can create a DataSource for Contentment in Contentment/GenreDataSource.cs:
using TwentyFourDays.Persistence.Repositories;
using Umbraco.Cms.Core.PropertyEditors;
using Umbraco.Community.Contentment.DataEditors;
namespace TwentyFourDays.Contentment;
public class GenreDataSource : IDataListSource
{
private readonly MovieRepository _movieRepository;
public string Name => "Movie Genre DataSource";
public string Description => "List of genres for movies";
public string Icon => "icon-movie-alt";
public Dictionary<string, object> DefaultValues => default;
public IEnumerable<ConfigurationField> Fields => default;
public string Group => "Custom data sources";
public OverlaySize OverlaySize => OverlaySize.Medium;
public GenreDataSource(MovieRepository movieRepository)
{
_movieRepository = movieRepository;
}
public IEnumerable<DataListItem> GetItems(Dictionary<string, object> config)
{
var genres = _movieRepository.GetAllGenres().GetAwaiter().GetResult();
var dataList = new List<DataListItem>();
foreach (var genre in genres)
{
dataList.Add(new DataListItem()
{
Name = genre.Name,
Value = genre.Id.ToString()
});
}
return dataList;
}
}
And then boot up the site and create the new Contentment dataeditor:
Contentment datalist
We can easily add this new datatype to the field in our UiBuilderConfigurator file:
.AddField(x => x.Genres).SetDataType("Movie Genre List Picker")
This gives us the prevalues for the checkboxes already, however to map the values from the Movie class with its List property to a value that the Contentment checkbox list can read we need to add a ValueMapper.
This is something that maps values from the dataeditor -> database and the other way (read more in the Umbraco docs).
So we add a new file:
using TwentyFourDays.Persistence.Models;
using Umbraco.UIBuilder.Mapping;
namespace TwentyFourDays.UiBuilder;
public class MovieGenreValueMapper : ValueMapper
{
public override object ModelToEditor(object input)
{
if (input is List<MovieGenre> movieGenres)
{
if (movieGenres.Count == 1)
{
return movieGenres.Single().Id;
}
var res = movieGenres.Select(x => x.Id).ToArray();
// Contentment checkbox list expects data as - ["val1", "val2"] - so have to convert it to this format
return $"[\"{string.Join("\", \"", res)}\"]";
}
return input;
}
public override object EditorToModel(object input)
{
var movieGenres = new List<MovieGenre>();
if (input is not string inputString)
{
return movieGenres;
}
var movieGenreStrings = inputString
.Trim('[')
.Trim(']')
.Replace("\"", "")
.Split(',');
if (!movieGenreStrings.Any())
{
return movieGenres;
}
foreach (var movieGenreString in movieGenreStrings)
{
movieGenres.Add(new MovieGenre
{
Name = movieGenreString.Trim()
});
}
return movieGenres;
}
}
We can then add it to the field:
.AddField(x => x.Genres).SetDataType("Movie Genre List Picker").SetValueMapper<MovieGenreValueMapper>()
And now it maps the values correctly - atleast when loading:
Editor fields with Contentment
Save a Movie
Next we need a method to save the Movie - it is important to note that the Ui Builder repository uses the save method for both saving existing- and adding new entities.
We already have an insert method from the data import - however it had to have a bunch of logic to handle inserting genres without Ids. So we will create a new InsertOrUpdate method:
public async Task<Movie?> InsertOrUpdate(Movie movie)
{
using var scope = _scopeProvider.CreateScope();
var result = await scope.ExecuteWithContextAsync<Movie>(async db =>
{
var movieGenresFromDb = db.MovieGenres;
var moviesFromDb = db.Movies.Include(x => x.Genres);
var movieFromDb = moviesFromDb.FirstOrDefault(x => x.Id == movie.Id);
var movieGenres = new List<MovieGenre>();
if (movieFromDb is not null)
{
// Set all property values to those of the incoming movie instead.
// Can't just save movie as it counts as a "new object", need to instead update the corresponding obj from the db
db.Entry(movieFromDb).CurrentValues.SetValues(movie);
foreach (var genre in movie.Genres)
{
// We need to add the genre objs from the database otherwise it will insert duplicates with different ids
var genreFromDb = movieGenresFromDb.FirstOrDefault(x => x.Id.ToString() == genre.Name);
movieGenres.Add(genreFromDb ?? genre);
}
movieFromDb.Genres = movieGenres;
db.Movies.Update(movieFromDb);
await db.SaveChangesAsync();
return movieFromDb;
}
else
{
foreach (var genre in movie.Genres)
{
// We need to add the genre objs from the database otherwise it will insert duplicates with different ids
var genreFromDb = movieGenresFromDb.FirstOrDefault(x => x.Id.ToString() == genre.Name);
movieGenres.Add(genreFromDb ?? genre);
}
movie.Genres = movieGenres;
db.Movies.Update(movie);
await db.SaveChangesAsync();
return movie;
}
});
scope.Complete();
return result;
}
Main thing to note here is that in EF Core they track their entities, which means if I create a new Movie entity and set its values to all of the same values - including the ID - of an existing one, then it will throw an error when trying to update it as an entity with the same ID exists.
That is why we need to instead match our incoming movie object to an existing one if its there, and then we can update all of the existing movie objects properties to the new ones, which is what happens in this line:
You may also wonder why we call db.Movies.Update(movieFromDb) when adding. In EF Core update does an update if the entity already exists, otherwise it does an add.
Now we can call this method from the UiBuilder repository:
protected override Movie SaveImpl(Movie entity)
{
return _movieRepository.InsertOrUpdate(entity).GetAwaiter().GetResult();
}
Now we can go in and see that we can save a movie, and that changes are carried through to the database:
Testing saving a movie
Delete a movie
To delete a movie we can get it by Id and then delete the Movie entity. We already have a method to get by id, so let's create another to delete from an entity:
public async Task Delete(Movie movie)
{
using var scope = _scopeProvider.CreateScope();
await scope.ExecuteWithContextAsync<Task>(async db =>
{
db.Movies.Remove(movie);
await db.SaveChangesAsync();
});
scope.Complete();
}
Now we can use this in our Ui Builder repository:
protected override void DeleteImpl(int id)
{
var movie = _movieRepository.GetById(id).GetAwaiter().GetResult();
if (movie is null)
{
return;
}
_movieRepository.Delete(movie).GetAwaiter().GetResult();
}
At this point we support all the functionality of the Ui Builder repository and can work with our movies as if they were native Umbraco content.
If you made it this far, I hope you enjoyed the example!