Shazwazza

An Examine fix for Umbraco index corruption

Wed, 31 Jul 2024 17:49:54 Z

A new Examine version 3.3.0 has been released to address a long awaited bug fix for Umbraco websites that use the SyncedFileSystemDirectoryFactory which is the default setting for Umbraco CMS.

The bug typically means that indexes cannot be used and log entries such as Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1 are present.

Understanding the problem:

The SyncedFileSystemDirectoryFactory directory is there to avoid performance implications of rebuilding indexes on startup when a site is moved to another worker in Azure. The reason an index rebuild would occur is because in Azure, Lucene files need to work off of the local 'fast drive' (C:\%temp%), not the default/shared network 'slow drive' (D:), and whenever a site is moved, or spawned on a new worker in Azure, the local 'fast drive' is empty, meaning no indexes exist. The SyncedFileSystemDirectoryFactory attempts to work around this challenge by continually synchronizing a copy of the indexes from the 'fast drive' to the 'slow drive' so that when a site is moved to another worker, it can sync (restore) from the 'slow drive' back to the 'fast drive' in order to avoid the index rebuild overhead.

The problem with SyncedFileSystemDirectoryFactory is that this implementation doesn't take into account what happens if the index files in your main storage ('slow drive') become corrupted which can happen for a number of reasons - misconfiguration, network latency, process termination, etc...

Understanding the solution:

The SyncedFileSystemDirectoryFactory has been updated to:

Check the health of the main index if it exists ('slow drive').
Check the health of the local index if it exists ('fast drive').
If the main index is unhealthy or doesn't exist and the local index is healthy, it will synchronize the local index to the main index. This can occur only if a site hasn't moved to a new worker.
If the main index is unhealthy and the local index doesn't exist or is unhealthy, then it will delete the main (corrupted) index.
Once health checks are done, the index from main is always synced to local. If the main index was deleted due to corruption, this will mean that the local index is empty and an index rebuild will occur.

This change will attempt to keep any healthy index that is available (main vs local), but if nothing can be read, the indexes will be deleted and an index rebuild will occur.

There's also a new option to fix a corrupted index but this is not enabled by default since it can mean a loss of documents.

Understanding the rebuilding overhead

The performance overhead of index rebuilding is due to the Umbraco database queries that need to be executed in order to populate the indexes. The only reason SyncedFileSystemDirectoryFactory exists is to prevent this overhead when hosting in Azure App Service (which is what Umbraco Cloud uses), and it can only be used on your Umbraco primary node. It does not prevent index rebuilding overhead for non-primary nodes when load balancing or scaling out because the main network 'slow drive' is shared between all workers and an index can only be read/written to be a single process.

This means that it's only useful if you are hosting in Azure App Service without any load balancing while keeping in mind that it does not always prevent index rebuilds (see above).

The index rebuilding overhead can be dramatic when load balancing or scaling out, for example: If you scale out to +5 nodes in a load balancing setup, that means that 5x nodes will be performing index rebuilds around the same time, this means that your DB is going to get hammered by queries to build all of those new indexes. The performance hit isn't the index building - it is the DB queries and this can lead to DB locks and lead to the dreaded SQL Lock Timeout issue in the Umbraco back office. Plus, if search is critical to your front-end, than for a while after your site has started up, there won't be any index which means there won't be any search until the background processing is done.

... Many of these reasons is why ExamineX was created:

Reliably and centrally persisted indexes means nothing is out of sync between nodes.
No index rebuilding when your site is moved or scaled = no rebuilding overhead.
Prevents SQL Timeout locks due to DB rebuilding queries.
Very easy to setup and seamlessly changes your Lucene based indexes to Azure/Elastic search indexes.
Ideal when hosting Umbraco on Azure Web Apps (or Umbraco Cloud) and a perfect solution for load balancing and scaling.
Automatically index Umbraco media file content without the need for additional indexes with support for PDFs, Microsoft Office documents and more.
(coming soon) Automatically generate Umbraco media image descriptions, tags, locations, and more using AI allowing your editors to quickly find the media/images they need.

Configuring a Suggester with ExamineX and Azure AI Search

Wed, 01 May 2024 14:51:52 Z

We recently had a customer ask about integrating the Azure AI Search suggester API with ExamineX and this turns out to be quite straight forward :)

First thing is to ensure that the fields you want to use the Suggester for are configured correctly:

cs // Configuration for the External Index for the City and Country fields. // The Azure AI Search Suggester requires fields to be 'string fields only' // and using the Standard Analyzer (default on the External Index) services.Configure<AzureSearchIndexOptions>( Constants.UmbracoIndexes.ExternalIndexName, options => { options.FieldDefinitions = new FieldDefinitionCollection( new FieldDefinition("City", AzureSearchFieldDefinitionTypes.FullText), new FieldDefinition("Country", AzureSearchFieldDefinitionTypes.FulText)); }); Next up, is to create the Suggester which is done as part of the CreatingOrUpdatingIndex event:

```cs externalIndex.CreatingOrUpdatingIndex += AzureIndexCreatingOrUpdatingIndexScoringProfile;

private void AzureIndexCreatingOrUpdatingIndexSuggester(object sender, CreatingOrUpdatingIndexEventArgs e) { switch (e.EventType) { case IndexModifiedEventType.Creating: case IndexModifiedEventType.Rebuilding: var index = e.AzureSearchIndexDefinition;

        var suggester = new SearchSuggester(
            "sg",
            new[] { "Country", "City" });

        index.Suggesters.Add(suggester);

        break;
}

} ``` When data has been indexed for these fields, using the Suggester API is simple. For example, lets assume that the following ValueSets were indexed:

cs externalIndex.IndexItems(new[] { ValueSet.FromObject(id1, new { City = "Calgary", Country = "Canada" }), ValueSet.FromObject(id2, new { City = "Sydney", Country = "Australia" }), ValueSet.FromObject(id3, new { City = "Copenhagen", Country = "Denmark" }), ValueSet.FromObject(id4, new { City = "Vienna", Country = "Austria" }), }); Then to get suggestions for "Au":

cs // cast to AzureSearchIndex to expose underlying Azure Search APIs var azureSearchIndex = (AzureSearchIndex)externalIndex; var result = await azureSearchIndex.IndexClient.SuggestAsync<ExamineDocument>( "Au", "sg");

The result will contain 2x entries, one for the Australia and one for the Austria documents. Super easy!

One thing to watch out for is if your index is not configured to use the Standard Analyzer by default (i.e. like the Internal Index in Umbraco). In that case, you'll need to define a custom field type with that indexer:

```cs services.Configure( Constants.UmbracoIndexes.ExternalIndexName, options => { // This adds a custom field type that uses the Standard Analyzer options.IndexValueTypesFactory = new Dictionary { ["standard"] = new AzureSearchFieldValueTypeFactory(s => new AzureSearchFieldValueType(s, SearchFieldDataType.String, LexicalAnalyzerName.StandardLucene.ToString())) };

    // Set these field types to the one defined above
    options.FieldDefinitions = new FieldDefinitionCollection(
        new FieldDefinition("City", "standard"),
        new FieldDefinition("Country", "standard"));
});

``` Happy Searching!

Can I disable Examine indexes on Umbraco front-end servers?

Thu, 23 Mar 2023 15:10:30 Z

In Umbraco v8, Examine and Lucene are only used for the back office searches, unless you specifically use those APIs for your front-end pages. I recently had a request to know if it’s possible to disable Examine/Lucene for front-end servers since they didn’t use Examine/Lucene APIs at all on their front-end pages… here’s the answer

Why would you want this?

If you are running a Load Balancing setup in Azure App Service then you have the option to scale out (and perhaps you do!). In this case, you need to have the Examine configuration option of:

<add key="Umbraco.Examine.LuceneDirectoryFactory" 
          value="Examine.LuceneEngine.Directories.TempEnvDirectoryFactory, Examine" />

This is because each scaled out worker is running from the same network share file system. Without this setting (or with the SyncTempEnvDirectoryFactory setting) it would mean that each worker will be trying to write Lucene file based indexes to the same location which will result in corrupt indexes and locked files. Using the TempEnvDirectoryFactory means that the indexes will only be stored on the worker's local 'fast drive' which is in it's %temp% folder on the local (non-network share) hard disk.

When a site is moved or provisioned on a new worker the local %temp% location will be empty so Lucene indexes will be rebuilt on startup for that worker. This will occur when Azure moves a site or when a new worker comes online from a scale out action. When indexes are rebuilt, the worker will query the database for the data and depending on how much data you have in your Umbraco installation, this could take a few minutes which can be problematic. Why? Because Umbraco v8 uses distributed SQL locks to ensure data integrity and during these queries a content lock will be created which means other back office operations on content will need to wait. This can end up with SQL Lock timeout issues. An important thing to realize is that these rebuild queries will occur for all new workers, so if you scaled out from 1 to 10, that is 9 new workers coming online at the same time.

How to avoid this problem?

If you use Examine APIs on your front-end, then you cannot just disable Examine/Lucene so the only reasonable solution is to use an Examine implementation that uses a hosted search service like ExamineX

If you don't use Examine APIs on your front-ends then it is a reasonable solution to disable Examine/Lucene on the front-ends to avoid this issue. To do that, you would change the default Umbraco indexes to use an in-memory only store and prohibit data from being put into the indexes. Then disable the queries that execute when Umbraco tries to re-populate the indexes.

Show me the code

First thing is to replace the default index factory. This new one will change the underlying Lucene directory for each index to be a RAMDirectory and will also disable the default Umbraco event handling that populates the indexes. This means Umbraco will not try to update the index based on content, media or member changes.

public class InMemoryExamineIndexFactory : UmbracoIndexesCreator
{
    public InMemoryExamineIndexFactory(
        IProfilingLogger profilingLogger,
        ILocalizationService languageService,
        IPublicAccessService publicAccessService,
        IMemberService memberService,
        IUmbracoIndexConfig umbracoIndexConfig)
        : base(profilingLogger, languageService, publicAccessService, memberService, umbracoIndexConfig)
    {
    }

    public override IEnumerable<IIndex> Create()
    {
        return new[]
        {
            CreateInternalIndex(),
            CreateExternalIndex(),
            CreateMemberIndex()
        };
    }

    // all of the below is the same as Umbraco defaults, except
    // we are using an in-memory Lucene directory.

    private IIndex CreateInternalIndex()
        => new UmbracoContentIndex(
            Constants.UmbracoIndexes.InternalIndexName,
            new RandomIdRamDirectory(), // in-memory dir
            new UmbracoFieldDefinitionCollection(),
            new CultureInvariantWhitespaceAnalyzer(),
            ProfilingLogger,
            LanguageService,
            UmbracoIndexConfig.GetContentValueSetValidator())
        {
            EnableDefaultEventHandler = false
        };

    private IIndex CreateExternalIndex()
        => new UmbracoContentIndex(
            Constants.UmbracoIndexes.ExternalIndexName,
            new RandomIdRamDirectory(), // in-memory dir
            new UmbracoFieldDefinitionCollection(),
            new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30),
            ProfilingLogger,
            LanguageService,
            UmbracoIndexConfig.GetPublishedContentValueSetValidator())
        {
            EnableDefaultEventHandler = false
        };

    private IIndex CreateMemberIndex()
        => new UmbracoMemberIndex(
            Constants.UmbracoIndexes.MembersIndexName,
            new UmbracoFieldDefinitionCollection(),
            new RandomIdRamDirectory(), // in-memory dir
            new CultureInvariantWhitespaceAnalyzer(),
            ProfilingLogger,
            UmbracoIndexConfig.GetMemberValueSetValidator())
        {
            EnableDefaultEventHandler = false
        };

    // required so that each ram dir has a different ID
    private class RandomIdRamDirectory : RAMDirectory
    {
        private readonly string _lockId = Guid.NewGuid().ToString();
        public override string GetLockId()
        {
            return _lockId;
        }
    }
}

The next thing to do is to create no-op index populators to replace the Umbraco default ones. All these do is ensure they are not associated with any index and then just to be sure, does not execute any logic for population.

public class DisabledMemberIndexPopulator : MemberIndexPopulator
{
    public DisabledMemberIndexPopulator(
        IMemberService memberService,
        IValueSetBuilder<IMember> valueSetBuilder)
        : base(memberService, valueSetBuilder)
    {
    }

    public override bool IsRegistered(IIndex index) => false;
    public override bool IsRegistered(IUmbracoMemberIndex index) => false;
    protected override void PopulateIndexes(IReadOnlyList<IIndex> indexes) { }
}

public class DisabledContentIndexPopulator : ContentIndexPopulator
{
    public DisabledContentIndexPopulator(
        IContentService contentService,
        ISqlContext sqlContext,
        IContentValueSetBuilder contentValueSetBuilder)
        : base(contentService, sqlContext, contentValueSetBuilder)
    {
    }

    public override bool IsRegistered(IIndex index) => false;
    public override bool IsRegistered(IUmbracoContentIndex2 index) => false;
    protected override void PopulateIndexes(IReadOnlyList<IIndex> indexes) { }
}

public class DisabledPublishedContentIndexPopulator : PublishedContentIndexPopulator
{
    public DisabledPublishedContentIndexPopulator(
        IContentService contentService,
        ISqlContext sqlContext,
        IPublishedContentValueSetBuilder contentValueSetBuilder)
        : base(contentService, sqlContext, contentValueSetBuilder)
    {
    }

    public override bool IsRegistered(IIndex index) => false;
    public override bool IsRegistered(IUmbracoContentIndex2 index) => false;
    protected override void PopulateIndexes(IReadOnlyList<IIndex> indexes) { }
}

public class DisabledMediaIndexPopulator : MediaIndexPopulator
{
    public DisabledMediaIndexPopulator(
        IMediaService mediaService,
        IValueSetBuilder<IMedia> mediaValueSetBuilder) : base(mediaService, mediaValueSetBuilder)
    {
    }

    public override bool IsRegistered(IIndex index) => false;
    public override bool IsRegistered(IUmbracoContentIndex index) => false;
    protected override void PopulateIndexes(IReadOnlyList<IIndex> indexes) { }
}

Lastly, we just need to enable these services:

public class DisabledExamineComposer : IUserComposer
{
    public void Compose(Composition composition)
    {
        // replace the default
        composition.RegisterUnique<IUmbracoIndexesCreator, InMemoryExamineIndexFactory>();

        // replace the default populators
        composition.Register<MemberIndexPopulator, DisabledMemberIndexPopulator>(Lifetime.Singleton);
        composition.Register<ContentIndexPopulator, DisabledContentIndexPopulator>(Lifetime.Singleton);
        composition.Register<PublishedContentIndexPopulator, DisabledPublishedContentIndexPopulator>(Lifetime.Singleton);
        composition.Register<MediaIndexPopulator, DisabledMediaIndexPopulator>(Lifetime.Singleton);
    }
}

With that all in place, it means that no data will ever be looked up to rebuild indexes and Umbraco will not send data to be indexed. There is nothing here preventing data from being indexed though. For example, if you use the Examine APIs to update the index directly, that data will be indexed in memory. If you wanted to absolutely make sure no data ever went into the index, you would have to override some methods on the RAMDirectory.

Can I run Examine with RAMDirectory with data?

You might have realized with the above that if you don't replace the populators, you will essentially have Examine indexes in Umbraco running from RAMDirectory. Is this ok? Yes absolutely, but that entirely depends on your data set. If you have a large index, that means it will consume large amounts of memory which is typically not a good idea. But if you have a small data set, or you filter the index so that it remains small enough, then yes! You can certainly run Examine with an in-memory directory but this would still only be advised on your front-end/replica servers.

Articulate 4.3.0 with support for markdown code snippets and syntax highlighting

Thu, 23 Mar 2023 15:10:26 Z

I'm happy to announce that Articulate 4.3.0 is shipped and includes a great new feature that I've been wanting/needing:

The ability to create markdown based posts with support for GitHub style code fences/snippets and syntax highlighting for (almost) any coding language. See #341.

Now I can finally do away with using Live Writer for my blog and having to manually add css classes to the html for syntax highlighting... yes that's what I've been doing ‍♂️

Upgrading

Once you've updated to the 4.3.0 release, you'll probably need to change the Property Editor of the Articulate Markdown Data Type to be Articulate Markdown editor since it's most likely currently configured to the default/basic Umbraco markdown editor.

You'll then need to update your Post.cshtml theme file to include the correct Prism dependencies. For example, here's the update/diff to the VAPOR theme which adds these dependencies. It's just ensuring that the Prism stylesheet is added to the header and the Prism.js dependencies are appended to the Post.cshtml file.

Once that's all done, your set!

Creating markdown posts

If you didn't already know, Articulate has always had a browser based markdown editor to create posts. You can simply go to your Articulate root URL and go to the path: a-new to load the markdown editor. Previously this editor would require you to authenticate at the end of writing your post (if you weren't already authenticated) but now it requires authentication up-front.

Once it's loaded, it's just a text editor and you can use your all the normal markdown syntax. You can even upload or take photos with the editor

Of course you can just use the back office markdown editor to create or update posts too but I find for quickly getting a post written and published it's faster to use the /a-new editor... and it works on mobile.

Using code fences with syntax highlighting

GitHub's documentation shows how this works. The typical code fence is 3x back-ticks above and below your code snippet. If you want to add language specific syntax highlighting you can use 3x back-ticks + the language name/alias. For example, c# would be: ```cs (or c# or csharp) and JavaScript would be ```js. GitHub's implementation is different from what Articulate uses so it may not be a perfect 1:1 result but should be fairly close. Articulate is using a combination of:

Rendered Examples

There are just rendered examples based on the default prism styles. I wrote this blog post with the Articulate markdown editor so you can see the results.

Here's an example of a rendered csharp code fence:

/// <summary>
/// This is an example of the ArticulateComposer
/// </summary>
[RuntimeLevel(MinLevel = RuntimeLevel.Run)]
public class ArticulateComposer : IUserComposer
{
    public void Compose(Composition composition)
    {
        composition.RegisterUnique<ArticulateRoutes>();
        composition.RegisterUnique<ContentUrls>();
        composition.RegisterUnique<ArticulateDataInstaller>();
        composition.RegisterUnique<ArticulateTempFileSystem>(
            x => new ArticulateTempFileSystem("~/App_Data/Temp/Articulate"));

        // TODO: Register remaining services....
    }
}

Here's an example of a rendered js code fence:

(function () {
    'use strict';

    /**
     * An example of the articulateOptionsManagementController
     * @param {any} $scope
     * @param {any} $element
     * @param {any} $timeout
     */
    function articulateOptionsManagementController($scope, $element, $timeout) {

        var vm = this;
        vm.viewState = "list";
        vm.selectedGroup = null;
        
        // TODO: Fill in the rest....
    }

    var articulateOptionsMgmtComponent = {
        templateUrl: '../../App_Plugins/Articulate/BackOffice/PackageOptions/articulatemgmt.html',        
        controllerAs: 'vm',
        controller: articulateOptionsManagementController
    };

    angular.module("umbraco")
        .component('articulateOptionsMgmt', articulateOptionsMgmtComponent);
})();

Here's an example of a rendered ruby code fence:

class Dog  
  def initialize(breed, name)  
    # Instance variables  
    @breed = breed  
    @name = name  
  end  
  
  def bark  
    puts 'Ruff! Ruff!'  
  end  
  
  def display  
    puts "I am of #{@breed} breed and my name is #{@name}"  
  end  
end

Auto upgrade your Nuget packages with Azure Pipelines or GitHub Actions

Thu, 23 Mar 2023 15:10:22 Z

Before we start I just want to preface this with some warnings

This works for me, it might not work for you
To get this working for you, you may need to tweak some of the code referenced
This is not under any support or warranty by anyone
Running Nuget.exe update command outside of Visual Studio will overwrite your files so there is a manual review process (more info below)
This is only for ASP.NET Framework using packages.config – Yes I know that is super old and I should get with the times, but this has been an ongoing behind the scenes project of mine for a long time. When I need this for Package Reference projects, ASP.NET Core/5, I’ll update it but there’s nothing stopping you from tweaking this to work for you
This only works for a specified csproj, not an entire sln – it could work for that but I’ve not tested, there would be a few tweaks to make that work
This does not yet work for GitHub actions but the concepts are all here and could probably very easily be converted UPDATE: This works now!

Now that’s out of the way …

How do I do it?

With a lot of PowerShell :) This also uses a few methods from the PowerShellForGitHub project.

The process is:

Run a pipeline/action on a schedule (i.e. each day)
This checks against your source code for the installed version for a particular package
Then it checks with Nuget (using your Nuget.config file) to see what the latest stable version is
If there’s a newer version:
Create a new branch
Run a Nuget update against your project
Build the project
Commit the changes
Push the changes
Create a PR for review

Azure Pipelines/GitHub Actions YAML

The only part of the YAML that needs editing is the variables, here's what they mean:

ProjectFile = The relative path to your csproj that you want to upgrade
PackageFile = The relative path to your packages.config file for this project
PackageName = The Nuget package name you want upgraded
GitBotUser = The name used for the Git commits
GitBotEmail = The email used for the Git commits

For Azure Pipelines, these are also required:

GitHubOwner = The repository owner (i.e. https://github.com/Shazwazza/Articulate )
GitHubRepository = The repository name (i.e. https://github.com/Shazwazza/Articulate )
There is one Secret setting you need to create in Azure Pipelines: GitHubAccessToken which is a GitHub PAT that you need to create that has access to this repository.

Then there are some variables to assist with testing:

DisableUpgradeStep = If true will just check if there’s an upgrade available and exit
DisableCommit = If true will run the upgrade and will exit after that (no commit, push or PR)
DisablePush = If true will run the upgrade + commit and will exit after that (no push or PR)
DisablePullRequest = If true will run the upgrade + commit + push and will exit after that (no PR)

Each step in the yaml build more or less either calls Git commands or PowerShell functions. The PowerShell functions are loaded as part of a PowerShell Module which is committed to the repository. This module’s functions are auto-loaded by PowerShell because the first step configures the PowerShell environment variable PSModulePath to include the custom path. Once that is in place, all functions exposed by the module are auto-loaded.

In these examples you’ll see that I’m referencing Umbraco Cloud names and that’s because I’m using this on Umbraco Cloud for my own website and the examples are for the UmbracoCms package. But this should in theory work for all packages!

Show me the code

The code for all of this is here in a new GitHub repo and here’s how you use it:

You can copy the folder structure in the repository as-is. Here's an example of what my site's repository folder structure is to make this work (everything except the src folder is in the GitHub repo above):

[root]
- auto-upgrader.devops.yml (If you are using Azure Pipelines)
- .github
  - workflows
    - auto-upgrader.gh.yml (If you are using GitHub Actions)
- build
  - PowershellModules
    - AutoUpgradeFunctions.psd1
    - AutoUpgradeFunctions.psm1
    - AutoUpgradeFunctions
- src
  - Shazwazza.Web
    - Shazwazza.Web.csproj
    - packages.config

All of the steps have descriptive display names and it should be reasonably self documenting.

The end result is a PR, here’s one that was generated by this process:

Nuget overwrites

Nuget.exe works differently than Nuget within Visual Studio’s Package Manager Console. All of those special commands like Install-Package, Update-Package, etc… are all PowerShell module commands built into Visual Studio and they are able to work with the context of Visual Studio. This allows those commands to try to be a little smarter when running Nuget updates and also allows the legacy Nuget features like running PowerShell scripts on install/update to run. This script just uses Nuget.exe and it’s less smart especially for these legacy .NET Framework projects. As such, it will just overwrite all files in most cases (it does detect file changes it seems but isn’t always accurate).

With that warning it is very important to make sure you review the changed files in the PR and revert or adjust any changes you need before applying the PR.

You’ll see a note in the PowerShell script about Nuget overwrites. There are other options that can be used like "Ignore" and "IgnoreAll" but all my tests have showed that for some reason this setting will end up deleting a whole bunch of files so the default overwrite setting is used.

Next steps

Get out there and try it! Would love some feedback on this if/when you get a change to test it.

PackageReference support with .NET Framework projects could also be done (but IMO this is low priority) along with being able to upgrade the entire SLN instead of just the csproj.

Then perhaps some attempts at getting a NET Core/5 version of this running. In theory that will be easier since it will mostly just be dotnet commands.

Searching with IPublishedContentQuery in Umbraco

Thu, 23 Mar 2023 15:10:02 Z

I recently realized that I don’t think Umbraco’s APIs on IPublishedContentQuery are documented so hopefully this post may inspire some docs to be written or at least guide some folks on some functionality they may not know about.

A long while back even in Umbraco v7 UmbracoHelper was split into different components and UmbracoHelper just wrapped these. One of these components was called ITypedPublishedContentQuery and in v8 is now called IPublishedContentQuery, and this component is responsible for executing queries for content and media on the front-end in razor templates. In v8 a lot of methods were removed or obsoleted from UmbracoHelper so that it wasn’t one gigantic object and tries to steer developers to use these sub components directly instead. For example if you try to access UmbracoHelper.ContentQuery you’ll see that has been deprecated saying:

Inject and use an instance of IPublishedContentQuery in the constructor for using it in classes or get it from Current.PublishedContentQuery in views

and the UmbracoHelper.Search methods from v7 have been removed and now only exist on IPublishedContentQuery.

There are API docs for IPublishedContentQuery which are a bit helpful, at least will tell you what all available methods and parameters are. The main one’s I wanted to point out are the Search methods.

Strongly typed search responses

When you use Examine directly to search you will get an Examine ISearchResults object back which is more or less raw data. It’s possible to work with that data but most people want to work with some strongly typed data and at the very least in Umbraco with IPublishedContent. That is pretty much what IPublishedContentQuery.Search methods are solving. Each of these methods will return an IEnumerable<PublishedSearchResult> and each PublishedSearchResult contains an IPublishedContent instance along with a Score value. A quick example in razor:

@inherits Umbraco.Web.Mvc.UmbracoViewPage
@using Current = Umbraco.Web.Composing.Current;
@{
    var search = Current.PublishedContentQuery.Search(Request.QueryString["query"]);
}

<div>
    <h3>Search Results</h3>
    <ul>
        @foreach (var result in search)
        {
            <li>
                Id: @result.Content.Id
                <br/>
                Name: @result.Content.Name
                <br />
                Score: @result.Score
            </li>
        }
    </ul>
</div>

The ordering of this search is by Score so the highest score is first. This makes searching very easy while the underlying mechanism is still Examine. The IPublishedContentQuery.Search methods make working with the results a bit nicer.

Paging results

You may have noticed that there’s a few overloads and optional parameters to these search methods too. 2 of the overloads support paging parameters and these take care of all of the quirks with Lucene paging for you. I wrote a previous post about paging with Examine and you need to make sure you do that correctly else you’ll end up iterating over possibly tons of search results which can have performance problems. To expand on the above example with paging is super easy:

@inherits Umbraco.Web.Mvc.UmbracoViewPage
@using Current = Umbraco.Web.Composing.Current;
@{
    var pageSize = 10;
    var pageIndex = int.Parse(Request.QueryString["page"]);
    var search = Current.PublishedContentQuery.Search(
        Request.QueryString["query"],
        pageIndex * pageSize,   // skip
        pageSize,               // take
        out var totalRecords);
}

<div>
    <h3>Search Results</h3>
    <ul>
        @foreach (var result in search)
        {
            <li>
                Id: @result.Content.Id
                <br/>
                Name: @result.Content.Name
                <br />
                Score: @result.Score
            </li>
        }
    </ul>
</div>

Simple search with cultures

Another optional parameter you might have noticed is the culture parameter. The docs state this about the culture parameter:

When the culture is not specified or is *, all cultures are searched. To search for only invariant documents and fields use null. When searching on a specific culture, all culture specific fields are searched for the provided culture and all invariant fields for all documents. While enumerating results, the ambient culture is changed to be the searched culture.

What this is saying is that if you aren’t using culture variants in Umbraco then don’t worry about it. But if you are, you will also generally not have to worry about it either! What?! By default the simple Search method will use the “ambient” (aka ‘Current’) culture to search and return data. So if you are currently browsing your “fr-FR” culture site this method will automatically only search for your data in your French culture but will also search on any invariant (non-culture) data. And as a bonus, the IPublishedContent returned also uses this ambient culture so any values you retrieve from the content item without specifying the culture will just be the ambient/default culture.

So why is there a “culture” parameter? It’s just there in case you want to search on a specific culture instead of relying on the ambient/current one.

Search with IQueryExecutor

IQueryExecutor is the resulting object created when creating a query with the Examine fluent API. This means you can build up any complex Examine query you want, even with raw Lucene, and then pass this query to one of the IPublishedContentQuery.Search overloads and you’ll get all the goodness of the above queries. There’s also paging overloads with IQueryExecutor too. To further expand on the above example:

@inherits Umbraco.Web.Mvc.UmbracoViewPage
@using Current = Umbraco.Web.Composing.Current;
@{
    // Get the external index with error checking
    if (ExamineManager.Instance.TryGetIndex(
        Constants.UmbracoIndexes.ExternalIndexName, out var index))
    {
        throw new InvalidOperationException(
            $"No index found with name {Constants.UmbracoIndexes.ExternalIndexName}");
    }

    // build an Examine query
    var query = index.GetSearcher().CreateQuery()
        .GroupedOr(new [] { "pageTitle", "pageContent"},
            Request.QueryString["query"].MultipleCharacterWildcard());


    var pageSize = 10;
    var pageIndex = int.Parse(Request.QueryString["page"]);
    var search = Current.PublishedContentQuery.Search(
        query,                  // pass the examine query in!
        pageIndex * pageSize,   // skip
        pageSize,               // take
        out var totalRecords);
}

<div>
    <h3>Search Results</h3>
    <ul>
        @foreach (var result in search)
        {
            <li>
                Id: @result.Content.Id
                <br/>
                Name: @result.Content.Name
                <br />
                Score: @result.Score
            </li>
        }
    </ul>
</div>

The base interface of the fluent parts of Examine’s queries are IQueryExecutor so you can just pass in your query to the method and it will work.

Recap

The IPublishedContentQuery.Search overloads are listed in the API docs, they are:

Search(String term, String culture, String indexName)
Search(String term, Int32 skip, Int32 take, out Int64 totalRecords, String culture, String indexName)
Search(IQueryExecutor query)
Search(IQueryExecutor query, Int32 skip, Int32 take, out Int64 totalRecords)

Should you always use this instead of using Examine directly? As always it just depends on what you are doing. If you need a ton of flexibility with your search results than maybe you want to use Examine’s search results directly but if you want simple and quick access to IPublishedContent results, then these methods will work great.

Does this all work with ExamineX ? Absolutely!! One of the best parts of ExamineX is that it’s completely seamless. ExamineX is just an index implementation of Examine itself so all Examine APIs and therefore all Umbraco APIs that use Examine will ‘just work’.

Filtering fields dynamically with Examine

Thu, 23 Mar 2023 15:09:58 Z

The index fields created by Umbraco in Examine by default can lead to quite a substantial amount of fields. This is primarily due in part by how Umbraco handles variant/culture data because it will create a different field per culture but there are other factors as well. Umbraco will create a “__Raw_” field for each rich text field and if you use the grid, it will create different fields for each grid row type. There are good reasons for all of these fields and this allows you by default to have the most flexibility when querying and retrieving your data from the Examine indexes. But in some cases these default fields can be problematic. Examine by default uses Lucene as it’s indexing engine and Lucene itself doesn’t have any hard limits on field count (as far as I know), however if you swap the indexing engine in Examine to something else like Azure Search with ExamineX then you may find your indexes are exceeding Azure Search’s limits.

Azure Search field count limits

Azure Search has varying limits for field counts based on the tier service level you have (strangely the Free tier allows more fields than the Basic tier). The absolute maximum however is 1000 fields and although that might seem like quite a lot when you take into account all of the fields created by Umbraco you might realize it’s not that difficult to exceed this limit. As an example, lets say you have an Umbraco site using language variants and you have 20 languages in use. Then let’s say you have 15 document types each with 5 fields (all with unique aliases) and each field is variant and you have content for each of these document types and languages created. This immediately means you are exceeding the field count limits: 20 x 15 x 10 = 1500 fields! And that’s not including the “__Raw_” fields or the extra grid fields or the required system fields like “id” and “nodeName”. I’m unsure why Azure Search even has this restriction in place

Why is Umbraco creating a field per culture?

When v8 was being developed a choice had to be made about how to handle multi-lingual data in Examine/Lucene. There’s a couple factors to consider with making this decision which mostly boils down to how Lucene’s analyzers work. The choice is either: language per field or language per index. Some folks might think, can’t we ‘just’ have a language per document? Unfortunately the answer is no because that would require you to apply a specific language analyzer for that document and then scoring would no longer work between documents. Elastic Search has a good write up about this. So either language per field or different indexes per language. Each has pros/cons but Umbraco went with language per field since it’s quite easy to setup, supports different analyzers per language and doesn’t require a ton of indexes which also incurs a lot more overhead and configuration.

Do I need all of these fields?

That really depends on what you are searching on but the answer is most likely ‘no’. You probably aren’t going to be searching on over 1000s fields, but who knows every site’s requirements are different. Umbraco Examine has something called an IValueSetValidator which you can configure to include/exclude certain fields or document types. This is synonymous with part of the old XML configuration in Examine. This is one of those things where configuration can make sense for Examine and @callumwhyte has done exactly that with his package “Umbraco Examine Config”. But the IValueSetValidator isn’t all that flexible and works based on exact naming which will work great for filtering content types but perhaps not field names. (Side note – I’m unsure if the Umbraco Examine Config package will work alongside ExamineX, need to test that out).

Since Umbraco creates fields with the same prefixed names for all languages it’s relatively easy to filter the fields based on a matching prefix for the fields you want to keep.

Here’s some code!

The following code is relatively straight forward with inline comments: A custom class “IndexFieldFilter” that does the filtering and can be applied different for any index by name, a Component to apply the filtering, a Composer to register services. This code will also ensure that all Umbraco required fields are retained so anything that Umbraco is reliant upon will still work.

/// <summary>
/// Register services
/// </summary>
public class MyComposer : ComponentComposer<MyComponent>
{
    public override void Compose(Composition composition)
    {
        base.Compose(composition);
        composition.RegisterUnique<IndexFieldFilter>();
    }
}

public class MyComponent : IComponent
{
    private readonly IndexFieldFilter _indexFieldFilter;

    public MyComponent(IndexFieldFilter indexFieldFilter)
    {
        _indexFieldFilter = indexFieldFilter;
    }

    public void Initialize()
    {
        // Apply an index field filter to an index
        _indexFieldFilter.ApplyFilter(
            // Filter the external index 
            Umbraco.Core.Constants.UmbracoIndexes.ExternalIndexName, 
            // Ensure fields with this prefix are retained
            new[] { "description", "title" },
            // optional: only keep data for these content types, else keep all
            new[] { "home" });
    }

    public void Terminate() => _indexFieldFilter.Dispose();
}

/// <summary>
/// Used to filter out fields from an index
/// </summary>
public class IndexFieldFilter : IDisposable
{
    private readonly IExamineManager _examineManager;
    private readonly IUmbracoTreeSearcherFields _umbracoTreeSearcherFields;
    private ConcurrentDictionary<string, (string[] internalFields, string[] fieldPrefixes, string[] contentTypes)> _fieldNames
        = new ConcurrentDictionary<string, (string[], string[], string[])>();
    private bool disposedValue;

    /// <summary>
    /// Constructor
    /// </summary>
    /// <param name="examineManager"></param>
    /// <param name="umbracoTreeSearcherFields"></param>
    public IndexFieldFilter(
        IExamineManager examineManager,
        IUmbracoTreeSearcherFields umbracoTreeSearcherFields)
    {
        _examineManager = examineManager;
        _umbracoTreeSearcherFields = umbracoTreeSearcherFields;
    }

    /// <summary>
    /// Apply a filter to the specified index
    /// </summary>
    /// <param name="indexName"></param>
    /// <param name="includefieldNamePrefixes">
    /// Retain all fields prefixed with these names
    /// </param>
    public void ApplyFilter(
        string indexName,
        string[] includefieldNamePrefixes,
        string[] includeContentTypes = null)
    {
        if (_examineManager.TryGetIndex(indexName, out var e)
            && e is BaseIndexProvider index)
        {
            // gather all internal index names used by Umbraco 
            // to ensure they are retained
            var internalFields = new[]
                {
                LuceneIndex.CategoryFieldName,
                LuceneIndex.ItemIdFieldName,
                LuceneIndex.ItemTypeFieldName,
                UmbracoExamineIndex.IconFieldName,
                UmbracoExamineIndex.IndexPathFieldName,
                UmbracoExamineIndex.NodeKeyFieldName,
                UmbracoExamineIndex.PublishedFieldName,
                UmbracoExamineIndex.UmbracoFileFieldName,
                "nodeName"
            }
                .Union(_umbracoTreeSearcherFields.GetBackOfficeFields())
                .Union(_umbracoTreeSearcherFields.GetBackOfficeDocumentFields())
                .Union(_umbracoTreeSearcherFields.GetBackOfficeMediaFields())
                .Union(_umbracoTreeSearcherFields.GetBackOfficeMembersFields())
                .ToArray();

            _fieldNames.TryAdd(indexName, (internalFields, includefieldNamePrefixes, includeContentTypes ?? Array.Empty<string>()));

            // Bind to the event to filter the fields
            index.TransformingIndexValues += TransformingIndexValues;
        }
        else
        {
            throw new InvalidOperationException(
                $"No index with name {indexName} found that is of type {typeof(BaseIndexProvider)}");
        }
    }

    private void TransformingIndexValues(object sender, IndexingItemEventArgs e)
    {
        if (_fieldNames.TryGetValue(e.Index.Name, out var fields))
        {
            // check if we should ignore this doc by content type
            if (fields.contentTypes.Length > 0 && !fields.contentTypes.Contains(e.ValueSet.ItemType))
            {
                e.Cancel = true;
            }
            else
            {
                // filter the fields
                e.ValueSet.Values.RemoveAll(x =>
                {
                    if (fields.internalFields.Contains(x.Key)) return false;
                    if (fields.fieldPrefixes.Any(f => x.Key.StartsWith(f))) return false;
                    return true;
                });
            }
        }
    }

    protected virtual void Dispose(bool disposing)
    {
        if (!disposedValue)
        {
            if (disposing)
            {
                // Unbind from the event for any bound indexes
                foreach (var keys in _fieldNames.Keys)
                {
                    if (_examineManager.TryGetIndex(keys, out var e)
                        && e is BaseIndexProvider index)
                    {
                        index.TransformingIndexValues -= TransformingIndexValues;
                    }
                }
            }
            disposedValue = true;
        }
    }

    public void Dispose()
    {
        Dispose(disposing: true);
        GC.SuppressFinalize(this);
    }
}

That should give you the tools you need to dynamically filter your index based on fields and content type’s if you need to get your field counts down. This would also be handy even if you aren’t using ExamineX and Azure Search since keeping an index size down and storing less data means less IO operations and storage size.

Examine and Azure Blob Storage

Thu, 23 Mar 2023 15:09:45 Z

Quite some time ago - probably close to 2 years - I created an alpha version of an extension library to Examine to allow storing Lucene indexes in Blob Storage called Examine.AzureDirectory. This idea isn’t new at all and in fact there’s been a library to do this for many years called AzureDirectory but it previously had issues and it wasn’t clear on exactly what it’s limitations are. The Examine.AzureDirectory implementation was built using a lot of the original code of AzureDirectory but has a bunch of fixes (which I contributed back to the project) and different ways of working with the data. Also since Examine 0.1.90 still worked with lucene 2.x, this also made this compatible with the older Lucene version.

… And 2 years later, I’ve actually released a real version

Why is this needed?

There’s a couple reasons – firstly Azure web apps storage run on a network share and Lucene absolutely does not like it’s files hosted on a network share, this will bring all sorts of strange performance issues among other things. The way AzureDirectory works is to store the ‘master’ index in Blob Storage and then sync the required Lucene files to the local ‘fast drive’. In Azure web apps there’s 2x drives: ‘slow drive’ (the network share) and the ‘fast drive’ which is the local server’s temp files on local storage with limited space. By syncing the Lucene files to the local fast drive it means that Lucene is no longer operating over a network share. When writes occur, it writes back to the local fast drive and then pushes those changes back to the master index in Blob Storage. This isn’t the only way to overcome this limitation of Lucene, in fact Examine has shipped a work around for many years which uses something called SyncDirectory which does more or less the same thing but instead of storing the master index in Blob Storage, the master index is just stored on the ‘slow drive’. Someone has actually taken this code and made a separate standalone project with this logic called SyncDirectory which is pretty cool!

Load balancing/Scaling

There’s a couple of ways to work around the network share storage in Azure web apps (as above), but in my opinion the main reason why this is important is for load balancing and being able to scale out. Since Lucene doesn’t work well over a network share, it means that Lucene files must exist local to the process it’s running in. That means that when you are load balancing or scaling out, each server that is handling requests will have it’s own local Lucene index. So what happens when you scale out further and another new worker goes online? This really depending on the hosting application… for example in Umbraco, this would mean that the new worker will create it’s own local indexes by rebuilding the indexes from the source data (i.e. database). This isn’t an ideal scenario especially in Umbraco v7 where requests won’t be served until the index is built and ready. A better scenario is that the new worker comes online and then syncs an existing index from master storage that is shared between all workers …. yes! like Blob Storage.

Read/Write vs Read only

Lucene can’t be written to concurrently by multiple processes. There are some workarounds here a there to try to achieve this by synchronizing processes with named mutex/semaphore locks and even AzureSearch tries to handle some of this by utilizing Blob Storage leases but it’s not a seamless experience. This is one of the reasons why Umbraco requires a ‘master’ web app for writing and a separate web app for scaling which guarantees that only one process writes to the indexes. This is the setup that Examine.AzureDirectory supports too and on the front-end/replica/slave web app that scales you will configure the provider to be readonly which guarantees it will never try to write back to the (probably locked) Blob Storage.

With this in place, when a new front-end worker goes online it doesn’t need to rebuild it’s own local indexes, it will just check if indexes exist and to do that will make sure the master index is there and then continue booting. At this stage there’s actually almost no performance overhead. Nothing actually happens with the local indexes until the index is referenced by this worker and when that happens Examine will lazily just sync the Lucene files that it needs locally.

How do I get it?

First thing to point out is that this first release is only for Examine 0.1.90 which is for Umbraco v7. Support for Examine 1.x and Umbraco 8.x will come out very soon with some slightly different install instructions.

The release notes of this are here, the install docs are here, and the Nuget package for this can be found here.

PM> Install-Package Examine.AzureDirectory -Version 0.1.90

To activate it, you need to add these settings to your web.config

<add key="examine:AzureStorageConnString" value="YOUR-STORAGE-CONNECTION-STRING" />
<add key="examine:AzureStorageContainer" value="YOUR-CONTAINER-NAME" />

Then for your master server/web app you’ll want to add a directoryFactory attribute to each of your indexers in ExamineSettings.config, for example:

<add name="InternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
      supportUnpublished="true"
      supportProtected="true"
      directoryFactory="Examine.AzureDirectory.AzureDirectoryFactory, Examine.AzureDirectory"
      analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"/>

For your front-end/replicate/slave server you’ll want a different readonly value for the directoryFactory like:

<add name="InternalIndexer" type="UmbracoExamine.UmbracoContentIndexer, UmbracoExamine"
      supportUnpublished="true"
      supportProtected="true"
      directoryFactory="Examine.AzureDirectory.ReadOnlyAzureDirectoryFactory, Examine.AzureDirectory"
      analyzer="Lucene.Net.Analysis.WhitespaceAnalyzer, Lucene.Net"/>

Does it work?

Great question :) With the testing that I’ve done it works and I’ve had this running on this site for all of last year without issue but I haven’t rigorously tested this at scale with high traffic sites, etc… I’ve decided to release a real version of this because having this as an alpha/proof of concept means that nobody will test or use it. So now hopefully a few of you will give this a whirl and let everyone know how it goes. Any bugs can be submitted to the Examine repo.

Web Application projects with Umbraco Cloud

Thu, 23 Mar 2023 15:09:41 Z

This is a common topic for developers when working with Umbraco Cloud because Umbraco Cloud simply hosts an ASP.Net Framework “Website”. The setup is quite simple, a website is stored in a Git repository and when it’s updated and pushed to Umbraco Cloud, all of the changes are live. You can think of this Git repository as a deployment repository (which is very similar to how Azure Web Apps can work with git deployments). When you create a new Umbraco Cloud site, the git repository will be pre-populated with a runnable website. You can clone the website and run it locally with IIS Express and it all just works. But this is not a compile-able website and it’s not part of a visual studio project or a solution and if you want to have that, there’s numerous work arounds that people have tried and use but in my personal opinion they aren’t the ideal working setup that I would like.

Ideal solution

In my opinion the ideal solution for building web apps in .NET Framework is:

A visual studio solution

A compile-able Web Application project (.csproj)
Additional class library projects (as needed)
Unit/Integration test projects (as needed)
All dependencies are managed via Nuget

Git source control for my code, probably stored in GitHub
A build server, CI/CD, I like Azure Pipelines

I think this is a pretty standard setup for building websites but trying to wrangle this setup to work with Umbraco Cloud isn’t as easy as you’d think. A wonderful Umbraco community member Paul Sterling has written about how to do this a couple of times, here and here and there’s certainly a few hoops you’d need to jump through. These posts were also written before the age of Azure YAML Pipelines which luckily has made this process a whole lot easier

Solution setup

NOTE: This is for Umbraco v8, there’s probably some other edge cases you’ll need to discover on your own for v7.

Setting up a Visual Studio solution with a web application compatible for Umbraco Cloud is pretty straight forward and should be very familiar. It will be much easier to do this starting from scratch with a new Umbraco Cloud website though it is more than possible to do this for an existing website (i.e. I did this for this website!) but most of those details are just migrating custom code, assets, etc… to your new solution.

I would suggest starting with a new Umbraco Cloud site that has no modifications to it but does have a content item or two that renders a template.

Create a new VS solution/project for a web application running .NET 4.7.2

Add this Nuget.config to the root folder (beside your .sln file)

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
	<add key="NuGet" value="https://api.nuget.org/v3/index.json" />
    <add key="UmbracoCloud" value="https://www.myget.org/F/uaas/api/v3/index.json" />
  </packageSources>
</configuration>

Install the Nuget package for the same Umbraco version that you are currently running on your Umbraco Cloud website. For example if you are running 8.4.0 then use Install-Package UmbracoCms –Version 8.4.0
Install Forms (generally the latest available): Install-Package UmbracoForms
Install Deploy (generally the latest available):
- Install-Package UmbracoDeploy
- Install-Package UmbracoDeploy.Forms
- Install-Package UmbracoDeploy.Contrib
Then you’ll need to install some additional Nuget packages that are required to run your site on Umbraco Cloud. This is undocumented but Umbraco Cloud adds a couple extra DLLs when it creates a website that are required.
- Install-Package Serilog.Sinks.MSSqlServer -Version 5.1.3-dev-00232
Copy these files from your Umbraco Cloud deployment repository to your web application project:
- ~/data/*
- ~/config/UmbracoDeploy.config
- ~/config/UmbracoDeploy.Settings.config
You then need to do something weird. These settings need to be filled in because Umbraco Deploy basically circumvents the normal Umbraco installation procedure and if you don’t have these settings populated you will get YSODs and things won’t work.
- Make sure that you have your Umbraco version specified in your web.config like: <add key="Umbraco.Core.ConfigurationStatus" value="YOURVERSIONGOESHERE" />
- Make sure your connectionStrings in your web.config is this:
  - ```
  <connectionStrings>
      <remove name="umbracoDbDSN" />
      <add name="umbracoDbDSN"
           connectionString="Data Source=|DataDirectory|\Umbraco.sdf"
           providerName="System.Data.SqlServerCe.4.0" />
  </connectionStrings>
```

But I don’t want to use SqlCE! Why do I need that connection string? In actual fact Umbraco Deploy will configure your web application to use Sql Express LocalDb if it’s available on your machine (which it most likely is). This is why when running Umbraco Cloud sites locally you’ll see .mdf and .ldf files in your App_Data folder instead of SqlCE files. Local Db operates just like Sql Server except the files are located locally, it’s really sql server under the hood. You can even use Sql Management Studio to look at these databases by connecting to the (localdb)\umbraco server locally with Windows Authentication. It is possible to have your local site run off of a normal Sql server database with a real connection string but I think you’d have to install Umbraco first before you install the UmbracoDeploy nuget package. Ideally UmbracoDeploy would allow the normal install screen to run if there was no Umbraco version detected in the web.config, but that’s a whole other story.

That should be it! In theory your web application is now configured to be able to publish a website output that is the same as what is on Umbraco Cloud.

Installation

At this stage you should be able to run your solution, it will show the typical Umbraco Deploy screen to restore from Cloud

In theory you should be able to restore your website and everything should ‘just work’

Working with code

Working with your code is now just the way you’re probably used to working. Now that you’ve got a proper Visual Studio solution with a Web Application Project, you can do all of the development that you are used to. You can add class libraries, unit test projects, etc… Then you commit all of these changes to your own source control like GitHub. This type of repository is not a deployment repository, this is a source code repository.

How do I get this to Umbraco Cloud?

So far there’s nothing too special going on but now we need to figure out how to get our Web Application Project to be deployed to Umbraco Cloud.

There’s a couple ways to do this, the first way is surprisingly simple:

Right click your web application project in VS
Click Publish
Choose Folder as a publish target
Select your cloned Umbraco Cloud project location
Click advanced and choose “Exclude files from App_Data folder’
Click Create Profile
Click Publish – you’ve just published a web application project to a website
Push these changes to Umbraco Cloud

The publish profile result created should match this one: https://github.com/umbraco/vsts-uaas-deploy-task/blob/master/PublishProfiles/ToFileSys.pubxml

This of course requires some manual work but if you’re ok with that then job done!

You should do this anyways before continuing since it will give you an idea of how in-sync your web application project and the output website is to the Umbraco Cloud website, you can then see what Git changes have been made and troubleshoot anything that might seem odd.

Azure Pipelines

I’m all for automation so instead I want Azure Pipelines to do my work. This is what I want to happen:

Whenever I commit to my source code repo Azure Pipelines will:

Build my solution
Run any unit tests that I have
Publish my web application project to a website
Zip the website
Publish my zipped website artifact

When I add a “release-*” tag to a commit I want Azure Pipelines to do all of the above and also:

Clone my Umbraco Cloud repository
Unzip my website artifact onto this cloned destination
Commit these changes to the Umbraco Cloud deployment repository
Push this commit to Umbraco Cloud

Luckily this work is all done for you :) and with YAML pipelines it’s fairly straight forward. Here’s how:

Go copy this PowerShell file and commit it to the /build folder of your source code repository (our friends Paul Sterling and Morten Christensen had previously done this work, thanks guys!). This PS script essentially does all of that Git work mentioned above, the cloning, committing and pushing files. It’s a bit more verbose than just running these git comands directly in your YAML file but it’s also a lot less error prone and handles character encoding properly along with piping the output of the git command to the log.
Go copy this azure-pipelines.yml file and commit it to the root of your git source code repository. This file contains a bunch of helpful notes so you know what it’s doing. (This pipelines file does run any tests, etc… that exercise will be left up to you.)
In Azure Pipelines, create a new pipeline, choose your Git source control option, choose “Existing Azure Pipelines YAML file”, select azure-pipelines.yml file in the drop down, click continue.
Click Variables and add these 3:

gitAddress = The full Git https endpoint for your Dev environment on Umbraco Cloud
gitUsername = Your Umbraco Cloud email address
gitPassword = Your Umbraco Cloud password - ensure this value is set to Secret

Click Run!

And that’s it! … Really? … In theory yes :)

Your pipeline should run and build your solution. The latest commit you made is probably the azure-pipelines.yml files so it didn’t contain a release-* tag so it’s not going to attempt to push any changes to Umbraco Cloud. So first thing to do is make sure that your your pipeline is building your solution and doing what its supposed to. Once that’s all good then it’s time to test an Umbraco Cloud deployment.

Deploying to Umbraco Cloud

A quick and easy test would be to change the output of a template so you can visibly see the change pushed.

Go ahead and make a change to your home page template
Run your site locally with your web application project and make sure the change is visible there
Commit this change to your source control Git repository
Create and push a release tag on this commit. For example, the tag name could be: “release-v1.0.0-beta01” … whatever suites your needs but based on the YAML script it needs to start with “release-“

Now you can sit back and watch Azure Pipelines build your solution and push it to Umbraco Cloud. Since this is a multi-stage pipeline, the result will look like:

And you should see a log output like this on the Deploy stage

Whoohoo! Automated deployments to Umbraco Cloud using Web Application Projects.

What about auto-upgrades?

All we’ve talked about so far is a one-way push to Umbraco Cloud but one thing we know and love about Umbraco Cloud is the automated upgrade process. So how do we deal with that? I actually have this working on my site but want to make the process even simpler so you’re going to have to be patient and wait for another blog post :)

The way it works is also using Azure Pipelines. Using a separate pipeline with a custom Git repo pointed at your Umbraco Cloud repository, this pipeline can be configured to poll for changes every day (or more often if you like). It then checks if changes have been made to the packages.config file to see if there’s been upgrades made to either the CMS, Forms or Deploy (in another solution I’m actually polling Nuget directly for this information). If an upgrade has been made, It clones down your source code repository, runs a Nuget update command to upgrade your solution. Then it creates a new branch, commits these changes, pushes it back GitHub and creates a Pull Request (currently this only works for GitHub).

This same solution can be used for Deploy files in case administrators are changing schema items directly on Umbraco Cloud so the /deploy/* files can be automatically kept in sync with your source code repository.

This idea is entirely inspired by Morten Christensen, thanks Morten! Hopefully I’ll find some time to finalize this.

Stay tuned!

How I upgraded my site to Umbraco 8 on Umbraco Cloud

Thu, 23 Mar 2023 15:09:37 Z

I have a Development site and a Live site on Umbraco Cloud. You might have some additional environments but in theory these steps should be more or less the same. This is just a guide that hopefully helps you, it’s by no means a fail-safe step by step guide, you’ll probably run into some other issues and edge cases that I didn’t. You will also need access to Kudu since you will most likely need to delete some left over files manually, you will probably also need to toggle debug and custom errors settings in your web.config to debug any YSODs you get along the way, you will need to manually change the Umbraco version number in the web.config during the upgrade process and you might need access to the Live Git repository endpoint in case you need to rollback.

… Good luck!

Make sure you can upgrade

Make sure you have no Obsolete data types

You cannot upgrade to v8 if you have any data types referencing old obsolete property editors. You will first need to migrate any properties using these to the non-obsolete version of these property editors. You should do this on your Dev (or lowest environment): Go to each data type and check if the property editor listed there is prefixed with the term “Obsolete”. If it is you will need to change this to a non-obsolete property editor. In some cases this might be tricky, for others it might be an easy switch. For example, I’m pretty sure you can switch from the Obsolete Content Picker to the normal Content Picker. Luckily for me I didn’t have any data relying on these old editors so I could just delete these data types.

Make sure you aren’t using incompatible packages

If you are using packages, make sure that any packages you are using also have a working v8 version of that package.

Make sure you aren’t using legacy technology

If you are using XSLT, master pages, user controls or other weird webforms things, you are going to need to migrate all of that to MVC before you can continue since Umbraco 8 doesn’t support any of these things.

Ensure all sites are in sync

Very important that all Cloud environments are in sync with all of your latest code and there’s no outstanding code that needs to be shipped between them. Then you need to make sure that all content and media are the same across your environments since each one will eventually be upgraded independently and you want to use your Dev site for testing along with pulling content/media to your local machine.

Clone locally, sync & backup

Once all of your cloud sites are in sync, you’ll need to clone locally – I would advise to start with a fresh clone. Then restore all of your content and media and ensure your site runs locally on your computer. Once that’s all running and your site is basically running and operating like your live site you’ll want to take a backup. This is just for piece of mind, when upgrading your actual live site you aren’t going to lose any data. To do this, close VS Code (or whatever tool you use to run your local site) and navigate to ~/App_Data/ and you’ll see Umbraco.mdf and Umbraco_log.mdf files. Make copies of those and put them someplace. Also make a zip of your ~/media folder.

Now to make things easy in case you need to start again, make a copy of this entire site folder which you can use for the work in progress/upgrade/migration. If you ever need to start again, you can just delete this copied wip folder and re-copy the original.

Create/clone a new v8 Cloud site

This can be a trial site, it’s just a site purely to be able to clone so we can copy some files over from it. Once you’ve cloned locally feel free to delete that project.

Update local site files

Many people will be using a visual studio web application solution with Nuget, etc… In fact I am too but for this migration it turned out to be simpler in my case to just upgrade/migrate the cloned website.

Next, I deleted all of the old files:

The entire /bin directly – we’ll put this back together with only the required parts, we can’t have any old left over DLLs hanging around
/Config
/App_Plugins/UmbracoForms, /App_Plugins/Deploy, /App_Plugins/DiploTraceLogViewer
/Umbraco & /Umbraco_Client
Old tech folders - /Xslt, /Masterpages, /UserControls, /App_Browsers
/web.config

If you use App_Code, then for now rename this to something else. You will probably have to refactor some of the code in there to work and for now the goal is to just get the site up and running and the database upgraded. So rename to _App_Code or whatever you like so long as it’s different.

Copy over the files from the cloned v8 sites:

/bin
/Config
/App_Plugins/UmbracoForms, /App_Plugins/Deploy
/Umbraco
/Views/Partials/Grid, /Views/MacroPartials, /Views/Partials/Forms – overwrite existing files, these are updated Forms and Umbraco files
/web.config

Merge any custom config

Create a git commit before continuing.

Now there’s some manual updates involved. You may have had some custom configuration in some of the /Config/* files and in your /web.config file. So it’s time to have a look in your git history. That last commit you just made will show all of the changes overwritten in any /config files and your web.config file so now you can copy any changes you want to maintain back to these files. Things like custom appSettings, etc…

One very important setting is the Umbraco.Core.ConfigurationStatus appSetting, you must change this to your previous v7 version so the upgrader knows it needs to upgrade and from where.

Upgrade the database

Create a git commit before continuing.

At this stage, you have all of the Umbraco files, config files and binary files needed to run Umbraco v8 based on the version that was provided to your from your cloned Cloud site. So go ahead and try to run the website, with any luck it will run and you will be prompted to login and upgrade. If not and you have some YSODs or something, then the only advise I can offer at this stage is to debug the error.

Now run the upgrader – this might also require a bit of luck and depends on what data is in your site, if you have some obscure property editors or if your site is super old and has some strange database configurations. My site is super old, from v4 and over the many years I’ve managed to wrangle it through the version upgrades and it also worked on v8 (after a few v8 patch releases were out to deal with old schema issues). If this doesn’t work, you may be prompted with a detailed error specifically telling you way (i.e. you have obsolete property editors installed), or it might just fail due to old schema problems. For the latter problem, perhaps some of these tickets might help you resolve it.

When you get this to work, it’s a good time to make a backup of your local DB. Close down the running website and tool you used to launch it, then make a backup of the Umbraco.mdf and Umbraco_log.mdf files.

Fix your own code

You will probably have noticed that the site now runs, you can probably access the back office (maybe?!) but your site probably has YSODs. This is most likely because:

Your views and c# code needs to be updated to work with the v8 APIs (remember to rename your _App_Code folder back to App_Code if you use it!)
Your packages need to be re-installed or upgraded or migrated into your new website with compatible v8 versions

This part of the migration process is going to be different for everyone. Basic sites will generally be pretty simple but if you are using lots of packages or custom code or a visual studio web application and/or additional class libraries, then there’s some manual work involved on your part. My recommendation is that each time you fix part of your sites you create a Git commit. You can always revert to previous commits if you want and you also have a backup of your v8 database if you ever need to revert that too. The API changes from v7 –> v8 aren’t too huge, you’ll have your local site up and running in no time!

Rebuild your deploy files

Create a git commit before continuing.

Now that your site is working locally in v8, it’s time to prep everything to be sent to Umbraco Cloud.

Since you are now running a newer version of Umbraco deploy you’ll want to re-generate all of the deploy files. You can do this by starting up your local site again, then open the command prompt and navigate to /data folder of your website. Then type in :

echo > deploy-export

All of your schema items will be re-exported to new deploy files.

Create a git commit before continuing.

Push to Dev

In theory if your site is working locally then there’s no reason why it won’t work on your Dev site once you push it up to Cloud. Don’t worry though, if all else fails, you can always revert back to a working commit for your site.

So… go ahead and push!

Once that is done, the status bar on the Cloud portal will probably be stuck at the end stage saying it’s trying to process Deploy files… but it will just hang there because it’s not able to. This is because your site is now in Upgrade mode since we’ve manually upgraded.

At this stage, you are going to need to login to Kudu. Go to the cmd prompt and navigate to /site/wwwroot/web.config and edit this file. The Umbraco.Core.ConfigurationStatus is going to be v8 already because that is what you committed to git and pushed to Cloud but we need Umbraco to detect an upgrade is required, so change this value to the v7 version you originally had (this is important!). While you are here, ensure that debug = false and CustomErrors = Off so you can see any errors that might occur.

Now visit the root of the site, you should be redirected to the login screen and then to the upgrade screen. With some more luck, this will ‘just work’!

Because the Deploy files couldn’t be processed when you first pushed because the site was in upgrade mode, you need to re-force the deploy files to be processed so go back to kudu cmd prompt and navigate to /site/wwwroot/data and type in:

echo > deploy

Test

Make sure your dev site is working as you would expect it to. There’s a chance you might have missed some code that needs changing in your views or other code. If that is the case, make sure you fix it first on your local site, test there and then push back up to Dev and then test again there. Don’t push to a higher environment until you are ready.

Push to Live

You might have other environments between Dev and Live so you need to follow the same steps as pushing to Dev (i.e. once you push you will need to go to Kudu, change the web.config version, debug and custom error mode). Pushing to Live is the same approach but of course your live site is going to incur some downtime. If you’ve practiced with a Staging site, you’ll know how much downtime to expect, in theory it could be as low as a couple minutes but of course if something goes wrong it could be for longer.

… And Hooray! You are live on v8 :)

Before you go, there’s a few things you’ll want to do:

log back into kudu on your live site and in your web.config turn off debug and change custom errors back to RemoteOnly
be sure to run “echo > deploy”
in kudu delete the temp file folder: App_Data/Temp
Rebuild your indexes via the back office dashboard
Rebuild your published caches via the back office dashboard

What if something goes wrong?

I mentioned above that you can revert to a working copy, but how? Well this happened to me since I don’t follow my own instructions and I forgot to get rid of the data types with Obsolete property editors on live which means all of my environments were not totally synced before I started since I had fixed that on Dev. When I pushed to live and then ran the upgrader, it told me that I had data types with old Obsolete property editors … well in that scenario there’s nothing I could do about it since I can’t login to the back office and change anything. So I had to revert the Live site to the commit before the merge from Dev –> Live. Luckily all database changes with the Upgrader are done in a transaction so your live data isn’t going to be changed unless the upgrader successfully completes.

To rollback, I logged into Kudu and on the home page there is a link to “Source control info” where you can get the git endpoint for your Live environment. Then I cloned that all down locally and reverted the merge, committed and pushed back up to the live site. Now the live site was just back to it’s previous v7 state and I could make the necessary changes. Once that was done, I reverted my revert commit locally and pushed back to Live, and went through the upgrade process again.

Next steps?

Now your site is live on v8 but there’s probably more to do for you solution. If you are like me, you will have a Visual Studio solution with a web application to power your website. I then run this locally and publish to my local file system – which just happens to be the location of my cloned git repo for my Umbraco Cloud Dev site, then I push those changes to Cloud. So now I needed to get my VS web application to produce the same binary output as Cloud. That took a little bit to figure out since Umbraco Cloud includes some extra DLLs/packages that are not included in the vanilla Umbraco Cms package, namely this one: “Serilog.Sinks.MSSqlServer - Version 5.1.3-dev-00232” so you’ll probably need to include that as a package reference to your site too.

That’s about as far as I’ve got myself, best of luck!

Articulate 4.0.0 released for Umbraco version 8

Thu, 23 Mar 2023 15:09:25 Z

It’s finally out in the wild! Articulate 4.0.0 is a pretty huge release so here’s the rundown…

Installation

As a developer, my recommendation is to install packages with Nuget

PM > Install-Package Articulate -Version 4.0.0

If you install from Nuget you will not automatically get the Articulate data structures installed because Nuget can’t talk to your live website/database so once you’ve installed the package and run your site, head over to the “Settings” section and you’ll see an “Articulate” dashboard there, click on the “Articulate Data Installer” tile and you’ll get all the data structures and a demo blog installed.

Alternatively you can install it directly from the Umbraco back office by searching for “Articulate” in the Packages section, or you can download the zip from https://our.umbraco.com/packages/starter-kits/articulate/ and install that in the Umbraco back office. If you install this way all of the data structures will be automatically installed.

Upgrading

I have no official documentation or way of doing this right now . I’ve written up some instructions on the GitHub release here but essentially it’s going to require you to do some investigations and manual updates yourselves. There’s very little schema changes and only small amount of model changes so it shouldn’t be too painful. Good luck!

(note: I have yet to give it a try myself)

Support for Umbraco 8

I think it will come as no surprise that Articulate 4.0.0 is not compatible with any Umbraco v7 version. Articulate 4.0.0 requires a minimum of Umbraco 8.0.2. Moving forward I will only release more Articulate 3.x versions to support v7 based on community pull requests, my future efforts will be solely focused on 4.x and above for Umbraco 8+.

Theme, Features + Bug fixes

There are several nice bug fixes in this release including a few PR sent in by the community – THANK YOU!

As for features, this is really all about updating the Themes. Previously Articulate shipped with 6 themes and all of them had a vast range of different features which I never really liked so I spent some time enhancing all of the ones I wanted to keep and made them look a bit prettier too. I’ve removed my own “Shazwazza” theme since it was way outdated to my own site here, plus I don’t really want other people to have the exact same site as me ;) But since that was the most feature rich theme I had to upgrade other ones. I also removed the old ugly Edictum them… pretty sure nobody used that one anyways.

Here’s the theme breakdown (it’s documented too)

I’ve also updated the default installation data to contain more than one blog post and an author profile so folks can see a better representation of the blog features on install. And I updated the default images and styling so it has a theme (which is Coffee ☕) and is less quirky (no more bloody rabbit or horse face photos )

Here’s the breakdown of what they look like now…

VAPOR

This is the default theme installed, it is a very clean & simple theme. Originally created by Seth Lilly

Material

This is based of of Google's material design lite and is based off their their blog template.

Phantom

Original theme for Ghost can be found here: https://github.com/Bartinger/phantom/. A nice simple responsive theme.

Mini

The original author's site can be found here: http://www.thyu.org/www/ but unfortunately their demo site for the Ghost theme is down. The theme's repository is here https://github.com/thyu/minighost.

Hope you enjoy the updates!

Need to remove an auto-routed controller in Umbraco?

Thu, 23 Mar 2023 15:09:21 Z

Umbraco will auto-route some controllers automatically. These controllers are any MVC SurfaceControllers or WebApi UmbracoApiController types discovered during startup. There might be some cases where you just don’t want these controllers to be routed at all, maybe a package installs a controller that you’d rather not have routable or maybe you want to control if your own plugin controllers are auto-routed based on some configuration.

The good news is that this is quite easy by just removing these routes during startup. There’s various ways you could do this but I’ve shown below one of the ways to interrogate the routes that have been created to remove the ones you don’t want:

Version 8


//This is required to ensure this composer runs after
//Umbraco's WebFinalComposer which is the component
//that creates all of the routes during startup    
[ComposeAfter(typeof(WebFinalComposer))]
public class MyComposer : ComponentComposer<MyComponent>{ }

//The component that runs after WebFinalComponent
//during startup to modify the routes
public class MyComponent : IComponent
{
    public void Initialize()
    {
        //list the routes you want removed, in this example
        //this will remove the built in Umbraco UmbRegisterController
        //and the TagsController from being routed.
        var removeRoutes = new[]
        {
            "/surface/umbregister",
            "/api/tags"
        };

        foreach (var route in RouteTable.Routes.OfType().ToList())
        {
            if (removeRoutes.Any(r => route.Url.InvariantContains(r)))
                RouteTable.Routes.Remove(route);
        }
    }

    public void Terminate() { }
}

Version 7

public class MyStartupHandler : ApplicationEventHandler
{
    protected override void ApplicationStarted(
        UmbracoApplicationBase umbracoApplication,
        ApplicationContext applicationContext)
    {

        //list the routes you want removed, in this example
        //this will remove the built in Umbraco UmbRegisterController
        //and the TagsController from being routed.
        var removeRoutes = new[]
        {
            "/surface/umbregister",
            "/api/tags"
        };

        foreach(var route in RouteTable.Routes.OfType<Route>().ToList())
        {
            if (removeRoutes.Any(r => route.Url.InvariantContains(r)))
                RouteTable.Routes.Remove(route);
        }
    }
}

Umbraco Down Under Festival 2019

Thu, 23 Mar 2023 15:09:13 Z

I had the pleasure of attending and speaking at this year’s Umbraco Down Under Festival which was fantastic! Thanks to everyone at KØBEN digital for putting on such a nice event as well to all of the sponsors Zero Seven, Tea Commerce and Luminary in helping make it all happen. And what great timing to have an Umbraco festival just after Umbraco v8 is launched! Big thanks to Niels Hartvig for coming all the way from Denmark, it certainly means a lot to us Australians (yes, I am one too even with this Canadian accent!).

Hackathon

We had quite a few people at the Hackathon this year (18!) and we were able to close 3 issues and merge 6 Pull Requests along with finding and submitting 4 other issues, all for Umbraco v8, great work! Looking forward to seeing the Australian community submit even more PRs for v8 and hope to see you all at the Australian Umbraco meetups :)

Slide Deck

My talk this year was on Umbraco Packages in v8 though much of it was really about transitioning to v8 in general.

Here is the rendered PDF version of my slides, of course it doesn’t come with all of the nice transitions but it’s better than nothing. My slides were done with the brilliant GitPitch service which I absolutely love. Not only does it make presenting code so much nicer/easier, it just makes sense to me as a developer since I can just write my slides in Mardown and style them with css. Plus having your slide deck in Git means making new slides out of old slides quite nice since all your history is there!

I tried to break down the talk into 3 sections: Migrating, Building and Packaging.

Migrating

“Migrating” was a bit of a walk through between some fundamental things that have changed between v7 and v8 that not only package developers will need to be aware of but anyone making the transition from v7 to v8.

Building

“Building” showcased some new features for packages and v8, though I didn’t talk about one of the major v8 features: Content Apps, because Robert Foster was already doing a talk all about them in the morning. Instead I focused on how Dashboards work in v8 and a couple currently undisclosed v8 features: Full Screen Sections (sans c#) and Package Options.

Packaging

“Packaging” may have been a bit rushed but I thought I was going to go overtime :P I talked about the new packager UI in v8 and that it is certainly possible to build packages for CI/CD integration with PowerShell scripts to build an Umbraco package from a command line. I’d like to make this far more straight forward than it is now which is something I’ll work on this year. You can find this PowerShell script here and a newer example in Articulate here. Lastly I mentioned that there is a disconnect between the Umbraco package format and the Nuget package format with regards to installing Umbraco data and that it would be nice to make both of these work seamlessly as one thing… and this is certainly possible. I created a PR a very long time ago to address this called Package Migrations (even though it says Morten made it … he actually just submitted the PR ;) ). I need to write a blog post about what this is and how it is intended to work so we can hopefully get some traction on this this year. The very brief overview is that package updates would work very similarly to Umbraco updates, if an update is detected that requires a migration, the installer will execute to guide the user through the process and to provide UI feedback if anything fails along the way. This way package developers can properly leverage the Migrations system built into Umbraco and Umbraco data will happily be installed on startup even if you install a package from Nuget.

The main barrier currently is that Umbraco Cloud will need to natively support it otherwise people will get the installer screen on every environment when they push a package update upstream which is not great, Umbraco Cloud should instead run the migration on the upstream environment in the background just like it does with Umbraco updates.

Lastly I talked about how Articulate currently manages this situation between the Umbraco package format and the Nuget package format.

UDUF 2020

Looks like UDUF is moving to Sydney next year, so we’ll so you all there!

Configuring Azure Active Directory login with Umbraco Members

Thu, 23 Mar 2023 15:09:08 Z

This post is about configuring Azure Active Directory with Umbraco Members (not Users), meaning this is for your front-end website, not the Umbraco back office. I did write up a post about Azure AD with back office users though, so if that is what you are looking for then this is the link.

Install the Nuget packages

First thing to do is get the UmbracoIdentity package installed.

PM > Install-Package UmbracoIdentity

(This will also install the UmbracoIdentity.Core base package)

This package installs some code snippets and updates your web.config to enable ASP.Net Identity for Umbraco members. Umbraco ships with the old and deprecated ASP.Net Membership Providers for members and not ASP.Net Identity so this package extends the Umbraco CMS and the Umbraco members implementation to use ASP.Net Identity APIs to interact with the built in members data store. Installing this package will remove the (deprecated) FormsAuthentication module from your web.config and it will no longer be used to authenticate members, so the typical members snippets built into Umbraco macros will not work. Instead use the supplied snippets shipped with this package.

To read more about this package see the GitHub repo here.

Next, the OpenIdConnect package needs to be installed

PM > Install-Package Microsoft.Owin.Security.OpenIdConnect

Configure Azure Active Directory

Head over to the Azure Active Directory section on the Azure portal, choose App Registrations (I’m using the Preview functionality for this) and create a New registration

Next fill out the app details

You may also need to enter other redirect URLs depending on how many different environments you have. All of these URLs can be added in the Authentication section of your app in the Azure portal.

For AAD configuration for front-end members, the redirect Urls are just your website’s root URL and it is advised to keep the trailing slash.

Next you will need to enable Id Tokens

Configure OpenIdConnect

The UmbracoIdentity package will have installed an OWIN startup class in ~/App_Start/UmbracoIdentityStartup.cs (or it could be in App_Code if you are using a website project). This is how ASP.Net Identity is configured for front-end members and where you can specify the configuration for different OAuth providers. There’s a few things you’ll need to do:

Allow external sign in cookies

If you scroll down to the ConfigureMiddleware method, there will be a link of code to uncomment: app.UseExternalSignInCookie(DefaultAuthenticationTypes.ExternalCookie); this is required for any OAuth providers to work.

Enable OpenIdConnect OAuth for AAD

You’ll need to add this extension method class to your code which is some boiler plate code to configure OpenIdConnect with AAD:

public static class UmbracoADAuthExtensions
{
    public static void ConfigureAzureActiveDirectoryAuth(this IAppBuilder app,
        string tenant, string clientId, string postLoginRedirectUri, Guid issuerId,
        string caption = "Active Directory")
    {
        var authority = string.Format(
            System.Globalization.CultureInfo.InvariantCulture,
            "https://login.windows.net/{0}",
            tenant);

        var adOptions = new OpenIdConnectAuthenticationOptions
        {
            ClientId = clientId,
            Authority = authority,
            RedirectUri = postLoginRedirectUri
        };

        adOptions.Caption = caption;
        //Need to set the auth type as the issuer path
        adOptions.AuthenticationType = string.Format(
            System.Globalization.CultureInfo.InvariantCulture,
            "https://sts.windows.net/{0}/",
            issuerId);
        app.UseOpenIdConnectAuthentication(adOptions);
    }
}

Next you’ll need to call this code, add the following line underneath the app.UseExternalSignInCookie method call:

app.ConfigureAzureActiveDirectoryAuth(
    ConfigurationManager.AppSettings["azureAd:tenantId"],
    ConfigurationManager.AppSettings["azureAd:clientId"],
    //The value of this will need to change depending on your current environment
    postLoginRedirectUri: ConfigurationManager.AppSettings["azureAd:redirectUrl"],
    //This is the same as the TenantId
    issuerId: new Guid(ConfigurationManager.AppSettings["azureAd:tenantId"]));

Then you’ll need to add a few appSettings to your web.config (based on your AAD info):

<add key="azureAd:tenantId" value="YOUR-TENANT-ID-GUID" />
<add key="azureAd:clientId" value="YOUR-CLIENT-ID-GUID" />
<add key="azureAd:redirectUrl" value="http://my-test-website/" />

Configure your Umbraco data

The UmbracoIdentity repository has the installation documentation and you must follow these 2 instructions, and they are very simple:

Once that is done you will have an Member account management page which is based off of the installed views and snippets of the UmbracoIdentity package. This account page will look like this:

As you can see the button text under the “Use another service to log in” is the login provider name which is a bit ugly. The good news is that this is easy to change since this is just a partial view that was installed with the UmbracoIdentity package. You can edit the file: ~/Views/UmbracoIdentityAccount/ExternalLoginsList.cshtml, the code to render that button text is using @p.Authentication provider but we can easily change this to @p.Caption which is actually the same caption text used in the extension method we created. So the whole button code can look like this instead:


<button type="submit" class="btn btn-default"
        id="@p.AuthenticationType"
        name="provider"
        value="@p.AuthenticationType"
        title="Log in using your @p.Caption account">
    @p.Caption
</button>

This is a bit nicer, now the button looks like:

The purpose of all of these snippets and views installed with UmbracoIdentity is for you to customize how the whole flow looks and works so you’ll most likely end up customizing a number of views found in this folder to suit your needs.

That’s it!

Once that’s all configured, if you click on the Active Directory button to log in as a member, you’ll be brought to the standard AAD permission screen:

Once you accept you’ll be redirect back to your Account page:

Any customizations is then up to you. You can modify how the flow works, whether or not you accepting auto-linking accounts (like in the above example), or if you require a member to exist locally before being able to link an OAuth account, etc… All of the views and controller code in UmbracoIdentity is there for you to manipulate. The main files are:

~/Views/Account.cshtml
~/Views/UmbracoIdentityAccount/*
~/Controllers/UmbracoIdentityAccountController.cs
~/App_Start/UmbracoIdentityStartup.cs

Happy coding!

Configuring Azure Active Directory login with Umbraco

Thu, 23 Mar 2023 15:09:04 Z

It’s been a while since I first set this up and back then not all of these settings were in the new Azure portal, but now that they are getting this all configured is quite easy so here’s the basic steps!

Install the Nuget package

First thing to do is get the UmbracoCms.IdentityExtensions.AzureActiveDirectory package installed

PM > Install-Package UmbracoCms.IdentityExtensions.AzureActiveDirectory

(This will also install the UmbracoCms.IdentityExtensions base package)

Configure Azure Active Directory

Head over to the Azure Active Directory section on the Azure portal, choose App Registrations (I’m using the Preview functionality for this) and create a New registration

Next fill out the app details

Add your redirect URLs for any additional environments

If you have local, dev, staging, live, etc… sites, you’ll need to configure those URLs here, always make sure it’s the Umbraco path with a trailing slash.

Make note of some settings

The settings you’ll need to copy are the Application/Client ID and the Tenant ID along with the redirect URLs.

Configure you application

Add the Client Id, Tenant Id and redirect URL to AppSettings. I’ve used the keys: azureAd:tenantId, azureAd:clientId, azureAd:redirectUrl but you can choose whatever you like.
Open the App_Start/UmbracoStandardOwinStartup.cs file that was installed with the IdentityExtensions pakage

Underneath the call to base.Configuration(app); add the block of code that was shown in the readme shown after you installed the Nuget package which looks like this:

app.ConfigureBackOfficeAzureActiveDirectoryAuth(
    //The Tenant can also be "YOURDIRECTORYNAME.onmicrosoft.com"
    tenant: ConfigurationManager.AppSettings["azureAd:tenantId"],
    clientId: ConfigurationManager.AppSettings["azureAd:clientId"],
    //The value of this will need to change depending on your current environment
    postLoginRedirectUri: ConfigurationManager.AppSettings["azureAd:redirectUrl"],
    //This is the same as the TenantId
    issuerId: new Guid(ConfigurationManager.AppSettings["azureAd:tenantId"]));

Ensure that this OWIN startup class is defined in your web.config: <add key="owin:appStartup" value="UmbracoStandardOwinStartup" />

That's it!

Once you’ve got this configured, and you login to the back office normally you can then link your AD account:

Once linked, you can login with this provider:

Auto linking?

If you configured your AD App to only authenticate “only accounts in this organization”, you might want to auto-link Umbraco back office accounts. Auto-linking will automatically ensure that a local Umbraco user account exists for any user that logs in via the AD provider. This is handy if you want to do all of your user administration via Azure AD. Auto-linking can be configured as part of your OWIN startup class.

I won’t cover auto-linking in this post but there are some docs available for that.

How I configure my Umbraco Cloud website to support Nuget packages

Thu, 23 Mar 2023 15:09:00 Z

Disclaimer: This post is about how I have my own website setup. This is based on my own personal opinions and is not meant to be an Umbraco Cloud ‘best practices’ guide. I’ve written this post since it might help others who have the same requirements as I do.

My Requirements

For my simple website, I would like to manage all of my dependencies with Nuget, I don’t need anything compiled, don’t need class libraries and I don’t mind putting any required custom code into App_Code.

Umbraco Cloud provisions a deployment Git repository which for all intensive purposes is meant for deploying code between environments, not so much for a source code repository. That said, since my website is ultra simple and doesn’t contain any class library projects (etc…) I figured having an ASP.Net website project configured in the Umbraco Cloud repository would be fine. A website project is different from the standard web application project; a website doesn’t compile, it runs as-is which is suitable for the Umbraco Cloud Git repository since that’s exactly what the repository is made for: to host a deployed website that runs based on the files as-is. I prefer working with web application projects normally but in this case my website is ultra simple and a website project will work just fine plus this allows me to have a Visual Studio project/solution that works fairly seamlessly with Umbraco Cloud.

How to set this up

There’s not a lot required to set this up but there are a couple ‘gotchas’, here’s the steps:

1) Clone your Umbraco Cloud Git repo

The first step is straight forward, you need to clone your Umbraco Cloud Git repository to your local machine

2) Open your Umbraco Cloud site as a Website in Visual Studio

Open Visual Studio, File –> Open –> Web site and choose the folder where your Umbraco Cloud site is cloned. This will open your Umbraco Cloud folder as a website project (at this point you could ctrl+F5 and it would run your site).

3) Save the sln file

You need to save the Visual Studio .sln file in order to have Nuget references, File –> Save localhost_1234.sln As…

This menu option is a bit odd and that’s because Visual Studio has created an in-memory .sln file which it’s auto-named to be localhost_PORT.sln

When you click on that, browse to your Umbraco Cloud Git repo folder and name the file something that makes more sense than localhost_PORT.sln.

4) Configure the Solution and Project to not build the website

This is optional but by default Visual Studio will try to build your website which means it’s going to try to precompile all views, etc… which not only can take some time but you will get false positive errors. So instead there’s 2 things to do: In Configuration Manager turn off the “Build” option for the website project and in the website project settings turn off building. Here’s how:

Build –> Configuration Manager opens a dialog, uncheck the Build checkbox for the website

Then right click the root of your website project and choose “Property Pages” and click on the “Build” navigation element. Then change the “Start action” to “No build” and un-check the “Build website as part of solution” checkbox

5) Create a Kudu .deployment file

Here’s one of the ‘gotchas’. Like Azure web apps, Umbraco Cloud also uses Kudu to perform some of it’s operations such as deploying a website from the Git repository to the hosting directory on the server. By default Kudu will copy the files in the deployment repository as-is to the hosting directory on the server (which is what we want)… that is until Kudu sees things like .sln or .csproj files in the root of the Git repository, then it tries to be clever and build things (which we don’t want).

So to tell Kudu to just deploy the files as-is, we create a special Kudu file at the repository root called .deployment (to be clear this file name starts with a dot!).

To create this, in your Visual Studio website project, right click the root click Add –> Add new item –> Choose Text File –> Enter the name .deployment

Then add the following to this file:

[config]
project = .

and this tells Kudu to just deploy the files that are found in the repo.

6) Now you can add Nuget references

Once all of this is setup, you can add Nuget references to this website project like you would normally add Nuget references. At this point you might need to make a choice: Do you want to manage your Umbraco installation based on Nuget? In some cases you might not have a choice if you need to reference a Nuget package that has a dependency on Umbraco.Core.

As it turns out this choice isn’t such a big deal but some things to be aware of. Since Umbraco Cloud auto-upgrades projects to the latest patch version, you might be concerned that your packages.config is going to get out of date… luckily Umbraco Cloud is clever and will auto-update this file to the version it just upgraded you too. This also goes for minor version upgrades that you perform on Umbraco Cloud. Since Umbraco Cloud auto-commits all of the upgraded files, it means you really don’t have to do anything.

7) You’ll need to commit the special *.dll.refresh files

Once you start using Nuget with a website project, you’ll notice a bunch of these *.dll.refresh files in your /bin directory. You’ll need to commit those. These are special marker files used by Visual Studio to know how to manage Nuget dependencies with a website project.

That's it!

The above setup is an easy way to setup a Visual Studio solution with a single website project that works seamlessly with Umbraco Cloud while allowing you to manage dependencies with Nuget.

But what if your solution is more complicated or you want to add class libraries, etc… ? There’s Umbraco documentation on how to configure this found here: https://our.umbraco.com/documentation/Umbraco-Cloud/Set-Up/Visual-Studio/, https://our.umbraco.com/documentation/Umbraco-Cloud/Set-Up/Working-With-Visual-Studio/ and the configuration there isn’t so different than the above except that the .sln file isn’t committed to the Umbraco Cloud Git deployment repository and instead exists in your own Git repository which in that case, the special .deployment file is not necessary.

Getting Umbraco to work with Azure Easy Auth

Thu, 23 Mar 2023 15:08:56 Z

There’s a nifty feature in your Azure App Service that allows you to very quickly add authentication and authorization to your Azure website. You’ll see most folks calling this “Easy Auth” and there’s quite a few articles on the subject such as:

The good news is this all works the way you’d expect with an Umbraco website for front-end requests but unfortunately this doesn’t play nicely with the Umbraco back office … but it’s easy to configure Umbraco to work!

The problem

The problem is that if you turn on Easy Auth and try to log in to your Umbraco back office, the login will succeed but you’ll get 401 responses for other back office requests and essentially you’ll see a blank screen. The reason this happens is due to the way that Easy Auth works:

It activates an HttpModule in your site called EasyAuthAspNetThreadPrincipalModule
During the HttpModule.AuthenticateRequest stage it replaces the Thread.CurrentPrincipal with it’s own ClaimsPrincipal/ClaimsIdentity instance

Umbraco also sets the Thread.CurrentPrincipal.Identity during this phase but at the OWIN level which executes before the EasyAuthAspNetThreadPrincipalModule. Because the Easy Auth module replaces the principal/identity, it wipes out the one created by Umbraco. What it should do instead is check if the current principal is a ClaimsPrincipal and then add it’s identity to the identity collection instead of wiping out anything that is already there. If that were the case, everything would ‘just work’ but since it is not we have to work around this issue.

The solution

UPDATE! (19/04/18) - Chris Gillum who created Easy Auth got in touch with me on Twitter to share some handy (and fairly hidden) documentation for Easy Auth. Looks like another work around is to use the WEBSITE_AUTH_DISABLE_IDENTITY_FLOW appSetting which will prevent Easy Auth from setting the thread identity at all.

To work around this problem we need to tell Umbraco to perform it’s authentication procedure after the Easy Auth module runs which is actually pretty easy to do.

Create a new OWIN startup class:


[assembly: OwinStartup("MyOwinStartup", typeof(MyOwinStartup))]
namespace MyProject
{
    public class MyOwinStartup : Umbraco.Web.UmbracoDefaultOwinStartup
    {
    }
}

Update an appSetting in your web.config to tell OWIN to use your class:


<add key="owin:appStartup" value="MyOwinStartup" />

Override the method that configures Umbraco's authentication procedure and tell it to execute after the default Authentication stage, notice this code block is using PipelineStage.PostAuthenticate:


public class MyOwinStartup : Umbraco.Web.UmbracoDefaultOwinStartup
{
    protected override void ConfigureUmbracoAuthentication(IAppBuilder app)
    {
        app
            .UseUmbracoBackOfficeCookieAuthentication(ApplicationContext, PipelineStage.PostAuthenticate)
            .UseUmbracoBackOfficeExternalCookieAuthentication(ApplicationContext, PipelineStage.PostAuthenticate)
            .UseUmbracoPreviewAuthentication(ApplicationContext, PipelineStage.Authorize);
    }
}

That's it! Now the Umbraco back office will authenticate correctly with Azure Easy Auth turned on.

Easily lock out Umbraco back office users

Thu, 23 Mar 2023 15:08:52 Z

Want an easy way to lock out all back office users?

Maybe you are performing an upgrade and want to make sure there’s no back office activity?

Here’s a handy script to do this


using System;
using Microsoft.Owin;
using Owin;
using Umbraco.Web;

[assembly: OwinStartup("AuthDisabledOwinStartup", typeof(MyWebsite.AuthDisabledOwinStartup))]

namespace MyWebsite
{
    public class AuthDisabledOwinStartup : UmbracoDefaultOwinStartup
    {
        protected override void ConfigureUmbracoAuthentication(IAppBuilder app)
        {
            //Don't do anything, this will mean all cookie authentication is disabled which means
            //that no requests from the back office user will be authenticated and therefore 
            //all requests will fail and the user will be logged out.
        }
    }
}

Now you can just update your web.config appSetting to


<add value="AuthDisabledOwinStartup" key="owin:appStartup"></add>

When you want to allow back office access again, just update your web.config with your original owin:appStartup value

Easily setup your Umbraco installation with IoC / Dependency Injection

Thu, 23 Mar 2023 15:08:48 Z

Umbraco supports allowing you to setup and configure any IoC container type that you want to use in your application. For a while now there’s been some sparse documentation on how to achieve this which you can find here: https://our.umbraco.org/Documentation/reference/using-ioc. As the Umbraco core codebase evolves, sometimes a new non-parameterless constructor is added to a class and sometimes this can confuse an existing container that you’ve setup. For many folks, fixing these errors after upgrading is a trial and error experience until they track down the dependency that is now missing from their container and finally add it.

Simone, a very helpful Umbracian, made a comment on the issue tracker and it’s something that is just so obvious (http://issues.umbraco.org/issue/U4-9562#comment=67-41855):

I think the point here is: as user of a framework, I shouldn't need to wire up dependencies for internals of the framework myself. I should only bother about my own dependencies.
Maybe Umbraco should ship a small extension method for each of the main IoC container out there which wires up all the internals.
Or come with a IoC container out of the box and then everyone using umbraco have to use that one.

Yes of course this should be done!

A new community project: Our.Umbraco.IoC

I decided to get the ball rolling with this one and have setup a new Git repo here: https://github.com/Shazwazza/Our.Umbraco.IoC

Currently there are 2 different container configurations committed and working for Autofac and LightInject.

I’ve added some notes to the readme on how to contribute and get started so I’m hoping that some folks can create some Pull Requests to add support for more containers. The project is very easy to navigate, it’s got a build script and nuget packages setup.

Give it a go!

I’ve published some beta’s to Nuget:

Install-Package Our.Umbraco.IoC.Autofac

Install-Package Our.Umbraco.IoC.LightInject

You can actually install both and test each one independently by disabling them by an appSetting:

<add key="Our.Umbraco.IoC.Autofac.Enabled" value="false" />

<add key="Our.Umbraco.IoC.LightInject.Enabled" value="false" />

If this config key doesn’t exist, it will assume the value is “true”

Using the container

Once you’ve got your desired package installed, it will be active in your solution (unless you disable it via config). At this stage you’ll want to add your own bits to the container, so here’s how you do that:

Create a custom Umbraco ApplicationEventHandler
Override ApplicationInitialized – we do this in this phase to bind to the container event before the container is built which occurs in the ApplicationStarted phase
Bind to the container event
add any custom services you want to the container

Here’s a full working example showing various techniques and includes the syntax for both LightInject and Autofac. In this example we’re registering a IServerInfoService as a request scoped object since it requires an HttpRequestBase. NOTE: That the basic web objects are already registered in the containers (such as HttpContextBase, HttpRequestBase, etc…)


public class MyUmbracoStartup : ApplicationEventHandler
{
    protected override void ApplicationInitialized(UmbracoApplicationBase umbracoApplication, ApplicationContext applicationContext)
    {
        //If you are using Autofac:
        AutofacStartup.ContainerBuilding += (sender, args) =>
        {
            //add our own services
            args.Builder.RegisterControllers(typeof(TestController).Assembly);
            args.Builder.RegisterType().As().InstancePerRequest();
        };

        //If you are using LightInject:
        LightInjectStartup.ContainerBuilding += (sender, args) =>
        {
            //add our own services
            args.Container.RegisterControllers(typeof(TestController).Assembly);
            args.Container.Register(new PerRequestLifeTime());
        };
    }
}

//service
public interface IServerInfoService
{
    string GetValue();
}

//implementation of the service
public class ServerInfoService : IServerInfoService
{
    private readonly HttpRequestBase _umbCtx;

    //requires a request based object so this must be scoped to a request
    public ServerInfoService(HttpRequestBase umbCtx)
    {
        _umbCtx = umbCtx;
    }

    public string GetValue()
    {
        var sb = new StringBuilder();
        sb.AppendLine("Server info!").AppendLine();
        foreach (var key in _umbCtx.ServerVariables.AllKeys)
        {
            sb.AppendLine($"{key} = {_umbCtx.ServerVariables[key]}");
        }
        return sb.ToString();
    }
}

public class TestController : SurfaceController
{
    private readonly IServerInfoService _serverInfoService;

    public TestController(IServerInfoService serverInfoService, UmbracoContext umbCtx): base(umbCtx)
    {
        _serverInfoService = serverInfoService;
    }

    //see /umbraco/surface/test/index to see the result
    public ActionResult Index()
    {
        return Content(_serverInfoService.GetValue(), "text/plain");
    }
}

Happy holidays!

Articulate 3.0.0 has finally arrived

Thu, 23 Mar 2023 15:08:40 Z

It’s taken me forever to get this release out, mostly due to lack of time, but it’s finally here! Huge thanks to many community members who helped with this release, here’s some shout outs – THANKS! (sorry many I don’t know real names)

Features features and more features

There’s a ton of new shiny things in this release. Here’s the milestone on GitHub for full details but I’ll run through some of the good ones here

Nuget

Yay finally! This has been asked for forever and I’ve finally done it. The blocking thing for this for me was that if you installed via Nuget you wouldn’t be able to install all of the demo data correctly. Package Migrations is a concept that would fix this but that hasn’t been integrated into the Umbraco core just yet. So in the meantime I’ve updated the Articulate dashboard to easily allow you to install the demo data. In fact this can be used to upgrade installations too and will put all the schema elements in sync like data types, document types, etc… with the latest release

Blog post feature image

Each blog post document now has a specific feature Image property and all of the default themes have been updated to use this image. Of course if you don’t want or need a feature image per post, you can easily just ignore it in your blog theme but these days every blog post has a feature image. This also give us the ability to have thumbnails for each blog post – which is now also a new feature. It gets even better though, both the Live Writer and the Markdown editor (the one you can use at the /a-new path) both support updating this feature image with the first image found in the post so you don’t have to worry about populating it afterwards – this is a toggle you can turn on/off

Edit Themes in the back office

Yes you can! You can even create them from the back office. There’s a new tree in the settings section that will allow you to duplicate an existing theme and then you can update, add and remove any theme files you want. It uses the great ACE editor that powers the template and script editors to edit the files.

Here’s creating a new theme:

Here’s creating a new theme file:

Date formatted blog post URLs

Normally the URLs in Articulate are formatted just by name,

for example: https://shazwazza.com/post/how-to-set-rewrite-method-on-queries-lazily-in-lucene/

but it’s been requested to format URLs with date parts such as: /blog/2014/10/17/my-article/

That can now be achieved with a simple configuration option on your Articulate root node:

Author pages

Each post now has a link to it’s author and if the theme allows it there’s an author listing page just like the categories/tags listing pages that show all posts by Author. Each author can also have their own bio page which is also just part of a theme.

There’s tons more so please have a look at the release notes for full details.

Umbraco passwords and ASP.NET Machine Keys

Thu, 23 Mar 2023 15:08:31 Z

Update! I’ve updated a few points below in bold and have corrected a few things too

This blog post is the result of a thread on Twitter which starts here: https://twitter.com/crumpled_jeavon/status/880522105795870720 and works its way into confusion. Suffice to say I can’t answer these questions in 140 chars so here’s re-cap in the form of Q and A about Machine Keys and Umbraco. Please note that I am not an expert in hashing algorithms, some of these answers are based on my own research. Hope this clears things up!

How is the password hashed?

It is hashed in the same way that the ASP.NET Universal membership provider (DefaultMembershipProvider) and SqlMembershipProvider hashes passwords which by default uses the HMACSHA256 algorithm.

Jeffrey Schoemaker has been discussing updating Umbraco’s default password hashing to use an even stronger hash algorithm and I’ve recently updated a new task on the issue tracker to research this but it really comes down to the fact that Microsoft does not offer the best stronger hashing method in it’s standard .NET Framework so we’ll see where we end up.

Update – we will be shipping umbraco with stronger password hashing, possibly in a 7.7.x release http://issues.umbraco.org/issue/U4-10089 and it will use HMACSHA1 + PBKDF2 which is what ASP.NET Identity uses by default.

Is the Machine Key used for password hashing?

Yes, In 7.6.0+ it is by default because useLegacyEncoding is false by default in this release. Previous to 7.6.0 the useLegacyEncoding value was true by default purely to preserve some backwards compatibility settings for other products but you’ve been able to set it to true from 7.0.0 (IIRC). Since those products support the more secure password formats, this is now false by default and should be kept as false for new installs. By default the hashing algorithm is HMACSHA256 which uses comes from the ASP.NET Machine Key to perform part of it’s hashing function ‘validation’ algorithm type. This ‘validation’ algorithm type is configurable via the Machine Key or it is configurable at the membership provider level which would override the algorithm specified in the Machine Key, but you really shouldn’t change this to be a lesser strength hashing algorithm.

The HMAC part of this algorithm means it’s derived from a keyed algorithm and uses a key to generate the hash and the machine key is used to create this key by default. There doesn’t seem to be any documentation or reference to this online that I can find but trying to look through the crypto source code (which isn’t nice to look at) it seems that the default key gets set based on some logic in the RSACryptoServiceProvider class which reads some info from the machine key.

Update – the key used to hash the passwords is the generated salt we produce it is not the key part of the Machine Key. This logic is exactly the same as the logic used in the (DefaultMembershipProvider) and SqlMembershipProvider and if the hashing algorithm key length doesn’t equal the generated salt key length then it is padded/trimmed to the correct length, see source here. The part of the Machine Key that is used to hash the passwords is specifically the algorithm type. As you can see on this machine key generator, there can be a handful of different algorithm types used for the ‘validation’ part of a machine key and the default of this value changes based on the ASP.NET version being used. In ASP.NET 4.5 the default is HMACSHA256. Also note that in ASP.NET 4.5 the following algorithms are no longer allowed in the Machine Key config: AES, 3DES, and MD5

Do machine keys change between environments?

If you explicitly generate and set your own machine key in your web.config then the answer is No.

If you don’t explicitly generate and set your own machine key than you will get an auto-generated machine key. The simple answer for this is: In most cases No, an auto-generated machine key will not change between environments.

To understand when it will change between environments is a little more complicated and comes down to a combination of IIS user, IIS website virtual path (i.e. if you are running a site in a virtual directory), and a combination of a few settings set at the machine config level: “IsolateApps” and “IsolateByAppId”. Unless a server administrator specifically changes these settings than chances are you won’t be affected by different auto-generated machine keys for your site. If you are really keen, here’s a series all about this topic and other cryptographic changes in .NET 4.5:

Part 1 – see the “A brief digression: auto-generated machine keys” for info on auto-generating keys
Part 2 – in-depth info about the machine key and hashing changes
Part 3 – interesting info especially with regards to PBKDF2 in .NET Framework

Update – another reason a machine key may change between environments is based on which version of ASP.NET is running and what it’s default ‘validation’ algorithm type is

Can I change my machine key?

No. YES

However, I realize In some cases you might need to change it or move from an auto-generated machine key to an explicit machine key. If that is the case there will may be a lot of some manual work you’ll need to do. If you simply just change the machine key or add an explicit one when you previously didn’t have one than your members/users will might not be able to log in! This really comes down to what hashing algorithm was used originally to hash the passwords and what hash algorithm is configured in your Machine Key. If you install or change a machine key to a different algorithm that was used to first hash your passwords, then your members/users will not be able to log in.

Update – Previously I was under the impression that the key in the hashing algorithm was directly affected by the key in the machine key but after some more investigation it is not actually the case. As mentioned in the above updates, the part of the Machine Key that is used in the password hashing is the algorithm type specified (if you haven’t overridden this algorithm type by specifying it directly on the membership provider). I’ve also detailed some of this investigation in this ticket: http://issues.umbraco.org/issue/U4-10222

Can I change from useLegacyEncoding ‘true’ to ‘false’?

Not easily.

This will require a bit of extra work just like the above question but it’s not quite as tricky as changing a Machine Key. When you have useLegacyEncoding=’true’ , the machine key validation algorithm is not used, so you could do something like:

Create a new password column in the database which will store the newly hashed password based on the new machine key validation algorithm
When a user logs in you can check if they have the new password format or not.

If not, then you can validate the password based on the old format then re-hash with the new format and store it and delete the old stored hash password.
If so, then you can validate the password based on the new format

To do this would probably require inheriting from the current Umbraco membership provider and implementing this logic yourself

Do I need to generate and store a custom Machine Key if I am planning on Load Balancing?

Yes.

This is also mentioned in the Umbraco docs for load balancing

Do I need to generate and install a Machine Key before I run the Umbraco installer?

Yes.

This is because during the Umbraco installation it will hash and store the password for the admin account and if you then add a Machine Key after the fact, you will no longer be able to log in.

Can Umbraco generate a custom Machine Key on install for me?

Yes!

but it doesn’t do that right now. I created that functionality in a PR but we decided to remove the machine key generation part when it was accepted. We have decided to bring it back though so that will be part of an upcoming Umbraco release, whether that is the 7.6.x series or 7.7 series is still being decided.

Update – this is shipping with 7.7 and new installs will have a Machine Key generated and installed for you. You can of course opt out of this in the advanced installer options if you like.

Do Machine Key’s exist in ASP.NET Core?

No.

Well sort of but not really. I haven’t been able to research too much into this but when speaking to a couple MS Dev’s about this a couple years ago the answer was that the way machine key’s work will be different. If keys need to be shared between apps in ASP.NET Core the data protection APIs need to be used and these keys would then be stored in the registry or the Cloud (i.e. Azure Key Vault), here’s a SO article on this.

Clear as mud?! ;)

ASP.Net Identity for Umbraco Members

Thu, 23 Mar 2023 15:08:15 Z

I’ve released version 1.0 of UmbracoIdentity which allows for ASP.Net Identity to be used with Umbraco for front-end members. I’ve tried to write enough documentation for you to get started, all of the code and docs are here: https://github.com/Shandem/UmbracoIdentity

It’s worth noting that this is not something that ‘everyone’ should just jump in with and start using. If you are not familiar with OWIN or ASP.Net Identity than none of this will really make any sense, I’ve added a bit of a disclaimer to the docs about this here: https://github.com/Shandem/UmbracoIdentity#owin-setup

There are some known issues and limitations that you should be aware of, and this will also not work currently with back office users (that will come eventually too though).

This package can be highly customized, it comes with many .cshtml views and c# classes that you can customize as you see fit – very similar to the Visual Studio 2013 Web Application templates that support ASP.Net Identity.

What about Asp.Net Membership with Umbraco?

The way that I’ve created this is to be 100% compatible with the current Membership structure in Umbraco. This means that this is not using EntityFramework to access data, it uses the Umbraco member services and providers using custom ASP.Net Identity user stores. This also means that the password formats are based on the current password formats of the membership provider. When the Nuget package is installed it will actually swap out the membership provider for a custom type: UmbracoIdentity.IdentityEnabledMembersMembershipProvider. This is required so that the password security can still be handled by the membership provider logic.

There is a note about passwords here, ASP.Net Identity normally will format and salt passwords different (slightly better) than how the membership providers current work, but if you need that functionality you’d have to implement your own IPasswordHasher.

Installation

It is absolutely essential you read the documentation before installing. Once you’ve done that, you can use Nuget:

PM> Install-Package UmbracoIdentity

Articulate 1.0.4 released

Thu, 23 Mar 2023 15:08:15 Z

I’ve finally got around to releasing Articulate 1.0.4 today. Want to say a big thanks to all those who submitted pull requests, you guys rock! There’s a few nice fixes in this release but most importantly it fixes the issues with multi-tenancy when domains are assigned in Umbraco.

Here’s the release notes on GitHub with links to each issue fixed:

https://github.com/Shandem/Articulate/releases/tag/1.0.4

You can download the Umbraco package from there or from Our here:

http://our.umbraco.org/projects/starter-kits/articulate

Unfortunately I haven’t got around to getting this on Nuget yet – That’s because it has it’s own challenges since I’d like to perform a full Umbraco data install via Nuget… coming soon I hope :)

Enjoy!

Powershell script to create an Umbraco package in Umbraco’s native file format

Thu, 23 Mar 2023 15:08:15 Z

Since I like using Powershell for my build scripts for various projects I thought it would be handy to create a Powershell script to create an Umbraco package in it’s native package format. I’ve added this script template to my GitHub Umbraco Scripts repository which you can see here: http://bit.ly/1kM9g9g

I’ve tried to add a bit of documentation to this script and in theory you should only need to set up the paths properly. You’ll also need to ensure that you initially create the package in the back office of Umbraco in order to generate the createdPackages.config as mentioned in the script notes.

I use this script in Articulate’s build script so at least I can also provide an example of using it which you can see here: https://github.com/Shazwazza/Articulate/blob/master/build/build.ps1

Isolated WebApi attribute routing

Thu, 23 Mar 2023 15:08:15 Z

Attribute routing in ASP.Net WebApi is great and makes routing your controllers quite a bit more elegant than writing routes manually. However one problem I have with it is that it is either “on” or “off” at an application level. There is no way for a library developer to tell ASP.Net to create routes based on attributes for specific controllers or assemblies without forcing the consumer of that library to enable Attribute Routing for the whole application. In many cases this might not matter, but if you are creating a package or library of that contains it’s own API routes, you probably don’t want to interfere with a developers’ normal application setup. There should be no reason why they need to be forced to turn on attribute routing in order for your product to work, and similarly they might not want your routes automatically enabled.

The good news is that this is possible. With a bit of code, you can route your own controllers with attribute routing and be able to turn them on or off without affecting the default application attribute routes. A full implementation of this has been created for the Umbraco RestApi project so I’ll reference that source in this post for the following code examples.

Show me the code

They key to getting this to work is: IDirectRouteProvider, IDirectRouteFactory

The first thing we need is a custom IDirectRouteFactory which is actually a custom attribute. I’ve called this CustomRouteAttribute but you could call it whatever you want.

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, AllowMultiple = true)]
public class CustomRouteAttribute : Attribute, IDirectRouteFactory

This custom attribute just wraps the default WebApi RouteAttribute’s IDirectRouteFactory implementation so we don’t have to re-write any code for that.

(see full implementation here)

Next we’ll create a custom IDirectRouteProvider:

/// <summary>
/// This is used to lookup our CustomRouteAttribute instead of the normal RouteAttribute so that 
/// we can use the CustomRouteAttribute instead of the RouteAttribute on our controlles so the normal
/// MapHttpAttributeRoutes method doesn't try to route our controllers - since the point of this is
/// to be able to map our controller routes with attribute routing explicitly without interfering
/// with default application routes.
/// </summary>
public class CustomRouteAttributeDirectRouteProvider : DefaultDirectRouteProvider
{
    private readonly bool _inherit;

    public CustomRouteAttributeDirectRouteProvider(bool inherit = false)
    {
        _inherit = inherit;
    }

    protected override IReadOnlyList<IDirectRouteFactory> GetActionRouteFactories(HttpActionDescriptor actionDescriptor)
    {
        return actionDescriptor.GetCustomAttributes<CustomRouteAttribute>(inherit: _inherit);
    }
}

So far this is all pretty straight forward so far but here’s where things start to get interesting. Because we only want to create routes for specific controllers, we need to use a custom IHttpControllerTypeResolver. However, since the HttpConfiguration instance only contains a single reference to the IHttpControllerTypeResolver we need to do some hacking. The route creation process for attribute routing happens during the HttpConfiguration initialization so we need to create an isolated instance of HttpConfiguration, set it up with the services we want to use, initialize it to create our custom routes and assign those custom routes back to the main application’s HttpConfiguration.

Up first, we create a custom IHttpControllerTypeResolver to only resolve the controller we’re looking for:

public class SpecificControllerTypeResolver : IHttpControllerTypeResolver
{
    private readonly IEnumerable<Type> _controllerTypes;

    public SpecificControllerTypeResolver(IEnumerable<Type> controllerTypes)
    {
        if (controllerTypes == null) throw new ArgumentNullException("controllerTypes");
        _controllerTypes = controllerTypes;
    }

    public ICollection<Type> GetControllerTypes(IAssembliesResolver assembliesResolver)
    {
        return _controllerTypes.ToList();
    }
}

Before we look at initializing a separate instance of HttpConfiguration, lets look at the code you’d use to enable all of this in your startup code:

//config = the main application HttpConfiguration instance
config.MapControllerAttributeRoutes(
    routeNamePrefix: "MyRoutes-",
    //Map these explicit controllers in the order they appear
    controllerTypes: new[]
    {                    
        typeof (MyProductController),
        typeof (MyStoreController)
    });

The above code will enable custom attribute routing for the 2 specific controllers. These controllers will be routed with attribute routing but instead of using the standard [Route] attribute, you’d use our custom [CustomRoute] attribute. The MapControllerAttributeRoutes extension method is where all of the magic happens, here’s what it does:

Iterates over each controller type
Creates an instance of HttpConfiguration
Sets it’s IHttpControllerTypeResolver instance to SpecificControllerTypeResolver for the current controller iteration (The reason an instance of HttpConfiguration is created for each controller is to ensure that the routes are created in the order of which they are specified in the above code snippet)
Initialize the HttpConfiguration instance to create the custom attribute routes
Copy these routes back to the main application’s HttpConfguration route table

You can see the full implementation of this extension method here which includes code comments and more details on what it’s doing. The actual implementation of this method also allows for some additional parameters and callbacks so that each of these routes could be customized if required when they are created.

There is obviously a bit of code involved to achieve this and there could very well be a simpler way, however this implementation does work rather well and offers quite a lot of flexibility. I’d certainly be interested to hear if other developers have figured this out and what their solutions were.