Shazwazza

Smidge 5 is out

Tue, 22 Sep 2026 19:28:07 Z

Smidge 5 is out

It's been about a year since the last release, so sorry 'bout that 😅. This one turned into a proper major version rather than another collection of small fixes: Smidge is now on .NET 10, NUglify is the only minifier, the MVC dependency is gone, the package structure is simpler, and a few long-running production bugs have finally been tracked down properly.

The full release notes have every detail. This is the story behind the parts that matter most.

One minifier instead of three

Smidge has historically shipped with its own hand-rolled JavaScript and CSS minifiers, JsMinifier and CssMinifier, while also offering NUglify through the separate Smidge.Nuglify package.

That made sense when those minifiers were first added, but over time it meant maintaining two different approaches to the same job. The built-in implementations had their own edge cases and bugs, while NUglify was already the better maintained and more capable option.

In v5 the old JSMin and CSSMin implementations are gone. NUglify is now included directly in the main Smidge package and is used automatically by the normal AddSmidge() and UseSmidge() setup. The separate Smidge.Nuglify package, along with AddSmidgeNuglify() and UseSmidgeNuglify(), is no longer needed.

This is a breaking change, but it's also a much simpler default. There is one minification pipeline, one set of behaviour to test, and source maps work without having to opt into another package.

.NET 10 and no more MVC dependency

Smidge now targets .NET 10.

The old MVC controllers used to serve bundle, composite-file and source-map requests. Those have been replaced with lightweight minimal API endpoints, with the old action-filter behaviour moved into endpoint filters.

That removes the MVC dependency entirely. If you're using Smidge in a small ASP.NET Core app you no longer need to pull MVC into the application just to serve a CSS or JavaScript bundle.

As part of that work, the old Smidge.Core project has also been folded into Smidge. There isn't much value in making consumers understand the internal package split, and having the public surface spread across assemblies made the dependency graph harder to follow than it needed to be.

The practical result is fewer packages, fewer setup methods and less framework machinery underneath it all.

The random bundle crash was actually a race condition

There have been a couple of reports over the years of Smidge suddenly throwing a NullReferenceException from BundleFileSetGenerator.ValidateFile. The frustrating part was that it only seemed to happen in production, often after an application had been running for a while, and recycling the app pool would make it disappear.

The immediate symptom was a null file entry inside a bundle, but there was no obvious path that intentionally added one. I added defensive checks so a bad entry could be logged and skipped instead of taking down the entire bundle response, but that still didn't explain how it got there.

The missing clue came from somebody who found that wrapping their view-based bundle registration in an application-level lock made the problem go away.

That pointed at the real problem. BundleManager is a singleton and its bundle dictionary was thread-safe, but each bundle stored its files in a normal List<IWebFile>. View-based declarations can run from multiple concurrent requests, which meant multiple threads could call List<T>.Add on the same list at the same time. List<T> doesn't support concurrent writers, and if two additions collided during an internal resize the list could lose entries or end up with null slots.

It wasn't the cache buster, app pool idle state or a missing source file. It was memory corruption inside a concurrently mutated list.

Bundle.Files is now backed by an immutable list and updated with ImmutableInterlocked.Update. Reads get a complete, unchanging snapshot with no locking cost, and concurrent writers use a compare-and-swap loop rather than corrupting the same backing array. There are also defensive null checks at bundle creation and file-set generation, because a bad input still shouldn't turn into a broken page.

A concurrency test that registers files from 50 tasks at once reliably lost entries against the old implementation. With the new implementation every file is present and there are no nulls.

This is probably the most important fix in the release even if it's not the flashiest one.

Missing cache files now return 404

The other production hardening work is around stale or invalid bundle URLs.

Composite bundle URLs contain references to already-processed cache files. If the cache was cleared, an in-memory cache disappeared after an application restart, or somebody deliberately requested a made-up file hash, Smidge used to call a throwing file lookup. That surfaced as a 500 and could be triggered repeatedly, which made it a fairly easy denial-of-service path.

The HTTP-facing composite and source-map handlers now use non-throwing lookups. If a requested cache file isn't there, Smidge returns a normal 404 instead of throwing. Internal invariants still throw where they should - a source file configured in a real bundle disappearing is different from an arbitrary client asking for a stale cache path.

The same work fixed an older source-map issue where an unexpected sourceMappingURL request could fail with Could not parse ... as a valid smidge path. Invalid or missing source-map requests now just return 404, which is what the browser expects anyway.

Globs and directory bundles

There are a couple of smaller file-matching fixes that are worth calling out.

Pointing a bundle at a bare directory used to expand it to *.*. That could pull generated files such as .js.gz or .map into a JavaScript bundle and then hand binary gzip data to NUglify, which ended about as well as you'd expect. Bare directory bundles are now constrained to the bundle type, so JavaScript bundles get .js files and CSS bundles get .css files.

Recursive glob patterns are supported as well:

bundles.CreateCss("site", "~/assets/css/**/*.css");

They already worked for the common physical-file-provider case through .NET's built-in globbing support, but the fallback used for embedded and composite file providers didn't understand the **.css form correctly. Both **.css and **/*.css now behave consistently across providers.

Breaking changes

This is a major version, so there are a few things to account for when upgrading:

Smidge now targets .NET 10.
Smidge.Core has been merged into Smidge and the separate assembly no longer exists.
Smidge.Nuglify, AddSmidgeNuglify() and UseSmidgeNuglify() are no longer needed. NUglify is built into the normal Smidge setup.
The old built-in JsMinifier and CssMinifier are gone.
A few old MVC handler and filter types are now internal implementation details.
Bundle.Files is now IReadOnlyList<IWebFile> rather than a mutable List<IWebFile>. Use Bundle.AddFile(IWebFile) if you need to add files programmatically.

For most applications the upgrade should mainly be removing the old NUglify-specific setup and package reference, then rebuilding against .NET 10. If you only used the standard AddSmidge() / UseSmidge() APIs and predefined bundles, there shouldn't be much else to change.

Build and release cleanup

The dependency set has been brought up to date, including the Microsoft.Extensions packages, SourceLink, xUnit, the test SDK and Moq.

I've also replaced GitVersion with MinVer. The version now comes directly from the Git tag: normal builds produce a 5.0.0-ci.<height> prerelease version and the v5.0.0 tag produces exactly 5.0.0. It's considerably less machinery for what Smidge needs.

Give it a go

Smidge 5.0.0 is available on NuGet, and the source and full release notes are on GitHub.

This release took longer than it should have, but it also ended up being more substantial than I'd originally planned. The package setup is simpler, the runtime surface is smaller, and the two worst classes of production failure - corrupted bundle registrations and attacker-reachable cache misses turning into 500s - are now properly fixed rather than worked around.

If you run into anything while upgrading, please open an issue. And thanks to everyone who kept reporting the weird, hard-to-reproduce failures over the years. Those are never the fun ones to investigate, but in this case they led to a much better fix than another null check 🙂

Auto upgrade your Nuget packages with Azure Pipelines or GitHub Actions

Thu, 25 Feb 2021 06:12:00 Z

Before we start I just want to preface this with some warnings

This works for me, it might not work for you
To get this working for you, you may need to tweak some of the code referenced
This is not under any support or warranty by anyone
Running Nuget.exe update command outside of Visual Studio will overwrite your files so there is a manual review process (more info below)
This is only for ASP.NET Framework using packages.config – Yes I know that is super old and I should get with the times, but this has been an ongoing behind the scenes project of mine for a long time. When I need this for Package Reference projects, ASP.NET Core/5, I’ll update it but there’s nothing stopping you from tweaking this to work for you
This only works for a specified csproj, not an entire sln – it could work for that but I’ve not tested, there would be a few tweaks to make that work
This does not yet work for GitHub actions but the concepts are all here and could probably very easily be converted UPDATE: This works now!

Now that’s out of the way …

How do I do it?

With a lot of PowerShell :) This also uses a few methods from the PowerShellForGitHub project.

The process is:

Run a pipeline/action on a schedule (i.e. each day)
This checks against your source code for the installed version for a particular package
Then it checks with Nuget (using your Nuget.config file) to see what the latest stable version is
If there’s a newer version:
Create a new branch
Run a Nuget update against your project
Build the project
Commit the changes
Push the changes
Create a PR for review

Azure Pipelines/GitHub Actions YAML

The only part of the YAML that needs editing is the variables, here's what they mean:

ProjectFile = The relative path to your csproj that you want to upgrade
PackageFile = The relative path to your packages.config file for this project
PackageName = The Nuget package name you want upgraded
GitBotUser = The name used for the Git commits
GitBotEmail = The email used for the Git commits

For Azure Pipelines, these are also required:

GitHubOwner = The repository owner (i.e. https://github.com/Shazwazza/Articulate )
GitHubRepository = The repository name (i.e. https://github.com/Shazwazza/Articulate )
There is one Secret setting you need to create in Azure Pipelines: GitHubAccessToken which is a GitHub PAT that you need to create that has access to this repository.

Then there are some variables to assist with testing:

DisableUpgradeStep = If true will just check if there’s an upgrade available and exit
DisableCommit = If true will run the upgrade and will exit after that (no commit, push or PR)
DisablePush = If true will run the upgrade + commit and will exit after that (no push or PR)
DisablePullRequest = If true will run the upgrade + commit + push and will exit after that (no PR)

Each step in the yaml build more or less either calls Git commands or PowerShell functions. The PowerShell functions are loaded as part of a PowerShell Module which is committed to the repository. This module’s functions are auto-loaded by PowerShell because the first step configures the PowerShell environment variable PSModulePath to include the custom path. Once that is in place, all functions exposed by the module are auto-loaded.

In these examples you’ll see that I’m referencing Umbraco Cloud names and that’s because I’m using this on Umbraco Cloud for my own website and the examples are for the UmbracoCms package. But this should in theory work for all packages!

Show me the code

The code for all of this is here in a new GitHub repo and here’s how you use it:

You can copy the folder structure in the repository as-is. Here's an example of what my site's repository folder structure is to make this work (everything except the src folder is in the GitHub repo above):

[root]
- auto-upgrader.devops.yml (If you are using Azure Pipelines)
- .github
  - workflows
    - auto-upgrader.gh.yml (If you are using GitHub Actions)
- build
  - PowershellModules
    - AutoUpgradeFunctions.psd1
    - AutoUpgradeFunctions.psm1
    - AutoUpgradeFunctions
- src
  - Shazwazza.Web
    - Shazwazza.Web.csproj
    - packages.config

All of the steps have descriptive display names and it should be reasonably self documenting.

The end result is a PR, here’s one that was generated by this process:

Nuget overwrites

Nuget.exe works differently than Nuget within Visual Studio’s Package Manager Console. All of those special commands like Install-Package, Update-Package, etc… are all PowerShell module commands built into Visual Studio and they are able to work with the context of Visual Studio. This allows those commands to try to be a little smarter when running Nuget updates and also allows the legacy Nuget features like running PowerShell scripts on install/update to run. This script just uses Nuget.exe and it’s less smart especially for these legacy .NET Framework projects. As such, it will just overwrite all files in most cases (it does detect file changes it seems but isn’t always accurate).

With that warning it is very important to make sure you review the changed files in the PR and revert or adjust any changes you need before applying the PR.

You’ll see a note in the PowerShell script about Nuget overwrites. There are other options that can be used like "Ignore" and "IgnoreAll" but all my tests have showed that for some reason this setting will end up deleting a whole bunch of files so the default overwrite setting is used.

Next steps

Get out there and try it! Would love some feedback on this if/when you get a change to test it.

PackageReference support with .NET Framework projects could also be done (but IMO this is low priority) along with being able to upgrade the entire SLN instead of just the csproj.

Then perhaps some attempts at getting a NET Core/5 version of this running. In theory that will be easier since it will mostly just be dotnet commands.

How to execute one Controller Action from another in ASP.NET 5

Mon, 15 Feb 2021 05:06:00 Z

This will generally be a rare thing to do but if you have your reasons to do it, then this is how…

In Umbraco one valid reason to do this is due to how HTTP POSTs are handled for forms. Traditionally an HTML form will POST to a specific endpoint, that endpoint will handle the validation (etc), and if all is successful it will redirect to another URL, else it will return a validation result on the current URL (i.e. PRG POST/REDIRECT/GET). In the CMS world this may end up a little bit weird because URLs are dynamic. POSTs in theory should just POST to the current URL so that if there is a validation result, this is still shown on the current URL and not a custom controller endpoint URL. This means that there can be multiple controllers handling the same URL, one for GET, another for POST and that’s exactly what Umbraco has been doing since MVC was enabled in it many years ago. For this to work, a controller is selected during the dynamic route to handle the POST (a SurfaceController in Umbraco) and if successful, typically the developer will use: return RedirectToCurrentUmbracoPage (of type RedirectToUmbracoPageResult) or if not successful will use: return CurrentUmbracoPage (of type UmbracoPageResult). The RedirectToUmbracoPageResult is easy to handle since this is just a redirect but the UmbracoPageResult is a little tricky because one controller has just handled the POST request but now it wants to return a page result for the current Umbraco page which is handled by a different controller.

IActionInvoker

The concept is actually pretty simple and the IActionInvoker does all of the work. You can create an IActionInvoker from the IActionInvokerFactory which needs an ActionContext. Here’s what the ExecuteResultAsync method of a custom IActionResult could look like to do this:

public async Task ExecuteResultAsync(ActionContext context)
{
    // Change the route values to match the action to be executed
    context.RouteData.Values["controller"] = "Page";
    context.RouteData.Values["action"] = "Index";

    // Create a new context and excute the controller/action
    // Copy the action context - this also copies the ModelState
    var renderActionContext = new ActionContext(context)
    {
        // Normally this would be looked up via the EndpointDataSource
        // or using the IActionSelector
        ActionDescriptor = new ControllerActionDescriptor
        {
            ActionName = "Index",
            ControllerName = "Page",
            ControllerTypeInfo = typeof(PageController).GetTypeInfo(),
            DisplayName = "PageController.Index"
        }
    };

    // Get the factory
    IActionInvokerFactory actionInvokerFactory = context.HttpContext
                .RequestServices
                .GetRequiredService<IActionInvokerFactory>();

    // Create the invoker
    IActionInvoker actionInvoker = actionInvokerFactory.CreateInvoker(renderActionContext);

    // Execute!
    await actionInvoker.InvokeAsync();
}

That’s pretty must the gist of it. The note about the ControllerActionDescriptor is important though, it’s probably best to not manually create these since they are already created with all of your routing. They can be queried and resolved in a few different ways such as interrogating the EndpointDataSource or using the IActionSelector. This execution will execute the entire pipeline for the other controller including all of it’s filters, etc…

Allowing dynamic SupportedCultures in RequestLocalizationOptions

Wed, 13 Jan 2021 03:30:00 Z

The documented usage of RequestLocalizationOptions in ASP.NET 5/Core is to assign a static list of SupportedCultures since ASP.NET is assuming you’ll know up-front what cultures your app is supporting. But what if you are creating a CMS or another web app that allows users to include cultures dynamically?

This isn’t documented anywhere but it’s certainly possible. RequestLocalizationOptions.SupportedCultures is a mutable IList which means that values can be added/removed at runtime if you really want.

Create a custom RequestCultureProvider

First thing you need is a custom RequestCultureProvider. The trick is to pass in the RequestLocalizationOptions into it’s ctor so you can dynamically modify the SupportedCultures when required.

public class MyCultureProvider : RequestCultureProvider
{
    private readonly RequestLocalizationOptions _localizationOptions;
    private readonly object _locker = new object();

    // ctor with reference to the RequestLocalizationOptions
    public MyCultureProvider(RequestLocalizationOptions localizationOptions)
        => _localizationOptions = localizationOptions;

    public override Task<ProviderCultureResult> DetermineProviderCultureResult(HttpContext httpContext)
    {
        // TODO: Implement GetCulture() to get a culture for the current request
        CultureInfo culture = GetCulture(); 

        if (culture is null)
        {
            return NullProviderCultureResult;
        }

        lock (_locker)
        {
            // check if this culture is already supported
            var cultureExists = _localizationOptions.SupportedCultures.Contains(culture);

            if (!cultureExists)
            {
                // If not, add this as a supporting culture
                _localizationOptions.SupportedCultures.Add(culture);
                _localizationOptions.SupportedUICultures.Add(culture);
            } 
        }

        return Task.FromResult(new ProviderCultureResult(culture.Name));
    }
}

Add your custom culture provider

You can configure RequestLocalizationOptions in a few different ways, this example registers a custom implementation of IConfigureOptions<RequestLocalizationOptions> into DI

public class MyRequestLocalizationOptions : IConfigureOptions<RequestLocalizationOptions>
{
    public void Configure(RequestLocalizationOptions options)
    {
        // TODO: Configure other options parameters

        // Add the custom provider,
        // in many cases you'll want this to execute before the defaults
        options.RequestCultureProviders.Insert(0, new MyCultureProvider(options));
    }
}

Then just register these options: Services.ConfigureOptions<MyRequestLocalizationOptions>();

That’s it, now you can have dynamic SupportedCultures in your app!

How to change HttpRequest.IsSecureConnection

Tue, 15 Dec 2020 13:27:00 Z

This post is for ASP.NET Framework, not ASP.NET Core

This is an old problem that is solved in dozens of different ways…

Typically in an ASP.NET Framework site when you need to check if the request is running in HTTPS you would do something like HttpContext.Request.IsSecureConnection. But this value won’t return the anticipated value if your site is running behind a proxy or load balancer which isn’t forwarding HTTPS traffic and instead is forwarding HTTP traffic. This is pretty common and the way people deal with it is to check some special headers, which will vary depending on the load balancer you are using. The primary one is HTTP_X_FORWARDED_PROTO but there are others too like: X-Forwarded-Protocol, X-Forwarded-Ssl, X-Url-Scheme , and the expected values for each are slightly different too.

So if you can’t rely on HttpContext.Request.IsSecureConnection then you’ll probably make an extension method, or better yet an abstraction to deal with checking if the request is HTTPS. That’s totally fine but unfortunately you can’t rely on all of the dependencies your website uses to deal with this scenario. You’d also need to be able to configure them all to possibly handle some weird header scheme that your proxy/load balancer uses.

Aha! HttpRequestBase is abstract

If you are using MVC you’ll probably know that the HttpContext.Request within a controller is HttpRequestBase which is an abstract class where it’s possible to re-implement IsSecureConnection. Great, that sounds like a winner! Unfortunately this will only get you as far as being able to replace that value within your own controllers, this will still not help you with the dependencies your website uses.

And controllers aren’t the only thing you might need to worry about. You might have some super old ASP.NET Webforms or other code lying around using HttpContext.Current.Request.IsSecureConnection which is based on the old HttpRequest object which cannot be overridden.

So how does the default HttpContext.Current get this value?

The HttpContextBase is based on HttpContextWrapper which simply wraps HttpContext.Current. This is what is passed to your MVC controllers. As for HttpContext.Current, this is the source of all of this data and with a bit (a LOT) of hacking you can actually change this value.

The HttpContext.Current is actually a set-able singleton value. The constructor of HttpContext has 2x overloads, one that takes in a request/response object and another that accepts something called HttpWorkerRequest. So it is possible (sort of) to construct a new HttpContext instance that wraps the old one … so can we change this value? Yes!

HttpWorkerRequest is an abstract class and all methods are overridable so if we can access the underlying default HttpWorkerRequest (which is IIS7WorkerRequest) from the current context, we could wrap it and return anything we want for IsSecureConnection. This is not a pretty endeavor but it can be done.

Wrapping the HttpContext

I created a gist to show this. The usage is easy, just create an IHttpModule and replace the HttpContext by using a custom HttpWorkerRequest instance that accepts a delegate to return whatever you want for IsSecureConnection:

Now to create the wrapper, there’s a ton of methods and properties to override:

So what about OWIN?

Owin is easy to take care of, it’s OwinContext.Request.IsSecure is purely based on whether or not the scheme is https and the .Scheme property is settable:

Does this all actually work?

This was an experiment to see how possible this is and based on my rudimentary tests, yes it works. I have no idea if this will have some strange side affects but in theory its all just wrapping what normally executes. There will be a small performance penalty because this is going to create a couple of new objects each request and use a reflection call but I think that will be very trivial.

Let me know if it works for you!

Controller Scoped Model Binding in ASP.NET Core

Tue, 12 May 2020 01:51:00 Z

Want to avoid [FromBody] attributes everywhere? Don’t want to use [ApiController] strict conventions? Don’t want to apply IInputFormatter’s globally?

ASP.NET Core MVC is super flexible but it very much caters towards configuring everything at a global level. Perhaps you are building a framework or library or a CMS in .NET Core? In which case you generally want to be as unobtrusive as possible so mucking around with global MVC configuration isn’t really acceptable. The traditional way of dealing with this is by applying configuration directly to controllers which generally means using controller base classes and attributes. This isn’t super pretty but it works in almost all cases from applying authorization/resource/action/exception/result filters to api conventions. However this doesn’t work for model binding.

Model binding vs formatters

Model binding comes in 2 flavors: formatters for deserializing the request body (like JSON) into models and value providers for getting data from other places like form body, query string, headers, etc… Both of these things internally in MVC use model binders though typically the language used for binding the request body are called formatters. The problem with formatters (which are of type IInputFormatter) is that they are only applied at the global level as part of MvcOptions which are in turn passed along to a special model binder called BodyModelBinder. Working with IInputFormatter at the controller level is almost impossible.

There seems to be a couple options that look like you might be able to apply a custom IInputFormatter to a specific controller:

Create a custom IModelBinderProvider – this unfortunately will not work because the ModelBinderProviderContext doesn’t provide the ControllerActionDescriptor executing so you cannot apply this provider to certain controllers/actions (though this should be possible).
Assign a custom IModelBinderFactory to the controller explicitly by assigning ControllerBase.ModelBinderFactory in the controllers constructor – this unfortunately doesn’t work because the ControllerBase.ModelBinderFactory isn’t used for body model binding

So how does [ApiController] attribute work?

The [ApiController] attribute does quite a lot of things and configures your controller in a very opinionated way. It almost does what I want and it somehow magically does this

[FromBody] is inferred for complex type parameters

That’s great! It’s what I want to do but I don’t want to use the [ApiController] attribute since it applies too many conventions and the only way to toggle these …. is again at the global level :/ This also still doesn’t solve the problem of applying a specific IInputFormatter to be used for the model binding but it’s a step in the right direction.

The way that the [ApiController] attribute works is by using MVC’s “application model” which is done by implementing IApplicationModelProvider.

A custom IApplicationModelProvider

Taking some inspiration from the way [ApiController] attribute works we can have a look at the source of the application model that makes this happen: ApiBehaviorApplicationModelProvider. This basically assigns a bunch of IActionModelConvention’s: ApiVisibilityConvention, ClientErrorResultFilterConvention, InvalidModelStateFilterConvention, ConsumesConstraintForFormFileParameterConvention, ApiConventionApplicationModelConvention, and InferParameterBindingInfoConvention. The last one InferParameterBindingInfoConvention is the important one that magically makes complex type parameters bind from the request body like JSON like good old WebApi used to do.

So we can make our own application model to target our own controllers and use a custom IActionModelConvention to apply a custom body model binder:


public class MyApplicationModelProvider : IApplicationModelProvider
{
    public MyApplicationModelProvider(IModelMetadataProvider modelMetadataProvider)
    {
        ActionModelConventions = new List<IActionModelConvention>()
        {
            // Ensure complex models are bound from request body
            new InferParameterBindingInfoConvention(modelMetadataProvider),
            // Apply custom IInputFormatter to the request body
            new MyModelBinderConvention()
        };
    }

    public List<IActionModelConvention> ActionModelConventions { get; }

    public int Order => 0;

    public void OnProvidersExecuted(ApplicationModelProviderContext context)
    {
    }

    public void OnProvidersExecuting(ApplicationModelProviderContext context)
    {
        foreach (var controller in context.Result.Controllers)
        {
            // apply conventions to all actions if attributed with [MyController]
            if (IsMyController(controller))
                foreach (var action in controller.Actions)
                    foreach (var convention in ActionModelConventions)
                        convention.Apply(action);
        }
    }

    // returns true if the controller is attributed with [MyController]
    private bool IsMyController(ControllerModel controller)
        => controller.Attributes.OfType<MyControllerAttribute>().Any();
}

And the custom convention:


public class MyModelBinderConvention : IActionModelConvention
{
    public void Apply(ActionModel action)
    {
        foreach (var p in action.Parameters
            // the InferParameterBindingInfoConvention must execute first,
            // which assigns this BindingSource, so if that is assigned
            // we can then assign a custom BinderType to be used.
            .Where(p => p.BindingInfo?.BindingSource == BindingSource.Body))
        {
            p.BindingInfo.BinderType = typeof(MyModelBinder);
        }
    }
}

Based on the above application model conventions, any controller attributed with our custom [MyController] attribute will have these conventions applied to all of it’s actions. With the above, any complex model that will be bound from the request body will use the IModelBinder type: MyModelBinder, so here’s how that implementation could look:


// inherit from BodyModelBinder - it does a bunch of magic like caching
// that we don't want to miss out on
public class MyModelBinder : BodyModelBinder
{
    // TODO: You can inject other dependencies to pass to GetInputFormatter
    public MyModelBinder(IHttpRequestStreamReaderFactory readerFactory)
        : base(GetInputFormatter(), readerFactory)
    {
    }

    private static IInputFormatter[] GetInputFormatter()
    {  
        return new IInputFormatter[]
        {
            // TODO: Return any IInputFormatter you want
            new MyInputFormatter()
        };
    }
}

The last thing to do is wire it up in DI:


services.TryAddSingleton<MyModelBinder>();            
services.TryAddEnumerable(
    ServiceDescriptor.Transient<IApplicationModelProvider,
    MyApplicationModelProvider>());

That’s a reasonable amount of plumbing!

It could certainly be simpler to configure a body model binder at the controller level but at least there’s actually a way to do it. For a single controller this is quite a lot of work but for a lot of controllers the MVC “application mode” is quite brilliant! … it just took a lot of source code reading to figure that out :)

How to register MVC controllers shipped with a class library in ASP.NET Core

Fri, 20 Sep 2019 03:20:00 Z

In many cases you’ll want to ship MVC controllers, possibly views or taghelpers, etc… as part of your class library. To do this correctly you’ll want to add your assembly to ASP.NET’s “Application Parts” on startup. Its quite simple to do but you might want to make sure you are not enabling all sorts of services that the user of your library doesn’t need.

The common way to do this on startup is to have your own extension method to “Add” your class library to the services. For example:

public static class MyLibStartup
{
    public static IServiceCollection AddMyLib(this IServiceCollection services)
    {
        //TODO: Add your own custom services to DI

        //Add your assembly to the ASP.NET application parts
        var builder = services.AddMvc();
        builder.AddApplicationPart(typeof(MyLibStartup).Assembly);
    }
}

This will work, but the call to AddMvc() is doing a lot more than you might think (also note in ASP.NET Core 3, it’s doing a similar amount of work). This call is adding all of the services to the application required for: authorization, controllers, views, taghelpers, razor, api explorer, CORS, and more… This might be fine if your library requires all of these things but otherwise unless the user of your library also wants all of these things, in my opinion it’s probably better to only automatically add the services that you know your library needs.

In order to add your assembly application part you need a reference to IMvcBuilder which you can resolve by calling any number of the extension methods to add the services you need. Depending on what your application requires will depend on what services you’ll want to add. It’s probably best to start with the lowest common feature-set which is a call to AddMvcCore(), the updated code might look like this:

//Add your assembly to the ASP.NET application parts
var builder = services.AddMvcCore();
builder.AddApplicationPart(typeof(MyLibStartup).Assembly);

From there you can add the other bits you need, for example, maybe you also need CORS:

//Add your assembly to the ASP.NET application parts
var builder = services.AddMvcCore().AddCors();
builder.AddApplicationPart(typeof(MyLibStartup).Assembly);

Benchmarking performance of your code between release versions

Fri, 22 Mar 2019 00:26:00 Z

A while ago in 2016 I posted a question on the BenchmarkDotNet repository about an official way to run benchmarks between Nuget releases. In 2018 I managed to find some time and with the with the help of Adam Sitnik (one of the project’s maintainers), was able to make that work!

I won’t go into detail about what BenchmarkDotNet is, but it’s a brilliant way to accurately run benchmarks against your code to see how it performs, how much memory is being used, etc…

With this new feature you can now easily see how your code changes effect performance between your releases.

Show me the code

Making this work is quite easy and there’s a quick start code snippet in the repo already. For the example below I’ll use ImageSharp as the library to be tested and we’ll see how well James and his team is doing with regards to improving it’s JPEG decoding performance.


[Config(typeof(Config))]
public class ImageTests
{
    private static readonly string _filePath = @"C:\temp\test.jpg";

    private class Config : ManualConfig
    {
        public Config()
        {
            var baseJob = Job.MediumRun.With(CsProjCoreToolchain.Current.Value);
            Add(baseJob.WithNuGet("SixLabors.ImageSharp", "1.0.0-beta0006").WithId("1.0.0-beta0006"));
            Add(baseJob.WithNuGet("SixLabors.ImageSharp", "1.0.0-beta0005").WithId("1.0.0-beta0005"));
            Add(baseJob.WithNuGet("SixLabors.ImageSharp", "1.0.0-beta0004").WithId("1.0.0-beta0004"));
        }
    }

    [Benchmark]
    public Size LoadJpg()
    {
        using (var img = Image.Load(_filePath))
        {
            var size = img.Size();
            return size;
        }
    }
}

The code above should seem pretty straightforward. It’s just setting up 3 BenchmarkDotNet jobs, each using a different version of the SixLabors.ImageSharp Nuget package. Then the actual benchmark test is loading in a JPEG extracting it’s size and returning it.

Running the benchmark is like running any other BenchmarkDotNet test, for example in a console app:


class Program
{
    static void Main(string[] args)
    {   
        var summary = BenchmarkRunner.Run();
    }
}

The results

Method	Job	NuGetReferences	Mean	Error	StdDev
LoadJpg	1.0.0-beta0004	SixLabors.ImageSharp 1.0.0-beta0004	297.5 ms	142.8 ms	7.827 ms
LoadJpg	1.0.0-beta0005	SixLabors.ImageSharp 1.0.0-beta0005	202.9 ms	466.6 ms	25.577 ms
LoadJpg	1.0.0-beta0006	SixLabors.ImageSharp 1.0.0-beta0006	148.8 ms	107.8 ms	5.910 ms

Looks good! from the beta0004 release to the beta0006 release there’s almost twice the performance boost.

API Surface Area

There is one caveat though… In order to run these tests between versions of your library, the same API surface area will need to exist otherwise you’ll get exceptions when running the benchmarks. This is the reason why versions beta0001 –> beta0003 are not included in the jobs listed above. Its because in the older versions either the APIs were different or the namespaces were different.

It is possible to work around this but you’d need to use some ugly reflection to do it and then you need to be careful that you are not testing the reflection performance hit either.

Now you should have a pretty easy way to know how the performance of your library is changing between versions. Happy coding!

Configuring Azure Active Directory login with Umbraco Members

Mon, 18 Feb 2019 02:09:00 Z

This post is about configuring Azure Active Directory with Umbraco Members (not Users), meaning this is for your front-end website, not the Umbraco back office. I did write up a post about Azure AD with back office users though, so if that is what you are looking for then this is the link.

Install the Nuget packages

First thing to do is get the UmbracoIdentity package installed.

PM > Install-Package UmbracoIdentity

(This will also install the UmbracoIdentity.Core base package)

This package installs some code snippets and updates your web.config to enable ASP.Net Identity for Umbraco members. Umbraco ships with the old and deprecated ASP.Net Membership Providers for members and not ASP.Net Identity so this package extends the Umbraco CMS and the Umbraco members implementation to use ASP.Net Identity APIs to interact with the built in members data store. Installing this package will remove the (deprecated) FormsAuthentication module from your web.config and it will no longer be used to authenticate members, so the typical members snippets built into Umbraco macros will not work. Instead use the supplied snippets shipped with this package.

To read more about this package see the GitHub repo here.

Next, the OpenIdConnect package needs to be installed

PM > Install-Package Microsoft.Owin.Security.OpenIdConnect

Configure Azure Active Directory

Head over to the Azure Active Directory section on the Azure portal, choose App Registrations (I’m using the Preview functionality for this) and create a New registration

Next fill out the app details

You may also need to enter other redirect URLs depending on how many different environments you have. All of these URLs can be added in the Authentication section of your app in the Azure portal.

For AAD configuration for front-end members, the redirect Urls are just your website’s root URL and it is advised to keep the trailing slash.

Next you will need to enable Id Tokens

Configure OpenIdConnect

The UmbracoIdentity package will have installed an OWIN startup class in ~/App_Start/UmbracoIdentityStartup.cs (or it could be in App_Code if you are using a website project). This is how ASP.Net Identity is configured for front-end members and where you can specify the configuration for different OAuth providers. There’s a few things you’ll need to do:

Allow external sign in cookies

If you scroll down to the ConfigureMiddleware method, there will be a link of code to uncomment: app.UseExternalSignInCookie(DefaultAuthenticationTypes.ExternalCookie); this is required for any OAuth providers to work.

Enable OpenIdConnect OAuth for AAD

You’ll need to add this extension method class to your code which is some boiler plate code to configure OpenIdConnect with AAD:

public static class UmbracoADAuthExtensions
{
    public static void ConfigureAzureActiveDirectoryAuth(this IAppBuilder app,
        string tenant, string clientId, string postLoginRedirectUri, Guid issuerId,
        string caption = "Active Directory")
    {
        var authority = string.Format(
            System.Globalization.CultureInfo.InvariantCulture,
            "https://login.windows.net/{0}",
            tenant);

        var adOptions = new OpenIdConnectAuthenticationOptions
        {
            ClientId = clientId,
            Authority = authority,
            RedirectUri = postLoginRedirectUri
        };

        adOptions.Caption = caption;
        //Need to set the auth type as the issuer path
        adOptions.AuthenticationType = string.Format(
            System.Globalization.CultureInfo.InvariantCulture,
            "https://sts.windows.net/{0}/",
            issuerId);
        app.UseOpenIdConnectAuthentication(adOptions);
    }
}

Next you’ll need to call this code, add the following line underneath the app.UseExternalSignInCookie method call:

app.ConfigureAzureActiveDirectoryAuth(
    ConfigurationManager.AppSettings["azureAd:tenantId"],
    ConfigurationManager.AppSettings["azureAd:clientId"],
    //The value of this will need to change depending on your current environment
    postLoginRedirectUri: ConfigurationManager.AppSettings["azureAd:redirectUrl"],
    //This is the same as the TenantId
    issuerId: new Guid(ConfigurationManager.AppSettings["azureAd:tenantId"]));

Then you’ll need to add a few appSettings to your web.config (based on your AAD info):

<add key="azureAd:tenantId" value="YOUR-TENANT-ID-GUID" />
<add key="azureAd:clientId" value="YOUR-CLIENT-ID-GUID" />
<add key="azureAd:redirectUrl" value="http://my-test-website/" />

Configure your Umbraco data

The UmbracoIdentity repository has the installation documentation and you must follow these 2 instructions, and they are very simple:

Once that is done you will have an Member account management page which is based off of the installed views and snippets of the UmbracoIdentity package. This account page will look like this:

As you can see the button text under the “Use another service to log in” is the login provider name which is a bit ugly. The good news is that this is easy to change since this is just a partial view that was installed with the UmbracoIdentity package. You can edit the file: ~/Views/UmbracoIdentityAccount/ExternalLoginsList.cshtml, the code to render that button text is using @p.Authentication provider but we can easily change this to @p.Caption which is actually the same caption text used in the extension method we created. So the whole button code can look like this instead:


<button type="submit" class="btn btn-default"
        id="@p.AuthenticationType"
        name="provider"
        value="@p.AuthenticationType"
        title="Log in using your @p.Caption account">
    @p.Caption
</button>

This is a bit nicer, now the button looks like:

The purpose of all of these snippets and views installed with UmbracoIdentity is for you to customize how the whole flow looks and works so you’ll most likely end up customizing a number of views found in this folder to suit your needs.

That’s it!

Once that’s all configured, if you click on the Active Directory button to log in as a member, you’ll be brought to the standard AAD permission screen:

Once you accept you’ll be redirect back to your Account page:

Any customizations is then up to you. You can modify how the flow works, whether or not you accepting auto-linking accounts (like in the above example), or if you require a member to exist locally before being able to link an OAuth account, etc… All of the views and controller code in UmbracoIdentity is there for you to manipulate. The main files are:

~/Views/Account.cshtml
~/Views/UmbracoIdentityAccount/*
~/Controllers/UmbracoIdentityAccountController.cs
~/App_Start/UmbracoIdentityStartup.cs

Happy coding!

Configuring Azure Active Directory login with Umbraco

Fri, 16 Nov 2018 03:47:00 Z

It’s been a while since I first set this up and back then not all of these settings were in the new Azure portal, but now that they are getting this all configured is quite easy so here’s the basic steps!

Install the Nuget package

First thing to do is get the UmbracoCms.IdentityExtensions.AzureActiveDirectory package installed

PM > Install-Package UmbracoCms.IdentityExtensions.AzureActiveDirectory

(This will also install the UmbracoCms.IdentityExtensions base package)

Configure Azure Active Directory

Head over to the Azure Active Directory section on the Azure portal, choose App Registrations (I’m using the Preview functionality for this) and create a New registration

Next fill out the app details

Add your redirect URLs for any additional environments

If you have local, dev, staging, live, etc… sites, you’ll need to configure those URLs here, always make sure it’s the Umbraco path with a trailing slash.

Make note of some settings

The settings you’ll need to copy are the Application/Client ID and the Tenant ID along with the redirect URLs.

Configure you application

Add the Client Id, Tenant Id and redirect URL to AppSettings. I’ve used the keys: azureAd:tenantId, azureAd:clientId, azureAd:redirectUrl but you can choose whatever you like.
Open the App_Start/UmbracoStandardOwinStartup.cs file that was installed with the IdentityExtensions pakage

Underneath the call to base.Configuration(app); add the block of code that was shown in the readme shown after you installed the Nuget package which looks like this:

app.ConfigureBackOfficeAzureActiveDirectoryAuth(
    //The Tenant can also be "YOURDIRECTORYNAME.onmicrosoft.com"
    tenant: ConfigurationManager.AppSettings["azureAd:tenantId"],
    clientId: ConfigurationManager.AppSettings["azureAd:clientId"],
    //The value of this will need to change depending on your current environment
    postLoginRedirectUri: ConfigurationManager.AppSettings["azureAd:redirectUrl"],
    //This is the same as the TenantId
    issuerId: new Guid(ConfigurationManager.AppSettings["azureAd:tenantId"]));

Ensure that this OWIN startup class is defined in your web.config: <add key="owin:appStartup" value="UmbracoStandardOwinStartup" />

That's it!

Once you’ve got this configured, and you login to the back office normally you can then link your AD account:

Once linked, you can login with this provider:

Auto linking?

If you configured your AD App to only authenticate “only accounts in this organization”, you might want to auto-link Umbraco back office accounts. Auto-linking will automatically ensure that a local Umbraco user account exists for any user that logs in via the AD provider. This is handy if you want to do all of your user administration via Azure AD. Auto-linking can be configured as part of your OWIN startup class.

I won’t cover auto-linking in this post but there are some docs available for that.

Getting Umbraco to work with Azure Easy Auth

Tue, 17 Apr 2018 01:23:00 Z

There’s a nifty feature in your Azure App Service that allows you to very quickly add authentication and authorization to your Azure website. You’ll see most folks calling this “Easy Auth” and there’s quite a few articles on the subject such as:

The good news is this all works the way you’d expect with an Umbraco website for front-end requests but unfortunately this doesn’t play nicely with the Umbraco back office … but it’s easy to configure Umbraco to work!

The problem

The problem is that if you turn on Easy Auth and try to log in to your Umbraco back office, the login will succeed but you’ll get 401 responses for other back office requests and essentially you’ll see a blank screen. The reason this happens is due to the way that Easy Auth works:

It activates an HttpModule in your site called EasyAuthAspNetThreadPrincipalModule
During the HttpModule.AuthenticateRequest stage it replaces the Thread.CurrentPrincipal with it’s own ClaimsPrincipal/ClaimsIdentity instance

Umbraco also sets the Thread.CurrentPrincipal.Identity during this phase but at the OWIN level which executes before the EasyAuthAspNetThreadPrincipalModule. Because the Easy Auth module replaces the principal/identity, it wipes out the one created by Umbraco. What it should do instead is check if the current principal is a ClaimsPrincipal and then add it’s identity to the identity collection instead of wiping out anything that is already there. If that were the case, everything would ‘just work’ but since it is not we have to work around this issue.

The solution

UPDATE! (19/04/18) - Chris Gillum who created Easy Auth got in touch with me on Twitter to share some handy (and fairly hidden) documentation for Easy Auth. Looks like another work around is to use the WEBSITE_AUTH_DISABLE_IDENTITY_FLOW appSetting which will prevent Easy Auth from setting the thread identity at all.

To work around this problem we need to tell Umbraco to perform it’s authentication procedure after the Easy Auth module runs which is actually pretty easy to do.

Create a new OWIN startup class:


[assembly: OwinStartup("MyOwinStartup", typeof(MyOwinStartup))]
namespace MyProject
{
    public class MyOwinStartup : Umbraco.Web.UmbracoDefaultOwinStartup
    {
    }
}

Update an appSetting in your web.config to tell OWIN to use your class:


<add key="owin:appStartup" value="MyOwinStartup" />

Override the method that configures Umbraco's authentication procedure and tell it to execute after the default Authentication stage, notice this code block is using PipelineStage.PostAuthenticate:


public class MyOwinStartup : Umbraco.Web.UmbracoDefaultOwinStartup
{
    protected override void ConfigureUmbracoAuthentication(IAppBuilder app)
    {
        app
            .UseUmbracoBackOfficeCookieAuthentication(ApplicationContext, PipelineStage.PostAuthenticate)
            .UseUmbracoBackOfficeExternalCookieAuthentication(ApplicationContext, PipelineStage.PostAuthenticate)
            .UseUmbracoPreviewAuthentication(ApplicationContext, PipelineStage.Authorize);
    }
}

That's it! Now the Umbraco back office will authenticate correctly with Azure Easy Auth turned on.

Using IExceptionLogger with per-controller IControllerConfiguration in WebApi

Wed, 13 Sep 2017 05:36:00 Z

There’s plenty of examples online about implementing a custom IExceptionLogger globally in WebApi and I’m sure they all work as expected but what if you don’t want to register it globally? In that case you’d use an instance of IControllerConfiguration to do a “per-controller” configuration and in theory you could just use this syntax:


public class MyExceptionLoggerConfigurationAttribute : Attribute, IControllerConfiguration
{
	public virtual void Initialize(
		HttpControllerSettings controllerSettings, 
		HttpControllerDescriptor controllerDescriptor)
	{
		controllerSettings.Services.Add(
			typeof(IExceptionLogger), 
			new UnhandledExceptionLogger());
	}	
}

That seems pretty simple, and then you attribute whatever controller you want to use this new IExceptionLogger … nope!

As it turns out, the custom UnhandledExceptionLogger instance will never get executed unless there is also an ExceptionFilter applied to the controller (even if it doesn’t do anything)! I can’t find documentation on this anywhere but it’s definitely the case. So to make the above work, we can just replace Attribute with ExceptionFilterAttribute like:


public class MyExceptionLoggerConfigurationAttribute : ExceptionFilterAttribute, IControllerConfiguration
{
	public virtual void Initialize(
		HttpControllerSettings controllerSettings, 
		HttpControllerDescriptor controllerDescriptor)
	{
		controllerSettings.Services.Add(
			typeof(IExceptionLogger), 
			new UnhandledExceptionLogger());
	}	
}

And voila! this works.

Umbraco passwords and ASP.NET Machine Keys

Mon, 03 Jul 2017 04:26:00 Z

Update! I’ve updated a few points below in bold and have corrected a few things too

This blog post is the result of a thread on Twitter which starts here: https://twitter.com/crumpled_jeavon/status/880522105795870720 and works its way into confusion. Suffice to say I can’t answer these questions in 140 chars so here’s re-cap in the form of Q and A about Machine Keys and Umbraco. Please note that I am not an expert in hashing algorithms, some of these answers are based on my own research. Hope this clears things up!

How is the password hashed?

It is hashed in the same way that the ASP.NET Universal membership provider (DefaultMembershipProvider) and SqlMembershipProvider hashes passwords which by default uses the HMACSHA256 algorithm.

Jeffrey Schoemaker has been discussing updating Umbraco’s default password hashing to use an even stronger hash algorithm and I’ve recently updated a new task on the issue tracker to research this but it really comes down to the fact that Microsoft does not offer the best stronger hashing method in it’s standard .NET Framework so we’ll see where we end up.

Update – we will be shipping umbraco with stronger password hashing, possibly in a 7.7.x release http://issues.umbraco.org/issue/U4-10089 and it will use HMACSHA1 + PBKDF2 which is what ASP.NET Identity uses by default.

Is the Machine Key used for password hashing?

Yes, In 7.6.0+ it is by default because useLegacyEncoding is false by default in this release. Previous to 7.6.0 the useLegacyEncoding value was true by default purely to preserve some backwards compatibility settings for other products but you’ve been able to set it to true from 7.0.0 (IIRC). Since those products support the more secure password formats, this is now false by default and should be kept as false for new installs. By default the hashing algorithm is HMACSHA256 which uses comes from the ASP.NET Machine Key to perform part of it’s hashing function ‘validation’ algorithm type. This ‘validation’ algorithm type is configurable via the Machine Key or it is configurable at the membership provider level which would override the algorithm specified in the Machine Key, but you really shouldn’t change this to be a lesser strength hashing algorithm.

The HMAC part of this algorithm means it’s derived from a keyed algorithm and uses a key to generate the hash and the machine key is used to create this key by default. There doesn’t seem to be any documentation or reference to this online that I can find but trying to look through the crypto source code (which isn’t nice to look at) it seems that the default key gets set based on some logic in the RSACryptoServiceProvider class which reads some info from the machine key.

Update – the key used to hash the passwords is the generated salt we produce it is not the key part of the Machine Key. This logic is exactly the same as the logic used in the (DefaultMembershipProvider) and SqlMembershipProvider and if the hashing algorithm key length doesn’t equal the generated salt key length then it is padded/trimmed to the correct length, see source here. The part of the Machine Key that is used to hash the passwords is specifically the algorithm type. As you can see on this machine key generator, there can be a handful of different algorithm types used for the ‘validation’ part of a machine key and the default of this value changes based on the ASP.NET version being used. In ASP.NET 4.5 the default is HMACSHA256. Also note that in ASP.NET 4.5 the following algorithms are no longer allowed in the Machine Key config: AES, 3DES, and MD5

Do machine keys change between environments?

If you explicitly generate and set your own machine key in your web.config then the answer is No.

If you don’t explicitly generate and set your own machine key than you will get an auto-generated machine key. The simple answer for this is: In most cases No, an auto-generated machine key will not change between environments.

To understand when it will change between environments is a little more complicated and comes down to a combination of IIS user, IIS website virtual path (i.e. if you are running a site in a virtual directory), and a combination of a few settings set at the machine config level: “IsolateApps” and “IsolateByAppId”. Unless a server administrator specifically changes these settings than chances are you won’t be affected by different auto-generated machine keys for your site. If you are really keen, here’s a series all about this topic and other cryptographic changes in .NET 4.5:

Part 1 – see the “A brief digression: auto-generated machine keys” for info on auto-generating keys
Part 2 – in-depth info about the machine key and hashing changes
Part 3 – interesting info especially with regards to PBKDF2 in .NET Framework

Update – another reason a machine key may change between environments is based on which version of ASP.NET is running and what it’s default ‘validation’ algorithm type is

Can I change my machine key?

No. YES

However, I realize In some cases you might need to change it or move from an auto-generated machine key to an explicit machine key. If that is the case there will may be a lot of some manual work you’ll need to do. If you simply just change the machine key or add an explicit one when you previously didn’t have one than your members/users will might not be able to log in! This really comes down to what hashing algorithm was used originally to hash the passwords and what hash algorithm is configured in your Machine Key. If you install or change a machine key to a different algorithm that was used to first hash your passwords, then your members/users will not be able to log in.

Update – Previously I was under the impression that the key in the hashing algorithm was directly affected by the key in the machine key but after some more investigation it is not actually the case. As mentioned in the above updates, the part of the Machine Key that is used in the password hashing is the algorithm type specified (if you haven’t overridden this algorithm type by specifying it directly on the membership provider). I’ve also detailed some of this investigation in this ticket: http://issues.umbraco.org/issue/U4-10222

Can I change from useLegacyEncoding ‘true’ to ‘false’?

Not easily.

This will require a bit of extra work just like the above question but it’s not quite as tricky as changing a Machine Key. When you have useLegacyEncoding=’true’ , the machine key validation algorithm is not used, so you could do something like:

Create a new password column in the database which will store the newly hashed password based on the new machine key validation algorithm
When a user logs in you can check if they have the new password format or not.

If not, then you can validate the password based on the old format then re-hash with the new format and store it and delete the old stored hash password.
If so, then you can validate the password based on the new format

To do this would probably require inheriting from the current Umbraco membership provider and implementing this logic yourself

Do I need to generate and store a custom Machine Key if I am planning on Load Balancing?

Yes.

This is also mentioned in the Umbraco docs for load balancing

Do I need to generate and install a Machine Key before I run the Umbraco installer?

Yes.

This is because during the Umbraco installation it will hash and store the password for the admin account and if you then add a Machine Key after the fact, you will no longer be able to log in.

Can Umbraco generate a custom Machine Key on install for me?

Yes!

but it doesn’t do that right now. I created that functionality in a PR but we decided to remove the machine key generation part when it was accepted. We have decided to bring it back though so that will be part of an upcoming Umbraco release, whether that is the 7.6.x series or 7.7 series is still being decided.

Update – this is shipping with 7.7 and new installs will have a Machine Key generated and installed for you. You can of course opt out of this in the advanced installer options if you like.

Do Machine Key’s exist in ASP.NET Core?

No.

Well sort of but not really. I haven’t been able to research too much into this but when speaking to a couple MS Dev’s about this a couple years ago the answer was that the way machine key’s work will be different. If keys need to be shared between apps in ASP.NET Core the data protection APIs need to be used and these keys would then be stored in the registry or the Cloud (i.e. Azure Key Vault), here’s a SO article on this.

Clear as mud?! ;)

OWIN Cookie Authentication with variable cookie paths

Fri, 17 Feb 2017 04:17:00 Z

By default OWIN Cookie Authentication let’s you specify a single configurable cookie path that does not change for the lifetime of the application, for example

app.UseCookieAuthentication(new CookieAuthenticationOptions
{                
    CookiePath = "/Client1",
    CookieSecure = CookieSecureOption.SameAsRequest,
    CookieHttpOnly = true
});

This is going to only allow cookie authentication to occur when the request is for any path under /Client1. But what if you wanted this same cookie and cookie authentication provider to work for other variable paths, what if we wanted it to execute for multiple configured paths like: /Client1, /Client2/Secured, /Client3/Private ? Or what if we wanted wanted this Cookie Authentication Provider to execute dynamically based on the request object?

ICookieManager to the rescue

The CookieAuthenticationOptions has a property: CookieManager which you can set to any instance of ICookieManager. ICookieManager contains these methods: GetRequestCookie, AppendResponseCookie, DeleteCookie but all we really need to worry about GetRequestCookie. It turns out that the CookieAuthenticationHandler will detect if this method returns null and if so it will just not continue trying to authenticate the request. To make things easy we’ll just inherit from the default OWIN ICookieManager which is ChunkingCookieManager, although it’s methods are not marked as virtual we can still override them by explicitly implementing the ICookieManager.GetRequestCookie method (Pro Tip! You can always override a non virtual method if you explicitly implement an interfaces method).

//explicit implementation
string ICookieManager.GetRequestCookie(IOwinContext context, string key)
{
    //TODO: Given what is in the context, we can check pretty much anything, if we don't want
    //this request to continue to be authenticated, just return null. Example:
    var toMatch = new[] {"/Client1", "/Client2/Secured", "/Client3/Private"};
    if (!toMatch.Any(m => context.Request.Uri.AbsolutePath.StartsWith(m, StringComparison.OrdinalIgnoreCase)))
    {
        return null;
    }

    //if we don't want to ignore it then continue as normal:
    return base.GetRequestCookie(context, key);            
}

The last step is to not worry about the CookiePath since the custom ICookieManager.GetRequestCookie is going to deal with whether the middleware receives a cookie value or not so in that case, the CookiePath for the CookieAuthenticationOptions will remain the default of “/”

We’ve been doing this in Umbraco CMS core for quite some time, I meant to blog about this then but just didn’t find the time. In Umbraco we have a few custom request paths that we need authenticated with our custom back office cookie, for reference the BackOfficeCookieManager source is found here: https://github.com/umbraco/Umbraco-CMS/blob/dev-v7/src/Umbraco.Web/Security/Identity/BackOfficeCookieManager.cs

Smidge 2.0 alpha is out

Fri, 30 Dec 2016 03:39:00 Z

What is Smidge? Smidge is a lightweight runtime bundling library (CSS/JavaScript file minification, combination, compression) for ASP.NET Core.

If you’ve come from ASP.NET 4.5 you would have been familiar with the bundling/minification API and other bundling options like ClientDependency, but that is no longer available in ASP.NET Core, instead it is advised to do all the bundling and pre-processing that you need as part of your build process …which certainly makes sense! So why create this library? A few reasons: some people just want to have a very simple bundling library and don’t want to worry about Gulp or Grunt or WebPack, in a lot of cases the overhead of runtime processing is not going to make any difference, and lastly, if you have created something like a CMS that dynamically loads in assets from 3rd party packages or plugins, you need a runtime bundler since these things don’t exist at build time.

Over the past few months I’ve been working on some enhancements to Smidge and have found a bit of time to get an alpha released. There’s loads of great new features in Smidge 2.0! You can install via Nuget and is targets .NET Standard 1.6 and .NET Framework 4.5.2

PM> Install-Package Smidge -Pre

New to Smidge?

It’s easy to get started with Smidge and there’s lots of docs available on GitHub that cover installation, configuration, creating bundles and rendering them.

New Features

Here’s a list of new features complete with lots of code examples

Customizable Debug and Production options

https://github.com/Shazwazza/Smidge/issues/58

Previous to version 2.0, you could only configure aspects of the Production options and the Debug assets that were returned were just the raw static files. With 2.0, you have full control over how your assets are processed in both Debug and Production configurations. For example, if you wanted you could have your assets combined but not minified in Debug mode. This will also allow for non native web assets such as TypeScript to have pre-processors running and able to work in Debug mode.

Example:

services.AddSmidge(_config)
    .Configure<SmidgeOptions>(options =>
    {
        //set the default e-tag options for Debug mode
        options.DefaultBundleOptions.DebugOptions.CacheControlOptions.EnableETag = false        
    });

Fluent syntax for declaring/configuring bundles

https://github.com/Shazwazza/Smidge/issues/55

If you want to customize Debug or Production options per bundle, you can do so with a fluent syntax, for example:

app.UseSmidge(bundles =>
{                
    //For this bundle, enable composite files for Debug mode, enable the file watcher so any changes
    //to the files are automatically re-processed and cache invalidated, disable cache control headers
    //and use a custom cache buster. You could of course use the .ForProduction options too 
    bundles.Create("test-bundle-2", WebFileType.Js, "~/Js/Bundle2")
        .WithEnvironmentOptions(BundleEnvironmentOptions.Create()
                .ForDebug(builder => builder
                    .EnableCompositeProcessing()
                    .EnableFileWatcher()
                    .SetCacheBusterType<AppDomainLifetimeCacheBuster>()
                    .CacheControlOptions(enableEtag: false, cacheControlMaxAge: 0))
                .Build()
        );                
});

Customizable Cache Buster

https://github.com/Shazwazza/Smidge/issues/51

In version 1.0 the only cache busting mechanism was Smidge’s version property which is set in config, in 2.0 Smidge allows you to control how cache busting is controlled at a global and bundle level. 2.0 ships with 2 ICacheBuster types:

ConfigCacheBuster – the default and uses Smidge’s version property in config
AppDomainLifetimeCacheBuster – if enabled will mean that the server/browser cache will be invalidated on every app domain recycle

If you want a different behavior, you can define you own ICacheBuster add it to the IoC container and then just use it globally or per bundle. For example:

//Set a custom MyCacheBuster as the default one for Debug assets:
services.AddSmidge(_config)
    .Configure<SmidgeOptions>(options =>
    {
        options.DefaultBundleOptions.DebugOptions.SetCacheBusterType<MyCustomCacheBuster>();       
    });

//Set a custom MyCacheBuster as the cache buster for a particular bundle in debug mode:
bundles.Create("test-bundle-2", WebFileType.Js, "~/Js/Bundle2")
    .WithEnvironmentOptions(BundleEnvironmentOptions.Create()
            .ForDebug(builder => builder
                .SetCacheBusterType<MyCacheBuster>()
            .Build()
    );

Customizable cache headers

https://github.com/Shazwazza/Smidge/issues/48

You can now control if you want the ETag header output and you can control the value set for max-age/s-maxage/Expires header at a global or bundle level, for example:

//This would set the max-age header for this bundle to expire in 5 days
bundles.Create("test-bundle-5", WebFileType.Js, "~/Js/Bundle5")
    .WithEnvironmentOptions(BundleEnvironmentOptions.Create()
            .ForProduction(builder => builder                                
                .CacheControlOptions(enableEtag: true, cacheControlMaxAge: (5 * 24)))
            .Build()
    );

Callback to customize the pre-processor pipeline per web file

https://github.com/Shazwazza/Smidge/issues/59

This is handy in case you want to modify the pipeline for a given web file at runtime based on some criteria, for example:

services.AddSmidge(_config)
    .Configure<SmidgeOptions>(options =>
    {
        //set the callback
        options.PipelineFactory.OnGetDefault = GetDefaultPipelineFactory;
    });

//The GetDefaultPipeline method could do something like modify the default pipeline to use Nuglify for JS processing:

private static PreProcessPipeline GetDefaultPipelineFactory(WebFileType fileType, IReadOnlyCollection<IPreProcessor> processors)
{
    switch (fileType)
    {
        case WebFileType.Js:
            return new PreProcessPipeline(new IPreProcessor[]
            {
                processors.OfType<NuglifyJs>().Single()
            });                
    }
    //returning null will fallback to the logic defined in the registered PreProcessPipelineFactory
    return null;
}

File watching with automatic cache invalidation

https://github.com/Shazwazza/Smidge/pull/42

During the development process it would be nice to be able to test composite files but have them auto re-process and invalidate the cache whenever one of the source files changes… in 2.0 this is possible! You can enable file watching at the global level or per bundle. Example:

//Enable file watching for all files in this bundle when in Debug mode
bundles.Create("test-bundle-7",
    new CssFile("~/Js/Bundle7/a1.js"),
    new CssFile("~/Js/Bundle7/a2.js"))
    .WithEnvironmentOptions(BundleEnvironmentOptions.Create()
            .ForDebug(builder => builder.EnableFileWatcher())
            .Build()
    );

What’s next?

This is an alpha release since there’s a few things that I need to complete. Most are already done but I just need to make Nuget packages for them:

More pre-processors

I’ve enabled support for a Nuglify pre-processor for both CSS and JS (Nuglify is a fork of the Microsoft Ajax Minifier for ASP.NET Core + additional features). I also enabled support for an Uglify NodeJs pre-processor which uses Microsoft.AspNetCore.NodeServices to invoke Node.js from ASP.NET and run the JS version of Uglify. I just need to get these on Nuget but haven’t got around to that yet.

A quick note on minifier performance

Though Nuglify and Uglify have a better minification engine (better/smarter size reduction) than JsMin because they create an AST (Abstract Syntax Tree) to perform it’s processing, they are actually much slower and consume more resources than JsMin. Since Smidge is a Runtime bundling engine, its generally important to ensure that the bundling/minification is performed quickly. Smidge has strict caching so the bundling/minification will only happen once (depending on your ICacheBuster you are using) but it is still recommended to understand the performance implications of replacing JsMin with another minifier. I’ve put together some benchmarks (NOTE: a smaller Minified % is better):

Method	Median	StdDev	Scaled	Scaled-SD	Minified %	Gen 0	Gen 1	Gen 2	Bytes Allocated/Op
JsMin	10.2008 ms	0.3102 ms	1.00	0.00	51.75%	-	-	-	155,624.67
Nuglify	69.0778 ms	0.0180 ms	6.72	0.16	32.71%	53.00	22.00	15.00	4,837,313.07
JsServicesUglify	1,548.3951 ms	7.6388 ms	150.95	3.73	32.63%	0.97	-	-	576,056.55

The last benchmark may be a bit misleading because the processing is done via NodeJs which executes in a separate process so I'm unsure if the actual memory usage of that can be properly captured by BenchmarkDotNet but you can see it's speed is much slower.

Thanks!

Big thanks to @dazinator for all the help, recommendations, testing, feedback, etc… and for the rest of the community for filing bugs, questions, and comments. Much appreciated :)

FCN (File Change Notification) Viewer for ASP.NET

Mon, 19 Dec 2016 05:04:00 Z

This is a follow up post about another article I previously wrote about FCN on ASP.NET and how it affects your application, performance, etc…

Since that post, I’ve discovered a few more tidbits about FCN and application restarts and have decided to release a Nuget package that you can install to generate a report of all file/directory change monitors that ASP.NET creates.

The code for the FCN report generator lives on GitHub and you can install it via nuget:

PM> Install-Package FCNViewer

Once that is installed you’ll get a readme on how to enable it, basically just this in your WebApi startup/route config:

config.Routes.MapFcnViewerRoute();

Then you can navigate to /fcn and you’ll get a nice report showing all of the files/folders being watched by ASP.NET.

FCN modes & app domain restarts

I won’t go into full details about all the FCN modes again (you can find those details on my previous post) but I want to provide some info about “Single” vs “NotSet” (the default).

When the default (“NotSet”) is used, ASP.NET will create a directory monitor inside of every folder that the ASP.NET runtime accesses. This includes accessing a folder to return an asset request like CSS or JS, or rendering a razor file or reading from a data file. Basically it means ASP.NET has the potential to create a directory monitor for every directory you have in your site. The good part about the default mode is that each directory monitor will have it’s own buffer to manage the files it is watching so there’s less chance these buffers can overflow. The problem with the default is file system performance especially if you are hosting your site from a remote file server – the more directory monitors that are created the more strain will be put on your file server/network which could cause unwanted app domain restarts.

When set to “Single”, ASP.NET will create one directory monitor and this single directory monitor will be used to monitor all folders that ASP.NET accesses. This means there is a single buffer that is used for monitoring all of the files and folders. This buffer is larger than the buffer used for each individual monitor created with the default is used, however the problem with “Single” is that it means that if you have tons of folders there’s a higher chance this buffer can overflow which could cause unwanted app domain restarts. The good part about “Single” is that its much better for performance for the file system especially when you are hosting your site from a remote file server.

As you can see there’s no perfect scenario and both can cause unwanted app domain restarts. These restarts will end up in your logs with very peculiar reasons like:

Application shutdown. Details: ConfigurationChange
_shutDownMessage=Overwhelming Change Notification in 
    HostingEnvironment initiated shutdown
    CONFIG change
    Overwhelming Change Notification in D:\inetpub\test
    CONFIG change
    Change Notification for critical directories.
    Overwhelming Change Notification in bin
    Change Notification for critical directories.
    Overwhelming Change Notification in App_LocalResources
    CONFIG change
    CONFIG change
    CONFIG change
    CONFIG change

Or other strange “ConfigurationChange” reasons that might tell you that all sorts of files have been changed – but you definitely know that is not the cause. The cause of this is very likely FCN issues.

Files outside of the web root

An interesting bit of info is that ASP.NET wont create directory/file monitors for files that exist outside of the web root even if ASP.NET accesses them. I’d recommend where possible that you store any sort of cache files, data files, etc… that ASP.NET doesn’t need to serve up as static file requests outside of the web root. This is actually the default way of working in ASP.NET Core too. As an example, you may have heard or use Image Processor which creates quite a substantial amount of cache folders for processed images. These files by default exist in /App_Data/cache and since that is in the web root, ASP.NET will create a directory watcher for each folder in there … and there can be literally thousands if you use Image Processor extensively! If you are using FCN “Single” mode and have that many folders in the Image Processor cache, there’s a good chance that you’ll have app domain restart issues - though changing it to “NotSet” could result in serious file system performance issues. The good news for Image Processor is I created a PR to allow for storing it’s cache outside of the web root: https://github.com/JimBobSquarePants/ImageProcessor/pull/521 so that functionality should be available in an upcoming release soon.

Virtual Directories

Updated 13/08/2018

As it turns out, Virtual Directories in IIS are treated differently as well! Any fcnMode that you configure has no affect on files within a Virtual Directory except for if you disable FCN. So if you are thinking of using “Single” and are using Virtual Directories, think again. The only FCN setting that affects Virtual Directories is “Disabled”. Here’s the actual code that creates directory watches for Virtual Directories: https://referencesource.microsoft.com/#System.Web/FileChangesMonitor.cs,0820c837402228ef and as you can see, if it’s not disabled, it will be creating a directory watcher for every sub directory in the Virtual Directory. This problem is compounded if the Virtual Directory is pointing to a network share.

FCN & ASP.NET CacheDependency

Another thing to be aware of is that if you use ASP.NET’s cache (i.e. HttpRuntime.Cache or HttpContext.Cache or MemoryCache) and you create cache dependencies on specific files, this effectively goes straight to the underlying FCN engine of ASP.NET and it will create these same file/directory monitors that ASP.NET creates by default … even if these files are stored outside of the web root! So if you are creating a ton of CacheDependency objects, you will be creating a ton of FCN directory/file monitors which could be directly causing FCN issues with app domain restarts.

The FCN Report

At least with this report viewer you can see how many files and folders are being watched. It’s important to know that these watchers are created lazily whenever ASP.NET accesses files so on first load you might not see too many but if you start browsing around your site, you’ll see the number grow.

You can also modify the default route by using an overload:

config.Routes.MapFcnViewerRoute("myfcnreport");

This can hopefully help debug these strange restart issues, give you an idea of how much ASP.NET is actually watching and show you how the FCN Modes affect these monitors.

Umbraco CLI running on ASP.NET Core

Wed, 26 Oct 2016 14:13:00 Z

TL;DR I’ve got Umbraco (the Core part) running on .NET Core (not just a .NET Core CLI wrapping a non .NET Core Umbraco). See below for a quick video of it working on Ubuntu and simple instructions on how to get it running yourself.

Over the past couple of years I’ve slowly been working on getting Umbraco to run on ASP.NET Core. Unlike many other ASP.NET frameworks and products that have rewritten their apps in ASP.NET Core, I’ve gone a different path and have updated Umbraco’s existing code to compile against both .Net Framework and .Net Core. This is a necessary transition for the Umbraco codebase, we don’t plan on rewriting Umbraco, just updating it to play nicely with both .Net Framework and .Net Core.

During my talk at CodeGarden this year I spoke about the Future of Umbraco Core. An important aspect of that talk was the fact that we need to build & release Umbraco version 8 before we can consider looking in to upgrading all of Umbraco to work with ASP.NET Core. A primary reason for this is because we need to remove all of the legacy code from Umbraco (including plenty of old Webforms views) and updated plenty of other things such as old libraries that are simply not going to work with ASP.NET Core.

I have been doing all of the work on updating Umbraco to work with ASP.NET Core on a fork on GitHub. It’s been a very tedious process made even more tedious due to the constant changes of ASP.NET Core over the last 2 years. I started this whole process by modifying the VS sln file to exclude all of the projects and only including the Umbraco.Core project, then starting with the most fundamental classes to include. I’d include one class at a time using the project.json file, compile, make changes if required until it built, include the next class, rinse/repeat. I did this up until the point where I could compile the Umbraco.Core project to include Umbraco’s ApplicationContext object and CoreBootManager. This basically meant I had everything I needed in order to bootstrap Umbraco and perform the business logic operations for Umbraco on ASP.NET Core :)

I did start looking at updating the Umbraco.Web project but this is going to be a much more involved process due to the way that MVC and WebApi have changed with regards to ASP.NET Core. It is worth noting that I do have the routing working, even things like hijacked routes, SurfaceController’s, etc… !

But before I continued down that road I wanted to see if I could get the core part of Umbraco running cross platform, so I tinkered around with making an Umbraco .NET Core CLI

… And it just works :)

On a side note, the Git branch that this live in is a fork of Umbraco’s current source code and the branch that it exists in is a branch from v8 so it is fully merge-able. This means that as we continue developing on v7 and v8 all of these fixes/changes will merge up into this branch.

Umbraco Interactive CLI

I didn’t want to reinvent the wheel with a CLI argument parser so a quick Googling on what was available on .Net Core pointed me to a great framework that Microsoft had already built. What I wanted to make was an interactive CLI so I could either execute a single command line statement to perform an operation, or I could start the CLI and be prompted to work with Umbraco data until I exited. This Microsoft framework didn’t quite support that OOTB but it wasn’t difficult to make it work the way I wanted without modifying it’s source. From there I wrote the code to boot Umbraco and started implementing some commands. Here’s the commands and sub-commands so far (each command has a help option: –h):

db – Used to configure the database
- install – used to install a new Umbraco db, schema and default data
- connect – used to connect to an existing Umbraco db
schema – Used for manipulating schema elements
- doctype – Used for Document type operations
  - create, del, list
  - groups – Used for property group operations (create + list)
  - props – Used for property type operations (create + list)
- medtype– Used for Media type operations
  - create, del , list
  - groups – Used for property group operations (create + list)
  - props – Used for property type operations (create + list)

See it in action

Cross Platform

I was very interested to see if this would work on Linux so I got Ubuntu up and running and put MySql on there and created a new db to use. Then I updated the solution to build a standalone .NET Core app, published that using

dotnet publish -c release -r ubuntu.14.04-x64

and unzipped that on my Ubuntu installation. Then I tried it by running

./Umbraco.Test.Console

and …. It worked !! There’s something quite magical about the whole thing, it really was very easy to get this to run on a non-windows environment. Very impressive :)

Try it out!

You should be able to get this working pretty easily – hopefully on any OS – here’s the steps:

I’ve not tested any of this with OSX, so hopefully it will ‘just work’ there too

You’ll need either an MS SQL or MySql empty database setup on your OS environment, then note the connection string since you’ll need to enter it in the CLI.

Clone the repo (It’s big, it’s the actual current Umbraco source):

git clone -b dev-v9 https://github.com/Shazwazza/Umbraco-CMS.git

Go to the project folder

cd Umbraco-CMS
cd src
cd Umbraco.Test.Console

Restore packages

dotnet restore

Build the project, depending on your OS

dotnet build -r win10-x64
dotnet build -r osx.10.10-x64
dotnet build -r ubuntu.16.04-x64

Publish it, depending on your OS

dotnet publish -c release -r win10-x64
dotnet publish -c release -r osx.10.10-x64
dotnet publish -c release -r ubuntu.16.04-x64

Run it, depending on your OS

bin\release\netcoreapp1.0\win10-x64\Umbraco.Test.Console.exe
./bin/release/netcoreapp1.0/osx.10.10-x64/Umbraco.Test.Console
./bin/release/netcoreapp1.0/ubuntu.16.04-x64/Umbraco.Test.Console

NOTE: On Linux you’ll probably have to mark it to be executable first by doing this:

chmod +x ./bin/release/netcoreapp1.0/ubuntu.16.04-x64/Umbraco.Test.Console

Next steps

I’m very excited about what has been achieved so far but there’s certainly a long way to go. As I mentioned above getting v8 completed is a requirement to getting a version of Umbraco fully working with ASP.NET Core. During that development time I do plan on continuing to tinker around with getting more stuff to work. I’d like to see some progress made with the web project, the first steps will require getting the website boot process working (in progress) and I think a good first milestone will be getting the installer all working. From there, there’s updating the controllers and authentication/authorization mechanisms for the back office and then looking into actually getting content rendered on the front-end ( this part is actually the easiest and mostly done already ).

Installing .NET Core 1.01 on Ubuntu 16.10

Sun, 23 Oct 2016 23:03:00 Z

TL;DR You’ll need to manually install libicu55

Warning: Linux noob content below

I’ve been testing out some .NET Core cross platform stuff and originally had been using Ubuntu 14.04 with .NET Core 1.0.0 and that all worked fine along with the installation instructions from https://www.microsoft.com/net/core#ubuntu , however some of the latest tests I’ve been doing needed a MySQL version later than 5.5. It would seem that when I installed MySQL on Ubuntu 14.04 by executing apt-get mysql-server that I got 5.5 which was not compatible with what I needed. So attempting to upgrade gave me other issues for which I would require a later version of Ubuntu. Long story short, I’m a linux noob and I couldn’t get anything to upgrade, ended up executing all sorts of commands I didn’t understand and probably shouldn’t have and ultimately killed my Linux install.

So a clean install of Ubuntu 16.04 it was … there’s a catch though, you can choose between LTS (Long Term Support) or not. I chose not to since It’s a VM and I don’t mind newer updates, etc… Turns out that was a bad idea with .NET Core installs! It would seem that once the non LTS is installed you end up with 16.10 which has installed some newer versions of required libraries, namely something called libicu which is now on 57 instead of a required 55.

Trying to run the normal installation procedure from the web instructions mentioned above for 16.04 ended up telling me this:

sudo apt-get install dotnet-dev-1.0.0-preview2-003131

…

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
dotnet-dev-1.0.0-preview2-003131 : Depends: dotnet-sharedframework-microsoft.netcore.app-1.0.1 but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

So what the heck does that mean?! So after some Googling, I tried to just install the dependency:

sudo apt-get install dotnet-sharedframework-microsoft.netcore.app-1.0.1

…

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
dotnet-sharedframework-microsoft.netcore.app-1.0.1 : Depends: libicu55 (>=55.1.1~) but it is not installable
E: Unable to correct problems, you have held broken packages.

Ok, not much further but I gather that I need libicu55 installed, so let’s try:

sudo apt-get install libicu55

…

Package libicu55 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package libicu55 has no installation candidate

Manual libicu55 installation

I suppose normal linux users would probably just know that you need to download and install libicu55 manually. Well it took a little bit of research for me to figure that out, but here’s what to do:

head over to http://packages.ubuntu.com/en/xenial/amd64/libicu55/download
click one of the mirror links to download the file
in Terminal, head to the folder you downloaded it (i.e. probably ~/Downloads)
install it using this command: sudo dpkg –i libicu55_55.1-7_amd64.deb (or whatever file name you saved it as)

That should install just fine, then you can run sudo apt-get install dotnet-dev-1.0.0-preview2-003131 and everything will be fine again :)

ASMX SOAP Webservices with abstract models without using XmlInclude

Fri, 22 Jul 2016 10:12:00 Z

I’m hoping this post might be useful to some folks out there that might be stuck using old ASMX/SOAP webservices in ASP.Net. If you’ve tried to return an abstract or superclass from an ASMX webservice without using XmlInclude or SoapInclude, you’ll get an error like:

System.InvalidOperationException: There was an error generating the XML document. ---> System.InvalidOperationException: The type MyAwesomeClass was not expected. Use the XmlInclude or SoapInclude attribute to specify types that are not known statically.
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriter1.Write6_Item(String n, String ns, Item o, Boolean isNullable, Boolean needType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriter1.Write10_Item(Object o)
   at Microsoft.Xml.Serialization.GeneratedAssembly.ItemSerializer.Serialize(Object objectToSerialize, XmlSerializationWriter writer)
   at System.Xml.Serialization.XmlSerializer.Serialize(XmlWriter xmlWriter, Object o, XmlSerializerNamespaces namespaces, String encodingStyle, String id)
   --- End of inner exception stack trace ---
   at System.Xml.Serialization.XmlSerializer.Serialize(XmlWriter xmlWriter, Object o, XmlSerializerNamespaces namespaces, String encodingStyle, String id)
   at System.Xml.Serialization.XmlSerializer.Serialize(TextWriter textWriter, Object o, XmlSerializerNamespaces namespaces)
   at System.Web.Services.Protocols.XmlReturnWriter.Write(HttpResponse response, Stream outputStream, Object returnValue)
   at System.Web.Services.Protocols.HttpServerProtocol.WriteReturns(Object[] returnValues, Stream outputStream)
   at System.Web.Services.Protocols.WebServiceHandler.WriteReturns(Object[] returnValues)
   at System.Web.Services.Protocols.WebServiceHandler.Invoke()

The normal way to work around this is to attribute your ASMX class with [XmlInclude(typeof(MyAwesomeClass)] and repeat this for every subclass that you might be returning. This essentially tells the SOAP handler what types it should expect to serialize so it can ‘warm’ up a list of XmlSerializers.

The problem with this is that you need to know about all of these types up-front, but what if you have a plugin system where other developers can define their own types? There would be no way of knowing up-front what types to register so this approach will not work.

IXmlSerializer to the rescue

To work around this problem you can define a wrapper class for your abstract/superclass. Working with IXmlSerializer is pretty annoying and I highly recommend this great article if you are going to use it since one mistake can cause all sorts of problems

The following class should work for any object. Also note the usage of the static dictionary to store references to created XmlSerializer instances since these are expensive to create per type.

public class SerializedObjectWrapper : IXmlSerializable
{
    /// <summary>
    /// The underlying Object reference that is being returned
    /// </summary>
    public object Object { get; set; }

    /// <summary>
    /// This is used because creating XmlSerializers are expensive
    /// </summary>
    private static readonly ConcurrentDictionary<Type, XmlSerializer> TypeSerializers 
        = new ConcurrentDictionary<Type, XmlSerializer>();

    public XmlSchema GetSchema()
    {
        return null;
    }

    public void ReadXml(XmlReader reader)
    {
        reader.MoveToContent();

        //Get the Item type attribute
        var itemType = reader.GetAttribute("ItemType");
        if (itemType == null) throw new InvalidOperationException("ItemType attribute cannot be null");
            
        //Ensure the type is found in the app domain
        var itemTypeType = Type.GetType(itemType);
        if (itemTypeType == null) throw new InvalidOperationException("Could not find the type " + itemType);

        var isEmptyElement = reader.IsEmptyElement;
                    
        reader.ReadStartElement();

        if (isEmptyElement == false)
        {
            var serializer = TypeSerializers.GetOrAdd(itemTypeType, t => new XmlSerializer(t));
            Object = serializer.Deserialize(reader);
            reader.ReadEndElement();
        }
    }

    public void WriteXml(XmlWriter writer)
    {
        var itemType = Object.GetType();
        var serializer = TypeSerializers.GetOrAdd(itemType, t => new XmlSerializer(t));
            
        //writes the object type so we can use that to deserialize later
        writer.WriteAttributeString("ItemType", 
            itemType.AssemblyQualifiedName ?? Object.GetType().ToString());

        serializer.Serialize(writer, Object);
    }
}

Usage

Here’s an example of the usage of the SerializedObjectWrapper class along with the example that would cause the above mentioned exception so you can see the difference:

public abstract class MyAbstractClass
{
}

public class MyAwesomeClass : MyAbstractClass
{
}

//WONT WORK
[WebMethod]
public MyAbstractClass GetStuff()
{
    return new MyAwesomeClass();
}

//WILL WORK
[WebMethod]
public SerializedObjectWrapper GetStuff()
{
    return new SerializedObjectWrapper
    {
        Object = new MyAwesomeClass()
    };
}

I know most people aren’t using AMSX web services anymore but in case your stuck on an old project or have inherited one, this might be of use :)

ASP.NET Core application shutdown events

Fri, 03 Jun 2016 09:39:00 Z

While porting an existing library to ASP.NET Core I had to find the equivalent functionality of IRegisteredObject which I use for graceful shutdowns of running tasks in background threads. The newer & nicer approach to this in ASP.NET Core is Microsoft.AspNetCore.Hosting.IApplicationLifetime:

  /// <summary>
  /// Allows consumers to perform cleanup during a graceful shutdown.
  /// </summary>
  public interface IApplicationLifetime
  {
    /// <summary>
    /// Triggered when the application host has fully started and is about to wait
    /// for a graceful shutdown.
    /// </summary>
    CancellationToken ApplicationStarted { get; }

    /// <summary>
    /// Triggered when the application host is performing a graceful shutdown.
    /// Requests may still be in flight. Shutdown will block until this event completes.
    /// </summary>
    CancellationToken ApplicationStopping { get; }

    /// <summary>
    /// Triggered when the application host is performing a graceful shutdown.
    /// All requests should be complete at this point. Shutdown will block
    /// until this event completes.
    /// </summary>
    CancellationToken ApplicationStopped { get; }

    /// <summary>Requests termination the current application.</summary>
    void StopApplication();
  }

Unlike the old IRegisteredObject this interface is pretty clear on it’s functionality.

Registering a method to be called for any of the three operations is simple:

//register the application shutdown handler
 applicationLifetime.ApplicationStopping.Register(DisposeResources);

protected void DisposeResources()
{
    //Cleanup stuff when the app is shutting down
}

Obtaining an instance of IApplicationLifetime can be done during Startup.cs in the Configure method

public void Configure(IApplicationBuilder app, IApplicationLifetime applicationLifetime)
{
    // start your app
}

Happy coding!

Custom Assembly loading with Asp.Net Core

Tue, 15 Mar 2016 14:58:00 Z

Building a plugin system in Asp.Net Core is a dream compared to previous Asp.Net versions!

In previous versions it was not really feasible to load Assemblies located outside of the /bin folder for a web application. I battled with this concept quite a long time ago and although it’s sort of possible, the notion of having a plugin system that supported loading DLLs from outside of the /bin folder was riddled with hacks/problems and not really supported OOTB.

A large part of the issues has to do with something called an ‘Assembly Load Context’. In traditional .Net there are 3 of these context types: “Load”, “LoadFrom” and “Neither”, here’s a very old but very relevant post about these contexts from Suzanne Cook. In traditional Asp.Net, the “Load” context is used as the default context and it is managed by something called Fusion (.Net’s normal Assembly Loader/Binder). The problem with this context is that it is difficult to load an assembly into it that isn’t located in Fusion’s probing paths (i.e. /bin folder). If you load in an Assembly with a different Assembly Load Context and then try to mix it’s Types with the Types from the default context … you’ll quickly see that it’s not going to work.

The “Neither” context

Here is the Neither context definition as defined by Suzanne Cook:

If the user generated or found the assembly instead of Fusion, it's in neither context. This applies to assemblies loaded by Assembly.Load(byte[]) and Reflection Emit assemblies (that haven't been loaded from disk). Assembly.LoadFile() assemblies are also generally loaded into this context, even though a path is given (because it doesn't go through Fusion).

In Asp.Net Core (targeting CoreCLR), the default Assembly Load Context is the “Neither” context. This is a flexible context because it doesn’t use Fusion and it allows for loading assemblies any way that you want - including loading an assembly from a byte array, from a path or by a name. Since all of Asp.Net Core uses this context it means that all of the types loaded in with this context can talk to each other without having the previous Asp.Net problems.

I would assume that Asp.Net Core targeting Desktop CLR would still operate the same as before and still have the 3 types of Assembly Load Context’s … Maybe someone over at Microsoft can elaborate on that one? (David Fowler… surely you know? :)

Finding referenced plugin assemblies

In many cases if you create a product that supports plugin types, developers will create plugins for your product and ship them via Nuget. This is a pretty standard approach since it allows developers that are using your product to install plugins from the Nuget command line or from within Visual Studio. In this case plugin types will be found in referenced assemblies to your application and will be automatically loaded. Asp.Net Core has an interface called Microsoft.Extensions.PlatformAbstractions.ILibraryManager that can be used to resolve your application’s currently referenced ‘Libraries’ (i.e Nuget packages) and then each ‘Library’ returned exposes the Assemblies that it includes. Asp.Net MVC 6 has an even more helpful interface called Microsoft.AspNet.Mvc.Infrastructure.IAssemblyProvider which returns a list of referenced assemblies that are filtered based on if they are assemblies that reference a subset of MVC assemblies. The default implementation of IAssemblyProvider (DefaultAssemblyProvider) is extensible and we can use it to override it’s property ReferenceAssemblies in order to supply our own product assembly names instead of the MVC ones. This is perfect since this allows us to get a list of candidate assemblies that might contain plugins for your product:

public class ReferencePluginAssemblyProvider : DefaultAssemblyProvider
{
    //NOTE: The DefaultAssemblyProvider uses ILibraryManager to do the library/assembly querying
    public ReferencePluginAssemblyProvider(ILibraryManager libraryManager) : base(libraryManager)
    {
    }

    protected override HashSet<string> ReferenceAssemblies 
        => new HashSet<string>(new[] {"MyProduct.Web", "MyProduct.Core"});
}

now if you want to get a list of candidate assemblies that your application is referencing you could do:

//returns all assemblies that reference your product Assemblies
var candidateReferenceAssemblies = referencedPluginAssemblyProvider.CandidateAssemblies;

Finding and loading non-referenced plugin assemblies

This is where things get fun since this is the type of thing that wasn’t really very feasible with traditional Asp.Net web apps. Lets say you have a plugin framework where a plugin is installed via your web app, not in Visual Studio and therefore not directly referenced in your project. For this example, the plugin is a self contained collection of files and folders which could consist of: Css, JavaScript, Razor Views, and Assemblies. This plugin model is pretty nice since to install the plugin would mean just dropping the plugin folder into the right directory in your app and similarly to uninstall it you can just remove the folder. The first step is to be able to load in these plugin Assemblies from custom locations. For an example, let’s assume the web app has the following folder structure:

App Root
- App_Plugins <—This will be the directory that contains plugin folders
  - MyPlugin1
    - bin <—by convention we’ll search for Assemblies in the /bin folder inside of a plugin
    - Views
  - MyPlugin2
    - bin <—by convention we’ll search for Assemblies in the /bin folder inside of a plugin
    - css
- Views
- wwwroot

IAssemblyLoader

The first thing we need is an ‘Microsoft.Extensions.PlatformAbstractions.IAssemblyLoader’, this is the thing that will do the assembly loading into the Assembly Load Context based on an AssemblyName and a location of a DLL:

public class DirectoryLoader : IAssemblyLoader
{
    private readonly IAssemblyLoadContext _context;
    private readonly DirectoryInfo _path;

    public DirectoryLoader(DirectoryInfo path, IAssemblyLoadContext context)
    {
        _path = path;
        _context = context;
    }

    public Assembly Load(AssemblyName assemblyName)
    {
        return _context.LoadFile(Path.Combine(_path.FullName, assemblyName.Name + ".dll"));
    }

    public IntPtr LoadUnmanagedLibrary(string name)
    {
        //this isn't going to load any unmanaged libraries, just throw
        throw new NotImplementedException();
    }
}

IAssemblyProvider

Next up we’ll need a custom IAssemblyProvider but instead of using the one MVC ships with, this one will be totally custom in order to load and resolve the assemblies based on the plugin’s /bin folders. The following code should be pretty straight forward, the CandidateAssemblies property iterates over each found /bin folder inside of a plugin’s folder inside of App_Plugins. For each /bin folder found it creates a DirectoryLoader mentioned above and loads in each DLL found by it’s AssemblyName into the current Assembly Load Context.

/// <summary>
/// This will return assemblies found in App_Plugins plugin's /bin folders
/// </summary>
public class CustomDirectoryAssemblyProvider : IAssemblyProvider
{
    private readonly IFileProvider _fileProvider;
    private readonly IAssemblyLoadContextAccessor _loadContextAccessor;
    private readonly IAssemblyLoaderContainer _assemblyLoaderContainer;

    public CustomDirectoryAssemblyProvider(
            IFileProvider fileProvider, 
            IAssemblyLoadContextAccessor loadContextAccessor, 
            IAssemblyLoaderContainer assemblyLoaderContainer)
    {
        _fileProvider = fileProvider;
        _loadContextAccessor = loadContextAccessor;
        _assemblyLoaderContainer = assemblyLoaderContainer;
    }

    public IEnumerable<Assembly> CandidateAssemblies
    {
        get
        {
            var content = _fileProvider.GetDirectoryContents("/App_Plugins");
            if (!content.Exists) yield break;
            foreach (var pluginDir in content.Where(x => x.IsDirectory))
            {
                var binDir = new DirectoryInfo(Path.Combine(pluginDir.PhysicalPath, "bin"));
                if (!binDir.Exists) continue;
                foreach (var assembly in GetAssembliesInFolder(binDir))
                {
                    yield return assembly;
                }
            }
        }
    }

    /// <summary>
    /// Returns assemblies loaded from /bin folders inside of App_Plugins
    /// </summary>
    /// <param name="binPath"></param>
    /// <returns></returns>
    private IEnumerable<Assembly> GetAssembliesInFolder(DirectoryInfo binPath)
    {
        // Use the default load context
        var loadContext = _loadContextAccessor.Default;

        // Add the loader to the container so that any call to Assembly.Load 
        // will call the load context back (if it's not already loaded)
        using (_assemblyLoaderContainer.AddLoader(
            new DirectoryLoader(binPath, loadContext)))
        {
            foreach (var fileSystemInfo in binPath.GetFileSystemInfos("*.dll"))
            {
                //// In theory you should be able to use Assembly.Load() here instead
                //var assembly1 = Assembly.Load(AssemblyName.GetAssemblyName(fileSystemInfo.FullName));
                var assembly2 = loadContext.Load(AssemblyName.GetAssemblyName(fileSystemInfo.FullName));
                yield return assembly2;
            }
        }
    }
}

That’s pretty much it! If you have an instance of CustomDirectoryAssemblyProvider then you can get Assembly references to all of the assemblies found in App_Plugins:

//returns all plugin assemblies found in App_Plugins
var candidatePluginAssemblies = customDirectoryAssemblyProvider.CandidateAssemblies;

Integrating non-referenced plugins/Assemblies with MVC

What if you had custom plugin types as MVC Controllers or other MVC types? By default MVC only knows about assemblies that your project has references to based on the DefaultAssemblyLoader. If we wanted MVC to know about Controllers that exist in a plugin not referenced by your project (i.e. in App_Plugins) then it’s a case of registering a custom IAssemblyProvider in IoC which will get resolved by MVC. To make this super flexible we can create a custom IAssemblyProvider that wraps multiple other ones and allows you to pass in a custom referenceAssemblies filter if you wanted to use this to resolve your own plugin types:

public class CompositeAssemblyProvider : DefaultAssemblyProvider
{
    private readonly IAssemblyProvider[] _additionalProviders;
    private readonly string[] _referenceAssemblies;

    /// <summary>
    /// Constructor
    /// </summary>
    /// <param name="libraryManager"></param>
    /// <param name="additionalProviders">
    /// If passed in will concat the assemblies returned from these 
    /// providers with the default assemblies referenced
    /// </param>
    /// <param name="referenceAssemblies">
    /// If passed in it will filter the candidate libraries to ones
    /// that reference the assembly names passed in. 
    /// (i.e. "MyProduct.Web", "MyProduct.Core" )
    /// </param>
    public CompositeAssemblyProvider(
        ILibraryManager libraryManager,
        IAssemblyProvider[] additionalProviders = null,
        string[] referenceAssemblies = null) : base(libraryManager)
    {
        _additionalProviders = additionalProviders;
        _referenceAssemblies = referenceAssemblies;
    }

    /// <summary>
    /// Uses the default filter if a custom list of reference
    /// assemblies has not been provided
    /// </summary>
    protected override HashSet<string> ReferenceAssemblies
        => _referenceAssemblies == null
            ? base.ReferenceAssemblies
            : new HashSet<string>(_referenceAssemblies);
    
    /// <summary>
    /// Returns the base Libraries referenced along with any DLLs/Libraries
    /// returned from the custom IAssemblyProvider passed in
    /// </summary>
    /// <returns></returns>
    protected override IEnumerable<Library> GetCandidateLibraries()
    {
        var baseCandidates = base.GetCandidateLibraries();
        if (_additionalProviders == null) return baseCandidates;
        return baseCandidates               
            .Concat(
            _additionalProviders.SelectMany(provider => provider.CandidateAssemblies.Select(
                x => new Library(x.FullName, null, Path.GetDirectoryName(x.Location), null, Enumerable.Empty<string>(),
                    new[] { new AssemblyName(x.FullName) }))));
    }
}

To register this in IoC you just need to make sure it’s registered after you register MVC so that it overrides the last registered IAssemblyProvider:

//Add MVC services
services.AddMvc();
//Replace the default IAssemblyProvider with the composite one
services.AddSingleton<IAssemblyProvider, CompositeAssemblyProvider>(provider =>
{
    //create the custom plugin directory provider
    var hosting = provider.GetRequiredService<IApplicationEnvironment>();
    var fileProvider = new PhysicalFileProvider(hosting.ApplicationBasePath);
    var pluginAssemblyProvider = new CustomDirectoryAssemblyProvider(
        fileProvider,         
        PlatformServices.Default.AssemblyLoadContextAccessor,
        PlatformServices.Default.AssemblyLoaderContainer);
    //return the composite one - this wraps the default MVC one
    return new CompositeAssemblyProvider(
        provider.GetRequiredService<ILibraryManager>(),
        new IAssemblyProvider[] {pluginAssemblyProvider});
});

Your all set! Now you have the ability to load in Assemblies from any location you want, you could even load them in as byte array’s from an external data source. What’s great about all of this is that it just works and you can integrate these external Assemblies into MVC.

Some things worth noting:

Parts of the assembly loading APIs are changing a bit in Asp.Net Core RC2: https://github.com/aspnet/Announcements/issues/149
The above code doesn’t take into account what happens if you load in the same Assembly from multiple locations. In this case, the last one in wins/is active AFAIK – I haven’t tested this yet but I’m pretty sure that’s how it works.
You may have some issues if load in the same Assembly more than once from multiple locations if those Assemblies have different strong names, or major versions applied to them – I also haven’t tested this yet

AppVeyor and ASP.Net Core (Previously ASP.Net 5)

Fri, 12 Feb 2016 16:23:00 Z

Last year I created a runtime Js/Css pre-processor for ASP.Net Core (Previously ASP.Net 5) called “Smidge” and have been meaning to blog about how I integrated this with AppVeyor – to run my tests, build the project and output the Nuget files I need, so here it goes…

The build script

I use Powershell for my build scripts for my projects since it’s reasonably easy to read and the same script format has worked quite well for ASP.Net Core projects too. You can see the whole build file here. Here’s the important things to note:

With AppVeyor (and probably other build servers), you need to ensure that it actually has the dnx version you need:

# ensure the correct version
& $DNVM install 1.0.0-rc1-update1

Next you need to make sure that the current process is using the version you need to build:

# use the correct version
& $DNVM use 1.0.0-rc1-update1

Then we need to use DNU to make sure that your project has everything it needs to build:

& $DNU restore "$ProjectJsonPath"

Lastly it’s just building and packaging the project:

& $DNU build "$ProjectJsonPath"
& $DNU pack "$ProjectJsonPath" --configuration Release --out "$ReleaseFolder"

The rest of the build file is normal Powershell bits.

The test script

I’m using xunit for unit tests in this project and similarly to the build script I’m using a simple Powershell script to execute the tests on the build server, the test runner file is here. The important parts are just like the above: Ensure the correct version is installed and being used by the current process and making sure that the project has everything it needs to build and finally to build it. The last missing piece is to actually run the tests:

& $DNX -p "$TestsFolder" test

Where ‘test’ is a command defined in my project.json as part of my unit test project.

AppVeyor configuration

The good news is that there’s really not a lot to setup, it’s super easy. In your AppVeyor settings just go to the ‘Build’ section and tell it to execute the Powershell script with its build version information:

Then for the unit tests is basically the same, click on the ‘Tests’ section and tell it to execute the Powershell test script:

And that’s pretty much it! The only other part I’ve had to setup is the paths to my Artifacts (Nuget files) based on the current build number.

Now whenever I commit, AppVeyor will execute the build script and test script and we can see the output:

And it’s smart enough to know that the test runner executed is for unit tests, so all the unit test output shows up in the ‘Tests’ tab of the build

Now that that’s all setup, AppVeyor even gives you a handy Nuget feed that you can use to test your packages based on each build, this can be configured on the ‘NuGet’ settings section, for example here’s the Smidge feed: https://ci.appveyor.com/nuget/smidge

Smidge 1.0.0-RC3

It’s worth noting that I’ve also released a new version of Smidge since I finally had some time to work on it. Couple of bugs fixed in this release and also a handy new feature too! You can see the release notes here: https://github.com/Shazwazza/Smidge/releases/tag/1.0.0-rc3. I’ve also updated a lot of the documentation, the main readme file was getting quite long so I’ve moved all of the docs over to the project’s Wiki on GitHub and condensed the readme for the most important bits. Have a look here: https://github.com/Shazwazza/Smidge

Making a DynamicObject implementation be case insensitive

Fri, 05 Feb 2016 13:25:00 Z

To me this seems to be a very straight forward request but I cannot find the ‘easy’ way to do this in .Net even though there seems to be a clear indication that this is supported Out of The Box. Here’s an example of what I want to do:

//Simple class
public class Foo
{
    public string Hello {get;set;}
    public string World {get;set;}
    public int AddNumbers(int num1, int num2)
    {
        return num1 + num2;
    }
}

//Use the simple class but allow case insensitive
// access to it's properties and members
var foo = new Foo{ Hello = "hi", World = "earth" };

//make it dynamic
dynamic dFoo = foo;

//access the dynamic object's props and methods 
// in a case insensitive way:
Assert.AreEqual(foo.Hello, dFoo.hello);
Assert.AreEqual(foo.World, dFoo.world);
Assert.AreEqual(foo.AddNumbers(1,2), dFoo.addNumbers(1,2));

DynamicObject

In order to do the above, you can implement ‘DynamicObject’ and override a couple of methods:

public class Foo : DynamicObject
{
    public string Hello {get;set;}
    public string World {get;set;}
    public int AddNumbers(int num1, int num2)
    {
        return num1 + num2;
    }
    
    public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result)
    {
        //Here we would need to match lower cased binder.Name
        // to a real method that our object has
        var name = binder.Name.ToLowerInvariant();
        //TODO: Do the matching
        
        return base.TryInvokeMember(binder, args, out result);
    }
    
    public override bool TryGetMember(GetMemberBinder binder, out object result)
    {
        //Here we would need to match lower cased binder.Name
        // to a real property that our object has
        var name = binder.Name.ToLowerInvariant();
        //TODO: Do the matching
        
        return base.TryGetMember(binder, out result);
    }
}

I don’t know about you, but I’d rather not have to write all the code to do the matching and executing. This seems like a pretty straight forward thing to want to do and after looking at the innards of some of this stuff, it turns out that the ‘GetMemberBinder’ object has a property conveniently called `IgnoreCase`… surely this functionality already exists!?

Well I unfortunately cannot find anything worthwhile on Google about ‘GetMemberBinder.IgnoreCase’, I also can’t really see where or how it gets used when looking at decompiled sources.

So I’ve reverted to handling this on my own but I would really really really like to know if there’s a built in and simple way to do this in .Net since it sure seems like it should exist.

CaseInsensitiveDynamicObject

This is a base class implementation to achieve what I want. It’s not error proof and I’m sure with some objects it won’t work 100% as expected but for simple objects this will work. Using this implementation is super easy:

public class Foo : CaseInsensitiveDynamicObject
{
    public string Hello {get;set;}
    public string World {get;set;}
    public int AddNumbers(int num1, int num2)
    {
        return num1 + num2;
    }
}

By doing that, my original requirement works! Here’s the code that does this … because I was lazy and didn’t want to create more classes than I needed I’m using Tuples so the code is fully of super generic fun ;)

/// <summary>
/// This will check enable dynamic access to properties and methods in a case insensitive manner
/// </summary>
/// <typeparam name="T"></typeparam>
/// <remarks>
/// This works by using reflection on the type - the reflection lookup is lazy so it will not execute unless a dynamic method needs to be accessed
/// </remarks>
public abstract class CaseInsensitiveDynamicObject<T> : DynamicObject
    where T: class
{
    /// <summary>
    /// Used for dynamic access for case insensitive property access
    /// </summary>`
    private static readonly Lazy<IDictionary<string, Func<T, object>>> CaseInsensitivePropertyAccess = new Lazy<IDictionary<string, Func<T, object>>>(() =>
    {
        var props = typeof(T).GetProperties(BindingFlags.Instance | BindingFlags.Public)
            .DistinctBy(x => x.Name);
        return props.Select(propInfo =>
        {
            var name = propInfo.Name.ToLowerInvariant();
            Func<T, object> getVal = propInfo.GetValue;
            return new KeyValuePair<string, Func<T, object>>(name, getVal);

        }).ToDictionary(x => x.Key, x => x.Value);
    });

    /// <summary>
    /// Used for dynamic access for case insensitive property access
    /// </summary>
    private static readonly Lazy<IDictionary<string, Tuple<ParameterInfo[], Func<T, object[], object>>>> CaseInsensitiveMethodAccess
        = new Lazy<IDictionary<string, Tuple<ParameterInfo[], Func<T, object[], object>>>>(() =>
        {
            var props = typeof(T).GetMethods(BindingFlags.Instance | BindingFlags.Public)
                .Where(x => x.IsSpecialName == false && x.IsVirtual == false)
                .DistinctBy(x => x.Name);
            return props.Select(methodInfo =>
            {
                var name = methodInfo.Name.ToLowerInvariant();
                Func<T, object[], object> getVal = methodInfo.Invoke;
                var val = new Tuple<ParameterInfo[], Func<T, object[], object>>(methodInfo.GetParameters(), getVal);
                return new KeyValuePair<string, Tuple<ParameterInfo[], Func<T, object[], object>>>(name, val);

            }).ToDictionary(x => x.Key, x => x.Value);
        });

    /// <summary>
    /// Used for dynamic access for case insensitive method access
    /// </summary>
    public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result)
    {
        var name = binder.Name.ToLowerInvariant();
        if (CaseInsensitiveMethodAccess.Value.ContainsKey(name) == false)
            return base.TryInvokeMember(binder, args, out result);

        var val = CaseInsensitiveMethodAccess.Value[name];
        var parameters = val.Item1;
        var callback = val.Item2;
        var fullArgs = new List<object>(args);
        if (args.Length <= parameters.Length)
        {
            //need to fill them up if they're optional
            for (var i = args.Length; i < parameters.Length; i++)
            {
                if (parameters[i].IsOptional)
                {
                    fullArgs.Add(parameters[i].DefaultValue);
                }
            }
            if (fullArgs.Count == parameters.Length)
            {
                result = callback((T)(object)this, fullArgs.ToArray());
                return true;
            }
        }
        return base.TryInvokeMember(binder, args, out result);
    }

    public override bool TryGetMember(GetMemberBinder binder, out object result)
    {
        var name = binder.Name.ToLowerInvariant();
        if (CaseInsensitivePropertyAccess.Value.ContainsKey(name) == false)
            return base.TryGetMember(binder, out result);

        result = CaseInsensitivePropertyAccess.Value[name]((T)(object)this);
        return true;
    }
}

If anyone know how to make this work OOTB with .Net, perhaps using the `GetMemberBinder.IgnoreCase` I’d love to hear it!

Isolated WebApi attribute routing

Sun, 17 Jan 2016 12:39:00 Z

Attribute routing in ASP.Net WebApi is great and makes routing your controllers quite a bit more elegant than writing routes manually. However one problem I have with it is that it is either “on” or “off” at an application level. There is no way for a library developer to tell ASP.Net to create routes based on attributes for specific controllers or assemblies without forcing the consumer of that library to enable Attribute Routing for the whole application. In many cases this might not matter, but if you are creating a package or library of that contains it’s own API routes, you probably don’t want to interfere with a developers’ normal application setup. There should be no reason why they need to be forced to turn on attribute routing in order for your product to work, and similarly they might not want your routes automatically enabled.

The good news is that this is possible. With a bit of code, you can route your own controllers with attribute routing and be able to turn them on or off without affecting the default application attribute routes. A full implementation of this has been created for the Umbraco RestApi project so I’ll reference that source in this post for the following code examples.

Show me the code

They key to getting this to work is: IDirectRouteProvider, IDirectRouteFactory

The first thing we need is a custom IDirectRouteFactory which is actually a custom attribute. I’ve called this CustomRouteAttribute but you could call it whatever you want.

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, AllowMultiple = true)]
public class CustomRouteAttribute : Attribute, IDirectRouteFactory

This custom attribute just wraps the default WebApi RouteAttribute’s IDirectRouteFactory implementation so we don’t have to re-write any code for that.

(see full implementation here)

Next we’ll create a custom IDirectRouteProvider:

/// <summary>
/// This is used to lookup our CustomRouteAttribute instead of the normal RouteAttribute so that 
/// we can use the CustomRouteAttribute instead of the RouteAttribute on our controlles so the normal
/// MapHttpAttributeRoutes method doesn't try to route our controllers - since the point of this is
/// to be able to map our controller routes with attribute routing explicitly without interfering
/// with default application routes.
/// </summary>
public class CustomRouteAttributeDirectRouteProvider : DefaultDirectRouteProvider
{
    private readonly bool _inherit;

    public CustomRouteAttributeDirectRouteProvider(bool inherit = false)
    {
        _inherit = inherit;
    }

    protected override IReadOnlyList<IDirectRouteFactory> GetActionRouteFactories(HttpActionDescriptor actionDescriptor)
    {
        return actionDescriptor.GetCustomAttributes<CustomRouteAttribute>(inherit: _inherit);
    }
}

So far this is all pretty straight forward so far but here’s where things start to get interesting. Because we only want to create routes for specific controllers, we need to use a custom IHttpControllerTypeResolver. However, since the HttpConfiguration instance only contains a single reference to the IHttpControllerTypeResolver we need to do some hacking. The route creation process for attribute routing happens during the HttpConfiguration initialization so we need to create an isolated instance of HttpConfiguration, set it up with the services we want to use, initialize it to create our custom routes and assign those custom routes back to the main application’s HttpConfiguration.

Up first, we create a custom IHttpControllerTypeResolver to only resolve the controller we’re looking for:

public class SpecificControllerTypeResolver : IHttpControllerTypeResolver
{
    private readonly IEnumerable<Type> _controllerTypes;

    public SpecificControllerTypeResolver(IEnumerable<Type> controllerTypes)
    {
        if (controllerTypes == null) throw new ArgumentNullException("controllerTypes");
        _controllerTypes = controllerTypes;
    }

    public ICollection<Type> GetControllerTypes(IAssembliesResolver assembliesResolver)
    {
        return _controllerTypes.ToList();
    }
}

Before we look at initializing a separate instance of HttpConfiguration, lets look at the code you’d use to enable all of this in your startup code:

//config = the main application HttpConfiguration instance
config.MapControllerAttributeRoutes(
    routeNamePrefix: "MyRoutes-",
    //Map these explicit controllers in the order they appear
    controllerTypes: new[]
    {                    
        typeof (MyProductController),
        typeof (MyStoreController)
    });

The above code will enable custom attribute routing for the 2 specific controllers. These controllers will be routed with attribute routing but instead of using the standard [Route] attribute, you’d use our custom [CustomRoute] attribute. The MapControllerAttributeRoutes extension method is where all of the magic happens, here’s what it does:

Iterates over each controller type
Creates an instance of HttpConfiguration
Sets it’s IHttpControllerTypeResolver instance to SpecificControllerTypeResolver for the current controller iteration (The reason an instance of HttpConfiguration is created for each controller is to ensure that the routes are created in the order of which they are specified in the above code snippet)
Initialize the HttpConfiguration instance to create the custom attribute routes
Copy these routes back to the main application’s HttpConfguration route table

You can see the full implementation of this extension method here which includes code comments and more details on what it’s doing. The actual implementation of this method also allows for some additional parameters and callbacks so that each of these routes could be customized if required when they are created.

There is obviously a bit of code involved to achieve this and there could very well be a simpler way, however this implementation does work rather well and offers quite a lot of flexibility. I’d certainly be interested to hear if other developers have figured this out and what their solutions were.

All about ASP.Net File Change Notification (FCN)

Thu, 15 Oct 2015 10:45:00 Z

There’s a chance you might not have heard of FCN (File Change Notification) in ASP.Net and there’s an even bigger chance you didn’t realize how much it might affect you.

What is FCN in ASP.Net?

As you know ASP.Net monitors a few files/folders such as the ~/web.config and ~/App_Code and will restart your app domain when it detects changes. This is part of FCN in ASP.Net but it goes much deeper than that. There are a few other files & folders that ASP.Net monitors which will also cause an app domain restart: bin, App_WebReferences, App_GlobalResources, App_Code, Global.asax, and others. However what you might not realize is that ASP.Net actually monitors every single folder (+ files) in your web app!

Update 29/07/2016! I found a nice MS article about FCN here. I’ve been looking for more resources about this since a few new issues have been cropping up and I’m wondering if another MS update has caused another problem. Recently we’ve seen an increase in the error: “Overwhelming Change Notification in …” which has everything to do with FCN. I’ve also added a few links to the bottom of this post.

Why do I care?

If you have a web app that contains a lot of folders there is a good chance that the performance of your file system is affected in one way or another. To give you an example of how many file system watchers ASP.Net generates I created a Razor view (.cshtml) (which you should definitely test on your own site!) to display what is actually happening behind the scenes. Here’s the output for the first run page from the Visual Studio 2015 MVC template site:

The table above lists all of the “DirectoryMonitor” instances and the folder they are attached to along with all of the “FileMonitor” instances attached to that DirectoryMonitor. It’s worth nothing that these are not simply just .Net’s FileSystemWatcher, but some sort of native windows IO using a delegate called NativeFileChangeNotification. The above table doesn’t seem too scary but these monitors are not static either, they grow with every directory and file that is accessed in your web app. For example, if I navigate to these pages: /Home/About, /Home/Contact, /Account/Register, /Account/Login and go back to view this table it looks like:

Things start to get interesting when you have a web application that has a lot of folders. Some examples of this might be:

You are using a dynamic image processor such as Image Resizer or Image Processor since these will create a lot of folders based on hashes to store these dynamic images
Maybe you have a lot of members/users on your site and you have one or more folders for each one
Maybe you use nodejs or bower and you store these generated folders in your web root and references the assets directly in those folders … there can be tons of folders in bower_components and node_modules
You could be using a web framework or CMS like Umbraco or Orchard (I’m sure there are plenty of others) that contain quite a lot of folders by default

Here’s the truncated result of an Orchard site after visiting the admin area, creating a page and displaying it:

Whoa!

ASP.Net’s FCNMode

At some stage in ASP.Net’s lifetime somebody must have complained about this to Microsoft and they released a hotfix (seen here: https://support.microsoft.com/en-us/kb/911272). Then I can only assume that other people complained about this and with .Net 4.5 a new configuration setting appeared: FCNMode. This documentation explains a little about what each option does:

Default - For each subdirectory, the application creates an object that monitors the subdirectory. This is the default behavior
Disabled - File change notification is disabled.
NotSet - File change notification is not set, so the application creates an object that monitors each subdirectory. This is the default behavior.
Single - The application creates one object to monitor the main directory and uses this object to monitor each subdirectory.

Unfortunately these docs don’t tell us the whole story. It’s obvious that Default/NotSet are the same thing and are the default. Disabled is fairly clear but what it doesn’t mention is that if you set it to Disabled, this will disable all FCN for your web app, so if you change the contents of /bin or /App_Code, the site will not restart. However, Disabled still means that the web.config file is monitored so if you use this setting you can still bump your web.config to restart your site.

What exactly is “Single” though?

The folks over at DNN seem to have quite a bit of experience with FCN and this article seems to be the only place that actually explains “Single” mode correctly:

“FCNMode creates a monitor object with a buffer size of 4KB for each folder. When FCNMode is set to Single, a single monitor object is created with a buffer size of 64KB. When there are file changes, the buffer is filled with file change information. If the buffer gets overwhelmed with too many file change notifications an “Overwhelming File Change Notifications” error will occur and the app domain will recycle. The likelihood of the buffer getting overwhelmed is higher in an environment where you are using separate file server because the folder paths are much larger.”

If I change fcnMode=”Single” in my web.config for the same Orchard site above, the results are:

That’s quite a bit less!

I’m sure there are pros to using “Default” instead of “Single” but I’m not actually sure what they are.

Real world problem

The circumstance where “Default” FCN mode becomes a real problem is when you have a website that is running off of a remote file share. As noted in Shawn Walker’s DNN article mentioned above:

“The likelihood of the buffer getting overwhelmed is higher in an environment where you are using separate file server because the folder paths are much larger.”

I can’t actually see a performance difference between Default or Single when running locally with an SSD HD, but I have seen some big issues running with Default when hosting on a remote file server:

Constant app domain restarts – I’ve seen constant app domain restarts even when a site is just serving requests without any IO activity apart from just reading. And by ‘Constant’ … I mean every few seconds all day long.
File server performance suffers severely – When multiple sites are active and are being hosted on a remote file server with Default FCNMode, the writing performance of the file server is drastically degraded even though when monitoring IOPS there doesn’t appear to be any issues

To solve this issue we changed fcnMode=”Single” in the machine.config so that all sites would effectively use “Single”… and the result was instant: No more constant app restarts, file server performance was instantly back to normal. And as far as I can tell, there has been no downside to running FCNMode in Single.

So I really wonder what the up-side of Default is when it seems that running in Single mode is perfectly stable running on any hosting environment… ?

FCN doesn’t appear to be a thing with aspnetcore!

More info?

Updated 13/08/2018 – I wrote an FCN Viewer tool and wrote some more in-depth info about FCN and more of it’s quirks here: https://shazwazza.com/post/fcn-file-change-notification-viewer-for-aspnet/

Here’s a list of helpful links about FCN in ASP.Net:

Updated 29/07/2016! – More links I’ve discovered. Turns out this has been an issue for IIS + ASP.Net for quite some time with various older hotfixes, some of these new links might shed some light on the particular problem you might be having.