Shannon Deminick's blog all about web development

Writing a DocFx markdown plugin

May 21, 2020 07:28
Writing a DocFx markdown plugin

What is DocFx? It’s a static site generator mainly used for creating API documentation for your code. But it can be used for any static sites. We use this for the Lucene.Net project’s website and documentation. The end result is API docs that look and feel a little bit familiar, kind of like Microsoft’s own API documentation website. I’m not entirely sure if their docs are built with DocFx but I suspect it is but with some highly customized builds and plugins … but that’s just my own assumption.

Speaking of customizing DocFx, it is certainly possible. That said the ironic part about DocFx is that it’s own documentation is not great. One of the markdown customizations we needed for the Lucene.Net project was to add a customized note that some APIs are experimental. This tag is based on the converted Java Lucene docs and looks like: “@ lucene.experimental ”. So we wanted to detect that string and convert it to a nice looking note similar to the DocFx markdown note. Luckily there is some docs on how to do that although they’re not at all succinct but the example pretty much covers exactly what we wanted to do.

Block markdown token

This example is a block level token since it exists on it’s own line and not within other text. This is also the example DocFx provides in it’s docs. It’s relatively easy to do:

  • Register a IDfmEngineCustomizer to insert/add a “Block Rule”
  • Create a new “Block Rule” which in it’s simplistic form is a regex that parses the current text block and if it matches it returns an instance of a custom “Token” class
  • Create a custom “Token” class to store the information about what you’ve parsed
  • Create a custom “Renderer” to write out actual HTML result you want
  • Register a IDfmCustomizedRendererPartProvider to expose your “Renderer”

This all uses MEF to wire everything up. You can see the Lucene.Net implementation of a custom markdown block token here: https://github.com/apache/lucenenet/tree/master/src/docs/LuceneDocsPlugins

Inline markdown token

The above was ‘easy’ because it’s more or less following the DocFx documentation example. So the next challenge is that I wanted to be able to render an Environment Variable value within the markdown… sounds easy enough? Well the code result is actually super simple but my journey to get there was absolutely not!

There’s zero documentation about customizing the markdown engine for inline markdown and there’s almost zero documentation in the codebase about what is going on too which makes things a little interesting. I tried following the same steps above for the block markdown token and realized in the code that it’s using a MarkdownBlockContext instance so I discovered there’s a MarkdownInlineContext so thought, we’ll just swap that out … but that doesn’t work. I tried inserting my inline rule at the beginning, end, middle, etc… of the DfmEngineBuilder.InlineInlineRules within my IDfmEngineCustomizer but nothing seemed to happen. Hrm. So I cloned the DocFx repo and started diving into the tests and breaking pointing, etc…

So here’s what I discovered:

  • Depending on the token and if a token can contain other tokens, its the tokens responsibility to recurse the parsing
  • There’s a sort of ‘catch all’ rule called MarkdownTextInlineRule and that will ‘eat’ characters that don’t match the very specific markdown chars that it’s not looking for.
    • This means that if you have an inline token that is delimited by chars that this doesn’t ‘eat’, then your rule will not match. So your rule can only begin with certain chars: \<!\[*`
  • Your rule must run before this one
  • For inline rules you don’t need a “Renderer” (i.e. IDfmCustomizedRendererPartProvider)
  • Inline rule regex needs to match at the beginning of the string with the hat ^ symbol. This is a pretty critical part of how DocFx parses it’s inline content.

Now that I know that, making this extension is super simple:

  • I’ll make a Markdown token: [EnvVar:MyEnvironmentVar] which will parse to just render the value of the environment variable with that name, in this example: MyEnvironmentVariable.
  • I’ll insert my rule to the top of the list so it doesn’t come after the catch-all rule
// customize the engine
public class LuceneDfmEngineCustomizer : IDfmEngineCustomizer
    public void Customize(DfmEngineBuilder builder, IReadOnlyDictionary<string, object> parameters)
        // insert inline rule at the top
        builder.InlineRules = builder.InlineRules.Insert(0, new EnvironmentVariableInlineRule());

// define the rule
public class EnvironmentVariableInlineRule : IMarkdownRule
    // give it a name
    public string Name => "EnvVarToken";

    // define my regex to match
    private static readonly Regex _envVarRegex = new Regex(@"^\[EnvVar:(\w+?)\]", RegexOptions.Compiled);

    // process the match
    public IMarkdownToken TryMatch(IMarkdownParser parser, IMarkdownParsingContext context)
        var match = _envVarRegex.Match(context.CurrentMarkdown);
        if (match.Length == 0) return null;

        var envVar = match.Groups[1].Value;
        var text = Environment.GetEnvironmentVariable(envVar);
        if (text == null) return null;

        // 'eat' the characters of the current markdown token so they aren't re-processed
        var sourceInfo = context.Consume(match.Length);

        // return a docfx token that just returns the text passed to it
        return new MarkdownTextToken(this, parser.Context, text, sourceInfo);

In the end, that’s actually pretty simple! But don’t go trying to create a fancy token that doesn’t start with those magic characters since it’s not going to work.

Finding success in OSS

September 26, 2019 09:30
Finding success in OSS

I’ve been working in open source for quite some time and wanted to share some of my experiences in regards to managing OSS projects and communities. I’ve been employed by Umbraco, an open source .Net based CMS, for over 7 years (wow! where has the time gone!?) and have been working with the Umbraco project for even longer. In the past 10 years I’ve also started a number of my own OSS projects, seen many other colleagues and companies do the same, seen many get stalled, sometimes fail, sometimes succeed and more rarely, actually thrive. We’ve seen an enormous rise in the OSS take-up within the .Net world in the past few years thanks to Microsoft really embracing and encouraging open source and the communities that make it work. But OSS is not an easy or straightforward task. So how does an open source project succeed? … I unfortunately don’t have a straightforward answer for that either but I would like to share some insight based on my own experience.

The main reason I’ve mentioned Umbraco is because it is one of these rarer open source projects that has actually thrived and it’s quite exciting to be part of such a large worldwide community. So how have they made it work? Like many other Umbraco community members, I started using it many moons ago, working in agency land and wanting a .Net based open source CMS. At that time, it turned out that this was really the only one out there that seemed to do the job. For me, what made this project so exciting back then was how easy it was to help make the software better. It was easy to submit patches, it was easy to get feedback on issues and proposed features, it was easy to get in touch with other community members and what was really great was that it didn’t take long for your fixes or features that you submitted to be part of a new release. This was all back before GitHub or even Git existed so all of this communication was done in email or on custom built forums, but it still worked. Skip ahead to today and the underlying principles that still make this project exciting haven’t changed, it’s now just infinitely easier to do all of these things with better tools. The community of Umbraco is what makes this project tick, it has always been the main focus of the project and with community growth, the software and ecosystem thrive.

But it’s free! … so how am I employed??

This is the most common question I get asked about working for an open source project. When I started, I had no idea how to make an open source project sustainable along with having paid employees and I was equally intrigued. Umbraco offered a few services and products that were paid in order to be able to pay full time staff members and keep the project growing. These were things like various levels of certified training courses, add-on licensed products and paid support plans. Today the story for Umbraco is similar and includes all of these things but now also includes subscription models for Software as a Service called Umbraco Cloud which helps to continue the cycle of re-investing into growing the community and the project.

Most OSS projects are made by individuals

But … most OSS projects are tiny and are run by a single person who is the sole developer on a project who dedicate their own time to writing and maintaining it. A lot of this time will be used to help people: investigating issues, answering questions, writing documentation, fixing bugs, releasing new versions, etc… and all of this is done for Free but these are the things that make a project successful. It is actually quite difficult to create a community that re-invests their time into helping write your software rather than just using your software. If you can convince one or more developers to get on board, you’ll then need to invest more of your time in reviewing code, documentation updates, bug fixes and features created by your community. This time is critical, you want to be able to provide feedback and to integrate your communities changes as fast as possible… this is what will help keep your community re-investing their own time in growing the project and hopefully drumming up support for even more developers to chip in.

In my experience, this is where most OSS projects plateau because the project’s founder doesn’t have enough time to manage it all. This generally leaves projects in a semi-stalled state and may end up losing active developers and community members since their pull requests and submitted issues will remain pending for too long. These projects will ebb and flow, developers get busy and then may be re-inspired to find some more time. This doesn’t mean these projects are unsuccessful and it is probably the state of most OSS projects out there.

So how do you keep a project’s momentum?

I try to keep my own projects ‘active’ by answering questions, reviewing PRs and releasing new versions as often as I can… but in reality, this doesn’t happen as often as I would like. Sometimes I get re-inspired and may invest my evenings or weekends on some features but ultimately, I don’t think this is sustainable if you do want your OSS project to become community driven. One of the first OSS projects I created was called “uComponents” which was a collection of plugins for Umbraco and I did something with that project that I haven’t done with any of my other ones – I asked another community member to take over the project. This meant providing write access to the code but more importantly trusting another developer to help move the project forward. Eventually there was a few devs with write access and helping to manage the project and it ended up being very successful. I think if you can find people to help manage your project and allow yourself to trust other devs with that task, it will massively help your chances of not stalling the project.

I think the key to keeping a project’s momentum is to get really good at setting up the fundamentals for establishing a community:

  • Make it as easy as possible to contribute and discuss
  • Respond to your community in a timely fashion – doesn’t need to be a solution, but some form of acknowledgement is needed
  • Have good documentation and make sure the community can contribute to it
  • Have small achievable tasks on your tracker and use tags like “up-for-grabs” or “help-needed” so it’s easy for the community to know where they can ship in
  • Have CI/CD setup with tests, feedback and build outputs for the community to use
  • Trust other people in helping you to manage your project

How do larger OSS projects do it?

I can’t answer this question for all larger OSS projects, but I can provide some insight with what we do at Umbraco. The more popular a project becomes, the more people there will be that that need help. Ideally, you’ll want to try to convert the people that are needing help, into people that are providing help. To do that you should have all of the above points working well to make it easy for your community to start helping each other instead of just relying on you. Trusting the community is certainly at the heart of it all and this allows for the community to manage several aspects of itself.

At Umbraco, we have a community PR team that dedicates their own time to helping out with the project and assisting with communication between the community and dev team and ensuring that contributors are receiving a timely response. We have a community documentation team that dedicates their own time to generate missing documentation, assessing documentation PRs and to help encourage more contributors to get involved. Our community forum is also open source and is where community members can ask and answer questions. We use GitHub for all of our code and our public issue tracker and use tags like up-for-grabs along with some bots to help automate the interaction with contributors on GitHub. We’ve also established some policies regarding issues and PRs to try to keep their numbers to a minimum, as one example: only allowing an up-for-grabs issue to be open for a certain amount of time and if it doesn’t get picked up it just gets closed. The premise for this is that it isn’t important enough for the community at the moment since nobody wanted to pick it up, so it gets closed with the option of being reopened if it becomes important again.

In order to manage all of this interaction we have a couple of full-time staff members that spend a significant amount of time working with the community and these community teams, on our forums and on our issue trackers. Several years ago, this was not the case and it’s now clear that managing a large community is a full time role and what quickly became obvious was that the more you invest in the community the more you get back.

CodeGarden conference

Are my own projects successful?

Sort of, yes and no. Some projects get old and I’d rather they didn’t exist anymore but people still use them so they need to be maintained. Some projects are newer and I’m excited about them but then lack the time to keep their momentum going. Then I have a problem of having too many projects! The important thing to me is that I keep learning and I’m generally happy to keep investing the time I can find into all of my projects. I think some of my projects would be more successful if I actually did all of the things I mentioned above :)

The fact is, unless you have a lot of spare time or you are getting paid to work on OSS (so that you can make time), establishing and fostering an active community is difficult and requires putting in a lot of time and effort. For me it’s entirely worth it, I think starting and managing your own OSS project is hugely rewarding and it’s a wonderful platform for learning and collaborating regardless of how big or small the project becomes.

Being a part of a community

You don’t need to be the founder of a project to reap the rewards of OSS. You can learn an enormous amount by collaborating on an OSS project. Of course there’s plenty of technical knowledge to gain but there’s a lot of other skills you can gain too and in my opinion the most important of all is communication skills. It’s easy to get frustrated by software or by other community members especially if you’ve run into some annoying bugs, somebody is being a jerk or just being difficult to communicate with. But, it’s important to not let these types of things get the better of you and that in itself is a difficult skill to master!

Communication is a 2 way street and it’s important for both community members and project owners to be friendly, patient and helpful.

A great example of this is in bug reports. Some reports are unhelpful, aggressive or even angry. When submitting bug reports or asking questions, keep in mind that you are generally asking people for help for free. Please be helpful, please be friendly, please be patient. Please think about how you would like to receive a bug report or be asked a question, include as much information as you can and remember that most people can’t read minds ;) And vice-versa, if you are responding to questions where people aren’t being as helpful as they should be, you’ll typically you’ll be able to turn a conversation into a friendly, patient and helpful one by just communicating that way yourself.

English is typically the language used in OSS projects and some people might not speak English as a first language and that’s ok! In this case, those 3 points are extremely important and it’s quite rewarding to see international community members who come from all sorts of backgrounds, countries and languages collaborating on a single project.

When both community members and project owners adopt these values, great things will happen and everyone wins!

Umbraco Friendly Hoodie

… now I need to go find some time to update my OSS projects :)

This blog is powered by Articulate, an OSS project that I maintain which is running on Umbraco, an OSS project that I work for and this post was written using Open Live Writer, an OSS project that could use some love.

Configuring Azure Active Directory login with Umbraco Members

February 18, 2019 02:09
Configuring Azure Active Directory login with Umbraco Members

This post is about configuring Azure Active Directory with Umbraco Members (not Users), meaning this is for your front-end website, not the Umbraco back office. I did write up a post about Azure AD with back office users though, so if that is what you are looking for then this is the link.

Install the Nuget packages

First thing to do is get the UmbracoIdentity package installed.

PM > Install-Package UmbracoIdentity

(This will also install the UmbracoIdentity.Core base package)

This package installs some code snippets and updates your web.config to enable ASP.Net Identity for Umbraco members. Umbraco ships with the old and deprecated ASP.Net Membership Providers for members and not ASP.Net Identity so this package extends the Umbraco CMS and the Umbraco members implementation to use ASP.Net Identity APIs to interact with the built in members data store. Installing this package will remove the (deprecated) FormsAuthentication module from your web.config and it will no longer be used to authenticate members, so the typical members snippets built into Umbraco macros will not work. Instead use the supplied snippets shipped with this package.

To read more about this package see the GitHub repo here.

Next, the OpenIdConnect package needs to be installed

PM > Install-Package Microsoft.Owin.Security.OpenIdConnect

Configure Azure Active Directory

Head over to the Azure Active Directory section on the Azure portal, choose App Registrations (I’m using the Preview functionality for this) and create a New registration


Next fill out the app details


You may also need to enter other redirect URLs depending on how many different environments you have. All of these URLs can be added in the Authentication section of your app in the Azure portal.

For AAD configuration for front-end members, the redirect Urls are just your website’s root URL and it is advised to keep the trailing slash.

Next you will need to enable Id Tokens


Configure OpenIdConnect

The UmbracoIdentity package will have installed an OWIN startup class in ~/App_Start/UmbracoIdentityStartup.cs (or it could be in App_Code if you are using a website project). This is how ASP.Net Identity is configured for front-end members and where you can specify the configuration for different OAuth providers. There’s a few things you’ll need to do:

Allow external sign in cookies

If you scroll down to the ConfigureMiddleware method, there will be a link of code to uncomment: app.UseExternalSignInCookie(DefaultAuthenticationTypes.ExternalCookie); this is required for any OAuth providers to work.

Enable OpenIdConnect OAuth for AAD

You’ll need to add this extension method class to your code which is some boiler plate code to configure OpenIdConnect with AAD:

public static class UmbracoADAuthExtensions
    public static void ConfigureAzureActiveDirectoryAuth(this IAppBuilder app,
        string tenant, string clientId, string postLoginRedirectUri, Guid issuerId,
        string caption = "Active Directory")
        var authority = string.Format(

        var adOptions = new OpenIdConnectAuthenticationOptions
            ClientId = clientId,
            Authority = authority,
            RedirectUri = postLoginRedirectUri

        adOptions.Caption = caption;
        //Need to set the auth type as the issuer path
        adOptions.AuthenticationType = string.Format(

Next you’ll need to call this code, add the following line underneath the app.UseExternalSignInCookie method call:

    //The value of this will need to change depending on your current environment
    postLoginRedirectUri: ConfigurationManager.AppSettings["azureAd:redirectUrl"],
    //This is the same as the TenantId
    issuerId: new Guid(ConfigurationManager.AppSettings["azureAd:tenantId"]));

Then you’ll need to add a few appSettings to your web.config (based on your AAD info):

<add key="azureAd:tenantId" value="YOUR-TENANT-ID-GUID" />
<add key="azureAd:clientId" value="YOUR-CLIENT-ID-GUID" />
<add key="azureAd:redirectUrl" value="http://my-test-website/" />

Configure your Umbraco data

The UmbracoIdentity repository has the installation documentation and you must follow these 2 instructions, and they are very simple:

  1. You need to update your member type with the securityStamp property
  2. Create the Account document type

Once that is done you will have an Member account management page which is based off of the installed views and snippets of the UmbracoIdentity package. This account page will look like this:


As you can see the button text under the “Use another service to log in” is the login provider name which is a bit ugly. The good news is that this is easy to change since this is just a partial view that was installed with the UmbracoIdentity package. You can edit the file: ~/Views/UmbracoIdentityAccount/ExternalLoginsList.cshtml, the code to render that button text is using @p.Authentication provider but we can easily change this to @p.Caption which is actually the same caption text used in the extension method we created. So the whole button code can look like this instead:

<button type="submit" class="btn btn-default"
        title="Log in using your @p.Caption account">

This is a bit nicer, now the button looks like:


The purpose of all of these snippets and views installed with UmbracoIdentity is for you to customize how the whole flow looks and works so you’ll most likely end up customizing a number of views found in this folder to suit your needs.

That’s it!

Once that’s all configured, if you click on the Active Directory button to log in as a member, you’ll be brought to the standard AAD permission screen:


Once you accept you’ll be redirect back to your Account page:


Any customizations is then up to you. You can modify how the flow works, whether or not you accepting auto-linking accounts (like in the above example), or if you require a member to exist locally before being able to link an OAuth account, etc… All of the views and controller code in UmbracoIdentity is there for you to manipulate. The main files are:

  • ~/Views/Account.cshtml
  • ~/Views/UmbracoIdentityAccount/*
  • ~/Controllers/UmbracoIdentityAccountController.cs
  • ~/App_Start/UmbracoIdentityStartup.cs

Happy coding!

Deploying to Azure from VSTS using publish profiles and msdeploy

October 26, 2017 06:39
Deploying to Azure from VSTS using publish profiles and msdeploy

In almost all of the examples online about how to deploy various services to Azure, they always list the super easy way to do it and that is to authenticate your current account to your Azure subscription which then grants your VSTS build to do all sorts of things… The problem is that not everyone has the security clearance to use the super easy tools in VSTS.

When you attempt to use these nice tools in VSTS you might get an error like this: “Failed to set Azure permission ‘RoleAssignmentId: some-guid-goes-here’ for the service principal … does not have authorizationto perform action ‘Microsoft.Authorization/roleAssignments/write’ over scope” This is because these nice VSTS tools actually creates a custom user behind the scenes in your azure subscription to use but your account doesn’t have access to authorize that.

Luckily there’s a work around

MS Deploy … sigh

Maybe there are other work arounds but this works, however it’s not the most elegant. I thought I’d post my findings here because it was a bit of a pain in the ass to get this all correct.

So here’s the steps:

1. Download the publish profile

You need to get the publish profile from your app service that you want to deploy to. This can be a website, a staging slot, an Azure function, (probably a bunch of others)


The file downloaded is an XML file containing a bunch of info you’ll need

2. Create a release definition and environment for your deployment

This assumes that you are pretty familiar with VSTS

You’ll want to create an empty environment in your release definition. Normally this is where you could choose the built in fancy VSTS deployment templates like “Azure App Service Deployment” … but as above, this doesn’t work if you don’t have security clearance. Instead, choose ‘Empty’


Then in your environment tasks, add Batch Script


3. Setup your batch script

There’s 2 ways to go about this and both depend on a msdeploy build output. This build output is generated by your build in VSTS if you are using a standard VSTS Visual Studio solution build. This will create msdeploy packages for you and will have put them in your artifacts folder. Along with msdeploy packages this will also generate a cmd batch file that executes msdeploy and a readme file to tell you how to execute it which contains some important info that you should read.

So here’s 2 options: Execute the cmd file, or execute msdeploy.exe directly

Execute the cmd file

There’s a bit of documentation about this online but most of it is based on using the SetParameters.xml file to adjust settings… but i just don’t want to use that.

Here’s the Path and Arguments that you need to run:

/y "/m:https://${publishUrl}/MSDeploy.axd?site=${msdeploySite}" /u:$(userName) /p:$(userPWD) /a:Basic -enableRule:DoNotDeleteRule "-setParam:name='IIS Web Application Name',value='${msdeploySite}'"

The parameters should be added to your VSTS Variables: ${msdeploySite}, $(userName), $(userPWD) and these variables correspond exactly to what is in your publish profile XML file that you downloaded. These parameters need to be pretty much exact, any misplaced quote or if you don’t include https, etc… will cause this to fail.

Important: the use of -enableRule:DoNotDeleteRule is totally optional, if you want to reset your site to exactly what is in the msdeploy package you do not want this. If however, you have user generated images, content or custom config files that exist on your site and you don’t want them deleted when you deploy, then you need to set this.

I’m unsure if this will work for Azure Functions deployment (it might!) … but I used the next option to do that:

Execute msdeploy.exe directly

If you execute the CMD file, you’ll see in the VSTS logs the exact msdeploy signature used which is:

"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -source:package='d:\a\r1\a\YOUR_PROJECT_NAME\drop\YOUR_MSDEPLOY_PACKAGE_FILE.zip' -dest:auto,computerName="https://YOUR_PUBLISH_URL/MSDeploy.axd?site=YOUR_PROFILE_NAME",userName=********,password=********,authtype="Basic",includeAcls="False" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -setParamFile:"d:\a\r1\a\YOUR_PROJECT_NAME\drop\YOUR_MSDEPLOY_PACKAGE_FILE.SetParameters.xml" -enableRule:DoNotDeleteRule -setParam:name='IIS Web Application Name',value='YOUR_PROFILE_NAME'

So if you wanted, you could take this and execute that directly instead of the CMD file. I use this method to deploy Azure Functions but the script is a little simpler since that deployment doesn’t require all of these parameters. For that I use this for the Path and Arguments:

C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe
-verb:sync -source:package='$(System.DefaultWorkingDirectory)/YOUR_BUILD_NAME/drop/YOUR_MSDEPLOY_PACKAGE.zip' -dest:auto,computerName="https://$(publishUrl)/msdeploy.axd?site=$(msdeploySite)",UserName='$(userName)',Password='$(userPWD)',AuthType='Basic' -setParam:name='IIS Web Application Name',value='$(msdeploySite)'

Hopefully this comes in handy for someone Winking smile

How to lazily set the multi-term rewrite method on queries in Lucene

September 15, 2017 02:21
How to lazily set the multi-term rewrite method on queries in Lucene

For wildcard queries in Lucene that you would like to have the results ordered by Score, there’s a trick that you need to do otherwise all of your scores will come back the same. The reason for this is because the default behavior of wildcard queries uses CONSTANT_SCORE_AUTO_REWRITE_DEFAULT which as the name describes is going to give a constant score. The code comments describe why this is the default:

a) Runs faster

b) Does not have the scarcity of terms unduly influence score

c) Avoids any "TooManyBooleanClauses" exceptions

Without fully understanding Lucene that doesn’t really mean a whole lot but the Lucene docs give a little more info

NOTE: if setRewriteMethod(org.apache.lucene.search.MultiTermQuery.RewriteMethod) is either CONSTANT_SCORE_BOOLEAN_QUERY_REWRITE or SCORING_BOOLEAN_QUERY_REWRITE, you may encounter a BooleanQuery.TooManyClauses exception during searching, which happens when the number of terms to be searched exceeds BooleanQuery.getMaxClauseCount(). Setting setRewriteMethod(org.apache.lucene.search.MultiTermQuery.RewriteMethod) to CONSTANT_SCORE_FILTER_REWRITE prevents this.

The recommended rewrite method is CONSTANT_SCORE_AUTO_REWRITE_DEFAULT: it doesn't spend CPU computing unhelpful scores, and it tries to pick the most performant rewrite method given the query. If you need scoring (like FuzzyQuery, use MultiTermQuery.TopTermsScoringBooleanQueryRewrite which uses a priority queue to only collect competitive terms and not hit this limitation. Note that org.apache.lucene.queryparser.classic.QueryParser produces MultiTermQueries using CONSTANT_SCORE_AUTO_REWRITE_DEFAULT by default.

So the gist is, unless you are ordering by Score this shouldn’t be changed because it will consume more CPU and depending on how many terms you are querying against you might get an exception (though I think that is rare).

So how do you change the default?

That’s super easy, it’s just this line of code:


But there’s a catch! You must set this flag before you parse any queries with the query parser otherwise it won’t work. All this really does is instruct the query parser to apply this scoring method to any MultiTermQuery or FuzzyQuery implementations it creates. So what if you don’t know if this change should be made before you use the query parser? One scenario might be: At the time of using the query parser, you are unsure if the user constructing the query is going to be sorting by score. In this case you want to change the scoring mechanism just before executing the search but after creating your query.

Setting the value lazily

The good news is that you can set this value lazily just before you execute the search even after you’ve used the query parser to create parts of your query. There’s only 1 class type that we need to check for that has this API: MultiTermQuery however not all implementations of it support rewriting so we have to check for that. So given an instance of a Query we can recursively update every query contained within it and manually apply the rewrite method like:

protected void SetScoringBooleanQueryRewriteMethod(Query query)
	if (query is MultiTermQuery mtq)
		catch (NotSupportedException)
			//swallow this, some implementations of MultiTermQuery don't support this like FuzzyQuery
	if (query is BooleanQuery bq)
		foreach (BooleanClause clause in bq.Clauses())
			var q = clause.GetQuery();

So you can call this method just before you execute your search and it will still work without having to eagerly use QueryParser.SetMultiTermRewriteMethod(MultiTermQuery.SCORING_BOOLEAN_QUERY_REWRITE); before you use the query parser methods.

Happy searching!

Using AspNet5 OptionsModel

January 12, 2015 04:21

If you’ve used AspNet5 then you’ve probably been using some MVC, in which case you’ve probably seen something like this in your Startup class:

// Add MVC services to the services container
    .Configure<MvcOptions>(options =>
        //Configure some MVC options like customizing the 
        // view engines, etc...
        options.ViewEngines.Insert(0, typeof(TestViewEngine));

It turns out this syntax for specifying ‘options’ for a given service is a generic pattern that you can use in your own code. In fact the OptionsModel framework is it’s own code repository: https://github.com/aspnet/Options

I’ve implemented custom options in my AspNet5 project called Smidge (a runtime JavaScript/CSS pre-processing engine) and wanted to share the details since as far as I’ve seen there isn’t any documentation about this.

What are options?

Probably the simplest way to describe the options framework is that: Options allow you to configure your application via code during startup.

Options are just a POCO class that can contain configuration options to customize the behavior of your library. These option classes can be injected into any of your services with IoC using an interface called Microsoft.Framework.OptionsModel.IOptions. There’s a caveat to this POCO class however: It must contain an parameter-less/empty constructor which means you cannot have services injected into the your options class via it’s constructor.  This options framework also allows for ‘named’ options. So for example, perhaps you have a single options class that you would like to have configured in 2 different ways, one for ‘staging’ and one for your ‘live’ website.

Creating options

Here’s a really simple example of a POCO options class:

public class CustomMessageOptions
    public CustomMessage()
        Message = "";

    public string Message { get; set; }

In order to use this options class you need to create an options configuration class. For example:

public class CustomMessageOptionsSetup : ConfigureOptions<CustomMessageOptions>
    public CustomMessageOptionsSetup() 
    : base(ConfigureMessageOptions)

    /// <summary>
    /// Set the default options
    /// </summary>
    public static void ConfigureMessageOptions(CustomMessageOptions options)
        options.Message = "Hello world";

Then you need to add this class to your IoC container of type Microsoft.Framework.OptionsModel.IConfigureOptions:

services.AddTransient<IConfigureOptions<CustomMessageOptions>, CustomMessageOptionsSetup>();

Using options

To configure your options during startup, you do so in the ConfigureServices method like:

services.Configure<CustomMessageOptions>(options =>
    options.Message = "Hello there!";

Now you can have these options injected into any of your services using the IOptions interface noted previously:

public class MyCoolService 
    public MyCoolService(IOptions<CustomMessageOptions> messageOptions)
        //IOptions exposes an 'Options' property which resolves an instance
        //of CustomMessageOptions
        ConfiguredMessage = messageOptions.Options.Message;

    public string ConfiguredMessage {get; private set;}

Named options

As an example, lets say that you want a different message configured for your ‘staging’ and ‘live’ websites. This can be done with named options, here’s an example:

    .Configure<CustomMessageOptions>(options =>
        options.Message = "Hi! This is the staging site";
    }, "staging")
    .Configure<CustomMessageOptions>(options =>
        options.Message = "Hi! This is the live site";
    }, "live");

Then in your service you can resolve the option instance by name:

public class MyCoolService 
    public MyCoolService(IOptions<CustomMessageOptions> messageOptions)
        //IRL This value would probably be set via some environment variable
        var configEnvironment = "staging";

        //IOptions exposes an 'GetNamedOptions' method which resolves an instance
        //of CustomMessageOptions based on a defined named configuration
        ConfiguredMessage = messageOptions.GetNamedOptions(configEnvironment);

    public string ConfiguredMessage {get; private set;}

Configuring options with other services

Since your options class is just a POCO object and must have a parameter-less/empty constructor, you cannot inject services into the options class. However, there is a way to use IoC services in your options classes by customizing the ConfigureOptions class created above.  In many cases this won’t be necessary but this really depends on how you are using options.  As a (bad) example, lets say we wanted to expose a custom helper service called SiteHelper on the CustomMessageOptions class that can be used by a developer to create the message. The end result syntax might look like:

services.Configure<CustomMessageOptions>(options =>
        var siteId = options.SiteHelper.GetSiteId();
        options.Message = "Hi! This is the staging site with id: " + siteId;

In order for that to work the options.SiteHelper property needs to be initialized. This is done with the CustomMessageOptionsSetup class (created above) which has been added to the IoC container, this means it can have other services injected into it. The resulting class would look like:

public class CustomMessageOptionsSetup : ConfigureOptions<CustomMessageOptions>
    //SiteHelper gets injected via IoC
    public CustomMessageOptionsSetup(SiteHelper siteHelper) 
    : base(ConfigureMessageOptions)
        SiteHelper = siteHelper;

    public SiteHelper SiteHelper { get; private set; }

    /// <summary>
    /// Set the default options
    /// </summary>
    public static void ConfigureMessageOptions(CustomMessageOptions options)
        options.Message = "Hello world";

    /// <summary>
    /// Allows for configuring the options instance before options are set
    /// </summary>
    public override void Configure(Bundles options, string name = "")
        //Assign the site helper instance
        options.SiteHelper = SiteHelper;

        base.Configure(options, name);

IRL to give you an example of why this might be useful, in my Smidge project I allow developers to create named JavaScript/CSS bundles during startup using options. In some cases a developer might want to manipulate the file processing pipeline for a given bundle and in that case they need access to a service called PreProcessPipelineFactory which needs to come from IoC. The usage might look like:

    .Configure<Bundles>(bundles =>
                //add as many processor types as you want
                typeof(DotLess), typeof(JsMin)), 

In the above, the bundles.PipelineFactory is a property on the bundles (options) class which gets initialized in my own ConfigureOptions class.


Hopefully this helps anyone looking to use custom options in their AspNet5 libraries!

Introducing ‘Smidge’ – an ASP.NET 5 runtime JS/CSS pre-processor

December 11, 2014 23:19

During the past month I decided to dive deep into learning ASP.NET 5, and what better way to learn than to start a new OSS project :)

I chose to make a new new simple and extensible Javascript/CSS runtime pre-processor for ASP.NET 5. It does file minification, combination and compression, has a nice file caching layer and it’s all done in async operations. I ported over a few ideas and code snippets from CDF (client dependency framework) but with a more modern approach. I’ve called it ‘Smidge’ = something really small.

The project is on GitHub, it’s still a work in progress but its functional and there’s even some documentation! In the next few weeks I’ll get more of the code and docs updated and hopefully have a beta release out. In the meantime, you can clone the source, browse the code, build it and of course use it if you like.

Project details

It’s currently only targeting aspnet50 and not the Core CLR… I didn’t start with Core CLR because there was some legacy code I had to port over and I wanted to get something up and working relatively quickly. It shouldn’t be too much work to convert to Core CLR and Mono, hopefully I’ll find time to do that soon. It’s referencing all of the beta-* libraries from the ASP.NET 5 nightly myget feeds since there’s some code I’m using that isn’t available in the current beta1 release (like Microsoft.AspNet.WebUtilities.UriHelper). The target KRE version is currently KRE-CLR-amd64 1.0.0-beta2-10760.


I’ve put up an Alpha 1 release on Nuget, so you can install it from there:

PM> Install-Package Smidge -Pre

There’s some installation instructions here, you’ll need to add the smidge.json file yourself for now, can’t figure out how to get VS 2015 (kpm pack) to package that up … more learning required!


There’s certainly a lot of detective work involved in learning ASP.NET 5 but with the code being open source and browse-able/searchable on GitHub, it makes finding what you need fairly easy.

Get JQuery requests to play nicely with AngularJS CSRF convention

December 6, 2013 19:06

Decided to write this quick post for anyone searching on this topic. AngularJS has it’s own convention for CSRF (Cross Site Request Forgery) protection but in some cases you’ll be calling these same server side services via JQuery so you might need to get JQuery requests to also follow Angular’s convention.

For information about Angular’s CSRF protection see the “Security Considerations” part of Angular’s $http documentation.

Luckily it’s pretty darn easy to get JQuery to follow this convention too and this will also work with 3rd party plugins that use JQuery for requests like Blueimp file uploader. The easiest way to get this done is to set the global JQuery $.ajax rules. Probably the best place to do this is in your Angular app.run statement:

app.run(function ($cookies) {

    //This sets the default jquery ajax headers to include our csrf token, we
    // need to user the beforeSend method because the token might change 
    // (different for each logged in user)
        beforeSend: function (xhr) {
            xhr.setRequestHeader("X-XSRF-TOKEN", $cookies["XSRF-TOKEN"]);

That’s it!

It’s important to note to set the header using beforeSend, if you just set $.ajax options ‘headers’ section directly that means the header cannot be dynamic – which you’ll probably want if you have users logging in/out.

Reference the current form controller in AngularJS

June 26, 2013 23:08

I previously wrote a post about Listening for validation changes in AngularJS which with my knowledge at that time required a handy hack to get a reference to the currently scoped form controller (ngForm) for a given input control. I also complained a bit that it seemed that angular didn’t really provide a way to reference the current form controller without this little hack… well, it turns out I was wrong! :)

AngularJS seems kind of like ASP.Net MVC in the early days when there wasn’t much documentation…  It definitely pays off to read through the source code to figure out how to do more complicated things. I had a bit of a ‘light bulb’ moment when I realized that ngForm was itself a directive/controller and had recently noticed that the ‘require’ parameter of setting up a directive allows you to search for controllers in the current directives ancestry (i.e. prefix the required controller with a hat: ^ )

What does the require parameter of a directive do?

Lets face it, the directive documentation for AngularJS is in desperate need of being updated so that human beings can understand it (as noted by the many comments at the bottom). So I’ll try to explain what the ‘require’ parameter actually does and how to use it.

We’ll create a simple custom validation directive which will invalidate a field if the value is “blah”

function blahValidator() {
    return {
        require: 'ngModel',
        link: function(scope, elm, attr, ctrl) {
            var validator = function(value) {
                if (ctrl.$viewValue == "blah") {
                    ctrl.$setValidity('blah', false);
                    return null;
                else {
                    ctrl.$setValidity('blah', true);
                    return value;


You’ll notice that we have a ‘require’ parameter specified for ‘ngModel’. What is happening here is that when we assign this directive to an input field, angular will ensure that the input field also has a defined ng-model attribute on it as well. Then angular will pass in the instance of the ng-model controller to the ‘ctrl’ parameter of the link function.

So, the ‘require’ parameter dictates what the ‘ctrl’ parameter of the link function equals.

You can also require multiple controllers:


NOTE: the ctrl/ctrls parameter in the above 2 examples can be called whatever you want

Special prefixes

Angular has 2 special prefixes for the ‘require’ parameter:

^ = search the current directives ancestry for the controller

? = don’t throw an exception if the required controller is not found, making it ‘optional’ not a requirement

You can also combine them so angular will search the ancestry but it can be optional too such as: ^?ngController'

In the above example, the blahValidator will only work if the directive is declared inside of an ng-controller block.

Referencing the current ng-form

Given the above examples, and knowing the ngForm itself is a controller we should be able to just make a requirement on ngForm and have it injected into the directive. BUT, it wont work the way you expect. For some reason angular references the ngForm controller by the name “form” which i discovered by browsing the source of angular.

So now its easy to get a reference to the containing ngForm controller, all you need to do is add a ‘require’ parameter to your directive that looks like:

require: '^form'
and it will be injected into your ctrl parameter of your link function.

NDepend review = pretty cool!

June 21, 2013 23:47

For a while I’ve been wanting to automate some reporting from our build process to show some fairly simple statistical information such as

  • Obsoleted code
  • Internal code planned to be made public
  • Code flagged to be obsoleted
  • Code flagged as experimental

NOTE: some of the above are based on our own internal c# attributes

Originally I was just going to write some code to show me this which would have been fairly straight forward but I knew that NDepend had a code query language and natively outputs reports and can be integrated into the build process, so I figured I’d give it a go.

So I went and installed NDepend and ran the analysis against my current project … I wasn’t quite prepared for the amount of information it gave me! :)


I quickly found out that the reports that I wanted to create were insanely simple to write. I then lost a large part of my day exploring the reports it generates OOTB and I realized I hadn’t even touched the surface of the information this tool could give me. I was mostly interested in statistical analysis but this will also happily tell you about how bad or good your code is, how much test coverage you’ve got, what backwards compatibility you broke with your latest release, what code is most used in your project and what code isn’t used at all… and the list goes on and on.

To give you an example of the first report I wanted to write, here’s all I wrote to give me what methods in my codebase were obsoleted:

from m in JustMyCode.Methods where m.HasAttribute("System.ObsoleteAttribute") select m

What’s cool about the report output is that you can output to HTML or XML, so if you want to publish it on your site you can just write some XSLT to make it look the way you want.

Project comparison

Another really great feature that I thought would be fun to automate is the code analysis against previous builds so you know straight away if you are breaking compatibility with previous releases. I think it has great potential but what I did discover is that in some cases the OOTB queries for this aren’t perfect. It will tell you what public types, methods, and fields have been removed or changed and the query is pretty straightforward but… In many cases when we’re trying to maintain backwards compatibility while creating new APIs we’ll remove a type’s methods/properties and just make the legacy type inherit from the new API class (or some other trickery similar to that). This might break compatibility if people are using reflection or specific type checking but it wont break the API usage which I’m mostly interested in. I haven’t figured it out yet but I’m sure there’s some zany NDepend query I can write to check that.


So now that I know I can extract all this info, it’s time to decide what to export. Thankfully NDepend comes fully equipped with it’s own MSBuild integration so automating the report output should be pretty straightforward. I also noticed that if you wanted to be ultra careful you can integrate it with your build server and have your builds fail if some of your queries cause errors, like the above: If you break compatibility the build will fail. Or anything you want really, like failing the build if we detect you’ve written a method that has too many lines of code.

Pretty cool indeed!

I was going to use NDepend just to output reports which it clearly does without issue. What I didn’t realize was how much detail you can get about your code with this tool. There’s some tools in there that were a bit over my head but I suppose if you are working on really huge projects and need anything from just a snapshot or insane details into what dependencies your code has on other code it’ll do that easily as well.

The UI is a bit overwhelming at first and could probably do with some work but TBH I’m not sure how else you’d display that much information nicely. Once I figured out how the whole query and reporting structures work then it was dead simple.

Happy reporting :)