Shazwazza

Writing a DocFx markdown plugin

Thu, 21 May 2020 07:28:00 Z

What is DocFx? It’s a static site generator mainly used for creating API documentation for your code. But it can be used for any static sites. We use this for the Lucene.Net project’s website and documentation. The end result is API docs that look and feel a little bit familiar, kind of like Microsoft’s own API documentation website. I’m not entirely sure if their docs are built with DocFx but I suspect it is but with some highly customized builds and plugins … but that’s just my own assumption.

Speaking of customizing DocFx, it is certainly possible. That said the ironic part about DocFx is that it’s own documentation is not great. One of the markdown customizations we needed for the Lucene.Net project was to add a customized note that some APIs are experimental. This tag is based on the converted Java Lucene docs and looks like: “@ lucene.experimental ”. So we wanted to detect that string and convert it to a nice looking note similar to the DocFx markdown note. Luckily there is some docs on how to do that although they’re not at all succinct but the example pretty much covers exactly what we wanted to do.

Block markdown token

This example is a block level token since it exists on it’s own line and not within other text. This is also the example DocFx provides in it’s docs. It’s relatively easy to do:

Register a IDfmEngineCustomizer to insert/add a “Block Rule”
Create a new “Block Rule” which in it’s simplistic form is a regex that parses the current text block and if it matches it returns an instance of a custom “Token” class
Create a custom “Token” class to store the information about what you’ve parsed
Create a custom “Renderer” to write out actual HTML result you want
Register a IDfmCustomizedRendererPartProvider to expose your “Renderer”

This all uses MEF to wire everything up. You can see the Lucene.Net implementation of a custom markdown block token here: https://github.com/apache/lucenenet/tree/master/src/docs/LuceneDocsPlugins

Inline markdown token

The above was ‘easy’ because it’s more or less following the DocFx documentation example. So the next challenge is that I wanted to be able to render an Environment Variable value within the markdown… sounds easy enough? Well the code result is actually super simple but my journey to get there was absolutely not!

There’s zero documentation about customizing the markdown engine for inline markdown and there’s almost zero documentation in the codebase about what is going on too which makes things a little interesting. I tried following the same steps above for the block markdown token and realized in the code that it’s using a MarkdownBlockContext instance so I discovered there’s a MarkdownInlineContext so thought, we’ll just swap that out … but that doesn’t work. I tried inserting my inline rule at the beginning, end, middle, etc… of the DfmEngineBuilder.InlineInlineRules within my IDfmEngineCustomizer but nothing seemed to happen. Hrm. So I cloned the DocFx repo and started diving into the tests and breaking pointing, etc…

So here’s what I discovered:

Depending on the token and if a token can contain other tokens, its the tokens responsibility to recurse the parsing
There’s a sort of ‘catch all’ rule called MarkdownTextInlineRule and that will ‘eat’ characters that don’t match the very specific markdown chars that it’s not looking for.

This means that if you have an inline token that is delimited by chars that this doesn’t ‘eat’, then your rule will not match. So your rule can only begin with certain chars: \<!\[*`

Your rule must run before this one
For inline rules you don’t need a “Renderer” (i.e. IDfmCustomizedRendererPartProvider)
Inline rule regex needs to match at the beginning of the string with the hat ^ symbol. This is a pretty critical part of how DocFx parses it’s inline content.

Now that I know that, making this extension is super simple:

I’ll make a Markdown token: [EnvVar:MyEnvironmentVar] which will parse to just render the value of the environment variable with that name, in this example: MyEnvironmentVariable.
I’ll insert my rule to the top of the list so it doesn’t come after the catch-all rule

// customize the engine
[Export(typeof(IDfmEngineCustomizer))]
public class LuceneDfmEngineCustomizer : IDfmEngineCustomizer
{
    public void Customize(DfmEngineBuilder builder, IReadOnlyDictionary<string, object> parameters)
        {
        // insert inline rule at the top
        builder.InlineRules = builder.InlineRules.Insert(0, new EnvironmentVariableInlineRule());
    }
}

// define the rule
public class EnvironmentVariableInlineRule : IMarkdownRule
{
    // give it a name
    public string Name => "EnvVarToken";

    // define my regex to match
    private static readonly Regex _envVarRegex = new Regex(@"^\[EnvVar:(\w+?)\]", RegexOptions.Compiled);

    // process the match
    public IMarkdownToken TryMatch(IMarkdownParser parser, IMarkdownParsingContext context)
    {
        var match = _envVarRegex.Match(context.CurrentMarkdown);
        if (match.Length == 0) return null;

        var envVar = match.Groups[1].Value;
        var text = Environment.GetEnvironmentVariable(envVar);
        if (text == null) return null;

        // 'eat' the characters of the current markdown token so they aren't re-processed
        var sourceInfo = context.Consume(match.Length);

        // return a docfx token that just returns the text passed to it
        return new MarkdownTextToken(this, parser.Context, text, sourceInfo);
    }
}

In the end, that’s actually pretty simple! But don’t go trying to create a fancy token that doesn’t start with those magic characters since it’s not going to work.

Finding success in OSS

Thu, 26 Sep 2019 09:30:00 Z

I’ve been working in open source for quite some time and wanted to share some of my experiences in regards to managing OSS projects and communities. I’ve been employed by Umbraco, an open source .Net based CMS, for over 7 years (wow! where has the time gone!?) and have been working with the Umbraco project for even longer. In the past 10 years I’ve also started a number of my own OSS projects, seen many other colleagues and companies do the same, seen many get stalled, sometimes fail, sometimes succeed and more rarely, actually thrive. We’ve seen an enormous rise in the OSS take-up within the .Net world in the past few years thanks to Microsoft really embracing and encouraging open source and the communities that make it work. But OSS is not an easy or straightforward task. So how does an open source project succeed? … I unfortunately don’t have a straightforward answer for that either but I would like to share some insight based on my own experience.

The main reason I’ve mentioned Umbraco is because it is one of these rarer open source projects that has actually thrived and it’s quite exciting to be part of such a large worldwide community. So how have they made it work? Like many other Umbraco community members, I started using it many moons ago, working in agency land and wanting a .Net based open source CMS. At that time, it turned out that this was really the only one out there that seemed to do the job. For me, what made this project so exciting back then was how easy it was to help make the software better. It was easy to submit patches, it was easy to get feedback on issues and proposed features, it was easy to get in touch with other community members and what was really great was that it didn’t take long for your fixes or features that you submitted to be part of a new release. This was all back before GitHub or even Git existed so all of this communication was done in email or on custom built forums, but it still worked. Skip ahead to today and the underlying principles that still make this project exciting haven’t changed, it’s now just infinitely easier to do all of these things with better tools. The community of Umbraco is what makes this project tick, it has always been the main focus of the project and with community growth, the software and ecosystem thrive.

But it’s free! … so how am I employed??

This is the most common question I get asked about working for an open source project. When I started, I had no idea how to make an open source project sustainable along with having paid employees and I was equally intrigued. Umbraco offered a few services and products that were paid in order to be able to pay full time staff members and keep the project growing. These were things like various levels of certified training courses, add-on licensed products and paid support plans. Today the story for Umbraco is similar and includes all of these things but now also includes subscription models for Software as a Service called Umbraco Cloud which helps to continue the cycle of re-investing into growing the community and the project.

Most OSS projects are made by individuals

But … most OSS projects are tiny and are run by a single person who is the sole developer on a project who dedicate their own time to writing and maintaining it. A lot of this time will be used to help people: investigating issues, answering questions, writing documentation, fixing bugs, releasing new versions, etc… and all of this is done for Free but these are the things that make a project successful. It is actually quite difficult to create a community that re-invests their time into helping write your software rather than just using your software. If you can convince one or more developers to get on board, you’ll then need to invest more of your time in reviewing code, documentation updates, bug fixes and features created by your community. This time is critical, you want to be able to provide feedback and to integrate your communities changes as fast as possible… this is what will help keep your community re-investing their own time in growing the project and hopefully drumming up support for even more developers to chip in.

In my experience, this is where most OSS projects plateau because the project’s founder doesn’t have enough time to manage it all. This generally leaves projects in a semi-stalled state and may end up losing active developers and community members since their pull requests and submitted issues will remain pending for too long. These projects will ebb and flow, developers get busy and then may be re-inspired to find some more time. This doesn’t mean these projects are unsuccessful and it is probably the state of most OSS projects out there.

So how do you keep a project’s momentum?

I try to keep my own projects ‘active’ by answering questions, reviewing PRs and releasing new versions as often as I can… but in reality, this doesn’t happen as often as I would like. Sometimes I get re-inspired and may invest my evenings or weekends on some features but ultimately, I don’t think this is sustainable if you do want your OSS project to become community driven. One of the first OSS projects I created was called “uComponents” which was a collection of plugins for Umbraco and I did something with that project that I haven’t done with any of my other ones – I asked another community member to take over the project. This meant providing write access to the code but more importantly trusting another developer to help move the project forward. Eventually there was a few devs with write access and helping to manage the project and it ended up being very successful. I think if you can find people to help manage your project and allow yourself to trust other devs with that task, it will massively help your chances of not stalling the project.

I think the key to keeping a project’s momentum is to get really good at setting up the fundamentals for establishing a community:

Make it as easy as possible to contribute and discuss
Respond to your community in a timely fashion – doesn’t need to be a solution, but some form of acknowledgement is needed
Have good documentation and make sure the community can contribute to it
Have small achievable tasks on your tracker and use tags like “up-for-grabs” or “help-needed” so it’s easy for the community to know where they can ship in
Have CI/CD setup with tests, feedback and build outputs for the community to use
Trust other people in helping you to manage your project

How do larger OSS projects do it?

I can’t answer this question for all larger OSS projects, but I can provide some insight with what we do at Umbraco. The more popular a project becomes, the more people there will be that that need help. Ideally, you’ll want to try to convert the people that are needing help, into people that are providing help. To do that you should have all of the above points working well to make it easy for your community to start helping each other instead of just relying on you. Trusting the community is certainly at the heart of it all and this allows for the community to manage several aspects of itself.

At Umbraco, we have a community PR team that dedicates their own time to helping out with the project and assisting with communication between the community and dev team and ensuring that contributors are receiving a timely response. We have a community documentation team that dedicates their own time to generate missing documentation, assessing documentation PRs and to help encourage more contributors to get involved. Our community forum is also open source and is where community members can ask and answer questions. We use GitHub for all of our code and our public issue tracker and use tags like up-for-grabs along with some bots to help automate the interaction with contributors on GitHub. We’ve also established some policies regarding issues and PRs to try to keep their numbers to a minimum, as one example: only allowing an up-for-grabs issue to be open for a certain amount of time and if it doesn’t get picked up it just gets closed. The premise for this is that it isn’t important enough for the community at the moment since nobody wanted to pick it up, so it gets closed with the option of being reopened if it becomes important again.

In order to manage all of this interaction we have a couple of full-time staff members that spend a significant amount of time working with the community and these community teams, on our forums and on our issue trackers. Several years ago, this was not the case and it’s now clear that managing a large community is a full time role and what quickly became obvious was that the more you invest in the community the more you get back.

Are my own projects successful?

Sort of, yes and no. Some projects get old and I’d rather they didn’t exist anymore but people still use them so they need to be maintained. Some projects are newer and I’m excited about them but then lack the time to keep their momentum going. Then I have a problem of having too many projects! The important thing to me is that I keep learning and I’m generally happy to keep investing the time I can find into all of my projects. I think some of my projects would be more successful if I actually did all of the things I mentioned above :)

The fact is, unless you have a lot of spare time or you are getting paid to work on OSS (so that you can make time), establishing and fostering an active community is difficult and requires putting in a lot of time and effort. For me it’s entirely worth it, I think starting and managing your own OSS project is hugely rewarding and it’s a wonderful platform for learning and collaborating regardless of how big or small the project becomes.

Being a part of a community

You don’t need to be the founder of a project to reap the rewards of OSS. You can learn an enormous amount by collaborating on an OSS project. Of course there’s plenty of technical knowledge to gain but there’s a lot of other skills you can gain too and in my opinion the most important of all is communication skills. It’s easy to get frustrated by software or by other community members especially if you’ve run into some annoying bugs, somebody is being a jerk or just being difficult to communicate with. But, it’s important to not let these types of things get the better of you and that in itself is a difficult skill to master!

Communication is a 2 way street and it’s important for both community members and project owners to be friendly, patient and helpful.

A great example of this is in bug reports. Some reports are unhelpful, aggressive or even angry. When submitting bug reports or asking questions, keep in mind that you are generally asking people for help for free. Please be helpful, please be friendly, please be patient. Please think about how you would like to receive a bug report or be asked a question, include as much information as you can and remember that most people can’t read minds ;) And vice-versa, if you are responding to questions where people aren’t being as helpful as they should be, you’ll typically you’ll be able to turn a conversation into a friendly, patient and helpful one by just communicating that way yourself.

English is typically the language used in OSS projects and some people might not speak English as a first language and that’s ok! In this case, those 3 points are extremely important and it’s quite rewarding to see international community members who come from all sorts of backgrounds, countries and languages collaborating on a single project.

When both community members and project owners adopt these values, great things will happen and everyone wins!

… now I need to go find some time to update my OSS projects :)

This blog is powered by Articulate, an OSS project that I maintain which is running on Umbraco, an OSS project that I work for and this post was written using Open Live Writer, an OSS project that could use some love.

Configuring Azure Active Directory login with Umbraco Members

Mon, 18 Feb 2019 02:09:00 Z

This post is about configuring Azure Active Directory with Umbraco Members (not Users), meaning this is for your front-end website, not the Umbraco back office. I did write up a post about Azure AD with back office users though, so if that is what you are looking for then this is the link.

Install the Nuget packages

First thing to do is get the UmbracoIdentity package installed.

PM > Install-Package UmbracoIdentity

(This will also install the UmbracoIdentity.Core base package)

This package installs some code snippets and updates your web.config to enable ASP.Net Identity for Umbraco members. Umbraco ships with the old and deprecated ASP.Net Membership Providers for members and not ASP.Net Identity so this package extends the Umbraco CMS and the Umbraco members implementation to use ASP.Net Identity APIs to interact with the built in members data store. Installing this package will remove the (deprecated) FormsAuthentication module from your web.config and it will no longer be used to authenticate members, so the typical members snippets built into Umbraco macros will not work. Instead use the supplied snippets shipped with this package.

To read more about this package see the GitHub repo here.

Next, the OpenIdConnect package needs to be installed

PM > Install-Package Microsoft.Owin.Security.OpenIdConnect

Configure Azure Active Directory

Head over to the Azure Active Directory section on the Azure portal, choose App Registrations (I’m using the Preview functionality for this) and create a New registration

Next fill out the app details

You may also need to enter other redirect URLs depending on how many different environments you have. All of these URLs can be added in the Authentication section of your app in the Azure portal.

For AAD configuration for front-end members, the redirect Urls are just your website’s root URL and it is advised to keep the trailing slash.

Next you will need to enable Id Tokens

Configure OpenIdConnect

The UmbracoIdentity package will have installed an OWIN startup class in ~/App_Start/UmbracoIdentityStartup.cs (or it could be in App_Code if you are using a website project). This is how ASP.Net Identity is configured for front-end members and where you can specify the configuration for different OAuth providers. There’s a few things you’ll need to do:

Allow external sign in cookies

If you scroll down to the ConfigureMiddleware method, there will be a link of code to uncomment: app.UseExternalSignInCookie(DefaultAuthenticationTypes.ExternalCookie); this is required for any OAuth providers to work.

Enable OpenIdConnect OAuth for AAD

You’ll need to add this extension method class to your code which is some boiler plate code to configure OpenIdConnect with AAD:

public static class UmbracoADAuthExtensions
{
    public static void ConfigureAzureActiveDirectoryAuth(this IAppBuilder app,
        string tenant, string clientId, string postLoginRedirectUri, Guid issuerId,
        string caption = "Active Directory")
    {
        var authority = string.Format(
            System.Globalization.CultureInfo.InvariantCulture,
            "https://login.windows.net/{0}",
            tenant);

        var adOptions = new OpenIdConnectAuthenticationOptions
        {
            ClientId = clientId,
            Authority = authority,
            RedirectUri = postLoginRedirectUri
        };

        adOptions.Caption = caption;
        //Need to set the auth type as the issuer path
        adOptions.AuthenticationType = string.Format(
            System.Globalization.CultureInfo.InvariantCulture,
            "https://sts.windows.net/{0}/",
            issuerId);
        app.UseOpenIdConnectAuthentication(adOptions);
    }
}

Next you’ll need to call this code, add the following line underneath the app.UseExternalSignInCookie method call:

app.ConfigureAzureActiveDirectoryAuth(
    ConfigurationManager.AppSettings["azureAd:tenantId"],
    ConfigurationManager.AppSettings["azureAd:clientId"],
    //The value of this will need to change depending on your current environment
    postLoginRedirectUri: ConfigurationManager.AppSettings["azureAd:redirectUrl"],
    //This is the same as the TenantId
    issuerId: new Guid(ConfigurationManager.AppSettings["azureAd:tenantId"]));

Then you’ll need to add a few appSettings to your web.config (based on your AAD info):

<add key="azureAd:tenantId" value="YOUR-TENANT-ID-GUID" />
<add key="azureAd:clientId" value="YOUR-CLIENT-ID-GUID" />
<add key="azureAd:redirectUrl" value="http://my-test-website/" />

Configure your Umbraco data

The UmbracoIdentity repository has the installation documentation and you must follow these 2 instructions, and they are very simple:

Once that is done you will have an Member account management page which is based off of the installed views and snippets of the UmbracoIdentity package. This account page will look like this:

As you can see the button text under the “Use another service to log in” is the login provider name which is a bit ugly. The good news is that this is easy to change since this is just a partial view that was installed with the UmbracoIdentity package. You can edit the file: ~/Views/UmbracoIdentityAccount/ExternalLoginsList.cshtml, the code to render that button text is using @p.Authentication provider but we can easily change this to @p.Caption which is actually the same caption text used in the extension method we created. So the whole button code can look like this instead:


<button type="submit" class="btn btn-default"
        id="@p.AuthenticationType"
        name="provider"
        value="@p.AuthenticationType"
        title="Log in using your @p.Caption account">
    @p.Caption
</button>

This is a bit nicer, now the button looks like:

The purpose of all of these snippets and views installed with UmbracoIdentity is for you to customize how the whole flow looks and works so you’ll most likely end up customizing a number of views found in this folder to suit your needs.

That’s it!

Once that’s all configured, if you click on the Active Directory button to log in as a member, you’ll be brought to the standard AAD permission screen:

Once you accept you’ll be redirect back to your Account page:

Any customizations is then up to you. You can modify how the flow works, whether or not you accepting auto-linking accounts (like in the above example), or if you require a member to exist locally before being able to link an OAuth account, etc… All of the views and controller code in UmbracoIdentity is there for you to manipulate. The main files are:

~/Views/Account.cshtml
~/Views/UmbracoIdentityAccount/*
~/Controllers/UmbracoIdentityAccountController.cs
~/App_Start/UmbracoIdentityStartup.cs

Happy coding!

Deploying to Azure from VSTS using publish profiles and msdeploy

Thu, 26 Oct 2017 06:39:00 Z

In almost all of the examples online about how to deploy various services to Azure, they always list the super easy way to do it and that is to authenticate your current account to your Azure subscription which then grants your VSTS build to do all sorts of things… The problem is that not everyone has the security clearance to use the super easy tools in VSTS.

When you attempt to use these nice tools in VSTS you might get an error like this: “Failed to set Azure permission ‘RoleAssignmentId: some-guid-goes-here’ for the service principal … does not have authorizationto perform action ‘Microsoft.Authorization/roleAssignments/write’ over scope” This is because these nice VSTS tools actually creates a custom user behind the scenes in your azure subscription to use but your account doesn’t have access to authorize that.

Luckily there’s a work around

MS Deploy … sigh

Maybe there are other work arounds but this works, however it’s not the most elegant. I thought I’d post my findings here because it was a bit of a pain in the ass to get this all correct.

So here’s the steps:

1. Download the publish profile

You need to get the publish profile from your app service that you want to deploy to. This can be a website, a staging slot, an Azure function, (probably a bunch of others)

The file downloaded is an XML file containing a bunch of info you’ll need

2. Create a release definition and environment for your deployment

This assumes that you are pretty familiar with VSTS

You’ll want to create an empty environment in your release definition. Normally this is where you could choose the built in fancy VSTS deployment templates like “Azure App Service Deployment” … but as above, this doesn’t work if you don’t have security clearance. Instead, choose ‘Empty’

Then in your environment tasks, add Batch Script

3. Setup your batch script

There’s 2 ways to go about this and both depend on a msdeploy build output. This build output is generated by your build in VSTS if you are using a standard VSTS Visual Studio solution build. This will create msdeploy packages for you and will have put them in your artifacts folder. Along with msdeploy packages this will also generate a cmd batch file that executes msdeploy and a readme file to tell you how to execute it which contains some important info that you should read.

So here’s 2 options: Execute the cmd file, or execute msdeploy.exe directly

Execute the cmd file

There’s a bit of documentation about this online but most of it is based on using the SetParameters.xml file to adjust settings… but i just don’t want to use that.

Here’s the Path and Arguments that you need to run:

$(System.DefaultWorkingDirectory)/YOUR_BUILD_NAME/drop/YOUR_MSBUILD_PACKAGE.deploy.cmd

/y "/m:https://${publishUrl}/MSDeploy.axd?site=${msdeploySite}" /u:$(userName) /p:$(userPWD) /a:Basic -enableRule:DoNotDeleteRule "-setParam:name='IIS Web Application Name',value='${msdeploySite}'"

The parameters should be added to your VSTS Variables: ${msdeploySite}, $(userName), $(userPWD) and these variables correspond exactly to what is in your publish profile XML file that you downloaded. These parameters need to be pretty much exact, any misplaced quote or if you don’t include https, etc… will cause this to fail.

Important: the use of -enableRule:DoNotDeleteRule is totally optional, if you want to reset your site to exactly what is in the msdeploy package you do not want this. If however, you have user generated images, content or custom config files that exist on your site and you don’t want them deleted when you deploy, then you need to set this.

I’m unsure if this will work for Azure Functions deployment (it might!) … but I used the next option to do that:

Execute msdeploy.exe directly

If you execute the CMD file, you’ll see in the VSTS logs the exact msdeploy signature used which is:

"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -source:package='d:\a\r1\a\YOUR_PROJECT_NAME\drop\YOUR_MSDEPLOY_PACKAGE_FILE.zip' -dest:auto,computerName="https://YOUR_PUBLISH_URL/MSDeploy.axd?site=YOUR_PROFILE_NAME",userName=********,password=********,authtype="Basic",includeAcls="False" -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension -setParamFile:"d:\a\r1\a\YOUR_PROJECT_NAME\drop\YOUR_MSDEPLOY_PACKAGE_FILE.SetParameters.xml" -enableRule:DoNotDeleteRule -setParam:name='IIS Web Application Name',value='YOUR_PROFILE_NAME'

So if you wanted, you could take this and execute that directly instead of the CMD file. I use this method to deploy Azure Functions but the script is a little simpler since that deployment doesn’t require all of these parameters. For that I use this for the Path and Arguments:

C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe

-verb:sync -source:package='$(System.DefaultWorkingDirectory)/YOUR_BUILD_NAME/drop/YOUR_MSDEPLOY_PACKAGE.zip' -dest:auto,computerName="https://$(publishUrl)/msdeploy.axd?site=$(msdeploySite)",UserName='$(userName)',Password='$(userPWD)',AuthType='Basic' -setParam:name='IIS Web Application Name',value='$(msdeploySite)'

Hopefully this comes in handy for someone

How to lazily set the multi-term rewrite method on queries in Lucene

Fri, 15 Sep 2017 02:21:00 Z

For wildcard queries in Lucene that you would like to have the results ordered by Score, there’s a trick that you need to do otherwise all of your scores will come back the same. The reason for this is because the default behavior of wildcard queries uses CONSTANT_SCORE_AUTO_REWRITE_DEFAULT which as the name describes is going to give a constant score. The code comments describe why this is the default:

a) Runs faster
b) Does not have the scarcity of terms unduly influence score
c) Avoids any "TooManyBooleanClauses" exceptions

Without fully understanding Lucene that doesn’t really mean a whole lot but the Lucene docs give a little more info

NOTE: if setRewriteMethod(org.apache.lucene.search.MultiTermQuery.RewriteMethod) is either CONSTANT_SCORE_BOOLEAN_QUERY_REWRITE or SCORING_BOOLEAN_QUERY_REWRITE, you may encounter a BooleanQuery.TooManyClauses exception during searching, which happens when the number of terms to be searched exceeds BooleanQuery.getMaxClauseCount(). Setting setRewriteMethod(org.apache.lucene.search.MultiTermQuery.RewriteMethod) to CONSTANT_SCORE_FILTER_REWRITE prevents this.
The recommended rewrite method is CONSTANT_SCORE_AUTO_REWRITE_DEFAULT: it doesn't spend CPU computing unhelpful scores, and it tries to pick the most performant rewrite method given the query. If you need scoring (like FuzzyQuery, use MultiTermQuery.TopTermsScoringBooleanQueryRewrite which uses a priority queue to only collect competitive terms and not hit this limitation. Note that org.apache.lucene.queryparser.classic.QueryParser produces MultiTermQueries using CONSTANT_SCORE_AUTO_REWRITE_DEFAULT by default.

So the gist is, unless you are ordering by Score this shouldn’t be changed because it will consume more CPU and depending on how many terms you are querying against you might get an exception (though I think that is rare).

So how do you change the default?

That’s super easy, it’s just this line of code:

QueryParser.SetMultiTermRewriteMethod(MultiTermQuery.SCORING_BOOLEAN_QUERY_REWRITE);

But there’s a catch! You must set this flag before you parse any queries with the query parser otherwise it won’t work. All this really does is instruct the query parser to apply this scoring method to any MultiTermQuery or FuzzyQuery implementations it creates. So what if you don’t know if this change should be made before you use the query parser? One scenario might be: At the time of using the query parser, you are unsure if the user constructing the query is going to be sorting by score. In this case you want to change the scoring mechanism just before executing the search but after creating your query.

Setting the value lazily

The good news is that you can set this value lazily just before you execute the search even after you’ve used the query parser to create parts of your query. There’s only 1 class type that we need to check for that has this API: MultiTermQuery however not all implementations of it support rewriting so we have to check for that. So given an instance of a Query we can recursively update every query contained within it and manually apply the rewrite method like:

protected void SetScoringBooleanQueryRewriteMethod(Query query)
{
	if (query is MultiTermQuery mtq)
	{
		try
		{
			mtq.SetRewriteMethod(MultiTermQuery.SCORING_BOOLEAN_QUERY_REWRITE);
		}
		catch (NotSupportedException)
		{
			//swallow this, some implementations of MultiTermQuery don't support this like FuzzyQuery
		}
	}
	if (query is BooleanQuery bq)
	{
		foreach (BooleanClause clause in bq.Clauses())
		{
			var q = clause.GetQuery();
			//recurse
			SetScoringBooleanQueryRewriteMethod(q);
		}
	}
}

So you can call this method just before you execute your search and it will still work without having to eagerly use QueryParser.SetMultiTermRewriteMethod(MultiTermQuery.SCORING_BOOLEAN_QUERY_REWRITE); before you use the query parser methods.

Happy searching!

Smidge + Nuglify bundling, minification and source maps

Mon, 08 May 2017 23:23:00 Z

Smidge 2.0.0 RTM is finally here!

What is Smidge? Smidge is a lightweight runtime bundling library (CSS/JavaScript file minification, combination, compression) for ASP.NET Core - Here’s some more background info on Smidge

There’s plenty of great new features in the version 2.0.0 , but in this post I’ll just highlight a couple of fun ones….

Nuglify and source maps

Nuglify if an ASP.NET Core port of the old AjaxMin library and this works very nicely with Smidge too. By default Smidge uses an updated version of JsMin that I maintain called JsMinSharp for it’s JavaScript minification engine. JsMin works fairly well, it’s primary feature is that it’s fast because it doesn’t come with any bells and whistles and doesn’t produce a Syntax Tree. Nuglify on the other hand does produce a Syntax Tree which gives it loads of control over how the minification is done so that it can produce much smaller files, has the ability to rename/reduce parts of the JS and to produce source maps. The downfall of all this control is that it becomes much slower but Smidge is smart enough to cache processed files on the server in a persistent state so it won’t re-process your files if it doesn’t need to.

Here’s a quick benchmark of JsMin vs Nuglify to minify jQuery using the default options (and no source map). As you can see JsMin is substantially faster and is far better for memory but Nuglify can reduce the file size quite a lot more (smaller Minified % is better)

Method	Median	StdDev	Scaled	Scaled-SD	Minified %	Gen 0	Gen 1	Gen 2	Bytes Allocated/Op
JsMin	10.2008 ms	0.3102 ms	1.00	0.00	51.75%	-	-	-	155,624.67
Nuglify	69.0778 ms	0.0180 ms	6.72	0.16	32.71%	53.00	22.00	15.00	4,837,313.07

Source maps you say? JsMin isn’t all that smart so it can’t produce source maps (yet) but Nuglify certainly can and by default the Smidge.Nuglify package has this turned on by default. Here’s how you can get this working:

Follow the super easy Smidge quick start installation guide
Install the Smidge.Nuglify package

PM> Install-Package Smidge.Nuglify
Add Smidge Nuglify to your services: services.AddSmidgeNuglify();
Use Smidge Nuglify in your Configure method: app.UseSmidgeNuglify();

Replace JsMin with Nuglify for the default pipeline:

services.Configure(opt =>
{
    opt.PipelineFactory.OnCreateDefault = (type, pipeline) => pipeline.Replace(opt.PipelineFactory);
});

That’s it! Nuglify is now the JS minifier for the default pipeline and will produce source maps. You can configure the Nuglify settings in the AddSmidgeNuglify method.

Now you can debug via the source maps in dev tools:

File watching

Smidge can now watch your source files and if any changes are detected it can auto cache bust the bundle and flag for re-processing.

Previous to version 2.0.0 if you were in release mode all of the bundling/minification would be active but you really just wanted to test this behavior but unfortunately it meant that if you updated a js/css file you would have to go bump the Smidge version in the config file to clear out the server side persistent cache. In 2.0.0, you can configure your bundle with different environment options for both Debug and Production. For example, perhaps in Debug mode you still want your bundles to be active so that the minification, combination, compression, etc… is still happening but you want to still actively edit files … you can easily do this! Here’s a nice fluent syntax to achieve this:

bundles.CreateJs("my-application", "~/js/site.js", "~/js/app")
    .WithEnvironmentOptions(BundleEnvironmentOptions.Create()
        .ForDebug(builder => builder
            .EnableCompositeProcessing()
            .EnableFileWatcher()
            .SetCacheBusterType())
        .Build()
    );

For this example the “my-application” bundle will do the following when being rendered in debug:

Still process the files in the bundle and minify, combine, compress into a composite file
Attach file watchers to all files in the bundle
Change the cache buster for this bundle to only survive the lifespan of the current app domain

And what’s cool is that if you edit any of the source files in this bundle it will auto cache bust to a new value and re-process the bundle when it’s next requested.

Debug vs Production?

There’s a few ways that you could render assets in either debug or production mode but this is probably the easiest and uses ASP.NET Core’s environment tag helpers. Notice the debug attribute on the script/link tags and that the src/href attributes just include the bundle name … this is using Smidge’s tag helpers to define which bundle to render and if it’s in debug mode.

<environment names="Development">
	<link rel="stylesheet" href="my-css" debug="true" />
	<script src="my-js" debug="true"></script>
</environment>

<environment names="Staging,Production">
	<link rel="stylesheet" href="my-css" />
	<script src="my-js"></script>
</environment>

Thanks!

Big thanks to @dazinator for all the help, recommendations, testing, feedback, etc… and for the rest of the community for filing bugs, questions, and comments. Much appreciated :)

Using AspNet5 OptionsModel

Mon, 12 Jan 2015 04:21:00 Z

If you’ve used AspNet5 then you’ve probably been using some MVC, in which case you’ve probably seen something like this in your Startup class:

// Add MVC services to the services container
services.AddMvc(configuration)
    .Configure<MvcOptions>(options =>
    {
        //Configure some MVC options like customizing the 
        // view engines, etc...
        options.ViewEngines.Insert(0, typeof(TestViewEngine));
    });

It turns out this syntax for specifying ‘options’ for a given service is a generic pattern that you can use in your own code. In fact the OptionsModel framework is it’s own code repository: https://github.com/aspnet/Options

I’ve implemented custom options in my AspNet5 project called Smidge (a runtime JavaScript/CSS pre-processing engine) and wanted to share the details since as far as I’ve seen there isn’t any documentation about this.

What are options?

Probably the simplest way to describe the options framework is that: Options allow you to configure your application via code during startup.

Options are just a POCO class that can contain configuration options to customize the behavior of your library. These option classes can be injected into any of your services with IoC using an interface called Microsoft.Framework.OptionsModel.IOptions. There’s a caveat to this POCO class however: It must contain an parameter-less/empty constructor which means you cannot have services injected into the your options class via it’s constructor. This options framework also allows for ‘named’ options. So for example, perhaps you have a single options class that you would like to have configured in 2 different ways, one for ‘staging’ and one for your ‘live’ website.

Creating options

Here’s a really simple example of a POCO options class:

public class CustomMessageOptions
{
    public CustomMessage()
    {
        Message = "";
    }

    public string Message { get; set; }
}

In order to use this options class you need to create an options configuration class. For example:

public class CustomMessageOptionsSetup : ConfigureOptions<CustomMessageOptions>
{
    public CustomMessageOptionsSetup() 
    : base(ConfigureMessageOptions)
    {
    }

    /// <summary>
    /// Set the default options
    /// </summary>
    public static void ConfigureMessageOptions(CustomMessageOptions options)
    {
        options.Message = "Hello world";
    }
}

Then you need to add this class to your IoC container of type Microsoft.Framework.OptionsModel.IConfigureOptions:

services.AddTransient<IConfigureOptions<CustomMessageOptions>, CustomMessageOptionsSetup>();

Using options

To configure your options during startup, you do so in the ConfigureServices method like:

services.Configure<CustomMessageOptions>(options =>
{
    options.Message = "Hello there!";
});

Now you can have these options injected into any of your services using the IOptions interface noted previously:

public class MyCoolService 
{
    public MyCoolService(IOptions<CustomMessageOptions> messageOptions)
    {
        //IOptions exposes an 'Options' property which resolves an instance
        //of CustomMessageOptions
        ConfiguredMessage = messageOptions.Options.Message;
    }

    public string ConfiguredMessage {get; private set;}
}

Named options

As an example, lets say that you want a different message configured for your ‘staging’ and ‘live’ websites. This can be done with named options, here’s an example:

services
    .Configure<CustomMessageOptions>(options =>
    {
        options.Message = "Hi! This is the staging site";
    }, "staging")
    .Configure<CustomMessageOptions>(options =>
    {
        options.Message = "Hi! This is the live site";
    }, "live");

Then in your service you can resolve the option instance by name:

public class MyCoolService 
{
    public MyCoolService(IOptions<CustomMessageOptions> messageOptions)
    {
        //IRL This value would probably be set via some environment variable
        var configEnvironment = "staging";

        //IOptions exposes an 'GetNamedOptions' method which resolves an instance
        //of CustomMessageOptions based on a defined named configuration
        ConfiguredMessage = messageOptions.GetNamedOptions(configEnvironment);
    }

    public string ConfiguredMessage {get; private set;}
}

Configuring options with other services

Since your options class is just a POCO object and must have a parameter-less/empty constructor, you cannot inject services into the options class. However, there is a way to use IoC services in your options classes by customizing the ConfigureOptions class created above. In many cases this won’t be necessary but this really depends on how you are using options. As a (bad) example, lets say we wanted to expose a custom helper service called SiteHelper on the CustomMessageOptions class that can be used by a developer to create the message. The end result syntax might look like:

services.Configure<CustomMessageOptions>(options =>
    {
        var siteId = options.SiteHelper.GetSiteId();
        options.Message = "Hi! This is the staging site with id: " + siteId;
    });

In order for that to work the options.SiteHelper property needs to be initialized. This is done with the CustomMessageOptionsSetup class (created above) which has been added to the IoC container, this means it can have other services injected into it. The resulting class would look like:

public class CustomMessageOptionsSetup : ConfigureOptions<CustomMessageOptions>
{
    //SiteHelper gets injected via IoC
    public CustomMessageOptionsSetup(SiteHelper siteHelper) 
    : base(ConfigureMessageOptions)
    {
        SiteHelper = siteHelper;
    }

    public SiteHelper SiteHelper { get; private set; }

    /// <summary>
    /// Set the default options
    /// </summary>
    public static void ConfigureMessageOptions(CustomMessageOptions options)
    {
        options.Message = "Hello world";
    }

    /// <summary>
    /// Allows for configuring the options instance before options are set
    /// </summary>
    public override void Configure(Bundles options, string name = "")
    {
        //Assign the site helper instance
        options.SiteHelper = SiteHelper;

        base.Configure(options, name);
    }
}

IRL to give you an example of why this might be useful, in my Smidge project I allow developers to create named JavaScript/CSS bundles during startup using options. In some cases a developer might want to manipulate the file processing pipeline for a given bundle and in that case they need access to a service called PreProcessPipelineFactory which needs to come from IoC. The usage might look like:

services.AddSmidge()
    .Configure<Bundles>(bundles =>
    {                   
        bundles.Create("test-bundle-3", 
            bundles.PipelineFactory.GetPipeline(
                //add as many processor types as you want
                typeof(DotLess), typeof(JsMin)), 
            WebFileType.Js, 
            "~/Js/Bundle2");
    });

In the above, the bundles.PipelineFactory is a property on the bundles (options) class which gets initialized in my own ConfigureOptions class.

Hopefully this helps anyone looking to use custom options in their AspNet5 libraries!

Introducing ‘Smidge’ – an ASP.NET 5 runtime JS/CSS pre-processor

Thu, 11 Dec 2014 23:19:00 Z

During the past month I decided to dive deep into learning ASP.NET 5, and what better way to learn than to start a new OSS project :)

I chose to make a new new simple and extensible Javascript/CSS runtime pre-processor for ASP.NET 5. It does file minification, combination and compression, has a nice file caching layer and it’s all done in async operations. I ported over a few ideas and code snippets from CDF (client dependency framework) but with a more modern approach. I’ve called it ‘Smidge’ = something really small.

The project is on GitHub, it’s still a work in progress but its functional and there’s even some documentation! In the next few weeks I’ll get more of the code and docs updated and hopefully have a beta release out. In the meantime, you can clone the source, browse the code, build it and of course use it if you like.

Project details

It’s currently only targeting aspnet50 and not the Core CLR… I didn’t start with Core CLR because there was some legacy code I had to port over and I wanted to get something up and working relatively quickly. It shouldn’t be too much work to convert to Core CLR and Mono, hopefully I’ll find time to do that soon. It’s referencing all of the beta-* libraries from the ASP.NET 5 nightly myget feeds since there’s some code I’m using that isn’t available in the current beta1 release (like Microsoft.AspNet.WebUtilities.UriHelper). The target KRE version is currently KRE-CLR-amd64 1.0.0-beta2-10760.

Installation

I’ve put up an Alpha 1 release on Nuget, so you can install it from there:

PM> Install-Package Smidge -Pre

There’s some installation instructions here, you’ll need to add the smidge.json file yourself for now, can’t figure out how to get VS 2015 (kpm pack) to package that up … more learning required!

There’s certainly a lot of detective work involved in learning ASP.NET 5 but with the code being open source and browse-able/searchable on GitHub, it makes finding what you need fairly easy.

PetaPoco may cause high memory usage with certain queries

Thu, 25 Sep 2014 09:00:00 Z

If you are using PetaPoco, or NPoco (which seams to be the most up-to-date fork of the project), the title of this post might be a bit scary… but hopefully you won’t have to worry. This really depends on how you create your queries and how many different query structures you are executing.

High memory usage

Here is the code in relation to the memory growth when using PetaPoco:

https://github.com/toptensoftware/PetaPoco/blob/master/PetaPoco/PetaPoco.cs#L1836

What is happening here is that every time a POCO needs to be mapped from a data source, this will add more values to a static cache, specifically this one:

https://github.com/toptensoftware/PetaPoco/blob/master/PetaPoco/PetaPoco.cs#L2126 (m_PocoDatas)

This isn’t a bad thing… but it can be if you are:

using non-parameterized where clauses
you have dynamically generated where clauses
you use a lot of sql ‘IN’ clauses – since the items in the array being passed to the ‘IN’ clauses is dynamic
you have tons of differently statically unique where clauses

Each time a unique SQL query is sent to PetaPoco it will store this SQL string and associate it to a delegate (which is also cached). Over time, as these unique SQL queries are executed, the internal static cache will grow. In some cases this could consume quite a lot of memory.

The other thing to note is how large the ‘key’ that PetaPoco/NPoco uses:

var key = string.Format("{0}:{1}:{2}:{3}:{4}", sql, connString, ForceDateTimesToUtc, firstColumn, countColumns);

Considering how many queries might be executing in your application, the storage for these keys alone could take up quite a lot of memory! An SQL statement combined with a connection string could be very long, and each of these combinations gets stored in memory for every unique SQL query executed that returns mapped POCO objects.

Parameterized queries vs. non-parameterized

Here’s some examples of why non-parameterized queries will cause lots of memory consumption. Lets say we have a simple query like:

db.Query<MyTable>("WHERE MyColumn=@myValue", new {myValue = "test"})

Which results in this SQL:

SELECT * FROM MyTable WHERE MyColumn = @myValue

This query can be used over and over again with a different value and PetaPoco will simply store a single SQL key in it’s internal cache. However, if you are executing queries without real parameters such as:

db.Query<MyTable>("WHERE MyColumn='hello'");
db.Query<MyTable>("WHERE MyColumn='world'");
db.Query<MyTable>("WHERE MyColumn='hello world'");

Which results in this SQL:

SELECT * FROM MyTable WHERE MyColumn = 'hello';
SELECT * FROM MyTable WHERE MyColumn = 'world';
SELECT * FROM MyTable WHERE MyColumn = 'hello world';

Then PetaPoco will store each of these statements against a delegate in it’s internal cache since each of these string statements are not equal to each other.

Depending on your application you still might have a very large number of unique parameterized queries, though I’d assume you’d have to have a terrifically huge amount for it to be a worry.

Order by queries

Unfortunately even if you use parameterized queries, PetaPoco will store the SQL query key with it’s Order By clause which isn’t necessary and will again mean more duplicate SQL keys and delegates being tracked. For example if you have these resulting queries:

SELECT * FROM MyTable WHERE MyColumn = @myValue ORDER BY SomeField;
SELECT * FROM MyTable WHERE MyColumn = @myValue ORDER BY AnotherField;

PetaPoco will store each of these statements in it’s internal cache separately since the strings don’t match, however the delegate that PetaPoco is storing against these SQL statements isn’t concerned about the ordering output, it’s only concerned about the column and table selection so in theory it should be stripping off the last Order By clause (and other irrelevant clauses) to avoid this duplication.

A slightly better implementation

First, if you are using PetaPoco/NPoco, you shouldn’t use dynamic queries for the point’s mentioned above. If you need this functionality then I suppose it might be worth these libraries adding some sort of property on the Database object or a parameter in either the Fetch or Query methods to specify that you don’t want to use the internal cache (this will be slower, but you won’t get unwanted memory growth). I’d really just suggest not using dynamically created where clauses ;-)

Next, there’s a few things that could be fixed in the PetaPoco/NPoco core to manage memory a little better:

The size the the key that is stored in memory doesn’t need to be that big. A better implementation would be to use a hash combiner class to combine the GetHashCode result of each of those parameters that make up the key. This is a very fast way to create a hash of some strings that will result in a much smaller key. An example of a hash combiner class is here (which is actually inspired by the various internal hash code combiner classes in .Net): https://github.com/umbraco/Umbraco-CMS/blob/7.2.0/src/Umbraco.Core/HashCodeCombiner.cs
Instead of storing all of this cache in static variables, have them stored in an ObjectCache/MemoryCache (http://msdn.microsoft.com/en-us/library/system.runtime.caching.objectcache(v=vs.110).aspx) with a sliding expiration so the memory can get collected when it’s unused
The Order By clause should be ignored based on the point mentioned above

I’ve created a PR for NPoco here, and also created an issue on the original PetaPoco source here.

Get JQuery requests to play nicely with AngularJS CSRF convention

Fri, 06 Dec 2013 19:06:00 Z

Decided to write this quick post for anyone searching on this topic. AngularJS has it’s own convention for CSRF (Cross Site Request Forgery) protection but in some cases you’ll be calling these same server side services via JQuery so you might need to get JQuery requests to also follow Angular’s convention.

For information about Angular’s CSRF protection see the “Security Considerations” part of Angular’s $http documentation.

Luckily it’s pretty darn easy to get JQuery to follow this convention too and this will also work with 3rd party plugins that use JQuery for requests like Blueimp file uploader. The easiest way to get this done is to set the global JQuery $.ajax rules. Probably the best place to do this is in your Angular app.run statement:

app.run(function ($cookies) {

    //This sets the default jquery ajax headers to include our csrf token, we
    // need to user the beforeSend method because the token might change 
    // (different for each logged in user)
    $.ajaxSetup({
        beforeSend: function (xhr) {
            xhr.setRequestHeader("X-XSRF-TOKEN", $cookies["XSRF-TOKEN"]);
        }
    }); 
});

That’s it!

It’s important to note to set the header using beforeSend, if you just set $.ajax options ‘headers’ section directly that means the header cannot be dynamic – which you’ll probably want if you have users logging in/out.

How to inspect assemblies with reflection before including them in your application

Fri, 25 Oct 2013 22:13:00 Z

If you application supports plugins or extensions in some cases it might be useful to scan a packages assemblies before importing them in to your app. Some reasons for this might be:

Checking if the package has missing assembly references
Checking if the assembly references obsolete types that might make the package unstable
Checking the .Net targeted framework of the assembly
Any other assembly inspection to determine it is compatible with your app

To do this you can load assemblies using Assembly.ReflectionOnlyLoadFrom and Assembly.ReflectionOnlyLoad methods which load assemblies into a special assembly load context called “reflection-only context” which will safely let you inspect these assemblies.

Further reading:

A great article on Reflection Only Assembly Loading and if you want to know more about assembly load contexts, here’s an explanation.

Loading in assemblies

For this example, we’ll assume that all of the assemblies for a package are in some folder outside of the normal /bin folder (not loaded in to the current app) and each assembly for the package will need to be inspected for type references that are not supported.

A common mistake when loading in assemblies with reflection (especially in the LoadFrom context) is to load them in one at a time, whereas they generally will need to be all loaded in before inspecting them since they probably have references to each other. Another thing that generally must be done is adding an event listener on AppDomain.ReflectionOnlyAssemblyResolve because even though all known referenced assemblies are loading into the context some assemblies might not be explicitly referenced but are need to load the assembly. This handler provides a way to resolve those missing references.

The first thing to do is setup the event handler

AppDomain.CurrentDomain.ReflectionOnlyAssemblyResolve += (s, e) =>
{
    var a = Assembly.ReflectionOnlyLoad(e.Name);
    if (a == null) throw new TypeLoadException("Could not load assembly " + e.Name);
    return a;
};

Next we need to load all of the assembly files in the folder

foreach (var f in files) Assembly.ReflectionOnlyLoadFrom(f);

Then load all of their referenced assemblies in to the reflection context

//Then load each referenced assembly into the context
var assembliesWithErrors = new List<string>();
foreach (var f in files)
{
    var reflectedAssembly = Assembly.ReflectionOnlyLoadFrom(f);
    foreach (var assemblyName in reflectedAssembly.GetReferencedAssemblies())
    {
        try
        {
            Assembly.ReflectionOnlyLoad(assemblyName.FullName);
        }
        catch (FileNotFoundException)
        {
            //if an exception occurs it means that a referenced assembly could not be found                        
            errors.Add(
                string.Concat("This package references the assembly '",
                    assemblyName.Name,
                    "' which was not found, this package may have problems running"));
            assembliesWithErrors.Add(f);
        }
    }
}

In the catch, I’m detecting assembly reference errors and adding an error message to the outgoing method response and also adding that assembly to the assembliesWithErrors list which is used later to ensure we’re not inspecting assemblies that couldn’t be loaded.

Now that all the assemblies are loaded we can inspect them (ignoring ones with errors). This example is looking for any assemblies that have types implementing ‘MyType’. If they do implement this type, add the assembly to a list to return from the current method.

//now that we have all referenced types into the context we can look up stuff
foreach (var f in files.Except(assembliesWithErrors))
{
    //now we need to see if they contain any type 'MyType'
    var reflectedAssembly = Assembly.ReflectionOnlyLoadFrom(f);
    var found = reflectedAssembly.GetExportedTypes()
        .Where(TypeHelper.IsTypeAssignableFrom<MyType>);
    if (found.Any())
    {
        dllsWithReference.Add(reflectedAssembly.FullName);
    }
}

Separate AppDomain

It’s best to execute all of this logic in a separate AppDomain because once assemblies are loaded in to a context, they cannot be unloaded and since we are loading in from files, those files will remain locked until the AppDomain is shutdown. Explaining how to create a separate AppDomain is outside the scope of this article but the code is included in the source below.

Source Code

Here’s a class the encapsulates all of this logic and of course if you can do much more when inspecting assemblies for various types.

https://gist.github.com/Shazwazza/7147978

Reference the current form controller in AngularJS

Wed, 26 Jun 2013 23:08:00 Z

I previously wrote a post about Listening for validation changes in AngularJS which with my knowledge at that time required a handy hack to get a reference to the currently scoped form controller (ngForm) for a given input control. I also complained a bit that it seemed that angular didn’t really provide a way to reference the current form controller without this little hack… well, it turns out I was wrong! :)

AngularJS seems kind of like ASP.Net MVC in the early days when there wasn’t much documentation… It definitely pays off to read through the source code to figure out how to do more complicated things. I had a bit of a ‘light bulb’ moment when I realized that ngForm was itself a directive/controller and had recently noticed that the ‘require’ parameter of setting up a directive allows you to search for controllers in the current directives ancestry (i.e. prefix the required controller with a hat: ^ )

What does the require parameter of a directive do?

Lets face it, the directive documentation for AngularJS is in desperate need of being updated so that human beings can understand it (as noted by the many comments at the bottom). So I’ll try to explain what the ‘require’ parameter actually does and how to use it.

We’ll create a simple custom validation directive which will invalidate a field if the value is “blah”

function blahValidator() {
    return {
        require: 'ngModel',
        link: function(scope, elm, attr, ctrl) {
            
            var validator = function(value) {
                if (ctrl.$viewValue == "blah") {
                    ctrl.$setValidity('blah', false);
                    return null;
                }
                else {
                    ctrl.$setValidity('blah', true);
                    return value;
                }
            };

            ctrl.$formatters.push(validator);
            ctrl.$parsers.push(validator);
        }
    };
}

You’ll notice that we have a ‘require’ parameter specified for ‘ngModel’. What is happening here is that when we assign this directive to an input field, angular will ensure that the input field also has a defined ng-model attribute on it as well. Then angular will pass in the instance of the ng-model controller to the ‘ctrl’ parameter of the link function.

So, the ‘require’ parameter dictates what the ‘ctrl’ parameter of the link function equals.

You can also require multiple controllers:

NOTE: the ctrl/ctrls parameter in the above 2 examples can be called whatever you want

Special prefixes

Angular has 2 special prefixes for the ‘require’ parameter:

^ = search the current directives ancestry for the controller

? = don’t throw an exception if the required controller is not found, making it ‘optional’ not a requirement

You can also combine them so angular will search the ancestry but it can be optional too such as: ^?ngController'

In the above example, the blahValidator will only work if the directive is declared inside of an ng-controller block.

Referencing the current ng-form

Given the above examples, and knowing the ngForm itself is a controller we should be able to just make a requirement on ngForm and have it injected into the directive. BUT, it wont work the way you expect. For some reason angular references the ngForm controller by the name “form” which i discovered by browsing the source of angular.

So now its easy to get a reference to the containing ngForm controller, all you need to do is add a ‘require’ parameter to your directive that looks like:

require: '^form'

and it will be injected into your ctrl parameter of your link function.

NDepend review = pretty cool!

Fri, 21 Jun 2013 23:47:00 Z

For a while I’ve been wanting to automate some reporting from our build process to show some fairly simple statistical information such as

Obsoleted code
Internal code planned to be made public
Code flagged to be obsoleted
Code flagged as experimental

NOTE: some of the above are based on our own internal c# attributes

Originally I was just going to write some code to show me this which would have been fairly straight forward but I knew that NDepend had a code query language and natively outputs reports and can be integrated into the build process, so I figured I’d give it a go.

So I went and installed NDepend and ran the analysis against my current project … I wasn’t quite prepared for the amount of information it gave me! :)

Reporting

I quickly found out that the reports that I wanted to create were insanely simple to write. I then lost a large part of my day exploring the reports it generates OOTB and I realized I hadn’t even touched the surface of the information this tool could give me. I was mostly interested in statistical analysis but this will also happily tell you about how bad or good your code is, how much test coverage you’ve got, what backwards compatibility you broke with your latest release, what code is most used in your project and what code isn’t used at all… and the list goes on and on.

To give you an example of the first report I wanted to write, here’s all I wrote to give me what methods in my codebase were obsoleted:

from m in JustMyCode.Methods where m.HasAttribute("System.ObsoleteAttribute") select m

What’s cool about the report output is that you can output to HTML or XML, so if you want to publish it on your site you can just write some XSLT to make it look the way you want.

Project comparison

Another really great feature that I thought would be fun to automate is the code analysis against previous builds so you know straight away if you are breaking compatibility with previous releases. I think it has great potential but what I did discover is that in some cases the OOTB queries for this aren’t perfect. It will tell you what public types, methods, and fields have been removed or changed and the query is pretty straightforward but… In many cases when we’re trying to maintain backwards compatibility while creating new APIs we’ll remove a type’s methods/properties and just make the legacy type inherit from the new API class (or some other trickery similar to that). This might break compatibility if people are using reflection or specific type checking but it wont break the API usage which I’m mostly interested in. I haven’t figured it out yet but I’m sure there’s some zany NDepend query I can write to check that.

Automation

So now that I know I can extract all this info, it’s time to decide what to export. Thankfully NDepend comes fully equipped with it’s own MSBuild integration so automating the report output should be pretty straightforward. I also noticed that if you wanted to be ultra careful you can integrate it with your build server and have your builds fail if some of your queries cause errors, like the above: If you break compatibility the build will fail. Or anything you want really, like failing the build if we detect you’ve written a method that has too many lines of code.

Pretty cool indeed!

I was going to use NDepend just to output reports which it clearly does without issue. What I didn’t realize was how much detail you can get about your code with this tool. There’s some tools in there that were a bit over my head but I suppose if you are working on really huge projects and need anything from just a snapshot or insane details into what dependencies your code has on other code it’ll do that easily as well.

The UI is a bit overwhelming at first and could probably do with some work but TBH I’m not sure how else you’d display that much information nicely. Once I figured out how the whole query and reporting structures work then it was dead simple.

Happy reporting :)

Listening for validation changes in AngularJS

Tue, 28 May 2013 06:18:00 Z

In some applications it can be really useful to have controllers listen for validation changes especially in more complicated AngularJS apps where ‘ng-repeat’ is used to render form controls. There’s plenty of cases where a parent scope might need to know about validation changes based on child scopes… one such case is a validation summary. There’s a couple ways to implement this (and probably more) but they all seem a bit hacky such as:

Apply a $watch to the current form object’s $valid property in the parent scope, then use jQuery to look for elements that have a class like ‘invalid’
- You could then use the scope() function on the DOM element that ng-repeat is used on to get any model information about the invalid item
In child scopes you could apply a $watch to individual form elements’ $valid property then change the $parent scope’s model values to indicate validation changes

Instead what I wanted to achieve was a re-usable way to ‘bubble’ up validation changes from any scope’s form element to ancestor scopes without having to do any of the following:

No jquery DOM selection
No hard coding of form names to access the validation objects
No requirement to modifying other scopes’ values

Implementation

The way I went about this was to create a very simple custom directive which I’ve called ‘val-bubble’ since it has to do with validation and it ‘bubbles’ up a message to any listening scopes. An input element might then look like this:

<input name="FirstName" type="text" required val-bubble />

Then in an outer scope I can then listen for validation changes and do whatever I want with the result:

scope.$on("valBubble", function(evt, args) {
    alert("Validation changed for field " + args.ctrl.$name + ". Valid? " + args.isValid);
});

The args object contains these properties:

isValid = is the field valid
ctrl = the current form controller object for the field
scope = the scope bound to the field being validated
element = the DOM element of the field being validated
expression = the current $watch expression used to watch this fields validation changes

With all of that information you can easily adds some additional functionality to your app based on the current validating inputs such as a validation summary or whatever.

Custom directive

The val-bubble custom directive is pretty simple, here’s the code and an explanation below:

app.directive('valBubble', function (formHelper) {
    return {
        require: 'ngModel',
        restrict: "A",
        link: function (scope, element, attr, ctrl) {

            if (!attr.name) {
                throw "valBubble must be set on an input element that has a 'name' attribute";
            }
                
            var currentForm = formHelper.getCurrentForm(scope);
            if (!currentForm || !currentForm.$name)
                throw "valBubble requires that a name is assigned to the ng-form containing the validated input";

            //watch the current form's validation for the current field name
            scope.$watch(currentForm.$name + "." + ctrl.$name + ".$valid", function (isValid, lastValue) {
                if (isValid != undefined) {
                    //emit an event upwards 
                    scope.$emit("valBubble", {
                        isValid: isValid,       // if the field is valid
                        element: element,       // the element that the validation applies to
                        expression: this.exp,   // the expression that was watched to check validity
                        scope: scope,           // the current scope
                        ctrl: ctrl              // the current controller
                    });
                }
            });
        }
    };
});

The first thing we’re doing here is limiting this directive to be used only as an attribute and ensuring the element has a model applied to it. Then we make sure that the element has a ‘name’ value applied. After that we are getting a reference to the current form object that this field is contained within using a custom method: formHelper.getCurrentForm … more on this below. Lastly we are applying a $watch to the current element’s $valid property and when this value changes we $emit an event upwards to parent/ancestor scopes to listen for.

formHelper

Above I mentioned that I wanted a re-usable solution where I didn’t need to hard code things like the current form name. Unfortunately Angular doesn’t really provide a way to do this OOTB (as far as I can tell!) (Update! see here on how to access the current form: http://shazwazza.com/post/Reference-the-current-form-controller-in-AngularJS) so I’ve just created a simple factory object that finds the current form object applied to the current scope. The type check is fairly rudimentary but it works, it’s simply checking each property that exists on the scope object and tries to detect the object that matches the definition of an Angular form object:

app.factory('formHelper', function() {
    return {
        getCurrentForm: function(scope) {
            var form = null;
            var requiredFormProps = ["$error", "$name", "$dirty", "$pristine", "$valid", "$invalid", "$addControl", "$removeControl", "$setValidity", "$setDirty"];
            for (var p in scope) {
                if (_.isObject(scope[p]) && !_.isFunction(scope[p]) && !_.isArray(scope[p]) && p.substr(0, 1) != "$") {
                    var props = _.keys(scope[p]);
                    if (props.length < requiredFormProps.length) continue;
                    if (_.every(requiredFormProps, function(item) {
                        return _.contains(props, item);
                    })) {
                        form = scope[p];
                        break;
                    }
                }
            }
            return form;
        }  
    };
});

NOTE: the above code has a dependency on UnderscoreJS

So now you can just apply the val-bubble attribute to any input element to ensure it’s validation changes are published to listening scopes!

Uploading files and JSON data in the same request with Angular JS

Sat, 25 May 2013 04:01:00 Z

I decided to write a quick blog post about this because much of the documentation and examples about this seems to be a bit scattered. What this achieves is the ability to upload any number of files with any other type of data in one request. For this example we’ll send up JSON data along with some files.

File upload directive

First we’ll create a simple custom file upload angular directive

app.directive('fileUpload', function () {
    return {
        scope: true,        //create a new scope
        link: function (scope, el, attrs) {
            el.bind('change', function (event) {
                var files = event.target.files;
                //iterate files since 'multiple' may be specified on the element
                for (var i = 0;i<files.length;i++) {
                    //emit event upward
                    scope.$emit("fileSelected", { file: files[i] });
                }                                       
            });
        }
    };
});

The usage of this is simple:

<input type="file" file-upload multiple/>

The ‘multiple’ parameter indicates that the user can select multiple files to upload which this example fully supports.

In the directive we ensure a new scope is created and then listen for changes made to the file input element. When changes are detected with emit an event to all ancestor scopes (upward) with the file object as a parameter.

Mark-up & the controller

Next we’ll create a controller to:

Create a model to bind to
Create a collection of files
Consume this event so we can assign the files to the collection
Create a method to post it all to the server

NOTE: I’ve put all this functionality in this controller for brevity, in most cases you’d have a separate factory to handle posting the data

With the controller in place, the mark-up might look like this (and will display the file names of all of the files selected):

<div ng-controller="Ctrl">
    <input type="file" file-upload multiple/>
    <ul>
        <li ng-repeat="file in files">{{file.name}}</li>
    </ul>
</div>

The controller code below contains some important comments relating to how the data gets posted up to the server, namely the ‘Content-Type’ header as the value that needs to be set is a bit quirky.

function Ctrl($scope, $http) {

    //a simple model to bind to and send to the server
    $scope.model = {
        name: "",
        comments: ""
    };

    //an array of files selected
    $scope.files = [];

    //listen for the file selected event
    $scope.$on("fileSelected", function (event, args) {
        $scope.$apply(function () {            
            //add the file object to the scope's files collection
            $scope.files.push(args.file);
        });
    });
    
    //the save method
    $scope.save = function() {
        $http({
            method: 'POST',
            url: "/Api/PostStuff",
            //IMPORTANT!!! You might think this should be set to 'multipart/form-data' 
            // but this is not true because when we are sending up files the request 
            // needs to include a 'boundary' parameter which identifies the boundary 
            // name between parts in this multi-part request and setting the Content-type 
            // manually will not set this boundary parameter. For whatever reason, 
            // setting the Content-type to 'false' will force the request to automatically
            // populate the headers properly including the boundary parameter.
            headers: { 'Content-Type': false },
            //This method will allow us to change how the data is sent up to the server
            // for which we'll need to encapsulate the model data in 'FormData'
            transformRequest: function (data) {
                var formData = new FormData();
                //need to convert our json object to a string version of json otherwise
                // the browser will do a 'toString()' on the object which will result 
                // in the value '[Object object]' on the server.
                formData.append("model", angular.toJson(data.model));
                //now add all of the assigned files
                for (var i = 0; i < data.files; i++) {
                    //add each file to the form data and iteratively name them
                    formData.append("file" + i, data.files[i]);
                }
                return formData;
            },
            //Create an object that contains the model and files which will be transformed
            // in the above transformRequest method
            data: { model: $scope.model, files: $scope.files }
        }).
        success(function (data, status, headers, config) {
            alert("success!");
        }).
        error(function (data, status, headers, config) {
            alert("failed!");
        });
    };
};

Handling the data server-side

This example shows how to handle the data on the server side using ASP.Net WebAPI, I’m sure it’s reasonably easy to do on other server-side platforms too.

public async Task<HttpResponseMessage> PostStuff()
{
    if (!Request.Content.IsMimeMultipartContent())
    {
        throw new HttpResponseException(HttpStatusCode.UnsupportedMediaType);
    }

    var root = HttpContext.Current.Server.MapPath("~/App_Data/Temp/FileUploads");
    Directory.CreateDirectory(root);
    var provider = new MultipartFormDataStreamProvider(root);
    var result = await Request.Content.ReadAsMultipartAsync(provider);
    if (result.FormData["model"] == null)
    {
        throw new HttpResponseException(HttpStatusCode.BadRequest);
    }

    var model = result.FormData["model"];
    //TODO: Do something with the json model which is currently a string


    //get the files
    foreach (var file in result.FileData)
    {                
        //TODO: Do something with each uploaded file
    }

    return Request.CreateResponse(HttpStatusCode.OK, "success!");
}

Razor + dynamic + internal + interface & the 'object' does not contain a definition for 'xxxx' exception

Thu, 11 Apr 2013 16:49:00 Z

I’ve come across this strange issue and decided to blog about it since I can’t figure out exactly why this is happening it is clearly something to do with the DLR and interfaces.

First, if you are getting the exception: 'object' does not contain a definition for 'xxxx' it is related to either an object you have being marked internal or you are are using an anonymous object type for your model (which .Net will always mark as internal).

Here’s a really easy way to replicate this:

1. Create an internal model

internal class MyModel
{
    public string Name {get;set;}
}

2. Return this model in your MVC action

public ActionResult Index()
{            
    return View(new InternalTestModel("Shannon"));
}

3. Make your view have a dynamic model and then try to render the model’s property

@model dynamic
<h1>@Model.Name</h1>

You’ll get the exception:

Server Error in '/' Application.

'object' does not contain a definition for 'Name'

So even though the error is not very informative, it makes sense since Razor is trying to access an internal class.

Try using a public interface

Ok so if we want to keep our class internal, we could expose it via a public interface. Our code might then look like this:

public interface IModel 
{    
    string Name {get;}
}
internal class MyModel : IModel 
{        
    public string Name {get;set;}
}

Then we can change our view to be strongly typed like:

@model IModel
<h1>@Model.Name</h1>

And it will work, however if you change your view back to be @model dynamic you will still get the exception. Pretty sure it’s because the DLR is just binding to the instance object and doesn’t really care about the interface… this makes sense.

Try using an abstract public class

For some reason if you make your internal class inherit from a public abstract class that implements the same interface you will not get this exception even though the object instance you are passing to razor is internal. For example with this structure:

public interface IModel 
{        
    string Name {get;}
}
public abstract class ModelBase : IModel 
{
    public abstract Name {get;}
}
internal class MyModel : IModel 
{            
    public override string Name {get;set;}
}

You will not get this error if your view is @model dynamic.

Strange!

I’m sure there’s something written in the DLR spec about this but still seems strange to me! If you are getting this error and you aren’t using anonymous objects for your model, hopefully this might help you figure out what is going on.

Taming the BuildManager, ASP.Net Temp files and AppDomain restarts

Wed, 28 Mar 2012 08:14:00 Z

I’ve recently had to do a bunch of research in to the BuildManager and how it deals with caching assemblies since my involvement with creating an ASP.Net plugin engine. Many times people will attempt to restart their AppDomain by ‘bumping’ their web.config files, meaning adding a space character or carriage return and then saving it. Sometimes you may have noticed that this does not always restart your AppDomain the way you’d expect and some of the new types you’d expect to have loaded in your AppDomain simply aren’t there. This situation has cropped up a few times when using the plugin engine that we have built in to Umbraco v5 and some people have resorted to manually clearing out their ASP.Net temp files and then forcing an IIS Restart, but this is hardly ideal. The good news is that we do have control over how these assemblies get refreshed, in fact the BuildManager reloads/refreshes/clears assemblies in different ways depending on how the AppDomain is restarted.

The hash.web file

An important step during the BuildManager initialization phase is a method call to BuildManager.CheckTopLevelFilesUpToDate which does some interesting things. First it checks if a special hash value is on disk and is not zero. You may have noticed a file in your ASP.Net temp files: /hash/hash.web and it will contain a value such as: 4f34227133de5346. This value represents a unique hash of many different combined objects that the BuildManager monitors. If this file exists and the value can be parsed properly then the BuildManager will call: diskCache.RemoveOldTempFiles(); What this does is the following:

removes the CodeGen temp resource folder
removes temp files that have been saved during CodeGen such as *.cs
removes the special .delete files and their associated original files which have been created when shutting down the app pool and when the .dll files cannot be removed (due to file locking)

Creating the hash value

The next step is to create this hash value based on the current objects in the AppDomain. This is done using an internal utility class in the .Net Framework called: HashCodeCombiner (pretty much everything that the BuildManager references is marked Internal! ). The process combines the following object values to create the hash (I’ve added in parenthesis the actual properties the BuildManager references):

the app's physical path, in case it changes (HttpRuntime.AppDomainAppPathInternal)
System.Web.dll (typeof(HttpRuntime).Module.FullyQualifiedName)
machine.config file name (HttpConfigurationSystem.MachineConfigurationFilePath)
root web.config file name, please note that this is not your web apps web.config (HttpConfigurationSystem.RootWebConfigurationFilePath)
the hash of the <compilation> section (compConfig.RecompilationHash)
the hash of the system.web/profile section (profileSection.RecompilationHash)
the file encoding set in config (appConfig.Globalization.FileEncoding)
the <trust> config section (appConfig.Trust.Level & appConfig.Trust.OriginUrl)
whether profile is enabled (ProfileManager.Enabled)
whether we are precompiling with debug info (but who precompiles :) (PrecompilingWithDebugInfo)

Then we do a check for something I didn’t know existed called Optimize Compilations which will not actual affect the hash file value for the following if it is set to true (by default is is false):

the ‘bin’ folder (HttpRuntime.BinDirectoryInternal)
App_WebReferences (HttpRuntime.WebRefDirectoryVirtualPath)
App_GlobalResources (HttpRuntime.ResourcesDirectoryVirtualPath)
App_Code (HttpRuntime.CodeDirectoryVirtualPath)
Global.asax (GlobalAsaxVirtualPath)

Refreshing the ASP.Net temp files (CodeGen files)

The last step of this process is to check if the persisted hash value in the hash.web file equals the generated hash value from the above process. If they do not match then a call is made to diskCache.RemoveAllCodegenFiles(); which will:

clear all codegen files, removes all files in folders but not the folders themselves,
removes all root level files except for temp files that are generated such as .cs files, etc...

This essentially clears your ASP.Net temp files completely, including the MVC controller cache file, etc…

Then the BuildManager simply resaves this calculated has back to the hash.web file.

What is the best way to restart your AppDomain?

There is really no ‘best’ way, it just depends on your needs. If you simply want to restart your AppDomain and not have the overhead of having your app recompile all of your views and other CodeGen classes then it’s best that you simply ‘bump’ your web.config by just adding a space, carriage return or whatever. As you can see above the hash value is not dependant on your local web.config file’s definition (timestamp, etc…). However, the hash value is dependent on some other stuff in your apps configuration such as the <compilation> section, system.web/profile section, the file encoding configured, and the <trust> section. If you update any value in any of these sections in your web.config it will force the BuildManager to clear all cached CodeGen files which is the same as clearing your ASP.Net temp files.

So long as you don’t have optimizeCompilations set to true, then the easiest way to clear your CodeGen files is to simply add a space or carriage return to your global.asax file or modify something in the 4 other places that the BuildManager checks locally: App_Code, App_GlobalResources, App_WebResources, or modify/add/remove a file in your bin folder.

How does this affect the ASP.Net plugin engine?

Firstly, i need to update that post as the code in the plugin engine has changed quite a bit but you can find the latest in the Umbraco source code on CodePlex. With the above knowledge its easy to clear out any stale plugins by perhaps bumping your global.asax file, however this is still not an ideal process. With the research I’ve done in to the BuildManager I’ve borrowed some concepts from it and learned of a special “.delete” file extension that the BuildManager looks for during AppDomain restarts. With this new info, I’ve already committed some new changes to the PluginManager so that you shouldn’t need to worry about stale plugin DLLs.

Consume Json REST service with WCF and dynamic object response

Fri, 11 Feb 2011 22:26:00 Z

This post was imported from FARMCode.org which has been discontinued. These posts now exist here as an archive. They may contain broken links and images.

This was my first shot at WCF and though I saw how nice it could be, i failed miserably… well, I’m actually not too sure. I tried all sorts of things but no matter what couldn’t get WCF to behave and return the actual response you would get if you just consumed the service with the WebClient object. Here’s what I tried and assumed that this ‘should’ work but the result was always an empty dictionary:

Note: this demo is based off of the @Task project management tool’s REST API.

Define the contract:

[ServiceContract]
public interface IAtTaskService
{        
    [OperationContract]
    [WebGet(
        UriTemplate = "/api/login?username={username}&password={password}", 
        RequestFormat = WebMessageFormat.Json)]
    IDictionary<string, object> Login(string username, string password);
}

Run the code:

public IDictionary<string, object> DoThis(string username, string pword)
{
    using (var cf = new WebChannelFactory<IAtTaskService>(
        new Uri("https://mycompaniesurl.attask-ondemand.com/attask/")))
    {
        var s = cf.CreateChannel();
        var token = s.Login(username, pword);
        return token;
    } 
}

The above seemed really nice to be able to consume services but the result was always empty. I tried using strongly typed objects in the result, made sure it wasn’t an HTTPS issue, used the normal WCF configuration approach, etc… but couldn’t get it to work.

The ‘Fix’

I quite like the way that WCF has UriTemplates and you can just attribute up an interface and then make strongly typed calls to access your services so i decided I’d just make that work with my own implementation. Again, perhaps I was ‘doing it wrong’ or whatever, but the below implementation is pretty cool and also lets you get JSON results as a dynamic object :)

Define the contract:

This is pretty much the same as above, but we have a dynamic response

[ServiceContract]
public interface IAtTaskService
{        
    [OperationContract]
    [WebGet(
        UriTemplate = "/api/login?username={username}&password={password}", 
        RequestFormat = WebMessageFormat.Json)]
    dynamic Login(string username, string password);
}

The JsonRestService base class

I created a class that lets you specify you contract service as a generic argument and then provides you with a method called “Send” which is defined a:

protected dynamic Send(Func<T, dynamic> func)

This class will manage all of the http requests and use the WCF attributes applied to your contract to generate the correct URLs and parameters (source code at the bottom of this article)

Define a standard service

So to consume the REST services, we just create a ‘real’ service class such as:

public class AtTaskService : JsonRestService<IAtTaskService>, IAtTaskService
{
    public AtTaskService(string serviceUrl)
        : base(serviceUrl)
    {
    }

    public dynamic Login(string username, string password)
    {
        return Send(x => x.Login(username, password));
    }
}

As you can see, the “Login” method just calls the underlying “Send” method which uses a strongly typed delegate. The Send method then inspects the delegate and generates the correct URL with parameters based on the WCF attributes.

Using the service

So now, to use the new service we can do something like:

[TestMethod]
public void AtAtask_Login()
{

    //Arrange
    var svc = new AtTaskService("https://mycompanyname.attask-ondemand.com/attask/");
            
    //Act
    var token = svc.Login("MyUserName", "mypassword");
            
    //Assert
    Assert.IsTrue(token != null);
    Assert.IsTrue(token.data != null);
    Assert.IsTrue(token.data.userID != null);        
}

Too easy!

As I mentioned earlier, I would have assumed that WCF should handle this out of the box, but for the life of my I couldn’t get it to return any results. Whether it was a request issue or a parsing issue, or whatever i have no idea. In the meantime, here’s the source code for the JsonRestService class which will allow you to do the above. The source requires references to Json.Net and Aaron-powell.dynamics

JsonRestService source

Disclaimer: I made this today in a couple of hours, I’m sure there’s some code in here that could be refactored to be nicer

public class JsonRestService<T>
{
    public string ServiceUrl { get; private set; }

    public JsonRestService(string serviceUrl)
    {
        ServiceUrl = serviceUrl;
    }

    /// <summary>
    /// Queries the WCF attributes of the method being called, builds the REST URL and sends the http request
    /// based on the WCF attribute parameters. When the JSON response is returned, it is changed to a dynamic object.
    /// </summary>
    /// <param name="func"></param>
    /// <returns></returns>
    protected dynamic Send(Func<T, dynamic> func)
    {
        //find the method on the main type that is being called... i'm sure there's a better way to do this,
        //but this does work.
        //This will not work if there are methods with overloads.
        var methodNameMatch = Regex.Match(func.Method.Name, "<(.*?)>");
        if (!methodNameMatch.Success && methodNameMatch.Groups.Count == 2)
        {
            throw new MissingMethodException("Could not find method " + func.Method.Name);
        }
        var realMethodName = methodNameMatch.Groups[1].Value;
        var m = typeof(T).GetMethods().Where(x => x.Name == realMethodName).SingleOrDefault();
        if (m == null)
        {
            throw new MissingMethodException("Could not find method" + realMethodName + " on type " +
                                                typeof(T).FullName);
        }

        //now that we have the method, find the wcf attributes
        var a = m.GetCustomAttributes(false);
        var webGet = a.OfType<WebGetAttribute>().SingleOrDefault();
        var webInvoke = a.OfType<WebInvokeAttribute>().SingleOrDefault();
        var httpMethod = webGet != null ? "GET" : webInvoke != null ? webInvoke.Method : string.Empty;
        if (string.IsNullOrEmpty(httpMethod))
        {
            throw new ArgumentNullException("The WebGet or WebInvoke attribute is missing from the method " +
                                            realMethodName);
        }

        //now that we have the WCF attributes, build the REST url based on the url template and the
        //method being called with it's parameters
        var urlTemplate = webGet != null ? webGet.UriTemplate : webInvoke.UriTemplate;
        var urlWithParams = GetUrlWithParams(urlTemplate, func);
        var url = ServiceUrl + urlWithParams;

        //Do the web requests
        string output;
        if (httpMethod == "GET")
        {
            output = HttpGet(url);
               
        }
        else
        {
            //need to move the query string params to the http parameters
            var parts = url.Split('?');
            output = HttpInvoke(parts[0], httpMethod, parts.Length > 1 ? parts[1] : string.Empty);
        }

        //change the response to json and then dynamic
        var stringReader = new StringReader(output);
        var jReader = new JsonTextReader(stringReader);
        var jsonSerializer = new JsonSerializer();
        var json = jsonSerializer.Deserialize<Dictionary<string, object>>(jReader);
        return json.AsDynamic();
    }

    /// <summary>
    /// Updates the url template with the correct parameters
    /// </summary>
    /// <param name="template"></param>
    /// <param name="expression"></param>
    /// <returns></returns>
    private static string GetUrlWithParams(string template, Func<T, dynamic> expression)
    {
        //parse the template, get the matches
        foreach (var m in Regex.Matches(template, @"\{(.*?)\}").Cast<Match>())
        {
            if (m.Groups.Count == 2)
            {
                var m1 = m;
                //find the fields based on the expression(Func<T>), get their values and replace the tokens in the url template
                var field = expression.Target.GetType().GetFields().Where(x => x.Name == m1.Groups[1].Value).Single();
                template = template.Replace(m.Groups[0].Value, field.GetValue(expression.Target).ToString());
            }
        }
        return template;
    }

    /// <summary>
    /// Do an Http GET
    /// </summary>
    /// <param name="uri"></param>
    /// <returns></returns>
    private static string HttpGet(string uri)
    {
        var webClient = new WebClient();
        var data = webClient.DownloadString(uri);
        return data;
    }

    /// <summary>
    /// Do an Http POST/PUT/etc...
    /// </summary>
    /// <param name="uri"></param>
    /// <param name="method"></param>
    /// <param name="parameters"></param>
    /// <returns></returns>
    private static string HttpInvoke(string uri, string method, string parameters)
    {
        var webClient = new WebClient();
        var data = webClient.UploadString(uri, method, parameters);
        return data;
    }
}

Vertically centring multi-line text with CSS

Thu, 18 Nov 2010 22:34:00 Z

This post was imported from FARMCode.org which has been discontinued. These posts now exist here as an archive. They may contain broken links and images.

So you have a fixed height container, and you want your text to be vertically centred, whether it be one line or many lines of text. In every browser. This is possible, as I just learned from this article.

For all modern browsers, and IE8 and IE9, the method uses display:table-cell. For IE6 and IE7, the method described in the above article is used, which takes advantage of a height bug.

You HTML looks like this:

<div id="container">
	<div id="wrapper">
		<div id="content">
			Stuff and things
		</div>
	</div>
</div>

Your CSS looks like this:

#container
{
	width:200px;
	height:50px;
	background-color:red;
	*position:relative;
}
#wrapper
{
	background-color:green;
	height:inherit;
	display:table-cell;
	vertical-align:middle;
	*height:auto;
	*position:absolute;
	*top:50%;
}
#content
{
	background-color:blue;
	*position:relative;
	*top:-50%;
	color:white;
}

The result looks like this:

IE6

IE7

IE8

IE9 Beta

Google Chrome

And of course, this is not just constrained to text. #content can be any variable-height block...

Mixing percentage and pixel dimensions in CSS

Tue, 09 Nov 2010 21:32:00 Z

This post was imported from FARMCode.org which has been discontinued. These posts now exist here as an archive. They may contain broken links and images.

Okay, so there is no trick to actually compute these dimensions (except in IE). However, there is a little known technique in CSS which is supported in all browsers and solves most of these problems.

Most usually, the desire to mix the two units comes about when one wants column A to fill the page except for column B, which has a fixed pixel width. The problem becomes more formidable when doing the same thing with rows, the reason being that while a block element will naturally expand to fit its container’s width, there is no parallel behaviour for height.

The misconception: when positioning absolutely, two positions and two dimensions should be set. I.E., top, left, width, height.

The truth: many combinations of position and dimension may be used. Want to fill the page except for a 300px bar on the right? Sure:

html, body {
    height: 100%;
}

#content {
    position:absolute;
    top:0;
    right:300px;
    bottom:0;
    left:0;
}

Turns out you don't even need to specify dimensions, they can be implied by positions. This is actually really old-school stuff which is supported by IE of old, back when it couldn't deal with flowing layouts very well. If you want to start combining percentages with pixels, just mess around with this technique and some containers.

Injecting JavaScript into other frames

Thu, 09 Sep 2010 20:44:00 Z

The beautiful part of JavaScript is that it is ridiculously flexible and lets you do things that ‘probably’ shouldn’t be done. Here’s a good example of that.

During uComponents development I stumbled upon a situation where I needed to attach a JavaScript method to the top-level frame from inside of an iframe. Well in fact it turns out this is quite easy, you can do something like this:

window.top.doThis = function() { alert("woot!"); }

However, since we’re attaching the ‘doThis’ method to the main frame from an inner iframe, when the inner iframenavigates to another page, this function will no longer exist on the main frame… So this clearly isn’t going to work if we want to be able to call the ‘doThis’ method from the inner frame no matter when and where it navigates to… Conundrum!

So the next possibility is to try to just inject a script block into the main frame from the iframewhich actually does work in Firefox and Chrome but fails in Internet Explorer and Safari. (This snippet of code requires that you have jQuery loaded in the main frame)

var js = "function doThis() { alert('woot!'); }";
var injectScript = window.top.$('<script>')
        .attr('type', 'text/javascript')
        .html(js);
window.top.$("head").append(injectScript);

In the above, we’re creating a string function, creating a <script> block with jQuery, appending the string function to the script block and then appending this script block to the <head> element of the main frame. But as i said before, Firefox and Chrome are ok with this but Internet Explorer and Safari will throw JavaScript exceptions such as: Unexpected call to method or property access

Ok, so unless you don’t want to be cross browser, this isn’t going to work. It took me a while to figure out that you can do this, but this does work. Yes it looks pretty dodgy, and it probably is. In reality, attempting to do something like this is pretty dodgy to begin with. So here it is (this works in Internet Explorer 8 (probably earlier ones too), Firefox 3.6 (probably earlier ones too) and Chrome 5 (probably earlier ones too) and i didn’t get around to testing Safari but I am assuming it works):

var iframe = window.top.$("#dummyIFrame");
if (iframe.length == 0) {
    var html = "<html><head><script type='text/javascript'>"
                + "this.window.doThis = function() { alert('woot'); };"
                + "</script></head><body></body></html>";
    iframe = window.top.$("<iframe id='dummyIFrame'>")
                .append(html)
                .hide()
                .css("width", "0px")
                .css("height", "0px");
    window.top.$("body").append(iframe);
}

So i guess this requires a bit of explanation. All browsers seem to let you create iframes dynamically which also means that you can put whatever content into the iframes while it’s being created, including script blocks. Here’s what we’re doing:

checking if our ‘dummy’ iframe already exists (since we don’t want to create multiple dummy iframes since we only need one), if it doesn't:
create an html text block including the script that will attach the ‘doThis’ method to the ‘this.window’ object (which for some reason will be referring to the window.top object)
next we create an iframe element and append the html text block, and then make sure the iframe is completely hidden
finally, we append the iframe to the main window’s body element

window.top.doThis();

Nice! So this pretty much means that you can create code from an inner frame and attach it to a different frame, then have that code run in the context of the main frame with the main frame’s objects and script set.

One last thing that i found out you can do, though i wouldn’t recommend it because i think it might start filling up your memory. But this is also possible:


var html = "<html><head><script type='text/javascript'>"
            + "this.window.doThis = function() { alert('woot'); };"
            + "this.window.doThis();"
            + "</script></head><body></body></html>";
iframe = window.top.$("<iframe id='dummyIFrame'>")
            .append(html);

All that is happening here is that I’m attaching the ‘doThis’ method to the main frame’s window object, calling it directly after and then creating an iframe in memory with this script block. The funny part is that the method executes straight away and I haven’t attached the iframe to the DOM anywhere!

VirtualPathUtility Fixed in .Net 4

Mon, 30 Aug 2010 22:13:00 Z

If you are not already aware there’s an issue with the VirtualPathUtility object in the .Net framework. If you are trying to use the IsAbsolute or ToAbsolute methods with a path that contains a query string, you’ll get an exception thrown. A quick Google search on this issue will show you a ton of posts and ‘work arounds’ for this problem and I’ve implemented a custom work around for this in the ClientDependency framework as well. In fact, I only noticed this problem when testing the ClientDependency framework with .Net 3.5 and trying to register embedded resources through it (as see in this blog post here). Since I’d been developing the latest builds of ClientDependency in .Net 4, i had no problems at all but as soon as I changed the build to .Net 3.5, the exceptions began to show since web resource URLs have query strings.

So it the good new is that it appears as though Microsoft have finally fixed this bug in .Net 4! The bad news is that you’ll still have to use your ‘work arounds’ if you are targeting framework versions below 4.

How to embed ASMX web services in a DLL

Fri, 13 Aug 2010 00:01:00 Z

The development of uComponents for Umbraco is still ongoing but is probably close to a v1.0 release. During the development of the Multi-node tree picker data type, I’ve spotted a bug in the Umbraco core in regards to the ASMX web service the existing tree uses to render out it’s initial node. The bug basically won’t allow the tree to render an individual tree, only a full app. Since the Multi-node tree picker requires rendering of an individual tree, i needed to fix this. To do this was a bit of a work around but i had to override the tree’s ASMX web service with my own (I’ll write a post about this one later). It then occurred to me that I’ve never embedded an ASMX web service into a DLL without a physical file and I didn’t want to go updating the web.config with custom handlers, etc… So I came up with a way to embed ASMX web services in your DLL without modifying web.config’s, etc… here’s how:

First, create your web service as a standard class and add all of the functionality that your require your web service to have:

In your class library project, create a new text file but give it the extension .asmx:

Set the file to be a non compiled file:

Create an resource file:

Double click on your new RESX file, then drag your ASMX file into the editor to add the file as a property of your RESX file:

Double click your ASMX file to edit it and set it to inherit from your custom web service class:

Now, we need to register this ASMX web service into the web application that it’s being deployed to. This is easily done by just checking if the ASMX file exists where it needs to be before it needs to be called and if it doesn’t, we just create the file. In the case of uComponents and the multi-node tree picker requires the ASMX web service for the CustomTreeControl, so we’ll do this check in the CustomTreeControl’s static constructor:

public CustomTreeControl()
{
    var filePath = HttpContext.Current
                    .Server.MapPath("~/CustomTreeService.asmx");
            
    //check if it exists
    if (!File.Exists(filePath))
    {
        //lock the thread
        lock (m_Locker)
        {
            //double check locking....
            if (!File.Exists(filePath))
            {
                //now create our new local web service
                using (var sw = new StreamWriter(File.Create(filePath)))
                {
                    //write the contents of our embedded resource to the file
                    sw.Write(CustomTreeServiceResource.CustomTreeService);
                }
            }
        }
    }
}

That’s it!

.Net Open Source Blog Comparison

Wed, 11 Aug 2010 11:09:00 Z

When I started creating this site this morning, the first thing i though to myself was that I wasn’t going to use BlogEngine.Net for yet another blog. So I went on a mission to find some other open source blogging systems written in ASP.Net. One of the things I needed the codebase to support was to work seamlessly with Live Writer, so after some quick research I found that the new Orchard project support Live Writer and so does SubText. So using WebPI, I decided to give them each a quick test run.

I’m sure the Umbracians out there are asking: Why wouldn’t you just use Blog 4 Umbraco considering I’m on the core development team. Well, i guess it’s because all I really want is a very simple blogging tool and really just wanted to try out some new tools.

So I started with Orchard and was kind of excited to see what it offered since it’s newcomer to the CMS scene and has a bit of hype behind it. The admin UI was nice and is was very easy to install. Setting up a new blog was pretty straight forward as well… just click on Create Blog. It' also has a couple nice skins you can test out. So far so good (even though this is still a alpha/beta product). So I tried plugging in Live Writer to it and it turns out that Orchard really hasn’t implemented some of the much needed features of the MetaWeblog API such as categories & tags! ouch. On a side note, i tried looking at creating new content types in Orchard, thinking it would be similar to how Umbraco works. Turns out it’s not quite the same and looks as though this functionality is in its infancy in Orchard. There was really only one data type to choose from which was text and it appeared as though i couldn’t delete a content type either. We’ll see how the project looks in a few months.

Next I went on to SubText. This was also pretty easy to install (not as easy as Orchard though) but the admin UI isn’t so pretty. SubText has pretty much every feature that you could want out of a blogging system and comes with a bunch of skins as well. One thing i noticed which looked pretty cool is that you can add text filters so the SubText will automatically create links for you in your posts. So I plugged in Live Writer and everything is pretty good, except that they haven’t implemented tagging in their MetaWeblog API either.

As you’ve probably noticed, I’ve settled back on BlogEngine.Net. It has better Live Writer integration than either of these other two blogging systems, the admin UI is nice enough and it’s pretty damn easy to use. I really wish they’d implement non ASPX page extensions sometime soon though!

And it begins

Wed, 11 Aug 2010 10:22:00 Z

For the last couple of years I've been blogging on FARMCode.org and today decided that I was going to start up my own blog as well. This doesn't mean that I won't still be blogging on FARMCode, it just means that I have a new place to post up information that may not directly relate to TheFARM.

In the coming weeks I'll start posting up some new content, code snippets, some info on the up and coming Umbraco project we've been working on called uComponents and maybe some portfolio work as well.