@Shazwazza

Shannon Deminick's blog all about web development

Auto upgrade your Nuget packages with Azure Pipelines or GitHub Actions

February 25, 2021 06:12
Auto upgrade your Nuget packages with Azure Pipelines or GitHub Actions

Before we start I just want to preface this with some 🔥 warnings 🔥

  • This works for me, it might not work for you
  • To get this working for you, you may need to tweak some of the code referenced
  • This is not under any support or warranty by anyone
  • Running Nuget.exe update command outside of Visual Studio will overwrite your files so there is a manual review process (more info below)
  • This is only for ASP.NET Framework using packages.config – Yes I know that is super old and I should get with the times, but this has been an ongoing behind the scenes project of mine for a long time. When I need this for Package Reference projects, ASP.NET Core/5, I’ll update it but there’s nothing stopping you from tweaking this to work for you
  • This only works for a specified csproj, not an entire sln – it could work for that but I’ve not tested, there would be a few tweaks to make that work
  • This does not yet work for GitHub actions but the concepts are all here and could probably very easily be converted UPDATE: This works now!

Now that’s out of the way …

How do I do it?

With a lot of PowerShell :) This also uses a few methods from the PowerShellForGitHub project.

The process is:

  • Run a pipeline/action on a schedule (i.e. each day)
  • This checks against your source code for the installed version for a particular package
  • Then it checks with Nuget (using your Nuget.config file) to see what the latest stable version is
  • If there’s a newer version:
  • Create a new branch
  • Run a Nuget update against your project
  • Build the project
  • Commit the changes
  • Push the changes
  • Create a PR for review

Azure Pipelines/GitHub Actions YAML

The only part of the YAML that needs editing is the variables, here's what they mean:

  • ProjectFile = The relative path to your csproj that you want to upgrade
  • PackageFile = The relative path to your packages.config file for this project
  • PackageName = The Nuget package name you want upgraded
  • GitBotUser = The name used for the Git commits
  • GitBotEmail = The email used for the Git commits

For Azure Pipelines, these are also required:

Then there are some variables to assist with testing:

  • DisableUpgradeStep = If true will just check if there’s an upgrade available and exit
  • DisableCommit = If true will run the upgrade and will exit after that (no commit, push or PR)
  • DisablePush = If true will run the upgrade + commit and will exit after that (no push or PR)
  • DisablePullRequest = If true will run the upgrade + commit + push and will exit after that (no PR)

Each step in the yaml build more or less either calls Git commands or PowerShell functions. The PowerShell functions are loaded as part of a PowerShell Module which is committed to the repository. This module’s functions are auto-loaded by PowerShell because the first step configures the PowerShell environment variable PSModulePath to include the custom path. Once that is in place, all functions exposed by the module are auto-loaded.

In these examples you’ll see that I’m referencing Umbraco Cloud names and that’s because I’m using this on Umbraco Cloud for my own website and the examples are for the UmbracoCms package. But this should in theory work for all packages!

Show me the code

The code for all of this is here in a new GitHub repo and here’s how you use it:

You can copy the folder structure in the repository as-is. Here's an example of what my site's repository folder structure is to make this work (everything except the src folder is in the GitHub repo above):

  • [root]
    • auto-upgrader.devops.yml (If you are using Azure Pipelines)
    • .github
      • workflows
        • auto-upgrader.gh.yml (If you are using GitHub Actions)
    • build
      • PowershellModules
        • AutoUpgradeFunctions.psd1
        • AutoUpgradeFunctions.psm1
        • AutoUpgradeFunctions
    • src
      • Shazwazza.Web
        • Shazwazza.Web.csproj
        • packages.config

All of the steps have descriptive display names and it should be reasonably self documenting.

The end result is a PR, here’s one that was generated by this process:

Nuget overwrites

Nuget.exe works differently than Nuget within Visual Studio’s Package Manager Console. All of those special commands like Install-Package, Update-Package, etc… are all PowerShell module commands built into Visual Studio and they are able to work with the context of Visual Studio. This allows those commands to try to be a little smarter when running Nuget updates and also allows the legacy Nuget features like running PowerShell scripts on install/update to run. This script just uses Nuget.exe and it’s less smart especially for these legacy .NET Framework projects. As such, it will just overwrite all files in most cases (it does detect file changes it seems but isn’t always accurate).

With that 🔥 warning 🔥 it is very important to make sure you review the changed files in the PR and revert or adjust any changes you need before applying the PR.

You’ll see a note in the PowerShell script about Nuget overwrites. There are other options that can be used like "Ignore" and "IgnoreAll" but all my tests have showed that for some reason this setting will end up deleting a whole bunch of files so the default overwrite setting is used.

Next steps

Get out there and try it! Would love some feedback on this if/when you get a change to test it.

PackageReference support with .NET Framework projects could also be done (but IMO this is low priority) along with being able to upgrade the entire SLN instead of just the csproj.

Then perhaps some attempts at getting a NET Core/5 version of this running. In theory that will be easier since it will mostly just be dotnet commands.

 

Web Application projects with Umbraco Cloud

January 8, 2020 05:12
Web Application projects with Umbraco Cloud

This is a common topic for developers when working with Umbraco Cloud because Umbraco Cloud simply hosts an ASP.Net Framework “Website”. The setup is quite simple, a website is stored in a Git repository and when it’s updated and pushed to Umbraco Cloud, all of the changes are live. You can think of this Git repository as a deployment repository (which is very similar to how Azure Web Apps can work with git deployments). When you create a new Umbraco Cloud site, the git repository will be pre-populated with a runnable website. You can clone the website and run it locally with IIS Express and it all just works. But this is not a compile-able website and it’s not part of a visual studio project or a solution and if you want to have that, there’s numerous work arounds that people have tried and use but in my personal opinion they aren’t the ideal working setup that I would like.

Ideal solution

In my opinion the ideal solution for building web apps in .NET Framework is:

  • A visual studio solution
    • A compile-able Web Application project (.csproj)
    • Additional class library projects (as needed)
    • Unit/Integration test projects (as needed)
    • All dependencies are managed via Nuget
  • Git source control for my code, probably stored in GitHub
  • A build server, CI/CD, I like Azure Pipelines

I think this is a pretty standard setup for building websites but trying to wrangle this setup to work with Umbraco Cloud isn’t as easy as you’d think. A wonderful Umbraco community member Paul Sterling has written about how to do this a couple of times, here and here and there’s certainly a few hoops you’d need to jump through. These posts were also written before the age of Azure YAML Pipelines which luckily has made this process a whole lot easier

Solution setup

NOTE: This is for Umbraco v8, there’s probably some other edge cases you’ll need to discover on your own for v7. 

Setting up a Visual Studio solution with a web application compatible for Umbraco Cloud is pretty straight forward and should be very familiar. It will be much easier to do this starting from scratch with a new Umbraco Cloud website though it is more than possible to do this for an existing website (i.e. I did this for this website!) but most of those details are just migrating custom code, assets, etc… to your new solution.

I would suggest starting with a new Umbraco Cloud site that has no modifications to it but does have a content item or two that renders a template.

  • Create a new VS solution/project for a web application running .NET 4.7.2
  • Add this Nuget.config to the root folder (beside your .sln file)
    • <?xml version="1.0" encoding="utf-8"?>
      <configuration>
        <packageSources>
      	<add key="NuGet" value="https://api.nuget.org/v3/index.json" />
          <add key="UmbracoCloud" value="https://www.myget.org/F/uaas/api/v3/index.json" />
        </packageSources>
      </configuration>
  • Install the Nuget package for the same Umbraco version that you are currently running on your Umbraco Cloud website. For example if you are running 8.4.0 then use Install-Package UmbracoCms –Version 8.4.0
  • Install Forms (generally the latest available): Install-Package UmbracoForms
  • Install Deploy (generally the latest available):
    • Install-Package UmbracoDeploy
    • Install-Package UmbracoDeploy.Forms
    • Install-Package UmbracoDeploy.Contrib
  • Then you’ll need to install some additional Nuget packages that are required to run your site on Umbraco Cloud. This is undocumented but Umbraco Cloud adds a couple extra DLLs when it creates a website that are required.
    • Install-Package Serilog.Sinks.MSSqlServer -Version 5.1.3-dev-00232
  • Copy these files from your Umbraco Cloud deployment repository to your web application project:
    • ~/data/*
    • ~/config/UmbracoDeploy.config
    • ~/config/UmbracoDeploy.Settings.config
  • You then need to do something weird. These settings need to be filled in because Umbraco Deploy basically circumvents the normal Umbraco installation procedure and if you don’t have these settings populated you will get YSODs and things won’t work.
    • Make sure that you have your Umbraco version specified in your web.config like: <add key="Umbraco.Core.ConfigurationStatus" value="YOURVERSIONGOESHERE" />
    • Make sure your connectionStrings in your web.config is this:
      • <connectionStrings>
            <remove name="umbracoDbDSN" />
            <add name="umbracoDbDSN"
                 connectionString="Data Source=|DataDirectory|\Umbraco.sdf"
                 providerName="System.Data.SqlServerCe.4.0" />
        </connectionStrings>

But I don’t want to use SqlCE! Why do I need that connection string? In actual fact Umbraco Deploy will configure your web application to use Sql Express LocalDb if it’s available on your machine (which it most likely is). This is why when running Umbraco Cloud sites locally you’ll see .mdf and .ldf files in your App_Data folder instead of SqlCE files. Local Db operates just like Sql Server except the files are located locally, it’s really sql server under the hood. You can even use Sql Management Studio to look at these databases by connecting to the (localdb)\umbraco server locally with Windows Authentication. It is possible to have your local site run off of a normal Sql server database with a real connection string but I think you’d have to install Umbraco first before you install the UmbracoDeploy nuget package. Ideally UmbracoDeploy would allow the normal install screen to run if there was no Umbraco version detected in the web.config, but that’s a whole other story.

That should be it! In theory your web application is now configured to be able to publish a website output that is the same as what is on Umbraco Cloud.

Installation

At this stage you should be able to run your solution, it will show the typical Umbraco Deploy screen to restore from Cloud

image

In theory you should be able to restore your website and everything should ‘just work’

Working with code

Working with your code is now just the way you’re probably used to working. Now that you’ve got a proper Visual Studio solution with a Web Application Project, you can do all of the development that you are used to. You can add class libraries, unit test projects, etc… Then you commit all of these changes to your own source control like GitHub. This type of repository is not a deployment repository, this is a source code repository.

How do I get this to Umbraco Cloud?

So far there’s nothing too special going on but now we need to figure out how to get our Web Application Project to be deployed to Umbraco Cloud.

There’s a couple ways to do this, the first way is surprisingly simple:

  • Right click your web application project in VS
  • Click Publish
  • Choose Folder as a publish target
  • Select your cloned Umbraco Cloud project location
  • Click advanced and choose “Exclude files from App_Data folder’
  • Click Create Profile
  • Click Publish – you’ve just published a web application project to a website
  • Push these changes to Umbraco Cloud

The publish profile result created should match this one: https://github.com/umbraco/vsts-uaas-deploy-task/blob/master/PublishProfiles/ToFileSys.pubxml

This of course requires some manual work but if you’re ok with that then job done!

You should do this anyways before continuing since it will give you an idea of how in-sync your web application project and the output website is to the Umbraco Cloud website, you can then see what Git changes have been made and troubleshoot anything that might seem odd.

Azure Pipelines

I’m all for automation so instead I want Azure Pipelines to do my work. This is what I want to happen:

  • Whenever I commit to my source code repo Azure Pipelines will:
    • Build my solution
    • Run any unit tests that I have
    • Publish my web application project to a website
    • Zip the website
    • Publish my zipped website artifact
  • When I add a “release-*” tag to a commit I want Azure Pipelines to do all of the above and also:
    • Clone my Umbraco Cloud repository
    • Unzip my website artifact onto this cloned destination
    • Commit these changes to the Umbraco Cloud deployment repository
    • Push this commit to Umbraco Cloud

Luckily this work is all done for you :) and with YAML pipelines it’s fairly straight forward. Here’s how:

  • Go copy this PowerShell file and commit it to the /build folder of your source code repository (our friends Paul Sterling and Morten Christensen had previously done this work, thanks guys!). This PS script essentially does all of that Git work mentioned above, the cloning, committing and pushing files. It’s a bit more verbose than just running these git comands directly in your YAML file but it’s also a lot less error prone and handles character encoding properly along with piping the output of the git command to the log.
  • Go copy this azure-pipelines.yml file and commit it to the root of your git source code repository. This file contains a bunch of helpful notes so you know what it’s doing. (This pipelines file does run any tests, etc… that exercise will be left up to you.)
  • In Azure Pipelines, create a new pipeline, choose your Git source control option, choose “Existing Azure Pipelines YAML file”, select azure-pipelines.yml file in the drop down, click continue.
  • Click Variables and add these 3:
    • gitAddress = The full Git https endpoint for your Dev environment on Umbraco Cloud
    • gitUsername = Your Umbraco Cloud email address
    • gitPassword = Your Umbraco Cloud password - ensure this value is set to Secret
  • Click Run!

And that’s it! … Really? … In theory yes :)

Your pipeline should run and build your solution. The latest commit you made is probably the azure-pipelines.yml files so it didn’t contain a release-* tag so it’s not going to attempt to push any changes to Umbraco Cloud. So first thing to do is make sure that your your pipeline is building your solution and doing what its supposed to. Once that’s all good then it’s time to test an Umbraco Cloud deployment.

Deploying to Umbraco Cloud

A quick and easy test would be to change the output of a template so you can visibly see the change pushed.

  • Go ahead and make a change to your home page template
  • Run your site locally with your web application project and make sure the change is visible there
  • Commit this change to your source control Git repository
  • Create and push a release tag on this commit. For example, the tag name could be: “release-v1.0.0-beta01” … whatever suites your needs but based on the YAML script it needs to start with “release-“

Now you can sit back and watch Azure Pipelines build your solution and push it to Umbraco Cloud. Since this is a multi-stage pipeline, the result will look like:

image

And you should see a log output like this on the Deploy stage

image

Whoohoo! Automated deployments to Umbraco Cloud using Web Application Projects.

What about auto-upgrades?

All we’ve talked about so far is a one-way push to Umbraco Cloud but one thing we know and love about Umbraco Cloud is the automated upgrade process. So how do we deal with that? I actually have this working on my site but want to make the process even simpler so you’re going to have to be patient and wait for another blog post :)

The way it works is also using Azure Pipelines. Using a separate pipeline with a custom Git repo pointed at your Umbraco Cloud repository, this pipeline can be configured to poll for changes every day (or more often if you like). It then checks if changes have been made to the packages.config file to see if there’s been upgrades made to either the CMS, Forms or Deploy (in another solution I’m actually polling Nuget directly for this information). If an upgrade has been made, It clones down your source code repository, runs a Nuget update command to upgrade your solution. Then it creates a new branch, commits these changes, pushes it back GitHub and creates a Pull Request (currently this only works for GitHub).

This same solution can be used for Deploy files in case administrators are changing schema items directly on Umbraco Cloud so the /deploy/* files can be automatically kept in sync with your source code repository.

This idea is entirely inspired by Morten Christensen, thanks Morten! Hopefully I’ll find some time to finalize this.

Stay tuned!