Shazwazza

Auto upgrade your Nuget packages with Azure Pipelines or GitHub Actions

Thu, 23 Mar 2023 15:10:22 Z

Before we start I just want to preface this with some warnings

This works for me, it might not work for you
To get this working for you, you may need to tweak some of the code referenced
This is not under any support or warranty by anyone
Running Nuget.exe update command outside of Visual Studio will overwrite your files so there is a manual review process (more info below)
This is only for ASP.NET Framework using packages.config – Yes I know that is super old and I should get with the times, but this has been an ongoing behind the scenes project of mine for a long time. When I need this for Package Reference projects, ASP.NET Core/5, I’ll update it but there’s nothing stopping you from tweaking this to work for you
This only works for a specified csproj, not an entire sln – it could work for that but I’ve not tested, there would be a few tweaks to make that work
This does not yet work for GitHub actions but the concepts are all here and could probably very easily be converted UPDATE: This works now!

Now that’s out of the way …

How do I do it?

With a lot of PowerShell :) This also uses a few methods from the PowerShellForGitHub project.

The process is:

Run a pipeline/action on a schedule (i.e. each day)
This checks against your source code for the installed version for a particular package
Then it checks with Nuget (using your Nuget.config file) to see what the latest stable version is
If there’s a newer version:
Create a new branch
Run a Nuget update against your project
Build the project
Commit the changes
Push the changes
Create a PR for review

Azure Pipelines/GitHub Actions YAML

The only part of the YAML that needs editing is the variables, here's what they mean:

ProjectFile = The relative path to your csproj that you want to upgrade
PackageFile = The relative path to your packages.config file for this project
PackageName = The Nuget package name you want upgraded
GitBotUser = The name used for the Git commits
GitBotEmail = The email used for the Git commits

For Azure Pipelines, these are also required:

GitHubOwner = The repository owner (i.e. https://github.com/Shazwazza/Articulate )
GitHubRepository = The repository name (i.e. https://github.com/Shazwazza/Articulate )
There is one Secret setting you need to create in Azure Pipelines: GitHubAccessToken which is a GitHub PAT that you need to create that has access to this repository.

Then there are some variables to assist with testing:

DisableUpgradeStep = If true will just check if there’s an upgrade available and exit
DisableCommit = If true will run the upgrade and will exit after that (no commit, push or PR)
DisablePush = If true will run the upgrade + commit and will exit after that (no push or PR)
DisablePullRequest = If true will run the upgrade + commit + push and will exit after that (no PR)

Each step in the yaml build more or less either calls Git commands or PowerShell functions. The PowerShell functions are loaded as part of a PowerShell Module which is committed to the repository. This module’s functions are auto-loaded by PowerShell because the first step configures the PowerShell environment variable PSModulePath to include the custom path. Once that is in place, all functions exposed by the module are auto-loaded.

In these examples you’ll see that I’m referencing Umbraco Cloud names and that’s because I’m using this on Umbraco Cloud for my own website and the examples are for the UmbracoCms package. But this should in theory work for all packages!

Show me the code

The code for all of this is here in a new GitHub repo and here’s how you use it:

You can copy the folder structure in the repository as-is. Here's an example of what my site's repository folder structure is to make this work (everything except the src folder is in the GitHub repo above):

[root]
- auto-upgrader.devops.yml (If you are using Azure Pipelines)
- .github
  - workflows
    - auto-upgrader.gh.yml (If you are using GitHub Actions)
- build
  - PowershellModules
    - AutoUpgradeFunctions.psd1
    - AutoUpgradeFunctions.psm1
    - AutoUpgradeFunctions
- src
  - Shazwazza.Web
    - Shazwazza.Web.csproj
    - packages.config

All of the steps have descriptive display names and it should be reasonably self documenting.

The end result is a PR, here’s one that was generated by this process:

Nuget overwrites

Nuget.exe works differently than Nuget within Visual Studio’s Package Manager Console. All of those special commands like Install-Package, Update-Package, etc… are all PowerShell module commands built into Visual Studio and they are able to work with the context of Visual Studio. This allows those commands to try to be a little smarter when running Nuget updates and also allows the legacy Nuget features like running PowerShell scripts on install/update to run. This script just uses Nuget.exe and it’s less smart especially for these legacy .NET Framework projects. As such, it will just overwrite all files in most cases (it does detect file changes it seems but isn’t always accurate).

With that warning it is very important to make sure you review the changed files in the PR and revert or adjust any changes you need before applying the PR.

You’ll see a note in the PowerShell script about Nuget overwrites. There are other options that can be used like "Ignore" and "IgnoreAll" but all my tests have showed that for some reason this setting will end up deleting a whole bunch of files so the default overwrite setting is used.

Next steps

Get out there and try it! Would love some feedback on this if/when you get a change to test it.

PackageReference support with .NET Framework projects could also be done (but IMO this is low priority) along with being able to upgrade the entire SLN instead of just the csproj.

Then perhaps some attempts at getting a NET Core/5 version of this running. In theory that will be easier since it will mostly just be dotnet commands.

Web Application projects with Umbraco Cloud

Thu, 23 Mar 2023 15:09:41 Z

This is a common topic for developers when working with Umbraco Cloud because Umbraco Cloud simply hosts an ASP.Net Framework “Website”. The setup is quite simple, a website is stored in a Git repository and when it’s updated and pushed to Umbraco Cloud, all of the changes are live. You can think of this Git repository as a deployment repository (which is very similar to how Azure Web Apps can work with git deployments). When you create a new Umbraco Cloud site, the git repository will be pre-populated with a runnable website. You can clone the website and run it locally with IIS Express and it all just works. But this is not a compile-able website and it’s not part of a visual studio project or a solution and if you want to have that, there’s numerous work arounds that people have tried and use but in my personal opinion they aren’t the ideal working setup that I would like.

Ideal solution

In my opinion the ideal solution for building web apps in .NET Framework is:

A visual studio solution

A compile-able Web Application project (.csproj)
Additional class library projects (as needed)
Unit/Integration test projects (as needed)
All dependencies are managed via Nuget

Git source control for my code, probably stored in GitHub
A build server, CI/CD, I like Azure Pipelines

I think this is a pretty standard setup for building websites but trying to wrangle this setup to work with Umbraco Cloud isn’t as easy as you’d think. A wonderful Umbraco community member Paul Sterling has written about how to do this a couple of times, here and here and there’s certainly a few hoops you’d need to jump through. These posts were also written before the age of Azure YAML Pipelines which luckily has made this process a whole lot easier

Solution setup

NOTE: This is for Umbraco v8, there’s probably some other edge cases you’ll need to discover on your own for v7.

Setting up a Visual Studio solution with a web application compatible for Umbraco Cloud is pretty straight forward and should be very familiar. It will be much easier to do this starting from scratch with a new Umbraco Cloud website though it is more than possible to do this for an existing website (i.e. I did this for this website!) but most of those details are just migrating custom code, assets, etc… to your new solution.

I would suggest starting with a new Umbraco Cloud site that has no modifications to it but does have a content item or two that renders a template.

Create a new VS solution/project for a web application running .NET 4.7.2

Add this Nuget.config to the root folder (beside your .sln file)

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
	<add key="NuGet" value="https://api.nuget.org/v3/index.json" />
    <add key="UmbracoCloud" value="https://www.myget.org/F/uaas/api/v3/index.json" />
  </packageSources>
</configuration>

Install the Nuget package for the same Umbraco version that you are currently running on your Umbraco Cloud website. For example if you are running 8.4.0 then use Install-Package UmbracoCms –Version 8.4.0
Install Forms (generally the latest available): Install-Package UmbracoForms
Install Deploy (generally the latest available):
- Install-Package UmbracoDeploy
- Install-Package UmbracoDeploy.Forms
- Install-Package UmbracoDeploy.Contrib
Then you’ll need to install some additional Nuget packages that are required to run your site on Umbraco Cloud. This is undocumented but Umbraco Cloud adds a couple extra DLLs when it creates a website that are required.
- Install-Package Serilog.Sinks.MSSqlServer -Version 5.1.3-dev-00232
Copy these files from your Umbraco Cloud deployment repository to your web application project:
- ~/data/*
- ~/config/UmbracoDeploy.config
- ~/config/UmbracoDeploy.Settings.config
You then need to do something weird. These settings need to be filled in because Umbraco Deploy basically circumvents the normal Umbraco installation procedure and if you don’t have these settings populated you will get YSODs and things won’t work.
- Make sure that you have your Umbraco version specified in your web.config like: <add key="Umbraco.Core.ConfigurationStatus" value="YOURVERSIONGOESHERE" />
- Make sure your connectionStrings in your web.config is this:
  - ```
  <connectionStrings>
      <remove name="umbracoDbDSN" />
      <add name="umbracoDbDSN"
           connectionString="Data Source=|DataDirectory|\Umbraco.sdf"
           providerName="System.Data.SqlServerCe.4.0" />
  </connectionStrings>
```

But I don’t want to use SqlCE! Why do I need that connection string? In actual fact Umbraco Deploy will configure your web application to use Sql Express LocalDb if it’s available on your machine (which it most likely is). This is why when running Umbraco Cloud sites locally you’ll see .mdf and .ldf files in your App_Data folder instead of SqlCE files. Local Db operates just like Sql Server except the files are located locally, it’s really sql server under the hood. You can even use Sql Management Studio to look at these databases by connecting to the (localdb)\umbraco server locally with Windows Authentication. It is possible to have your local site run off of a normal Sql server database with a real connection string but I think you’d have to install Umbraco first before you install the UmbracoDeploy nuget package. Ideally UmbracoDeploy would allow the normal install screen to run if there was no Umbraco version detected in the web.config, but that’s a whole other story.

That should be it! In theory your web application is now configured to be able to publish a website output that is the same as what is on Umbraco Cloud.

Installation

At this stage you should be able to run your solution, it will show the typical Umbraco Deploy screen to restore from Cloud

In theory you should be able to restore your website and everything should ‘just work’

Working with code

Working with your code is now just the way you’re probably used to working. Now that you’ve got a proper Visual Studio solution with a Web Application Project, you can do all of the development that you are used to. You can add class libraries, unit test projects, etc… Then you commit all of these changes to your own source control like GitHub. This type of repository is not a deployment repository, this is a source code repository.

How do I get this to Umbraco Cloud?

So far there’s nothing too special going on but now we need to figure out how to get our Web Application Project to be deployed to Umbraco Cloud.

There’s a couple ways to do this, the first way is surprisingly simple:

Right click your web application project in VS
Click Publish
Choose Folder as a publish target
Select your cloned Umbraco Cloud project location
Click advanced and choose “Exclude files from App_Data folder’
Click Create Profile
Click Publish – you’ve just published a web application project to a website
Push these changes to Umbraco Cloud

The publish profile result created should match this one: https://github.com/umbraco/vsts-uaas-deploy-task/blob/master/PublishProfiles/ToFileSys.pubxml

This of course requires some manual work but if you’re ok with that then job done!

You should do this anyways before continuing since it will give you an idea of how in-sync your web application project and the output website is to the Umbraco Cloud website, you can then see what Git changes have been made and troubleshoot anything that might seem odd.

Azure Pipelines

I’m all for automation so instead I want Azure Pipelines to do my work. This is what I want to happen:

Whenever I commit to my source code repo Azure Pipelines will:

Build my solution
Run any unit tests that I have
Publish my web application project to a website
Zip the website
Publish my zipped website artifact

When I add a “release-*” tag to a commit I want Azure Pipelines to do all of the above and also:

Clone my Umbraco Cloud repository
Unzip my website artifact onto this cloned destination
Commit these changes to the Umbraco Cloud deployment repository
Push this commit to Umbraco Cloud

Luckily this work is all done for you :) and with YAML pipelines it’s fairly straight forward. Here’s how:

Go copy this PowerShell file and commit it to the /build folder of your source code repository (our friends Paul Sterling and Morten Christensen had previously done this work, thanks guys!). This PS script essentially does all of that Git work mentioned above, the cloning, committing and pushing files. It’s a bit more verbose than just running these git comands directly in your YAML file but it’s also a lot less error prone and handles character encoding properly along with piping the output of the git command to the log.
Go copy this azure-pipelines.yml file and commit it to the root of your git source code repository. This file contains a bunch of helpful notes so you know what it’s doing. (This pipelines file does run any tests, etc… that exercise will be left up to you.)
In Azure Pipelines, create a new pipeline, choose your Git source control option, choose “Existing Azure Pipelines YAML file”, select azure-pipelines.yml file in the drop down, click continue.
Click Variables and add these 3:

gitAddress = The full Git https endpoint for your Dev environment on Umbraco Cloud
gitUsername = Your Umbraco Cloud email address
gitPassword = Your Umbraco Cloud password - ensure this value is set to Secret

Click Run!

And that’s it! … Really? … In theory yes :)

Your pipeline should run and build your solution. The latest commit you made is probably the azure-pipelines.yml files so it didn’t contain a release-* tag so it’s not going to attempt to push any changes to Umbraco Cloud. So first thing to do is make sure that your your pipeline is building your solution and doing what its supposed to. Once that’s all good then it’s time to test an Umbraco Cloud deployment.

Deploying to Umbraco Cloud

A quick and easy test would be to change the output of a template so you can visibly see the change pushed.

Go ahead and make a change to your home page template
Run your site locally with your web application project and make sure the change is visible there
Commit this change to your source control Git repository
Create and push a release tag on this commit. For example, the tag name could be: “release-v1.0.0-beta01” … whatever suites your needs but based on the YAML script it needs to start with “release-“

Now you can sit back and watch Azure Pipelines build your solution and push it to Umbraco Cloud. Since this is a multi-stage pipeline, the result will look like:

And you should see a log output like this on the Deploy stage

Whoohoo! Automated deployments to Umbraco Cloud using Web Application Projects.

What about auto-upgrades?

All we’ve talked about so far is a one-way push to Umbraco Cloud but one thing we know and love about Umbraco Cloud is the automated upgrade process. So how do we deal with that? I actually have this working on my site but want to make the process even simpler so you’re going to have to be patient and wait for another blog post :)

The way it works is also using Azure Pipelines. Using a separate pipeline with a custom Git repo pointed at your Umbraco Cloud repository, this pipeline can be configured to poll for changes every day (or more often if you like). It then checks if changes have been made to the packages.config file to see if there’s been upgrades made to either the CMS, Forms or Deploy (in another solution I’m actually polling Nuget directly for this information). If an upgrade has been made, It clones down your source code repository, runs a Nuget update command to upgrade your solution. Then it creates a new branch, commits these changes, pushes it back GitHub and creates a Pull Request (currently this only works for GitHub).

This same solution can be used for Deploy files in case administrators are changing schema items directly on Umbraco Cloud so the /deploy/* files can be automatically kept in sync with your source code repository.

This idea is entirely inspired by Morten Christensen, thanks Morten! Hopefully I’ll find some time to finalize this.

Stay tuned!

How I configure my Umbraco Cloud website to support Nuget packages

Thu, 23 Mar 2023 15:09:00 Z

Disclaimer: This post is about how I have my own website setup. This is based on my own personal opinions and is not meant to be an Umbraco Cloud ‘best practices’ guide. I’ve written this post since it might help others who have the same requirements as I do.

My Requirements

For my simple website, I would like to manage all of my dependencies with Nuget, I don’t need anything compiled, don’t need class libraries and I don’t mind putting any required custom code into App_Code.

Umbraco Cloud provisions a deployment Git repository which for all intensive purposes is meant for deploying code between environments, not so much for a source code repository. That said, since my website is ultra simple and doesn’t contain any class library projects (etc…) I figured having an ASP.Net website project configured in the Umbraco Cloud repository would be fine. A website project is different from the standard web application project; a website doesn’t compile, it runs as-is which is suitable for the Umbraco Cloud Git repository since that’s exactly what the repository is made for: to host a deployed website that runs based on the files as-is. I prefer working with web application projects normally but in this case my website is ultra simple and a website project will work just fine plus this allows me to have a Visual Studio project/solution that works fairly seamlessly with Umbraco Cloud.

How to set this up

There’s not a lot required to set this up but there are a couple ‘gotchas’, here’s the steps:

1) Clone your Umbraco Cloud Git repo

The first step is straight forward, you need to clone your Umbraco Cloud Git repository to your local machine

2) Open your Umbraco Cloud site as a Website in Visual Studio

Open Visual Studio, File –> Open –> Web site and choose the folder where your Umbraco Cloud site is cloned. This will open your Umbraco Cloud folder as a website project (at this point you could ctrl+F5 and it would run your site).

3) Save the sln file

You need to save the Visual Studio .sln file in order to have Nuget references, File –> Save localhost_1234.sln As…

This menu option is a bit odd and that’s because Visual Studio has created an in-memory .sln file which it’s auto-named to be localhost_PORT.sln

When you click on that, browse to your Umbraco Cloud Git repo folder and name the file something that makes more sense than localhost_PORT.sln.

4) Configure the Solution and Project to not build the website

This is optional but by default Visual Studio will try to build your website which means it’s going to try to precompile all views, etc… which not only can take some time but you will get false positive errors. So instead there’s 2 things to do: In Configuration Manager turn off the “Build” option for the website project and in the website project settings turn off building. Here’s how:

Build –> Configuration Manager opens a dialog, uncheck the Build checkbox for the website

Then right click the root of your website project and choose “Property Pages” and click on the “Build” navigation element. Then change the “Start action” to “No build” and un-check the “Build website as part of solution” checkbox

5) Create a Kudu .deployment file

Here’s one of the ‘gotchas’. Like Azure web apps, Umbraco Cloud also uses Kudu to perform some of it’s operations such as deploying a website from the Git repository to the hosting directory on the server. By default Kudu will copy the files in the deployment repository as-is to the hosting directory on the server (which is what we want)… that is until Kudu sees things like .sln or .csproj files in the root of the Git repository, then it tries to be clever and build things (which we don’t want).

So to tell Kudu to just deploy the files as-is, we create a special Kudu file at the repository root called .deployment (to be clear this file name starts with a dot!).

To create this, in your Visual Studio website project, right click the root click Add –> Add new item –> Choose Text File –> Enter the name .deployment

Then add the following to this file:

[config]
project = .

and this tells Kudu to just deploy the files that are found in the repo.

6) Now you can add Nuget references

Once all of this is setup, you can add Nuget references to this website project like you would normally add Nuget references. At this point you might need to make a choice: Do you want to manage your Umbraco installation based on Nuget? In some cases you might not have a choice if you need to reference a Nuget package that has a dependency on Umbraco.Core.

As it turns out this choice isn’t such a big deal but some things to be aware of. Since Umbraco Cloud auto-upgrades projects to the latest patch version, you might be concerned that your packages.config is going to get out of date… luckily Umbraco Cloud is clever and will auto-update this file to the version it just upgraded you too. This also goes for minor version upgrades that you perform on Umbraco Cloud. Since Umbraco Cloud auto-commits all of the upgraded files, it means you really don’t have to do anything.

7) You’ll need to commit the special *.dll.refresh files

Once you start using Nuget with a website project, you’ll notice a bunch of these *.dll.refresh files in your /bin directory. You’ll need to commit those. These are special marker files used by Visual Studio to know how to manage Nuget dependencies with a website project.

That's it!

The above setup is an easy way to setup a Visual Studio solution with a single website project that works seamlessly with Umbraco Cloud while allowing you to manage dependencies with Nuget.

But what if your solution is more complicated or you want to add class libraries, etc… ? There’s Umbraco documentation on how to configure this found here: https://our.umbraco.com/documentation/Umbraco-Cloud/Set-Up/Visual-Studio/, https://our.umbraco.com/documentation/Umbraco-Cloud/Set-Up/Working-With-Visual-Studio/ and the configuration there isn’t so different than the above except that the .sln file isn’t committed to the Umbraco Cloud Git deployment repository and instead exists in your own Git repository which in that case, the special .deployment file is not necessary.