Examine v4 is out

Examine v4.0.0 has been released to NuGet! It's been in beta since October 2023, so it feels great to finally have this one out the door.

The headline feature is faceting, which people have been asking me about for years, along with taxonomy indexes, deep paging, and a fair amount of modernization of the codebase.

What took so long

The first facet commit landed in November 2022 from Nikolaj Brask-Nielsen who did the original implementation, and then Chad Currie built taxonomy index support, deep paging and a lot of the surrounding infrastructure on top of that. Between them that's well over a hundred commits of genuinely hard Lucene work, and honestly the community contributions on this release have been fantastic.

After that it sat in beta across nine releases. Part of that is simply that Examine is a side project I maintain around a full time job, and nothing was broken enough in the betas to force the issue.

The other part is that Examine v4 is built on Lucene.NET 4.8, which is itself still in beta, and I didn't want to ship a final release that depended on a beta of something else. So the plan was to wait for Lucene.NET 4.8 to go stable first.

Lucene.NET 4.8.0-beta00001 was released in May 2017, and the last stable Lucene.NET release was 3.0.3 back in 2012, so in hindsight that was a fairly optimistic plan 😊

That's not a criticism by the way - porting Lucene to .NET is an enormous amount of work done by volunteers and the project is very much active. beta00018 shipped in June this year and the hope is for a final 4.8 release before the end of the year. In practice these betas have been running in production on a very large number of Umbraco sites for years now, so the beta label was never really the risk that it sounds like.

What changed is that Umbraco needed v4.

What's new in v4

Faceting is the main one. You declare facet fields in your field definitions like any other field, request facets on a query with WithFacets(...), and read them back with GetFacet():

var results = searcher.CreateQuery("content")
    .Field("nodeName", "product")
    .WithFacets(facets => facets.FacetString("category"))
    .Execute();

foreach (var value in results.GetFacet("category")!)
{
    Console.WriteLine($"{value.Label}: {value.Value}");
}

There are facet types for full text, numerics, dates and ranges, and each has a taxonomy variant. Being able to do this natively in Examine, rather than dropping down to Lucene yourself, is the bit I'm most pleased about in this release.

The rest of it:

• Taxonomy indexes - a sidecar index enabled with UseTaxonomyIndex, which gives you hierarchical facets and faster faceting generally. This started out as a separate index type and ended up merged into LuceneIndex, which is a much better place for it.
• Deep paging - SearchAfterOptions lets you page a long way into a result set without Lucene having to materialize everything up to your offset. It works with faceted queries too.
• Boost factors and phrase queries - WithBoost() on individual search values, and a proper Phrase() instead of the old escaped value approach.
• Nullable reference types throughout, with warnings as errors, so every public API is annotated. You'll likely get some new warnings when you upgrade, all of which were true before, you just couldn't see them.
• .NET 8, 9 and 10. .NET 6 support is gone.

The replicator has also had a lot of work done to it. Taxonomy replication is handled properly, transient lock failures are retried instead of thrown, and persistent commit failures now surface through IsReplicationHealthy instead of being silently swallowed. If you're running Examine on Azure App Service, or anything else with a network file system, that's the part that matters to you.

There are breaking changes and they're mostly constructors. The release notes have a summary and the full API change report lists every one of them.

Umbraco and the new search abstraction

Umbraco HQ have been rebuilding search from the ground up in Umbraco.Cms.Search, based on the Future of Search RFC written by Bjarke Berg and Kenn Jacobsen. It's a provider based abstraction, so in principle you can put Elasticsearch, Algolia or anything else behind it.

For the initial release there is one provider, and it's Examine. From the RFC:

We plan to ship a single implementation of the search abstraction, which will be based on Examine to be backward compatible.

Their README says it will replace the current search implementation "at the earliest starting from Umbraco v19" so I'm not going to put a date on that, but it ships as an add-on for v17 and v18 first and Examine is what sits underneath it either way.

Which meant that v4 suddenly had a real consumer with real deadlines, and a beta wasn't going to cut it any more. It's a nice position to be in though - Examine has been quietly doing its job under Umbraco for years, and it's great to see it become the foundation for the new search stack too.

Backwards compatibility

Most of the work in the final stretch wasn't features at all, it was making sure v4 could be a drop-in replacement for v3.

Examine underpins search in every Umbraco install, so telling everyone to upgrade and fix up their code isn't a realistic answer. I spent a good while on this with Kenn Jacobsen and Nikolaj Geisle at Umbraco HQ - they would run v4 against the CMS, something would break, and we'd work out whether it was a genuine bug, an API I'd removed too eagerly, or something that needed a compatibility shim. Then repeat. There was a lot of testing on both sides of this.

A good example of the kind of thing that catches you out: I had removed some old positional overloads of AddExamineLuceneIndex and replaced them with optional parameters. That's source compatible, so anything recompiling against v4 is fine. But optional parameters are a compile time feature, and Umbraco's already published DLLs still had IL referencing the original method signatures, so it fell over on startup with a MissingMethodException. Source compatible, binary incompatible, and the only way to find it is to actually run it.

Automating the compatibility testing

Doing that verification manually over and over is pretty tedious, so I stopped doing it manually.

I wrote a set of Copilot skills that live in the Examine repo and automate the whole loop - clone the downstream consumer, rewrite its Examine NuGet package references into project references pointing at my local working copy, build it, run its tests, and report back exactly what broke. There's one each for Umbraco CMS, Umbraco.Cms.Search and ExamineX, plus a compat validator agent that picks the right one and drives it.

That turned "have I broken Umbraco?" from an afternoon's work into something I could run as a routine check while working on a branch. Combined with the Roslyn public API analyzer, which tracks every public member in a checked-in file and fails the build when one disappears, breaking changes had to be a deliberate decision rather than something discovered later by somebody else.

If you maintain a library with downstream consumers you care about, this is honestly worth setting up. It's a lot more approachable to build than it was even a year ago, and it takes a whole category of worry off your plate.

Umbraco.Cms.Search is currently pinned to 4.0.0-beta.9 and can now move to the final release.

Documentation

The documentation site hadn't been published since March 2023. Not for lack of anyone writing any - the publishing pipeline itself had quietly been broken for years. It only triggered on a branch nobody was pushing to, it used versions of the GitHub Pages actions that have since been removed, and it built with .NET 6, which can't compile a project targeting .NET 8 or later. So documentation kept getting written and committed and never actually went anywhere.

That's all fixed now and the docs rebuild and publish automatically on every push, which means about three years of writing has gone live in one go. Faceting, taxonomy indexes, deep paging and replication are all documented, so it's well worth a look if the last version you saw was the old site.

A number of the older code samples turned out to be wrong as well, including every faceting example, which called a method that doesn't exist. This time around every sample was compiled against the real v4 assemblies rather than just eyeballed, which is how those got caught.

The public API change report is published there too - 246 additions, 19 changed signatures and 14 removals between v3.10.0 and v4.0.0. It's generated from the API surface files that the analyzer keeps in the repo rather than written by hand, so it's the complete picture rather than what I remembered to write down.

What this means for ExamineX

I've been asked this a few times now so it's worth spelling out.

ExamineX is my commercial product that swaps Examine's Lucene implementation for Azure AI Search or Elasticsearch, so that you don't have to deal with file locks, index corruption and index rebuilds on Azure App Service. It has always been an implementation of the Examine abstractions rather than a fork or a competing API.

That's the important part here. Umbraco Search has a provider model and its default provider talks to Examine. Examine has its own implementation model underneath that, and that's the layer ExamineX plugs into. They're two separate seams stacked on top of each other, so ExamineX continues to work under Umbraco Search without needing to be an Umbraco Search provider at all - it just replaces what sits underneath the Examine one.

So the new Umbraco search stack doesn't replace ExamineX, it sits on top of the same abstraction it always has. The v4 support including faceting is already written, I just haven't cut the release yet - that's next on my list.

Upgrading

If you're on v3 and not using Umbraco then it should be reasonably straight forward. Retarget to net8.0 or later, rebuild, work through whatever the nullable annotations tell you, and check the API change report if you were constructing searchers or directory factories by hand.

If you're on Umbraco, wait for Umbraco to pull it in rather than upgrading Examine underneath it yourself.

Everything is on GitHub and issues and PRs are always welcome. Give it a go!

Huge thanks to Chad, both Nikolajs and Kenn, and to everyone who ran the betas in production and told me what broke. Seventeen years in and Examine is in the best shape it's ever been, which is a pretty good feeling 🙂