Shazwazza

Examine v4 is out

Tue, 25 Aug 2026 20:30:30 Z

Examine v4 is out

Examine v4.0.0 has been released to NuGet! It's been in beta since October 2023, so it feels great to finally have this one out the door.

The headline feature is faceting, which people have been asking me about for years, along with taxonomy indexes, deep paging, and a fair amount of modernization of the codebase.

What took so long

The first facet commit landed in November 2022 from Nikolaj Brask-Nielsen who did the original implementation, and then Chad Currie built taxonomy index support, deep paging and a lot of the surrounding infrastructure on top of that. Between them that's well over a hundred commits of genuinely hard Lucene work, and honestly the community contributions on this release have been fantastic.

After that it sat in beta across nine releases. Part of that is simply that Examine is a side project I maintain around a full time job, and nothing was broken enough in the betas to force the issue.

The other part is that Examine v4 is built on Lucene.NET 4.8, which is itself still in beta, and I didn't want to ship a final release that depended on a beta of something else. So the plan was to wait for Lucene.NET 4.8 to go stable first.

Lucene.NET 4.8.0-beta00001 was released in May 2017, and the last stable Lucene.NET release was 3.0.3 back in 2012, so in hindsight that was a fairly optimistic plan 😊

That's not a criticism by the way - porting Lucene to .NET is an enormous amount of work done by volunteers and the project is very much active. beta00018 shipped in June this year and the hope is for a final 4.8 release before the end of the year. In practice these betas have been running in production on a very large number of Umbraco sites for years now, so the beta label was never really the risk that it sounds like.

What changed is that Umbraco needed v4.

What's new in v4

Faceting is the main one. You declare facet fields in your field definitions like any other field, request facets on a query with WithFacets(...), and read them back with GetFacet():

var results = searcher.CreateQuery("content")
    .Field("nodeName", "product")
    .WithFacets(facets => facets.FacetString("category"))
    .Execute();

foreach (var value in results.GetFacet("category")!)
{
    Console.WriteLine($"{value.Label}: {value.Value}");
}

There are facet types for full text, numerics, dates and ranges, and each has a taxonomy variant. Being able to do this natively in Examine, rather than dropping down to Lucene yourself, is the bit I'm most pleased about in this release.

The rest of it:

Taxonomy indexes - a sidecar index enabled with UseTaxonomyIndex, which gives you hierarchical facets and faster faceting generally. This started out as a separate index type and ended up merged into LuceneIndex, which is a much better place for it.
Deep paging - SearchAfterOptions lets you page a long way into a result set without Lucene having to materialize everything up to your offset. It works with faceted queries too.
Boost factors and phrase queries - WithBoost() on individual search values, and a proper Phrase() instead of the old escaped value approach.
Nullable reference types throughout, with warnings as errors, so every public API is annotated. You'll likely get some new warnings when you upgrade, all of which were true before, you just couldn't see them.
.NET 8, 9 and 10. .NET 6 support is gone.

The replicator has also had a lot of work done to it. Taxonomy replication is handled properly, transient lock failures are retried instead of thrown, and persistent commit failures now surface through IsReplicationHealthy instead of being silently swallowed. If you're running Examine on Azure App Service, or anything else with a network file system, that's the part that matters to you.

There are breaking changes and they're mostly constructors. The release notes have a summary and the full API change report lists every one of them.

Umbraco and the new search abstraction

Umbraco HQ have been rebuilding search from the ground up in Umbraco.Cms.Search, based on the Future of Search RFC written by Bjarke Berg and Kenn Jacobsen. It's a provider based abstraction, so in principle you can put Elasticsearch, Algolia or anything else behind it.

For the initial release there is one provider, and it's Examine. From the RFC:

We plan to ship a single implementation of the search abstraction, which will be based on Examine to be backward compatible.

Their README says it will replace the current search implementation "at the earliest starting from Umbraco v19" so I'm not going to put a date on that, but it ships as an add-on for v17 and v18 first and Examine is what sits underneath it either way.

Which meant that v4 suddenly had a real consumer with real deadlines, and a beta wasn't going to cut it any more. It's a nice position to be in though - Examine has been quietly doing its job under Umbraco for years, and it's great to see it become the foundation for the new search stack too.

Backwards compatibility

Most of the work in the final stretch wasn't features at all, it was making sure v4 could be a drop-in replacement for v3.

Examine underpins search in every Umbraco install, so telling everyone to upgrade and fix up their code isn't a realistic answer. I spent a good while on this with Kenn Jacobsen and Nikolaj Geisle at Umbraco HQ - they would run v4 against the CMS, something would break, and we'd work out whether it was a genuine bug, an API I'd removed too eagerly, or something that needed a compatibility shim. Then repeat. There was a lot of testing on both sides of this.

A good example of the kind of thing that catches you out: I had removed some old positional overloads of AddExamineLuceneIndex and replaced them with optional parameters. That's source compatible, so anything recompiling against v4 is fine. But optional parameters are a compile time feature, and Umbraco's already published DLLs still had IL referencing the original method signatures, so it fell over on startup with a MissingMethodException. Source compatible, binary incompatible, and the only way to find it is to actually run it.

Automating the compatibility testing

Doing that verification manually over and over is pretty tedious, so I stopped doing it manually.

I wrote a set of Copilot skills that live in the Examine repo and automate the whole loop - clone the downstream consumer, rewrite its Examine NuGet package references into project references pointing at my local working copy, build it, run its tests, and report back exactly what broke. There's one each for Umbraco CMS, Umbraco.Cms.Search and ExamineX, plus a compat validator agent that picks the right one and drives it.

That turned "have I broken Umbraco?" from an afternoon's work into something I could run as a routine check while working on a branch. Combined with the Roslyn public API analyzer, which tracks every public member in a checked-in file and fails the build when one disappears, breaking changes had to be a deliberate decision rather than something discovered later by somebody else.

If you maintain a library with downstream consumers you care about, this is honestly worth setting up. It's a lot more approachable to build than it was even a year ago, and it takes a whole category of worry off your plate.

Umbraco.Cms.Search is currently pinned to 4.0.0-beta.9 and can now move to the final release.

Documentation

The documentation site hadn't been published since March 2023. Not for lack of anyone writing any - the publishing pipeline itself had quietly been broken for years. It only triggered on a branch nobody was pushing to, it used versions of the GitHub Pages actions that have since been removed, and it built with .NET 6, which can't compile a project targeting .NET 8 or later. So documentation kept getting written and committed and never actually went anywhere.

That's all fixed now and the docs rebuild and publish automatically on every push, which means about three years of writing has gone live in one go. Faceting, taxonomy indexes, deep paging and replication are all documented, so it's well worth a look if the last version you saw was the old site.

A number of the older code samples turned out to be wrong as well, including every faceting example, which called a method that doesn't exist. This time around every sample was compiled against the real v4 assemblies rather than just eyeballed, which is how those got caught.

The public API change report is published there too - 246 additions, 19 changed signatures and 14 removals between v3.10.0 and v4.0.0. It's generated from the API surface files that the analyzer keeps in the repo rather than written by hand, so it's the complete picture rather than what I remembered to write down.

What this means for ExamineX

I've been asked this a few times now so it's worth spelling out.

ExamineX is my commercial product that swaps Examine's Lucene implementation for Azure AI Search or Elasticsearch, so that you don't have to deal with file locks, index corruption and index rebuilds on Azure App Service. It has always been an implementation of the Examine abstractions rather than a fork or a competing API.

That's the important part here. Umbraco Search has a provider model and its default provider talks to Examine. Examine has its own implementation model underneath that, and that's the layer ExamineX plugs into. They're two separate seams stacked on top of each other, so ExamineX continues to work under Umbraco Search without needing to be an Umbraco Search provider at all - it just replaces what sits underneath the Examine one.

So the new Umbraco search stack doesn't replace ExamineX, it sits on top of the same abstraction it always has. The v4 support including faceting is already written, I just haven't cut the release yet - that's next on my list.

Upgrading

If you're on v3 and not using Umbraco then it should be reasonably straight forward. Retarget to net8.0 or later, rebuild, work through whatever the nullable annotations tell you, and check the API change report if you were constructing searchers or directory factories by hand.

If you're on Umbraco, wait for Umbraco to pull it in rather than upgrading Examine underneath it yourself.

Everything is on GitHub and issues and PRs are always welcome. Give it a go!

Huge thanks to Chad, both Nikolajs and Kenn, and to everyone who ran the betas in production and told me what broke. Seventeen years in and Examine is in the best shape it's ever been, which is a pretty good feeling 🙂

Filtering fields dynamically with Examine

Thu, 23 Mar 2023 15:09:58 Z

The index fields created by Umbraco in Examine by default can lead to quite a substantial amount of fields. This is primarily due in part by how Umbraco handles variant/culture data because it will create a different field per culture but there are other factors as well. Umbraco will create a “__Raw_” field for each rich text field and if you use the grid, it will create different fields for each grid row type. There are good reasons for all of these fields and this allows you by default to have the most flexibility when querying and retrieving your data from the Examine indexes. But in some cases these default fields can be problematic. Examine by default uses Lucene as it’s indexing engine and Lucene itself doesn’t have any hard limits on field count (as far as I know), however if you swap the indexing engine in Examine to something else like Azure Search with ExamineX then you may find your indexes are exceeding Azure Search’s limits.

Azure Search field count limits

Azure Search has varying limits for field counts based on the tier service level you have (strangely the Free tier allows more fields than the Basic tier). The absolute maximum however is 1000 fields and although that might seem like quite a lot when you take into account all of the fields created by Umbraco you might realize it’s not that difficult to exceed this limit. As an example, lets say you have an Umbraco site using language variants and you have 20 languages in use. Then let’s say you have 15 document types each with 5 fields (all with unique aliases) and each field is variant and you have content for each of these document types and languages created. This immediately means you are exceeding the field count limits: 20 x 15 x 10 = 1500 fields! And that’s not including the “__Raw_” fields or the extra grid fields or the required system fields like “id” and “nodeName”. I’m unsure why Azure Search even has this restriction in place

Why is Umbraco creating a field per culture?

When v8 was being developed a choice had to be made about how to handle multi-lingual data in Examine/Lucene. There’s a couple factors to consider with making this decision which mostly boils down to how Lucene’s analyzers work. The choice is either: language per field or language per index. Some folks might think, can’t we ‘just’ have a language per document? Unfortunately the answer is no because that would require you to apply a specific language analyzer for that document and then scoring would no longer work between documents. Elastic Search has a good write up about this. So either language per field or different indexes per language. Each has pros/cons but Umbraco went with language per field since it’s quite easy to setup, supports different analyzers per language and doesn’t require a ton of indexes which also incurs a lot more overhead and configuration.

Do I need all of these fields?

That really depends on what you are searching on but the answer is most likely ‘no’. You probably aren’t going to be searching on over 1000s fields, but who knows every site’s requirements are different. Umbraco Examine has something called an IValueSetValidator which you can configure to include/exclude certain fields or document types. This is synonymous with part of the old XML configuration in Examine. This is one of those things where configuration can make sense for Examine and @callumwhyte has done exactly that with his package “Umbraco Examine Config”. But the IValueSetValidator isn’t all that flexible and works based on exact naming which will work great for filtering content types but perhaps not field names. (Side note – I’m unsure if the Umbraco Examine Config package will work alongside ExamineX, need to test that out).

Since Umbraco creates fields with the same prefixed names for all languages it’s relatively easy to filter the fields based on a matching prefix for the fields you want to keep.

Here’s some code!

The following code is relatively straight forward with inline comments: A custom class “IndexFieldFilter” that does the filtering and can be applied different for any index by name, a Component to apply the filtering, a Composer to register services. This code will also ensure that all Umbraco required fields are retained so anything that Umbraco is reliant upon will still work.

/// <summary>
/// Register services
/// </summary>
public class MyComposer : ComponentComposer<MyComponent>
{
    public override void Compose(Composition composition)
    {
        base.Compose(composition);
        composition.RegisterUnique<IndexFieldFilter>();
    }
}

public class MyComponent : IComponent
{
    private readonly IndexFieldFilter _indexFieldFilter;

    public MyComponent(IndexFieldFilter indexFieldFilter)
    {
        _indexFieldFilter = indexFieldFilter;
    }

    public void Initialize()
    {
        // Apply an index field filter to an index
        _indexFieldFilter.ApplyFilter(
            // Filter the external index 
            Umbraco.Core.Constants.UmbracoIndexes.ExternalIndexName, 
            // Ensure fields with this prefix are retained
            new[] { "description", "title" },
            // optional: only keep data for these content types, else keep all
            new[] { "home" });
    }

    public void Terminate() => _indexFieldFilter.Dispose();
}

/// <summary>
/// Used to filter out fields from an index
/// </summary>
public class IndexFieldFilter : IDisposable
{
    private readonly IExamineManager _examineManager;
    private readonly IUmbracoTreeSearcherFields _umbracoTreeSearcherFields;
    private ConcurrentDictionary<string, (string[] internalFields, string[] fieldPrefixes, string[] contentTypes)> _fieldNames
        = new ConcurrentDictionary<string, (string[], string[], string[])>();
    private bool disposedValue;

    /// <summary>
    /// Constructor
    /// </summary>
    /// <param name="examineManager"></param>
    /// <param name="umbracoTreeSearcherFields"></param>
    public IndexFieldFilter(
        IExamineManager examineManager,
        IUmbracoTreeSearcherFields umbracoTreeSearcherFields)
    {
        _examineManager = examineManager;
        _umbracoTreeSearcherFields = umbracoTreeSearcherFields;
    }

    /// <summary>
    /// Apply a filter to the specified index
    /// </summary>
    /// <param name="indexName"></param>
    /// <param name="includefieldNamePrefixes">
    /// Retain all fields prefixed with these names
    /// </param>
    public void ApplyFilter(
        string indexName,
        string[] includefieldNamePrefixes,
        string[] includeContentTypes = null)
    {
        if (_examineManager.TryGetIndex(indexName, out var e)
            && e is BaseIndexProvider index)
        {
            // gather all internal index names used by Umbraco 
            // to ensure they are retained
            var internalFields = new[]
                {
                LuceneIndex.CategoryFieldName,
                LuceneIndex.ItemIdFieldName,
                LuceneIndex.ItemTypeFieldName,
                UmbracoExamineIndex.IconFieldName,
                UmbracoExamineIndex.IndexPathFieldName,
                UmbracoExamineIndex.NodeKeyFieldName,
                UmbracoExamineIndex.PublishedFieldName,
                UmbracoExamineIndex.UmbracoFileFieldName,
                "nodeName"
            }
                .Union(_umbracoTreeSearcherFields.GetBackOfficeFields())
                .Union(_umbracoTreeSearcherFields.GetBackOfficeDocumentFields())
                .Union(_umbracoTreeSearcherFields.GetBackOfficeMediaFields())
                .Union(_umbracoTreeSearcherFields.GetBackOfficeMembersFields())
                .ToArray();

            _fieldNames.TryAdd(indexName, (internalFields, includefieldNamePrefixes, includeContentTypes ?? Array.Empty<string>()));

            // Bind to the event to filter the fields
            index.TransformingIndexValues += TransformingIndexValues;
        }
        else
        {
            throw new InvalidOperationException(
                $"No index with name {indexName} found that is of type {typeof(BaseIndexProvider)}");
        }
    }

    private void TransformingIndexValues(object sender, IndexingItemEventArgs e)
    {
        if (_fieldNames.TryGetValue(e.Index.Name, out var fields))
        {
            // check if we should ignore this doc by content type
            if (fields.contentTypes.Length > 0 && !fields.contentTypes.Contains(e.ValueSet.ItemType))
            {
                e.Cancel = true;
            }
            else
            {
                // filter the fields
                e.ValueSet.Values.RemoveAll(x =>
                {
                    if (fields.internalFields.Contains(x.Key)) return false;
                    if (fields.fieldPrefixes.Any(f => x.Key.StartsWith(f))) return false;
                    return true;
                });
            }
        }
    }

    protected virtual void Dispose(bool disposing)
    {
        if (!disposedValue)
        {
            if (disposing)
            {
                // Unbind from the event for any bound indexes
                foreach (var keys in _fieldNames.Keys)
                {
                    if (_examineManager.TryGetIndex(keys, out var e)
                        && e is BaseIndexProvider index)
                    {
                        index.TransformingIndexValues -= TransformingIndexValues;
                    }
                }
            }
            disposedValue = true;
        }
    }

    public void Dispose()
    {
        Dispose(disposing: true);
        GC.SuppressFinalize(this);
    }
}

That should give you the tools you need to dynamically filter your index based on fields and content type’s if you need to get your field counts down. This would also be handy even if you aren’t using ExamineX and Azure Search since keeping an index size down and storing less data means less IO operations and storage size.