Shazwazza

An Examine fix for Umbraco index corruption

Wed, 31 Jul 2024 17:49:54 Z

A new Examine version 3.3.0 has been released to address a long awaited bug fix for Umbraco websites that use the SyncedFileSystemDirectoryFactory which is the default setting for Umbraco CMS.

The bug typically means that indexes cannot be used and log entries such as Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1 are present.

Understanding the problem:

The SyncedFileSystemDirectoryFactory directory is there to avoid performance implications of rebuilding indexes on startup when a site is moved to another worker in Azure. The reason an index rebuild would occur is because in Azure, Lucene files need to work off of the local 'fast drive' (C:\%temp%), not the default/shared network 'slow drive' (D:), and whenever a site is moved, or spawned on a new worker in Azure, the local 'fast drive' is empty, meaning no indexes exist. The SyncedFileSystemDirectoryFactory attempts to work around this challenge by continually synchronizing a copy of the indexes from the 'fast drive' to the 'slow drive' so that when a site is moved to another worker, it can sync (restore) from the 'slow drive' back to the 'fast drive' in order to avoid the index rebuild overhead.

The problem with SyncedFileSystemDirectoryFactory is that this implementation doesn't take into account what happens if the index files in your main storage ('slow drive') become corrupted which can happen for a number of reasons - misconfiguration, network latency, process termination, etc...

Understanding the solution:

The SyncedFileSystemDirectoryFactory has been updated to:

Check the health of the main index if it exists ('slow drive').
Check the health of the local index if it exists ('fast drive').
If the main index is unhealthy or doesn't exist and the local index is healthy, it will synchronize the local index to the main index. This can occur only if a site hasn't moved to a new worker.
If the main index is unhealthy and the local index doesn't exist or is unhealthy, then it will delete the main (corrupted) index.
Once health checks are done, the index from main is always synced to local. If the main index was deleted due to corruption, this will mean that the local index is empty and an index rebuild will occur.

This change will attempt to keep any healthy index that is available (main vs local), but if nothing can be read, the indexes will be deleted and an index rebuild will occur.

There's also a new option to fix a corrupted index but this is not enabled by default since it can mean a loss of documents.

Understanding the rebuilding overhead

The performance overhead of index rebuilding is due to the Umbraco database queries that need to be executed in order to populate the indexes. The only reason SyncedFileSystemDirectoryFactory exists is to prevent this overhead when hosting in Azure App Service (which is what Umbraco Cloud uses), and it can only be used on your Umbraco primary node. It does not prevent index rebuilding overhead for non-primary nodes when load balancing or scaling out because the main network 'slow drive' is shared between all workers and an index can only be read/written to be a single process.

This means that it's only useful if you are hosting in Azure App Service without any load balancing while keeping in mind that it does not always prevent index rebuilds (see above).

The index rebuilding overhead can be dramatic when load balancing or scaling out, for example: If you scale out to +5 nodes in a load balancing setup, that means that 5x nodes will be performing index rebuilds around the same time, this means that your DB is going to get hammered by queries to build all of those new indexes. The performance hit isn't the index building - it is the DB queries and this can lead to DB locks and lead to the dreaded SQL Lock Timeout issue in the Umbraco back office. Plus, if search is critical to your front-end, than for a while after your site has started up, there won't be any index which means there won't be any search until the background processing is done.

... Many of these reasons is why ExamineX was created:

Reliably and centrally persisted indexes means nothing is out of sync between nodes.
No index rebuilding when your site is moved or scaled = no rebuilding overhead.
Prevents SQL Timeout locks due to DB rebuilding queries.
Very easy to setup and seamlessly changes your Lucene based indexes to Azure/Elastic search indexes.
Ideal when hosting Umbraco on Azure Web Apps (or Umbraco Cloud) and a perfect solution for load balancing and scaling.
Automatically index Umbraco media file content without the need for additional indexes with support for PDFs, Microsoft Office documents and more.
(coming soon) Automatically generate Umbraco media image descriptions, tags, locations, and more using AI allowing your editors to quickly find the media/images they need.

Configuring a Suggester with ExamineX and Azure AI Search

Wed, 01 May 2024 14:51:52 Z

We recently had a customer ask about integrating the Azure AI Search suggester API with ExamineX and this turns out to be quite straight forward :)

First thing is to ensure that the fields you want to use the Suggester for are configured correctly:

cs // Configuration for the External Index for the City and Country fields. // The Azure AI Search Suggester requires fields to be 'string fields only' // and using the Standard Analyzer (default on the External Index) services.Configure<AzureSearchIndexOptions>( Constants.UmbracoIndexes.ExternalIndexName, options => { options.FieldDefinitions = new FieldDefinitionCollection( new FieldDefinition("City", AzureSearchFieldDefinitionTypes.FullText), new FieldDefinition("Country", AzureSearchFieldDefinitionTypes.FulText)); }); Next up, is to create the Suggester which is done as part of the CreatingOrUpdatingIndex event:

```cs externalIndex.CreatingOrUpdatingIndex += AzureIndexCreatingOrUpdatingIndexScoringProfile;

private void AzureIndexCreatingOrUpdatingIndexSuggester(object sender, CreatingOrUpdatingIndexEventArgs e) { switch (e.EventType) { case IndexModifiedEventType.Creating: case IndexModifiedEventType.Rebuilding: var index = e.AzureSearchIndexDefinition;

        var suggester = new SearchSuggester(
            "sg",
            new[] { "Country", "City" });

        index.Suggesters.Add(suggester);

        break;
}

} ``` When data has been indexed for these fields, using the Suggester API is simple. For example, lets assume that the following ValueSets were indexed:

cs externalIndex.IndexItems(new[] { ValueSet.FromObject(id1, new { City = "Calgary", Country = "Canada" }), ValueSet.FromObject(id2, new { City = "Sydney", Country = "Australia" }), ValueSet.FromObject(id3, new { City = "Copenhagen", Country = "Denmark" }), ValueSet.FromObject(id4, new { City = "Vienna", Country = "Austria" }), }); Then to get suggestions for "Au":

cs // cast to AzureSearchIndex to expose underlying Azure Search APIs var azureSearchIndex = (AzureSearchIndex)externalIndex; var result = await azureSearchIndex.IndexClient.SuggestAsync<ExamineDocument>( "Au", "sg");

The result will contain 2x entries, one for the Australia and one for the Austria documents. Super easy!

One thing to watch out for is if your index is not configured to use the Standard Analyzer by default (i.e. like the Internal Index in Umbraco). In that case, you'll need to define a custom field type with that indexer:

```cs services.Configure( Constants.UmbracoIndexes.ExternalIndexName, options => { // This adds a custom field type that uses the Standard Analyzer options.IndexValueTypesFactory = new Dictionary { ["standard"] = new AzureSearchFieldValueTypeFactory(s => new AzureSearchFieldValueType(s, SearchFieldDataType.String, LexicalAnalyzerName.StandardLucene.ToString())) };

    // Set these field types to the one defined above
    options.FieldDefinitions = new FieldDefinitionCollection(
        new FieldDefinition("City", "standard"),
        new FieldDefinition("Country", "standard"));
});

``` Happy Searching!

Searching with IPublishedContentQuery in Umbraco

Thu, 23 Mar 2023 15:10:02 Z

I recently realized that I don’t think Umbraco’s APIs on IPublishedContentQuery are documented so hopefully this post may inspire some docs to be written or at least guide some folks on some functionality they may not know about.

A long while back even in Umbraco v7 UmbracoHelper was split into different components and UmbracoHelper just wrapped these. One of these components was called ITypedPublishedContentQuery and in v8 is now called IPublishedContentQuery, and this component is responsible for executing queries for content and media on the front-end in razor templates. In v8 a lot of methods were removed or obsoleted from UmbracoHelper so that it wasn’t one gigantic object and tries to steer developers to use these sub components directly instead. For example if you try to access UmbracoHelper.ContentQuery you’ll see that has been deprecated saying:

Inject and use an instance of IPublishedContentQuery in the constructor for using it in classes or get it from Current.PublishedContentQuery in views

and the UmbracoHelper.Search methods from v7 have been removed and now only exist on IPublishedContentQuery.

There are API docs for IPublishedContentQuery which are a bit helpful, at least will tell you what all available methods and parameters are. The main one’s I wanted to point out are the Search methods.

Strongly typed search responses

When you use Examine directly to search you will get an Examine ISearchResults object back which is more or less raw data. It’s possible to work with that data but most people want to work with some strongly typed data and at the very least in Umbraco with IPublishedContent. That is pretty much what IPublishedContentQuery.Search methods are solving. Each of these methods will return an IEnumerable<PublishedSearchResult> and each PublishedSearchResult contains an IPublishedContent instance along with a Score value. A quick example in razor:

@inherits Umbraco.Web.Mvc.UmbracoViewPage
@using Current = Umbraco.Web.Composing.Current;
@{
    var search = Current.PublishedContentQuery.Search(Request.QueryString["query"]);
}

<div>
    <h3>Search Results</h3>
    <ul>
        @foreach (var result in search)
        {
            <li>
                Id: @result.Content.Id
                <br/>
                Name: @result.Content.Name
                <br />
                Score: @result.Score
            </li>
        }
    </ul>
</div>

The ordering of this search is by Score so the highest score is first. This makes searching very easy while the underlying mechanism is still Examine. The IPublishedContentQuery.Search methods make working with the results a bit nicer.

Paging results

You may have noticed that there’s a few overloads and optional parameters to these search methods too. 2 of the overloads support paging parameters and these take care of all of the quirks with Lucene paging for you. I wrote a previous post about paging with Examine and you need to make sure you do that correctly else you’ll end up iterating over possibly tons of search results which can have performance problems. To expand on the above example with paging is super easy:

@inherits Umbraco.Web.Mvc.UmbracoViewPage
@using Current = Umbraco.Web.Composing.Current;
@{
    var pageSize = 10;
    var pageIndex = int.Parse(Request.QueryString["page"]);
    var search = Current.PublishedContentQuery.Search(
        Request.QueryString["query"],
        pageIndex * pageSize,   // skip
        pageSize,               // take
        out var totalRecords);
}

<div>
    <h3>Search Results</h3>
    <ul>
        @foreach (var result in search)
        {
            <li>
                Id: @result.Content.Id
                <br/>
                Name: @result.Content.Name
                <br />
                Score: @result.Score
            </li>
        }
    </ul>
</div>

Simple search with cultures

Another optional parameter you might have noticed is the culture parameter. The docs state this about the culture parameter:

When the culture is not specified or is *, all cultures are searched. To search for only invariant documents and fields use null. When searching on a specific culture, all culture specific fields are searched for the provided culture and all invariant fields for all documents. While enumerating results, the ambient culture is changed to be the searched culture.

What this is saying is that if you aren’t using culture variants in Umbraco then don’t worry about it. But if you are, you will also generally not have to worry about it either! What?! By default the simple Search method will use the “ambient” (aka ‘Current’) culture to search and return data. So if you are currently browsing your “fr-FR” culture site this method will automatically only search for your data in your French culture but will also search on any invariant (non-culture) data. And as a bonus, the IPublishedContent returned also uses this ambient culture so any values you retrieve from the content item without specifying the culture will just be the ambient/default culture.

So why is there a “culture” parameter? It’s just there in case you want to search on a specific culture instead of relying on the ambient/current one.

Search with IQueryExecutor

IQueryExecutor is the resulting object created when creating a query with the Examine fluent API. This means you can build up any complex Examine query you want, even with raw Lucene, and then pass this query to one of the IPublishedContentQuery.Search overloads and you’ll get all the goodness of the above queries. There’s also paging overloads with IQueryExecutor too. To further expand on the above example:

@inherits Umbraco.Web.Mvc.UmbracoViewPage
@using Current = Umbraco.Web.Composing.Current;
@{
    // Get the external index with error checking
    if (ExamineManager.Instance.TryGetIndex(
        Constants.UmbracoIndexes.ExternalIndexName, out var index))
    {
        throw new InvalidOperationException(
            $"No index found with name {Constants.UmbracoIndexes.ExternalIndexName}");
    }

    // build an Examine query
    var query = index.GetSearcher().CreateQuery()
        .GroupedOr(new [] { "pageTitle", "pageContent"},
            Request.QueryString["query"].MultipleCharacterWildcard());


    var pageSize = 10;
    var pageIndex = int.Parse(Request.QueryString["page"]);
    var search = Current.PublishedContentQuery.Search(
        query,                  // pass the examine query in!
        pageIndex * pageSize,   // skip
        pageSize,               // take
        out var totalRecords);
}

<div>
    <h3>Search Results</h3>
    <ul>
        @foreach (var result in search)
        {
            <li>
                Id: @result.Content.Id
                <br/>
                Name: @result.Content.Name
                <br />
                Score: @result.Score
            </li>
        }
    </ul>
</div>

The base interface of the fluent parts of Examine’s queries are IQueryExecutor so you can just pass in your query to the method and it will work.

Recap

The IPublishedContentQuery.Search overloads are listed in the API docs, they are:

Search(String term, String culture, String indexName)
Search(String term, Int32 skip, Int32 take, out Int64 totalRecords, String culture, String indexName)
Search(IQueryExecutor query)
Search(IQueryExecutor query, Int32 skip, Int32 take, out Int64 totalRecords)

Should you always use this instead of using Examine directly? As always it just depends on what you are doing. If you need a ton of flexibility with your search results than maybe you want to use Examine’s search results directly but if you want simple and quick access to IPublishedContent results, then these methods will work great.

Does this all work with ExamineX ? Absolutely!! One of the best parts of ExamineX is that it’s completely seamless. ExamineX is just an index implementation of Examine itself so all Examine APIs and therefore all Umbraco APIs that use Examine will ‘just work’.