Shazwazza

PetaPoco may cause high memory usage with certain queries

Thu, 23 Mar 2023 15:08:15 Z

If you are using PetaPoco, or NPoco (which seams to be the most up-to-date fork of the project), the title of this post might be a bit scary… but hopefully you won’t have to worry. This really depends on how you create your queries and how many different query structures you are executing.

High memory usage

Here is the code in relation to the memory growth when using PetaPoco:

https://github.com/toptensoftware/PetaPoco/blob/master/PetaPoco/PetaPoco.cs#L1836

What is happening here is that every time a POCO needs to be mapped from a data source, this will add more values to a static cache, specifically this one:

https://github.com/toptensoftware/PetaPoco/blob/master/PetaPoco/PetaPoco.cs#L2126 (m_PocoDatas)

This isn’t a bad thing… but it can be if you are:

using non-parameterized where clauses
you have dynamically generated where clauses
you use a lot of sql ‘IN’ clauses – since the items in the array being passed to the ‘IN’ clauses is dynamic
you have tons of differently statically unique where clauses

Each time a unique SQL query is sent to PetaPoco it will store this SQL string and associate it to a delegate (which is also cached). Over time, as these unique SQL queries are executed, the internal static cache will grow. In some cases this could consume quite a lot of memory.

The other thing to note is how large the ‘key’ that PetaPoco/NPoco uses:

var key = string.Format("{0}:{1}:{2}:{3}:{4}", sql, connString, ForceDateTimesToUtc, firstColumn, countColumns);

Considering how many queries might be executing in your application, the storage for these keys alone could take up quite a lot of memory! An SQL statement combined with a connection string could be very long, and each of these combinations gets stored in memory for every unique SQL query executed that returns mapped POCO objects.

Parameterized queries vs. non-parameterized

Here’s some examples of why non-parameterized queries will cause lots of memory consumption. Lets say we have a simple query like:

db.Query<MyTable>("WHERE MyColumn=@myValue", new {myValue = "test"})

Which results in this SQL:

SELECT * FROM MyTable WHERE MyColumn = @myValue

This query can be used over and over again with a different value and PetaPoco will simply store a single SQL key in it’s internal cache. However, if you are executing queries without real parameters such as:

db.Query<MyTable>("WHERE MyColumn='hello'");
db.Query<MyTable>("WHERE MyColumn='world'");
db.Query<MyTable>("WHERE MyColumn='hello world'");

Which results in this SQL:

SELECT * FROM MyTable WHERE MyColumn = 'hello';
SELECT * FROM MyTable WHERE MyColumn = 'world';
SELECT * FROM MyTable WHERE MyColumn = 'hello world';

Then PetaPoco will store each of these statements against a delegate in it’s internal cache since each of these string statements are not equal to each other.

Depending on your application you still might have a very large number of unique parameterized queries, though I’d assume you’d have to have a terrifically huge amount for it to be a worry.

Order by queries

Unfortunately even if you use parameterized queries, PetaPoco will store the SQL query key with it’s Order By clause which isn’t necessary and will again mean more duplicate SQL keys and delegates being tracked. For example if you have these resulting queries:

SELECT * FROM MyTable WHERE MyColumn = @myValue ORDER BY SomeField;
SELECT * FROM MyTable WHERE MyColumn = @myValue ORDER BY AnotherField;

PetaPoco will store each of these statements in it’s internal cache separately since the strings don’t match, however the delegate that PetaPoco is storing against these SQL statements isn’t concerned about the ordering output, it’s only concerned about the column and table selection so in theory it should be stripping off the last Order By clause (and other irrelevant clauses) to avoid this duplication.

A slightly better implementation

First, if you are using PetaPoco/NPoco, you shouldn’t use dynamic queries for the point’s mentioned above. If you need this functionality then I suppose it might be worth these libraries adding some sort of property on the Database object or a parameter in either the Fetch or Query methods to specify that you don’t want to use the internal cache (this will be slower, but you won’t get unwanted memory growth). I’d really just suggest not using dynamically created where clauses ;-)

Next, there’s a few things that could be fixed in the PetaPoco/NPoco core to manage memory a little better:

The size the the key that is stored in memory doesn’t need to be that big. A better implementation would be to use a hash combiner class to combine the GetHashCode result of each of those parameters that make up the key. This is a very fast way to create a hash of some strings that will result in a much smaller key. An example of a hash combiner class is here (which is actually inspired by the various internal hash code combiner classes in .Net): https://github.com/umbraco/Umbraco-CMS/blob/7.2.0/src/Umbraco.Core/HashCodeCombiner.cs
Instead of storing all of this cache in static variables, have them stored in an ObjectCache/MemoryCache (http://msdn.microsoft.com/en-us/library/system.runtime.caching.objectcache(v=vs.110).aspx) with a sliding expiration so the memory can get collected when it’s unused
The Order By clause should be ignored based on the point mentioned above

I’ve created a PR for NPoco here, and also created an issue on the original PetaPoco source here.

Umbraco 4.1 Benchmarks Part 2 (Back Office Database Queries)

Thu, 23 Mar 2023 15:08:14 Z

This post was imported from FARMCode.org which has been discontinued. These posts now exist here as an archive. They may contain broken links and images.

This is part 2 in a series of Umbraco 4.1 benchmarks created by various members of the core team in the lead up to launch. See Part 1 here on request/response peformance in the Umbraco back office.

This benchmark report looks at the data layer improvements in 4.1 by comparing query counts in 4.1 to 4.0.3. Not only has the data layer improved but there’s been significant improvements in the consumption of the data layer API made by many of the 4.1 pages and controls.

The stats below are represented as a percentage of the total calls of 4.0.3 where the number of queries in 4.0.3 are 100% and the number of queries in 4.1 are a percentage in relation to this. These results are based on the procedures listed at the bottom of this post and on averages run over 3 separate trials.

Step	4.0.3	4.1.0
Login	100%	68%
Expand all Content nodes	100%	23%
Edit Home node	100%	49%
Publishing Home node	100%	55%
Edit About Umbraco node	100%	49%
Go to Settings App	100%	100%
Expand Document types tree	100%	100%
Edit Home document type	100%	61%
Save Home document type	100%	67%
Go to Media app	100%	50%
Create new folder labeled ‘Test’	100%	88%
Create new image under new ‘Test’ folder labeled ‘test1’	100%	64%
Upload new image file to ‘test1’ and save the node	100%	49%
Go to Content app (and in the case of 4.0.3, expand the tree and select the About Umbraco node since in 4.1 this will already be selected and loaded)	100%	41%
Edit Home node	100%	43%
Add ‘test1’ image to the ‘Text’ WYSIWYG property with the image picker and Publish node	100%	49%
Average of averages above		60%
Complete run through of the above steps	100%	66%

So based on averages, Umbraco 4.1 is looking to have around 40% less queries made than 4.0.3!!! Thats HUGE!

The following steps were taken on each trial of the above steps:

New instances of both 4.0.3 and 4.1
Install CWS package on both instances
Log out of both instances
Bump web.config for both instances (clear out all data cache)
Use SQL Profiler to determine query counts for each step listed above

Also, SQL debugging has been added to 4.1 for MS SQL instances. If you compile the source in Debug mode you can get the SQL command output by adding a trace listener to your web.config. Underneath the configuration node you can add this xml block:

<system.diagnostics>
	<trace autoflush="true">
	  <listeners>
		<add name="SqlListener" 
			type="System.Diagnostics.TextWriterTraceListener" 
			initializeData="trace.log" />
	  </listeners>
	</trace>
</system.diagnostics>

This will create a trace.log file in the root of your web app SQL debugging.

Umbraco 4.1 Database Structure

Thu, 23 Mar 2023 15:08:14 Z

This post was imported from FARMCode.org which has been discontinued. These posts now exist here as an archive. They may contain broken links and images.

We’re finally shipping Umbraco 4.1 with all of the real database constraints, indexes and keys that it so much deserves! This issue was reported on over 3 years ago (http://umbraco.codeplex.com/workitem/8162) and it’s now a reality! So without further adieu, here’s the complete database diagram. If you stare at it long enough you might notice some constraints missing which may be true, but this is by design (I assure you :). I’ve attempted to compact the diagram to as small as possible but no matter which way you cut it, it’s a bit crazy. Enjoy! (click the image to see the full size)