Use Azure Storage Tables as Long Term Cache

Azure Storage Tables is an excellent service, not only just to be used as tables, but it is also great as long term cache for exact key-value type of query scenarios.

Tables have limitations, so before we begin, we should be aware of them.

According to Azure Storage Scalability and Performance Targets,

Total Request Rate (assuming 1KB object size) per storage account is Up to 20,000 IOPS, entities per second, or messages per second.

Target throughput for single table partition (1 KB entities) is Up to 2000 entities per second.

When these limits are reached, Azure Storage service starts to throttle requests and may return 503 errors, which can be retried properly.

So from this we can assume that a single storage account can hold 10 partitions at maximum load, when each partition is limited and actually serving 2000 requests per second.

If you still haven’t, I suggest making yourself familiar with Azure Storage Performance Targets document, mentioned above and also Microsoft Azure Storage Performance and Scalability Checklist. There’s also plenty of very useful information about Azure Storage in Azure Storage Documentation.

For my scenario this is a perfect unit, since all the objects I store in tables are in average at about 500 bytes.

These objects are Protobuf serialized data that are used to track incoming service requests validity and not used in a service critical path, so we can use Tables to process them in a separate thread or on an entirely different server.

I still use Azure Redis cache for critical path and most recent data storage, Azure Storage Tables are mostly used as fallback when Redis is unavailable or data in Redis Cache is expired.

Azure Storage Tables is a very cheap service, for a large number of operations and per GB cost of storing data, it is a no brainer.

I was very curious if I can squeeze everything from Storage Tables as limitations document specified.

Here’s how I tested it.

Partition Key was evenly distributed between 10 partitions per storage account and Row Key was unique for each record.

For my tests I’ve randomly took one the objects to be of the same size just replaced Partition Key and Row Key before inserting into table

Partition Key = max 2 chars

Row Key = 54 chars

Data = 418 bytes

At the moment of testing I’ve used Windows Azure Storage Nuget 6.0.0

If you are using Azure Tables for objects smaller than 2Kb, don’t forget to turn off Nagle through ServicePointManager class. Remember, this setting must be set before any operation is performed.

Add this method:

private static void SetServicePointUriSettings(Uri uri)

{

ServicePoint tableServicePoint = ServicePointManager.FindServicePoint(uri);

tableServicePoint.UseNagleAlgorithm = false;

tableServicePoint.Expect100Continue = false;

tableServicePoint.ConnectionLimit = 100;

}
Use it for both Primary and Secondary Storage Table Uri.
var storageAccount = CloudStorageAccount.Parse(connectionString);
SetServicePointUriSettings(storageAccount.TableStorageUri.PrimaryUri);
if (storageAccount.TableStorageUri.SecondaryUri != null)
{
SetServicePointUriSettings(storageAccount.TableStorageUri.SecondaryUri);
}

In this test I’ve tested specific scenario where I needed to check insert operations.

Create class that inherit from TableEntity class
public class CacheTableEntry<TItem> : TableEntity
{
public byte[] Data { get; set; }
}

Create Insert operation method
public async Task InsertAsync<TItem>(string partitionKey, string rowKey, TItem item)
{
var entry = new CacheTableEntry<TItem>(partitionKey, rowKey, item);
TableOperation operation = TableOperation.InsertOrReplace(entry);
CloudTable table = this.tableClient.GetTableReference(this.tableName);
TableResult tableResult = await table.ExecuteAsync(operation);
// Here you can check tableResult response codes and act accordingly if response is 503
}

I was testing 2 storage accounts and my goal was to hit 40000 requests per second without getting 503 errors.

I’ve created in Azure 30xA3 VM instances on which I’ve uploaded a simple console application tester. Both test machines and storage account was in the same Azure Region, in this case it was West US.

The client application was executing “Parallel.For” from 0 to 10 million insert operations.

PartitionKey was calculated as simple mod 10 operation on iteration index and RowKey was generated on each iteration.

Important to mention that on average InsertOrReplace operation took around 12-15ms, and is much heavier than Retrieve operation, which is usually takes around 5-7ms in my case.

Of course it depends on size of data, and other factors that impact machine, network and Azure Storage utilization.

Here’s the output from these 30 machines:

[TABLETEST01] Actions executed: 1268809. Current Rate: 1364.7 msg/ps.

[TABLETEST02] Actions executed: 1179312. Current Rate: 1314.6 msg/ps.

[TABLETEST03] Actions executed: 1182596. Current Rate: 1291.2 msg/ps.

[TABLETEST04] Actions executed: 1140343. Current Rate: 1264.9 msg/ps.

[TABLETEST05] Actions executed: 1196147. Current Rate: 1409.7 msg/ps.

[TABLETEST06] Actions executed: 1173047. Current Rate: 1417.3 msg/ps.

[TABLETEST07] Actions executed: 1162402. Current Rate: 1376.9 msg/ps.

[TABLETEST08] Actions executed: 1182636. Current Rate: 1233.7 msg/ps.

[TABLETEST09] Actions executed: 1150454. Current Rate: 1169.7 msg/ps.

[TABLETEST10] Actions executed: 1222454. Current Rate: 1414.3 msg/ps.

[TABLETEST11] Actions executed: 1249945. Current Rate: 1412.2 msg/ps.

[TABLETEST12] Actions executed: 1199391. Current Rate: 1230.9 msg/ps.

[TABLETEST13] Actions executed: 1211910. Current Rate: 1242.4 msg/ps.

[TABLETEST14] Actions executed: 1150387. Current Rate: 1262.3 msg/ps.

[TABLETEST15] Actions executed: 1187726. Current Rate: 1418.6 msg/ps.

[TABLETEST16] Actions executed: 1153287. Current Rate: 1368.1 msg/ps.

[TABLETEST17] Actions executed: 1173985. Current Rate: 1093.3 msg/ps.

[TABLETEST18] Actions executed: 1148697. Current Rate: 1374 msg/ps.

[TABLETEST19] Actions executed: 1156103. Current Rate: 1409.2 msg/ps.

[TABLETEST20] Actions executed: 1154751. Current Rate: 1371.4 msg/ps.

[TABLETEST21] Actions executed: 1200523. Current Rate: 1325.2 msg/ps.

[TABLETEST22] Actions executed: 1156883. Current Rate: 1280.5 msg/ps.

[TABLETEST23] Actions executed: 1158917. Current Rate: 1400.7 msg/ps.

[TABLETEST24] Actions executed: 1163229. Current Rate: 1405.3 msg/ps.

[TABLETEST25] Actions executed: 1135691. Current Rate: 1325.3 msg/ps.

[TABLETEST26] Actions executed: 1154157. Current Rate: 1406.1 msg/ps.

[TABLETEST27] Actions executed: 1117446. Current Rate: 1266.8 msg/ps.

[TABLETEST28] Actions executed: 1159560. Current Rate: 1254 msg/ps.

[TABLETEST29] Actions executed: 1145130. Current Rate: 1266.4 msg/ps.

[TABLETEST30] Actions executed: 1193902. Current Rate: 1389.1 msg/ps.

This output is it taken at about 15 minutes into test execution.

If we sum all Current Rate outputs, it will be 39758.8 request per second.

For me that is pretty much enough to estimate that single storage account can live up to the promise of 20000 IOPS with 1K entities.

I also made a simple test of partition limit at which I threw more than 2000 requests, and as promised, I’ve seen throttling of requests and after a bit started to get 503 Service Busy errors.

In my tests I did achieve the promised SLA of Azure Tables, which for me, was very satisfying and provided me the metric for scaling our services accordingly.

Written by Anton Troshin
Lead developer, IronSource
anton