Sampling Redis keys for memory consumption

We run a farm of Redis instances for storing real time analytics. Because Redis stores objects in memory, it happens to be an extremely fast way to retrieve data; thus, most of our charts and graphs pull data from various Redis instances that contain desired data.

Our Redis boxes are running in AWS on m2.2xlarge instances, which contain a bit over 30GB of memory. Every once in awhile, it’s helpful to get an idea of how much memory various key patterns are consuming because we hold over 20GB across several hundred thousand keys in some cases. There’s a nifty gem dubbed redis-audit (my fork of it adds Bundler) that is quite helpful in painting a broad picture of memory usage across a sampling of key patterns.

Redis-audit works by sampling a configurable portion of keys residing in a Redis database. It then prints out a report of key patterns that delineates, among other things, memory usage. The summary portion of the output report looks something like:

redis-audit summary
<span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>

<code class='bash'><span class='line'>------------------------------------+--------------+-------------------+---------------------------------------------------
</span><span class='line'>Key                                 | Memory Usage | Expiry Proportion | Last Access Time
</span><span class='line'>------------------------------------+--------------+-------------------+---------------------------------------------------
</span><span class='line'>se::4f733211e::02/03/13::model::SGT | 19.03%    | 14.81%   | 1 days, 8 hours, 8 minutes, 40 seconds
</span><span class='line'>se::4f7332a3e::03/08/13::manufactur | 8.76%     | 71.43%   | 6 hours, 14 minutes, 50 seconds
</span><span class='line'><span class="c"># more keys omitted</span>
</span></code>

You can also get more details on individual key patterns; for example, the key above using 19% of Redis’s memory is detailed like so:

redis-audit details for a key pattern
<span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>

<code class='bash'><span class='line'><span class="o">==============================================================================</span>
</span><span class='line'>Found 27 keys containing hashs, like:
</span><span class='line'>se::4f7332a300011e::02/03/13::model::SGT SMA2+::aggregations,
</span><span class='line'>se::4f7332a300011e::06/06/13::model::SGH-M919V::aggregations,
</span><span class='line'>se::4f7332a300011e::05/19/13::model::vivo S7::aggregations,
</span><span class='line'>se::4f7332a300011e::02/15/13::model::PAD707::aggregations,
</span><span class='line'>se::4f7332a300011e::06/12/13::model::Rise::aggregations,
</span><span class='line'>se::4f7332a300011e::04/12/13::model::Micromax A91::aggregations,
</span><span class='line'>se::4f7332a300011e::06/11/13::model::900TPCII::aggregations,
</span><span class='line'>se::4f7332a300011e::07/01/13::model::HUAWEI U8825D::aggregations,
</span><span class='line'>se::4f7332a300011e::02/20/13::model::AT7D-TE25DA::aggregations,
</span><span class='line'>se::4f7332a300011e::02/14/13::model::HUAWEI G510-0010::aggregations
</span><span class='line'>
</span><span class='line'>These keys use 19.03% of the total sampled memory <span class="o">(</span>990 bytes<span class="o">)</span>
</span><span class='line'>14.81% of these keys expire <span class="o">(</span>4<span class="o">)</span>, with maximum ttl of 28 days, 15 hours, 51 minutes, 20 seconds
</span><span class='line'>Average last accessed <span class="nb">time</span>: 69 days, 19 hours, 14 minutes, 15 seconds - <span class="o">(</span>Max: 95 days, 23 hours, 10 minutes, 10 seconds Min:1 days, 8 hours, 8 minutes, 40 seconds<span class="o">)</span>
</span></code>

This data is helpful in a number of ways; for instance, we discovered that a significant portion of memory was being consumed by keys containing old time series data that for some reason did not have associated TTLs. Thus, we were able to achieve that particular data (into MongoDB) and free up memory.

If you want to understand how memory is distributed across key patterns in a Redis instance, then I think you’ll find Redis-audit quite helpful!

This story, "Sampling Redis keys for memory consumption" was originally published by JavaWorld.

Next read this:

When Andrew Glover isn't listening to “Funkytown” or “Le Freak” he enjoys speaking on the No Fluff Just Stuff Tour. He also writes articles for multiple online publications including IBM's developerWorks and O'Reilly’s ONJava and ONLamp portals.

Copyright © 2013 IDG Communications, Inc.