Increasing Retention and Trial Period

Up to now the data retention of request measurement data was limited to seven days. We didn’t aggregate the data so far, which means we currently save all the raw data on 160 million requests that we profiled in the last week in our Elasticsearch cluster.

Consequently aggregation has been on our to-do list for months now to allow you to keep request information for longer time spans (retention), improve the speed of all charts and lists and prepare for future growth.

Today we have deployed a first version. We have already started to aggregate data into one hour buckets. A bucket aggregates data for the request count, response time (95% percentile), error rate and impact on the whole application for every transaction. Please be patient while we reindex the last week of data, it should not take more than some hours.

As a result of the change response time charts for all date intervals that are greater than 24 hours will now be rendered from the aggregate data. If you monitor an application with several hundred thousand requests per day you will experience a nice increase in performance. In addition, the charts are also much more detailed and show the performance for every hour instead of the much larger intervals of 6-8 hours we used before.

See this chart of the Profiler performance in the last 7 days:

Seven Days of DataSeven Days of Data

Using this aggregated data we will begin to slowly increase the data retention, to 14 days in the first step.

Trial Period is now 30 days

We have been a bit over-ambitious with the short trial period of just 14 days, especially during beta. Sorry for that! We have increased the trial period to 30 days to allow you more time for setting up the Profiler and testing it in production.

Toby Toby 05.11.2014