RelStorage

From Zenoss Wiki
This is the approved revision of this page, as well as being the most recent.
Jump to: navigation, search

What is RelStorage?

The following synopsis from the RelStorage site (http://pypi.python.org/pypi/RelStorage) provides the best description.

RelStorage is a storage implementation for ZODB that stores pickles in a relational database. PostgreSQL 8.1 and above (via psycopg2), MySQL 5.0.32+ / 5.1.34+ (via MySQLdb 1.2.2 and above), and Oracle 10g (via cx_Oracle) are currently supported. RelStorage replaces the PGStorage project.

What are the Alternatives?

There are three major storage implementations for ZODB that should be considered.

  • Direct FileStorage: This is the fastest possible storage implementation because it doesn't have to worry about conflict resolution or invalidations. Unfortunately it is not a possibility for Zenoss because it can only be used by a single simultaneous Zope client.
  • ZEO & FileStorage: This is current implementation in use by Zenoss. ZEO provides a layer on top of FileStorage that handles conflict resolution and invalidations along with the ability to support multiple simultaneous Zope clients.
  • RelStorage & Memcached: RelStorage is comparable to the combination of ZEO & FileStorage in that it provides conflict resolution, invalidates and support for multiple simultaneous client access.

Why RelStorage?

The real question is why would we want to use the combination of RelStorage and Memcached instead of the current combination of ZEO and FileStorage. I'll attempt to outline the advantages each approach has vs. the other.

ZEO & FileStorage Advantages

1. More mature. It has been around longer and many people are using it. 1. Well-known. We're already using it. 1. More compact. Requires roughly 50% of the disk space.

RelStorage & Memcached Advantages

1. Roughly 50% better read performance and 20% better write performance. 1. Shared L2 memory cache for all Zope clients. 1. Database level (L3?) caching handled by MySQL. This makes it tunable. 1. Database connections are authenticated using standard MySQL protocol.

Architecture

The following image illustrates the major architectural differences between ZEO+FileStorage and RelStorage+Memcached. RelStorage.png

Performance Comparisons

The following table illustrates how long specific long-running Zenoss calls took to run using RelStorage in comparison with ZEO. The exact same ZODB was used in all cases. RelStorage performed roughly twice as fast as ZEO across the board. Calls which are more CPU bound than storage bound don't show much improvement. The getAllProducts result indicates that RelStorage's raw read performance may be ten times better.

Call Cold Memcache Hot Memcache
countIpAddresses 88% 64%

countServices

43% 14%
getAllProducts 40% 13%
getMonitoredComponents 60% 42%
getPingTree 96% 96%
getSubDevices 57% 35%

These numbers were gathered from a test system. This Zenoss installation has the following parameters.

  • 948 devices
  • 18,331 monitored components
  • 405,345 objects in ZODB

Modeling is one trouble spot for our ZODB performance at the moment so I used a test system to do a very high-level test of how fast an entire modeling run would complete using ZEO and RelStorage. During the modeling only zeo, zenhub and zenmodeler were running. Results indicate that modeling performance is three times faster with RelStorage.

  • ZEO: ZenModeler: Scan time: 8446.16 seconds
  • RelStorage: ZenModeler: Scan time: 2770.65 seconds