What is RelStorage?
The following synopsis from the RelStorage site (http://pypi.python.org/pypi/RelStorage) provides the best description.
RelStorage is a storage implementation for ZODB that stores pickles in a relational database. PostgreSQL 8.1 and above (via psycopg2), MySQL 5.0.32+ / 5.1.34+ (via MySQLdb 1.2.2 and above), and Oracle 10g (via cx_Oracle) are currently supported. RelStorage replaces the PGStorage project.
What are the Alternatives?
There are three major storage implementations for ZODB that should be considered.
- Direct FileStorage: This is the fastest possible storage implementation because it doesn't have to worry about conflict resolution or invalidations. Unfortunately it is not a possibility for Zenoss because it can only be used by a single simultaneous Zope client.
- ZEO & FileStorage: This is current implementation in use by Zenoss. ZEO provides a layer on top of FileStorage that handles conflict resolution and invalidations along with the ability to support multiple simultaneous Zope clients.
- RelStorage & Memcached: RelStorage is comparable to the combination of ZEO & FileStorage in that it provides conflict resolution, invalidates and support for multiple simultaneous client access.
The real question is why would we want to use the combination of RelStorage and Memcached instead of the current combination of ZEO and FileStorage. I'll attempt to outline the advantages each approach has vs. the other.
ZEO & FileStorage Advantages
1. More mature. It has been around longer and many people are using it. 1. Well-known. We're already using it. 1. More compact. Requires roughly 50% of the disk space.
RelStorage & Memcached Advantages
1. Roughly 50% better read performance and 20% better write performance. 1. Shared L2 memory cache for all Zope clients. 1. Database level (L3?) caching handled by MySQL. This makes it tunable. 1. Database connections are authenticated using standard MySQL protocol.
The following table illustrates how long specific long-running Zenoss calls took to run using RelStorage in comparison with ZEO. The exact same ZODB was used in all cases. RelStorage performed roughly twice as fast as ZEO across the board. Calls which are more CPU bound than storage bound don't show much improvement. The getAllProducts result indicates that RelStorage's raw read performance may be ten times better.
|Call||Cold Memcache||Hot Memcache|
These numbers were gathered from a test system. This Zenoss installation has the following parameters.
- 948 devices
- 18,331 monitored components
- 405,345 objects in ZODB
Modeling is one trouble spot for our ZODB performance at the moment so I used a test system to do a very high-level test of how fast an entire modeling run would complete using ZEO and RelStorage. During the modeling only zeo, zenhub and zenmodeler were running. Results indicate that modeling performance is three times faster with RelStorage.
- ZEO: ZenModeler: Scan time: 8446.16 seconds
- RelStorage: ZenModeler: Scan time: 2770.65 seconds