Revision as of 18:56, 27 August 2015

PythonCollector ZenPack

Warning

The ZenPack Catalog has moved to its new home at https://www.zenoss.com/product/zenpacks as of January 17, 2017. The following information may be out of date, and this page will eventually be removed.

Support

This is an Open Source ZenPack developed by Zenoss, Inc. Enterprise support for this ZenPack is available to commercial customers with an active subscription.

Releases

Version 1.7.2- Download

Released on 2015/08/27

Compatible with Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x, Zenoss Resource Manager 5.0.x

Version 1.6.4- Download

Released on 2015/03/30

Compatible with Zenoss Core 4.2.x, Zenoss Core 5.0.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x, Zenoss Resource Manager 5.0.x

Version 1.5.3- Download

Released on 2014/09/29

Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x

Version 1.4.0- Download

Released on 2014/04/02

Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x

Version 1.3.0- Download

Released on 2014/03/20

Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x

Version 1.2.0- Download

Released on 2013/11/25

Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x

Version 1.1.1- Download

Released on 2013/09/19

Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x

Version 1.0.2- Download

Released on 2013/07/29

Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x

Background

This ZenPack provides a new Python data source type. It also provides a new zenpython collector daemon that is responsible for collecting these data sources.

Usage

This zenpack adds a new zenpython collector daemon. In most cases there is nothing you need to know about to make use of this additional collector functionality. The following sections describe available configuration, troubleshooting and tuning information.

ZenPython Configuration Options

The following options can be specified in the zenpython configuration file. Typically the default values for these options are appropriate and don't need to be adjusted.

blockingwarning

The zenpython collector daemon executes plugin code provided by other ZenPacks. If this plugin code blocks for too long it will prevent zenpython from performing other tasks including collecting other datasources while the plugin code is executed. The blockingwarning option will cause zenpython to log a warning for any plugin code that blocks for the configured number of seconds or more. The default value is 30. Decimal precision such as 0.5 can be used.

twistedthreadpoolsize

Controls size of threads pool. Datasources can use multi-threading to run multiple requests in parallel. Increasing this value may boost performance at the cost of system memory used. The default value is 10.

collect

Allows only specific plugins to run. This is primarily a developer option to help reduce the noise while developing plugins. The default is to collect all configured plugins. The value for this option is a regular expression. Only plugins which class name matches the regular expression will be run.

ignore

Prevents specific plugins from running. This is primarily a developer option to help reduce the noise while developing plugins. The default is to not ignore any configured plugins. The value for this option is a regular expression. Only plugins which class name doesn't match the regular expression will be run.

ZenPython Statistics

There are two kinds of statistics available from the zenpython collector daemon. There are datapoints which can be plotted on graphs and have thresholds defined, and there are detailed task statistics that can be logged.

Datapoints

The following standard collector datapoints are available for zenpython. These datapoints are provided by the Zenoss platform and may differ depending on which version of Zenoss is being used.

• devices: Number of devices being collected.
• dataPoints: Number of datapoints collected per second.
• eventCount: Number of events sent per second.
• discardedEvents: Number of events discarded per second.
• eventQueueLength: Number of events queued to be set to zenhub.
• taskCount: Total number of configured tasks.
• runningTasks: Number of tasks in the running state.
• queuedTasks: Number of tasks queued waiting to be run.
• missedRuns: Number of tasks that missed their scheduled run time.

The following additional collector datapoonts are available for zenpython. These datapoints are provided by the collector daemon and may differ depending on which version of PythonCollector is being used.

• percentBlocked: Percent of the time blocked by plugin code execution.

Task Statistics

The datapoints above provide for a good high-level understanding of the overall trends for the zenpython collector daemon. When certain datapoints show a potential problem it can often be useful to capture detailed per-task statistics to identify which tasks are contributing to the problem.

The zenpython process will dump detailed per-task information to its regular log file if it sent the SIGUSR2 signal. Depending on the Zenoss version being used, different utilities are provided for invoking this signal.

• Zenoss 5: serviced service action zenpython stats
• Zenoss 4: zenpython stats

Running those commands won't produce any direct output. You must look at the zenpython log to see the resulting statistics.

Developing Plugins

The goal of the Python data source type is to replicate some of the standard COMMAND data source type's functionality without requiring a new shell and shell subprocess to be spawned each time the data source is collected. The COMMAND data source type is infinitely flexible, but because of the shell and subprocess spawning, it's performance and ability to pass data into the collection script are limited. The Python data source type circumvents the need to spawn subprocesses by forcing the collection code to be asynchronous using the Twisted library. It circumvents the problem with passing data into the collection logic by being able to pass any basic Python data type without the need to worry about shell escaping issues.

The Python data source type is intended to be used in one of two ways. The first way is directly through the creation of Python data sources through the web interface or in a ZenPack. When used in this way, it is the responsibility of the data source creator to implement the required Python class specified in the data source's Python Class Name property field. The second way the Python data source can be used is as a base class for another data source type. Used in this way, the ZenPack author will create a subclass of PythonDataSource to provide a higher-level functionality to the user. The user is then not responsible for writing a Python class to collect and process data.

Using the Python Data Source Type Directly

To create a Python data source directly you should first implement the Python class you'll eventually use for the data source's Plugin Class Name. It is recommended to implement this class in a ZenPack so that it is portable from one Zenoss system to another and differentiates your custom code from Zenoss code.

Assuming you have a ZenPack named ZenPacks.example.PackName you would create a ZenPacks/example/PackName/dsplugins.py file with contents like the following.

import time
 
from Products.ZenEvents import ZenEventClasses
 
from ZenPacks.zenoss.PythonCollector.datasources.PythonDataSource \
    import PythonDataSourcePlugin
 
class MyPlugin(PythonDataSourcePlugin):
    """Explanation of what MyPlugin does."""
 
    # List of device attributes you'll need to do collection.
    proxy_attributes = (
        'zCommandUsername',
        'zCommandPassword',
        )
 
    @classmethod
    def config_key(cls, datasource, context):
        """
        Return a tuple defining collection uniqueness.
 
        This is a classmethod that is executed in zenhub. The datasource and
        context parameters are the full objects.
 
        This example implementation is the default. Split configurations by
        device, cycle time, template id, datasource id and the Python data
        source's plugin class name.
 
        You can omit this method from your implementation entirely if this
        default uniqueness behavior fits your needs. In many cases it will.
        """
        return (
            context.device().id,
            datasource.getCycleTime(context),
            datasource.rrdTemplate().id,
            datasource.id,
            datasource.plugin_classname,
            )
 
    @classmethod
    def params(cls, datasource, context):
        """
        Return params dictionary needed for this plugin.
 
        This is a classmethod that is executed in zenhub. The datasource and
        context parameters are the full objects.
 
        This example implementation will provide no extra information for
        each data source to the collect method.
 
        You can omit this method from your implementation if you don't require
        any additional information on each of the datasources of the config
        parameter to the collect method below. If you only need extra
        information at the device level it is easier to just use
        proxy_attributes as mentioned above.
        """
        return {}
 
    def collect(self, config):
        """
        No default collect behavior. You must implement this method.
 
        This method must return a Twisted deferred. The deferred results will
        be sent to the onResult then either onSuccess or onError callbacks
        below.
        """
        ds0 = config.datasources[0]
        return somethingThatReturnsADeferred(
            username=ds0.zCommandUsername,
            password=ds0.zCommandPassword)
 
    def onResult(self, result, config):
        """
        Called first for success and error.
 
        You can omit this method if you want the result of the collect method
        to be used without further processing.
        """
        return result
 
    def onSuccess(self, result, config):
        """
        Called only on success. After onResult, before onComplete.
 
        You should return a data structure with zero or more events, values
        and maps.
        """
        collectionTime = time.time()
        return {
            'events': [{
                'summary': 'successful collection',
                'eventKey': 'myPlugin_result',
                'severity': ZenEventClasses.Clear,
                },{
                'summary': 'first event summary',
                'eventKey': 'myPlugin_result',
                'severity': ZenEventClasses.Info,
                },{
                'summary': 'second event summary',
                'eventKey': 'myPlugin_result',
                'severity': ZenEventClasses.Warning,
                }],
 
            'values': {
                None: {
                    # datapoints for the device (no component)
                    'datasource1_datapoint1': (123.4, collectionTime),
                    'datasource1_datapoint2': (5.678, collectionTime),
                    },
                'cpu1': {
                    # datapoints can be specified per datasource...
                    'datasource1_user': (12.1, collectionTime),
                    'datasource2_user': (13.2, collectionTime),
                    # or just by id
                    'datasource1_system': (1.21, collectionTime),
                    'io': (23, collectionTime),
                    }
                },
 
            'maps': [
                ObjectMap(...),
                RelationshipMap(..),
                ]
            }
 
    def onError(self, result, config):
        """
        Called only on error. After onResult, before onComplete.
 
        You can omit this method if you want the error result of the collect
        method to be used without further processing. It recommended to
        implement this method to capture errors.
        """
        return {
            'events': [{
                'summary': 'error: %s' % result,
                'eventKey': 'myPlugin_result',
                'severity': 4,
                }],
            }
 
    def onComplete(self, result, config):
        """
        Called last for success and error.
 
        You can omit this method if you want the result of either the
        onSuccess or onError method to be used without further processing.
        """
        return result
 
    def cleanup(self, config):
        """
        Called when collector exits, or task is deleted or changed.
        """
        return

Extending the Python Data Source Type

To extend the Python data source type to create a new data source type you will absolutely need to create a ZenPack to contain your new data source type. Assuming you have a ZenPack named ZenPacks.example.PackName you would create a ZenPacks/example/PackName/datasources/MyDataSource.py file with contents like the following.

from zope.component import adapts
from zope.interface import implements
 
from Products.Zuul.form import schema
from Products.Zuul.infos import ProxyProperty
from Products.Zuul.infos.template import RRDDataSourceInfo
from Products.Zuul.interfaces import IRRDDataSourceInfo
from Products.Zuul.utils import ZuulMessageFactory as _t
 
from ZenPacks.zenoss.PythonCollector.datasources.PythonDataSource \
    import PythonDataSource, PythonDataSourcePlugin
 
 
class MyDataSource(PythonDataSource):
    """Explanation of what MyDataSource does."""
 
    ZENPACKID = 'ZenPacks.example.PackName'
 
    # Friendly name for your data source type in the drop-down selection.
    sourcetypes = ('MyDataSource',)
    sourcetype = sourcetypes[0]
 
    # Collection plugin for this type. Defined below in this file.
    plugin_classname = 'ZenPacks.example.PackName.datasources.MyDataSource.MyDataSourcePlugin'
 
    # Extra attributes for my type.
    extra1 = ''
    extra2 = ''
 
    # Registering types for my attributes.
    _properties = PythonDataSource._properties + (
        {'id': 'extra1', 'type': 'string'},
        {'id': 'extra2', 'type': 'string'},
        )
 
 
class IMyDataSourceInfo(IRRDDataSourceInfo):
    """Interface that creates the web form for this data source type."""
 
    cycletime = schema.TextLine(
        title=_t(u'Cycle Time (seconds)'))
 
    extra1 = schema.TextLine(
        group=_t('MyDataSource'),
        title=_t('Extra 1'))
 
    extra2 = schema.TextLine(
        group=_t('MyDataSource'),
        title=_t('Extra 1'))
 
 
class MyDataSourceInfo(RRDDataSourceInfo):
    """Adapter between IMyDataSourceInfo and MyDataSource."""
 
    implements(IMyDataSourceInfo)
    adapts(MyDataSource)
 
    testable = False
 
    cycletime = ProxyProperty('cycletime')
 
    extra1 = ProxyProperty('extra1')
    extra2 = ProxyProperty('extra2')
 
 
class MyDataSourcePlugin(PythonDataSourcePlugin):
    """
    Collection plugin class for MyDataSource.
 
    See the "Using the Python Data Source Type Directly" section above for
    an example implementation.
    """
    pass

Changes

1.7.2 (2015-08-27)

• Add "blockingwarning" option to zenpython.
• Add detailed task state tracking to zenpython plugin execution.
• Add "percentBlocked" metric to zenpython.
• Restore compatibility with Zenoss 4.1.

1.7.1 (2015-07-30)

• Avoid a tight loop when writing metrics and events. (ZEN-18956)
• Switch back to epoll reactor. No select-dependent plugins left.

1.7.0 (2015-07-06)

• Fix datapoint format for Control Center metrics.
• Fix potential applyDataMaps traceback. (ZEN-17249)
• Add twistedthreadpoolsize configuration option to zenpython.
• Add optional startDelay property to PythonDataSourcePlugin.

1.6.4 (2015-03-30)

• Fix serviced datapoint format syntax. (ZEN-17255)

1.6.3 (2015-02-10)

• Revert to select reactor. Some plugins require it. (ZEN-16542)

1.6.2 (2015-01-27)

• Optimize datasource plugin loading. (ZEN-16344)
• Use epoll reactor to support >1024 descriptors. (ZEN-16164)

1.6.1 (2015-01-13)

• Add container Support for Zenoss 5X (Europa) services including RabbitMQ.

1.6.0 (2014-11-04)

• Provide PythonDataSourcePlugin instances access to hub services.
• Add --ignore and --collect options for zenpython.
• Handle Decimal performance values.
• Fix indexing bug when adding components with only an "id" property.

1.5.3 (2014-09-29)

• Switch to Zenoss 5 writeMetricWithMetadata() API.

1.5.2 (2014-09-25)

• Fix bug that causes device corruption on retried model transactions.
• Add support for Zenoss 5 writeMetric() API.

1.5.1 (2014-07-24)

• Fix bug in handling of long-typed values.

1.5.0 (2014-07-03)

• Fix application of maps in "run" mode.
• Support TALES evaluation of datapoint properties.
• Cleanup irrelevant errors when zenpython is forcefully stopped.
• Convert value timestamps to int for Zenoss 4.1 compatibility.
• Fix illegal update errors when attempting to write old values.
• Support collection of data for multiple timestamps in one interval.

1.4.0 (2014-04-02)

• Support callables in PythonDataSourcePlugin.proxy_attributes.

1.3.0 (2014-03-20)

• Optionally pass config into datasource plugins' __init__.
• Support ds_dp syntax for data['values'] keys.
• Support None return from datasource plugins' collect method.

1.2.0 (2013-11-25)

• Add cleanup hook for datasource plugins.

1.1.1 (2013-09-19)

• Improve incremental modeling support.

1.1.0 (2013-08-22)

• Support model updates from datasource plugins.
• Support incremental modeling.

1.0.2 (2013-07-29)

• Initial release.

Installation

Normal Installation (packaged egg)

1. Download the appropriate egg file for the version of Zenoss you are running.
2. Ensure you are logged in as the zenoss user:

   $ sudo su - zenoss

3. Install the ZenPack:

   $ zenpack --install ZenPacks.zenoss.PythonCollector-*.egg

4. Restart these services:

   $ zenoss restart

Developer Mode Installation

In order to do a development mode installation you will want to clone the existing git repository, and then use the --link flag with the zenpack command:

1. Ensure you are logged in as the zenoss user:

   $ sudo su - zenoss

2. Start by cloning the upstream repository:

   $ git clone git://github.com/zenoss/ZenPacks.zenoss.PythonCollector.git

3. Next, perform the installation:

   $ zenpack --link --install ZenPacks.zenoss.PythonCollector

4. Finally, restart these serivices:

   $ zenoss restart

Discuss

New: Don't forget to add yourself to the Zenoss User Map!