ZenPack:PythonCollector

From Zenoss Wiki
Revision as of 20:11, 13 January 2015 by Pcarinhas (Talk | contribs)$7

Jump to: navigation, search
Current Maintainer(s)
Zenoss,Inc.
Organization
Zenoss, Inc.
License
GNU General Public License, Version 2, or later
ZenPack name
ZenPacks.zenoss.PythonCollector
More Information
GitHub page/HomePage
Git sources (for cloning)
Link




PythonCollector ZenPack

Adds a "Python" data source type for asynchronous collection using a custom Python module.

Warning

The ZenPack Catalog has moved to its new home at https://www.zenoss.com/product/zenpacks as of January 17, 2017. The following information may be out of date, and this page will eventually be removed.

Support

This is an Open Source ZenPack developed by Zenoss, Inc. Enterprise support for this ZenPack is available to commercial customers with an active subscription.

Releases

Version 1.6.1- Download
Summary of changes: Adds RabbitMQ connectivity for Zenoss 5X (Europa)
Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
Version 1.6.0- Download
Released on 2014/11/04
Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
Version 1.5.3- Download
Released on 2014/09/29
Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
Version 1.4.0- Download
Released on 2014/04/02
Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
Version 1.3.0- Download
Released on 2014/03/20
Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
Version 1.2.0- Download
Released on 2013/11/25
Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
Version 1.1.1- Download
Summary of changes: Better incremental modeling support.
Released on 2013/09/19
Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
Version 1.0.2- Download
Released on 2013/07/29
Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x

Background

This ZenPack provides a new Python data source type. It also provides a new zenpython collector daemon that is responsible for collecting these data sources.

The goal of the Python data source type is to replicate some of the standard COMMAND data source type's functionality without requiring a new shell and shell subprocess to be spawned each time the data source is collected. The COMMAND data source type is infinitely flexible, but because of the shell and subprocess spawning, it's performance and ability to pass data into the collection script are limited. The Python data source type circumvents the need to spawn subprocesses by forcing the collection code to be asynchronous using the Twisted library. It circumvents the problem with passing data into the collection logic by being able to pass any basic Python data type without the need to worry about shell escaping issues.

The Python data source type is intended to be used in one of two ways. The first way is directly through the creation of Python data sources through the web interface or in a ZenPack. When used in this way, it is the responsibility of the data source creator to implement the required Python class specified in the data source's Python Class Name property field. The second way the Python data source can be used is as a base class for another data source type. Used in this way, the ZenPack author will create a subclass of PythonDataSource to provide a higher-level functionality to the user. The user is then not responsible for writing a Python class to collect and process data.

Using the Python Data Source Type Directly

To create a Python data source directly you should first implement the Python class you'll eventually use for the data source's Plugin Class Name. It is recommended to implement this class in a ZenPack so that it is portable from one Zenoss system to another and differentiates your custom code from Zenoss code.

Assuming you have a ZenPack named ZenPacks.example.PackName you would create a ZenPacks/example/PackName/dsplugins.py file with contents like the following.

import time
 
from Products.ZenEvents import ZenEventClasses
 
from ZenPacks.zenoss.PythonCollector.datasources.PythonDataSource \
    import PythonDataSourcePlugin
 
class MyPlugin(PythonDataSourcePlugin):
    """Explanation of what MyPlugin does."""
 
    # List of device attributes you'll need to do collection.
    proxy_attributes = (
        'zCommandUsername',
        'zCommandPassword',
        )
 
    @classmethod
    def config_key(cls, datasource, context):
        """
        Return a tuple defining collection uniqueness.
 
        This is a classmethod that is executed in zenhub. The datasource and
        context parameters are the full objects.
 
        This example implementation is the default. Split configurations by
        device, cycle time, template id, datasource id and the Python data
        source's plugin class name.
 
        You can omit this method from your implementation entirely if this
        default uniqueness behavior fits your needs. In many cases it will.
        """
        return (
            context.device().id,
            datasource.getCycleTime(context),
            datasource.rrdTemplate().id,
            datasource.id,
            datasource.plugin_classname,
            )
 
    @classmethod
    def params(cls, datasource, context):
        """
        Return params dictionary needed for this plugin.
 
        This is a classmethod that is executed in zenhub. The datasource and
        context parameters are the full objects.
 
        This example implementation will provide no extra information for
        each data source to the collect method.
 
        You can omit this method from your implementation if you don't require
        any additional information on each of the datasources of the config
        parameter to the collect method below. If you only need extra
        information at the device level it is easier to just use
        proxy_attributes as mentioned above.
        """
        return {}
 
    def collect(self, config):
        """
        No default collect behavior. You must implement this method.
 
        This method must return a Twisted deferred. The deferred results will
        be sent to the onResult then either onSuccess or onError callbacks
        below.
        """
        ds0 = config.datasources[0]
        return somethingThatReturnsADeferred(
            username=ds0.zCommandUsername,
            password=ds0.zCommandPassword)
 
    def onResult(self, result, config):
        """
        Called first for success and error.
 
        You can omit this method if you want the result of the collect method
        to be used without further processing.
        """
        return result
 
    def onSuccess(self, result, config):
        """
        Called only on success. After onResult, before onComplete.
 
        You should return a data structure with zero or more events, values
        and maps.
        """
        collectionTime = time.time()
        return {
            'events': [{
                'summary': 'successful collection',
                'eventKey': 'myPlugin_result',
                'severity': ZenEventClasses.Clear,
                },{
                'summary': 'first event summary',
                'eventKey': 'myPlugin_result',
                'severity': ZenEventClasses.Info,
                },{
                'summary': 'second event summary',
                'eventKey': 'myPlugin_result',
                'severity': ZenEventClasses.Warning,
                }],
 
            'values': {
                None: {
                    # datapoints for the device (no component)
                    'datasource1_datapoint1': (123.4, collectionTime),
                    'datasource1_datapoint2': (5.678, collectionTime),
                    },
                'cpu1': {
                    # datapoints can be specified per datasource...
                    'datasource1_user': (12.1, collectionTime),
                    'datasource2_user': (13.2, collectionTime),
                    # or just by id
                    'datasource1_system': (1.21, collectionTime),
                    'io': (23, collectionTime),
                    }
                },
 
            'maps': [
                ObjectMap(...),
                RelationshipMap(..),
                ]
            }
 
    def onError(self, result, config):
        """
        Called only on error. After onResult, before onComplete.
 
        You can omit this method if you want the error result of the collect
        method to be used without further processing. It recommended to
        implement this method to capture errors.
        """
        return {
            'events': [{
                'summary': 'error: %s' % result,
                'eventKey': 'myPlugin_result',
                'severity': 4,
                }],
            }
 
    def onComplete(self, result, config):
        """
        Called last for success and error.
 
        You can omit this method if you want the result of either the
        onSuccess or onError method to be used without further processing.
        """
        return result
 
    def cleanup(self, config):
        """
        Called when collector exits, or task is deleted or changed.
        """
        return

Extending the Python Data Source Type

To extend the Python data source type to create a new data source type you will absolutely need to create a ZenPack to contain your new data source type. Assuming you have a ZenPack named ZenPacks.example.PackName you would create a ZenPacks/example/PackName/datasources/MyDataSource.py file with contents like the following.

from zope.component import adapts
from zope.interface import implements
 
from Products.Zuul.form import schema
from Products.Zuul.infos import ProxyProperty
from Products.Zuul.infos.template import RRDDataSourceInfo
from Products.Zuul.interfaces import IRRDDataSourceInfo
from Products.Zuul.utils import ZuulMessageFactory as _t
 
from ZenPacks.zenoss.PythonCollector.datasources.PythonDataSource \
    import PythonDataSource, PythonDataSourcePlugin
 
 
class MyDataSource(PythonDataSource):
    """Explanation of what MyDataSource does."""
 
    ZENPACKID = 'ZenPacks.example.PackName'
 
    # Friendly name for your data source type in the drop-down selection.
    sourcetypes = ('MyDataSource',)
    sourcetype = sourcetypes[0]
 
    # Collection plugin for this type. Defined below in this file.
    plugin_classname = 'ZenPacks.example.PackName.datasources.MyDataSource.MyDataSourcePlugin'
 
    # Extra attributes for my type.
    extra1 = ''
    extra2 = ''
 
    # Registering types for my attributes.
    _properties = PythonDataSource._properties + (
        {'id': 'extra1', 'type': 'string'},
        {'id': 'extra2', 'type': 'string'},
        )
 
 
class IMyDataSourceInfo(IRRDDataSourceInfo):
    """Interface that creates the web form for this data source type."""
 
    cycletime = schema.TextLine(
        title=_t(u'Cycle Time (seconds)'))
 
    extra1 = schema.TextLine(
        group=_t('MyDataSource'),
        title=_t('Extra 1'))
 
    extra2 = schema.TextLine(
        group=_t('MyDataSource'),
        title=_t('Extra 1'))
 
 
class MyDataSourceInfo(RRDDataSourceInfo):
    """Adapter between IMyDataSourceInfo and MyDataSource."""
 
    implements(IMyDataSourceInfo)
    adapts(MyDataSource)
 
    testable = False
 
    cycletime = ProxyProperty('cycletime')
 
    extra1 = ProxyProperty('extra1')
    extra2 = ProxyProperty('extra2')
 
 
class MyDataSourcePlugin(PythonDataSourcePlugin):
    """
    Collection plugin class for MyDataSource.
 
    See the "Using the Python Data Source Type Directly" section above for
    an example implementation.
    """
    pass

Changes

1.6.0 (2014-11-04)
  • Provide PythonDataSourcePlugin instances access to hub services.
  • Add --ignore and --collect options for zenpython.
  • Handle Decimal performance values.
  • Fix indexing bug when adding components with only an "id" property.
1.5.3 (2014-09-29)
  • Switch to Zenoss 5 writeMetricWithMetadata() API.
1.5.2 (2014-09-25)
  • Fix bug that causes device corruption on retried model transactions.
  • Add support for Zenoss 5 writeMetric() API.
1.5.1 (2014-07-24)
  • Fix bug in handling of long-typed values.
1.5.0 (2014-07-03)
  • Fix application of maps in "run" mode.
  • Support TALES evaluation of datapoint properties.
  • Cleanup irrelevant errors when zenpython is forcefully stopped.
  • Convert value timestamps to int for Zenoss 4.1 compatibility.
  • Fix illegal update errors when attempting to write old values.
  • Support collection of data for multiple timestamps in one interval.
1.4.0 (2014-04-02)
  • Support callables in PythonDataSourcePlugin.proxy_attributes.
1.3.0 (2014-03-20)
  • Optionally pass config into datasource plugins' __init__.
  • Support ds_dp syntax for data['values'] keys.
  • Support None return from datasource plugins' collect method.
1.2.0 (2013-11-25)
  • Add cleanup hook for datasource plugins.
1.1.1 (2013-09-19)
  • Improve incremental modeling support.
1.1.0 (2013-08-22)
  • Support model updates from datasource plugins.
  • Support incremental modeling.
1.0.2 (2013-07-29)
  • Initial release.

Installation

Normal Installation (packaged egg)

  1. Download the appropriate egg file for the version of Zenoss you are running.
  2. Ensure you are logged in as the zenoss user:
    $ sudo su - zenoss
  3. Install the ZenPack:
    $ zenpack --install ZenPacks.zenoss.PythonCollector-*.egg
  4. Restart these services:
    $ zenoss restart

Developer Mode Installation

In order to do a development mode installation you will want to clone the existing git repository, and then use the --link flag with the zenpack command:

  1. Ensure you are logged in as the zenoss user:
    $ sudo su - zenoss
  2. Start by cloning the upstream repository:
    $ git clone git://github.com/zenoss/ZenPacks.zenoss.PythonCollector.git
  3. Next, perform the installation:
    $ zenpack --link --install ZenPacks.zenoss.PythonCollector
  4. Finally, restart these serivices:
    $ zenoss restart

Discuss

Purplemarker.png New: Don't forget to add yourself to the Zenoss User Map!

blog comments powered by Disqus