Difference between revisions of "ZenPack:PythonCollector"
Chet Luther (Talk | contribs) (Replace 1.7.0 release with 1.7.1.) |
Chet Luther (Talk | contribs) (Replace 1.7.1 release with 1.7.2.) |
||
Line 9: | Line 9: | ||
|Flavor=free | |Flavor=free | ||
|Releases={{Release | |Releases={{Release | ||
− | |Version=1.7. | + | |Version=1.7.2 |
− | |Tag=1.7. | + | |Tag=1.7.2 |
− | |Release date=2015/ | + | |Release date=2015/08/27 |
|Compatible with=Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x, Zenoss Resource Manager 5.0.x | |Compatible with=Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x, Zenoss Resource Manager 5.0.x | ||
}}{{Release | }}{{Release | ||
Line 55: | Line 55: | ||
}} | }} | ||
This ZenPack provides a new ''Python'' data source type. It also provides a new ''zenpython'' collector daemon that is responsible for collecting these data sources. | This ZenPack provides a new ''Python'' data source type. It also provides a new ''zenpython'' collector daemon that is responsible for collecting these data sources. | ||
+ | |||
+ | == Usage == | ||
+ | |||
+ | This zenpack adds a new ''zenpython'' collector daemon. In most cases there is nothing you need to know about to make use of this additional collector functionality. The following sections describe available configuration, troubleshooting and tuning information. | ||
+ | |||
+ | === ZenPython Configuration Options === | ||
+ | |||
+ | The following options can be specified in the zenpython configuration file. Typically the default values for these options are appropriate and don't need to be adjusted. | ||
+ | |||
+ | ;blockingwarning | ||
+ | : The zenpython collector daemon executes plugin code provided by other ZenPacks. If this plugin code blocks for too long it will prevent zenpython from performing other tasks including collecting other datasources while the plugin code is executed. The ''blockingwarning'' option will cause zenpython to log a warning for any plugin code that blocks for the configured number of seconds or more. The default value is 30. Decimal precision such as 0.5 can be used. | ||
+ | |||
+ | ;twistedthreadpoolsize | ||
+ | : Controls size of threads pool. Datasources can use multi-threading to run multiple requests in parallel. Increasing this value may boost performance at the cost of system memory used. The default value is 10. | ||
+ | |||
+ | ;collect | ||
+ | : Allows only specific plugins to run. This is primarily a developer option to help reduce the noise while developing plugins. The default is to collect all configured plugins. The value for this option is a regular expression. Only plugins which class name matches the regular expression will be run. | ||
+ | |||
+ | ;ignore | ||
+ | : Prevents specific plugins from running. This is primarily a developer option to help reduce the noise while developing plugins. The default is to not ignore any configured plugins. The value for this option is a regular expression. Only plugins which class name doesn't match the regular expression will be run. | ||
+ | |||
+ | === ZenPython Statistics === | ||
+ | |||
+ | There are two kinds of statistics available from the ''zenpython'' collector daemon. There are datapoints which can be plotted on graphs and have thresholds defined, and there are detailed task statistics that can be logged. | ||
+ | |||
+ | ==== Datapoints ==== | ||
+ | |||
+ | The following standard collector datapoints are available for ''zenpython''. These datapoints are provided by the Zenoss platform and may differ depending on which version of Zenoss is being used. | ||
+ | |||
+ | * ''devices'': Number of devices being collected. | ||
+ | * ''dataPoints'': Number of datapoints collected per second. | ||
+ | * ''eventCount'': Number of events sent per second. | ||
+ | * ''discardedEvents'': Number of events discarded per second. | ||
+ | * ''eventQueueLength'': Number of events queued to be set to zenhub. | ||
+ | * ''taskCount'': Total number of configured tasks. | ||
+ | * ''runningTasks'': Number of tasks in the running state. | ||
+ | * ''queuedTasks'': Number of tasks queued waiting to be run. | ||
+ | * ''missedRuns'': Number of tasks that missed their scheduled run time. | ||
+ | |||
+ | The following additional collector datapoonts are available for ''zenpython''. These datapoints are provided by the collector daemon and may differ depending on which version of PythonCollector is being used. | ||
+ | |||
+ | * ''percentBlocked'': Percent of the time blocked by plugin code execution. | ||
+ | |||
+ | ==== Task Statistics ==== | ||
+ | |||
+ | The datapoints above provide for a good high-level understanding of the overall trends for the zenpython collector daemon. When certain datapoints show a potential problem it can often be useful to capture detailed per-task statistics to identify which tasks are contributing to the problem. | ||
+ | |||
+ | The zenpython process will dump detailed per-task information to its regular log file if it sent the SIGUSR2 signal. Depending on the Zenoss version being used, different utilities are provided for invoking this signal. | ||
+ | |||
+ | * Zenoss 5: ''serviced service action zenpython stats'' | ||
+ | * Zenoss 4: ''zenpython stats'' | ||
+ | |||
+ | Running those commands won't produce any direct output. You must look at the zenpython log to see the resulting statistics. | ||
+ | |||
+ | == Developing Plugins == | ||
The goal of the ''Python'' data source type is to replicate some of the standard ''COMMAND'' data source type's functionality without requiring a new shell and shell subprocess to be spawned each time the data source is collected. The ''COMMAND'' data source type is infinitely flexible, but because of the shell and subprocess spawning, it's performance and ability to pass data into the collection script are limited. The ''Python'' data source type circumvents the need to spawn subprocesses by forcing the collection code to be asynchronous using the Twisted library. It circumvents the problem with passing data into the collection logic by being able to pass any basic Python data type without the need to worry about shell escaping issues. | The goal of the ''Python'' data source type is to replicate some of the standard ''COMMAND'' data source type's functionality without requiring a new shell and shell subprocess to be spawned each time the data source is collected. The ''COMMAND'' data source type is infinitely flexible, but because of the shell and subprocess spawning, it's performance and ability to pass data into the collection script are limited. The ''Python'' data source type circumvents the need to spawn subprocesses by forcing the collection code to be asynchronous using the Twisted library. It circumvents the problem with passing data into the collection logic by being able to pass any basic Python data type without the need to worry about shell escaping issues. | ||
Line 304: | Line 359: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
− | == | + | == Changes == |
− | ; | + | ;1.7.2 (2015-08-27) |
− | * | + | * Add "blockingwarning" option to zenpython. |
− | + | * Add detailed task state tracking to zenpython plugin execution. | |
− | + | * Add "percentBlocked" metric to zenpython. | |
+ | * Restore compatibility with Zenoss 4.1. | ||
;1.7.1 (2015-07-30) | ;1.7.1 (2015-07-30) |
Revision as of 18:56, 27 August 2015
- Current Maintainer(s)
- Zenoss,Inc.
- Organization
- Zenoss, Inc.
- License
- GNU General Public License, Version 2, or later
- ZenPack name
- ZenPacks.zenoss.PythonCollector
- More Information
- GitHub page/HomePage
- Git sources (for cloning)
- Link
PythonCollector ZenPack
Warning
The ZenPack Catalog has moved to its new home at https://www.zenoss.com/product/zenpacks as of January 17, 2017. The following information may be out of date, and this page will eventually be removed.
Support
This is an Open Source ZenPack developed by Zenoss, Inc. Enterprise support for this ZenPack is available to commercial customers with an active subscription.
Releases
- Version 1.7.2- Download
- Released on 2015/08/27
- Compatible with Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x, Zenoss Resource Manager 5.0.x
- Version 1.6.4- Download
- Released on 2015/03/30
- Compatible with Zenoss Core 4.2.x, Zenoss Core 5.0.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x, Zenoss Resource Manager 5.0.x
- Version 1.5.3- Download
- Released on 2014/09/29
- Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
- Version 1.4.0- Download
- Released on 2014/04/02
- Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
- Version 1.3.0- Download
- Released on 2014/03/20
- Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
- Version 1.2.0- Download
- Released on 2013/11/25
- Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
- Version 1.1.1- Download
- Released on 2013/09/19
- Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
- Version 1.0.2- Download
- Released on 2013/07/29
- Compatible with Zenoss Core 4.2.x, Zenoss Resource Manager 4.1.x, Zenoss Resource Manager 4.2.x
Background
This ZenPack provides a new Python data source type. It also provides a new zenpython collector daemon that is responsible for collecting these data sources.
Usage
This zenpack adds a new zenpython collector daemon. In most cases there is nothing you need to know about to make use of this additional collector functionality. The following sections describe available configuration, troubleshooting and tuning information.
ZenPython Configuration Options
The following options can be specified in the zenpython configuration file. Typically the default values for these options are appropriate and don't need to be adjusted.
- blockingwarning
- The zenpython collector daemon executes plugin code provided by other ZenPacks. If this plugin code blocks for too long it will prevent zenpython from performing other tasks including collecting other datasources while the plugin code is executed. The blockingwarning option will cause zenpython to log a warning for any plugin code that blocks for the configured number of seconds or more. The default value is 30. Decimal precision such as 0.5 can be used.
- twistedthreadpoolsize
- Controls size of threads pool. Datasources can use multi-threading to run multiple requests in parallel. Increasing this value may boost performance at the cost of system memory used. The default value is 10.
- collect
- Allows only specific plugins to run. This is primarily a developer option to help reduce the noise while developing plugins. The default is to collect all configured plugins. The value for this option is a regular expression. Only plugins which class name matches the regular expression will be run.
- ignore
- Prevents specific plugins from running. This is primarily a developer option to help reduce the noise while developing plugins. The default is to not ignore any configured plugins. The value for this option is a regular expression. Only plugins which class name doesn't match the regular expression will be run.
ZenPython Statistics
There are two kinds of statistics available from the zenpython collector daemon. There are datapoints which can be plotted on graphs and have thresholds defined, and there are detailed task statistics that can be logged.
Datapoints
The following standard collector datapoints are available for zenpython. These datapoints are provided by the Zenoss platform and may differ depending on which version of Zenoss is being used.
- devices: Number of devices being collected.
- dataPoints: Number of datapoints collected per second.
- eventCount: Number of events sent per second.
- discardedEvents: Number of events discarded per second.
- eventQueueLength: Number of events queued to be set to zenhub.
- taskCount: Total number of configured tasks.
- runningTasks: Number of tasks in the running state.
- queuedTasks: Number of tasks queued waiting to be run.
- missedRuns: Number of tasks that missed their scheduled run time.
The following additional collector datapoonts are available for zenpython. These datapoints are provided by the collector daemon and may differ depending on which version of PythonCollector is being used.
- percentBlocked: Percent of the time blocked by plugin code execution.
Task Statistics
The datapoints above provide for a good high-level understanding of the overall trends for the zenpython collector daemon. When certain datapoints show a potential problem it can often be useful to capture detailed per-task statistics to identify which tasks are contributing to the problem.
The zenpython process will dump detailed per-task information to its regular log file if it sent the SIGUSR2 signal. Depending on the Zenoss version being used, different utilities are provided for invoking this signal.
- Zenoss 5: serviced service action zenpython stats
- Zenoss 4: zenpython stats
Running those commands won't produce any direct output. You must look at the zenpython log to see the resulting statistics.
Developing Plugins
The goal of the Python data source type is to replicate some of the standard COMMAND data source type's functionality without requiring a new shell and shell subprocess to be spawned each time the data source is collected. The COMMAND data source type is infinitely flexible, but because of the shell and subprocess spawning, it's performance and ability to pass data into the collection script are limited. The Python data source type circumvents the need to spawn subprocesses by forcing the collection code to be asynchronous using the Twisted library. It circumvents the problem with passing data into the collection logic by being able to pass any basic Python data type without the need to worry about shell escaping issues.
The Python data source type is intended to be used in one of two ways. The first way is directly through the creation of Python data sources through the web interface or in a ZenPack. When used in this way, it is the responsibility of the data source creator to implement the required Python class specified in the data source's Python Class Name property field. The second way the Python data source can be used is as a base class for another data source type. Used in this way, the ZenPack author will create a subclass of PythonDataSource to provide a higher-level functionality to the user. The user is then not responsible for writing a Python class to collect and process data.
Using the Python Data Source Type Directly
To create a Python data source directly you should first implement the Python class you'll eventually use for the data source's Plugin Class Name. It is recommended to implement this class in a ZenPack so that it is portable from one Zenoss system to another and differentiates your custom code from Zenoss code.
Assuming you have a ZenPack named ZenPacks.example.PackName you would create a ZenPacks/example/PackName/dsplugins.py file with contents like the following.
import time from Products.ZenEvents import ZenEventClasses from ZenPacks.zenoss.PythonCollector.datasources.PythonDataSource \ import PythonDataSourcePlugin class MyPlugin(PythonDataSourcePlugin): """Explanation of what MyPlugin does.""" # List of device attributes you'll need to do collection. proxy_attributes = ( 'zCommandUsername', 'zCommandPassword', ) @classmethod def config_key(cls, datasource, context): """ Return a tuple defining collection uniqueness. This is a classmethod that is executed in zenhub. The datasource and context parameters are the full objects. This example implementation is the default. Split configurations by device, cycle time, template id, datasource id and the Python data source's plugin class name. You can omit this method from your implementation entirely if this default uniqueness behavior fits your needs. In many cases it will. """ return ( context.device().id, datasource.getCycleTime(context), datasource.rrdTemplate().id, datasource.id, datasource.plugin_classname, ) @classmethod def params(cls, datasource, context): """ Return params dictionary needed for this plugin. This is a classmethod that is executed in zenhub. The datasource and context parameters are the full objects. This example implementation will provide no extra information for each data source to the collect method. You can omit this method from your implementation if you don't require any additional information on each of the datasources of the config parameter to the collect method below. If you only need extra information at the device level it is easier to just use proxy_attributes as mentioned above. """ return {} def collect(self, config): """ No default collect behavior. You must implement this method. This method must return a Twisted deferred. The deferred results will be sent to the onResult then either onSuccess or onError callbacks below. """ ds0 = config.datasources[0] return somethingThatReturnsADeferred( username=ds0.zCommandUsername, password=ds0.zCommandPassword) def onResult(self, result, config): """ Called first for success and error. You can omit this method if you want the result of the collect method to be used without further processing. """ return result def onSuccess(self, result, config): """ Called only on success. After onResult, before onComplete. You should return a data structure with zero or more events, values and maps. """ collectionTime = time.time() return { 'events': [{ 'summary': 'successful collection', 'eventKey': 'myPlugin_result', 'severity': ZenEventClasses.Clear, },{ 'summary': 'first event summary', 'eventKey': 'myPlugin_result', 'severity': ZenEventClasses.Info, },{ 'summary': 'second event summary', 'eventKey': 'myPlugin_result', 'severity': ZenEventClasses.Warning, }], 'values': { None: { # datapoints for the device (no component) 'datasource1_datapoint1': (123.4, collectionTime), 'datasource1_datapoint2': (5.678, collectionTime), }, 'cpu1': { # datapoints can be specified per datasource... 'datasource1_user': (12.1, collectionTime), 'datasource2_user': (13.2, collectionTime), # or just by id 'datasource1_system': (1.21, collectionTime), 'io': (23, collectionTime), } }, 'maps': [ ObjectMap(...), RelationshipMap(..), ] } def onError(self, result, config): """ Called only on error. After onResult, before onComplete. You can omit this method if you want the error result of the collect method to be used without further processing. It recommended to implement this method to capture errors. """ return { 'events': [{ 'summary': 'error: %s' % result, 'eventKey': 'myPlugin_result', 'severity': 4, }], } def onComplete(self, result, config): """ Called last for success and error. You can omit this method if you want the result of either the onSuccess or onError method to be used without further processing. """ return result def cleanup(self, config): """ Called when collector exits, or task is deleted or changed. """ return
Extending the Python Data Source Type
To extend the Python data source type to create a new data source type you will absolutely need to create a ZenPack to contain your new data source type. Assuming you have a ZenPack named ZenPacks.example.PackName you would create a ZenPacks/example/PackName/datasources/MyDataSource.py file with contents like the following.
from zope.component import adapts from zope.interface import implements from Products.Zuul.form import schema from Products.Zuul.infos import ProxyProperty from Products.Zuul.infos.template import RRDDataSourceInfo from Products.Zuul.interfaces import IRRDDataSourceInfo from Products.Zuul.utils import ZuulMessageFactory as _t from ZenPacks.zenoss.PythonCollector.datasources.PythonDataSource \ import PythonDataSource, PythonDataSourcePlugin class MyDataSource(PythonDataSource): """Explanation of what MyDataSource does.""" ZENPACKID = 'ZenPacks.example.PackName' # Friendly name for your data source type in the drop-down selection. sourcetypes = ('MyDataSource',) sourcetype = sourcetypes[0] # Collection plugin for this type. Defined below in this file. plugin_classname = 'ZenPacks.example.PackName.datasources.MyDataSource.MyDataSourcePlugin' # Extra attributes for my type. extra1 = '' extra2 = '' # Registering types for my attributes. _properties = PythonDataSource._properties + ( {'id': 'extra1', 'type': 'string'}, {'id': 'extra2', 'type': 'string'}, ) class IMyDataSourceInfo(IRRDDataSourceInfo): """Interface that creates the web form for this data source type.""" cycletime = schema.TextLine( title=_t(u'Cycle Time (seconds)')) extra1 = schema.TextLine( group=_t('MyDataSource'), title=_t('Extra 1')) extra2 = schema.TextLine( group=_t('MyDataSource'), title=_t('Extra 1')) class MyDataSourceInfo(RRDDataSourceInfo): """Adapter between IMyDataSourceInfo and MyDataSource.""" implements(IMyDataSourceInfo) adapts(MyDataSource) testable = False cycletime = ProxyProperty('cycletime') extra1 = ProxyProperty('extra1') extra2 = ProxyProperty('extra2') class MyDataSourcePlugin(PythonDataSourcePlugin): """ Collection plugin class for MyDataSource. See the "Using the Python Data Source Type Directly" section above for an example implementation. """ pass
Changes
- 1.7.2 (2015-08-27)
- Add "blockingwarning" option to zenpython.
- Add detailed task state tracking to zenpython plugin execution.
- Add "percentBlocked" metric to zenpython.
- Restore compatibility with Zenoss 4.1.
- 1.7.1 (2015-07-30)
- Avoid a tight loop when writing metrics and events. (ZEN-18956)
- Switch back to epoll reactor. No select-dependent plugins left.
- 1.7.0 (2015-07-06)
- Fix datapoint format for Control Center metrics.
- Fix potential applyDataMaps traceback. (ZEN-17249)
- Add twistedthreadpoolsize configuration option to zenpython.
- Add optional startDelay property to PythonDataSourcePlugin.
- 1.6.4 (2015-03-30)
- Fix serviced datapoint format syntax. (ZEN-17255)
- 1.6.3 (2015-02-10)
- Revert to select reactor. Some plugins require it. (ZEN-16542)
- 1.6.2 (2015-01-27)
- Optimize datasource plugin loading. (ZEN-16344)
- Use epoll reactor to support >1024 descriptors. (ZEN-16164)
- 1.6.1 (2015-01-13)
- Add container Support for Zenoss 5X (Europa) services including RabbitMQ.
- 1.6.0 (2014-11-04)
- Provide PythonDataSourcePlugin instances access to hub services.
- Add --ignore and --collect options for zenpython.
- Handle Decimal performance values.
- Fix indexing bug when adding components with only an "id" property.
- 1.5.3 (2014-09-29)
- Switch to Zenoss 5 writeMetricWithMetadata() API.
- 1.5.2 (2014-09-25)
- Fix bug that causes device corruption on retried model transactions.
- Add support for Zenoss 5 writeMetric() API.
- 1.5.1 (2014-07-24)
- Fix bug in handling of long-typed values.
- 1.5.0 (2014-07-03)
- Fix application of maps in "run" mode.
- Support TALES evaluation of datapoint properties.
- Cleanup irrelevant errors when zenpython is forcefully stopped.
- Convert value timestamps to int for Zenoss 4.1 compatibility.
- Fix illegal update errors when attempting to write old values.
- Support collection of data for multiple timestamps in one interval.
- 1.4.0 (2014-04-02)
- Support callables in PythonDataSourcePlugin.proxy_attributes.
- 1.3.0 (2014-03-20)
- Optionally pass config into datasource plugins' __init__.
- Support ds_dp syntax for data['values'] keys.
- Support None return from datasource plugins' collect method.
- 1.2.0 (2013-11-25)
- Add cleanup hook for datasource plugins.
- 1.1.1 (2013-09-19)
- Improve incremental modeling support.
- 1.1.0 (2013-08-22)
- Support model updates from datasource plugins.
- Support incremental modeling.
- 1.0.2 (2013-07-29)
- Initial release.
Installation
Normal Installation (packaged egg)
- Download the appropriate egg file for the version of Zenoss you are running.
- Ensure you are logged in as the zenoss user:
$ sudo su - zenoss
- Install the ZenPack:
$ zenpack --install ZenPacks.zenoss.PythonCollector-*.egg
- Restart these services:
$ zenoss restart
Developer Mode Installation
In order to do a development mode installation you will want to clone the existing git repository, and then use the --link flag with the zenpack command:
- Ensure you are logged in as the zenoss user:
$ sudo su - zenoss
- Start by cloning the upstream repository:
$ git clone git://github.com/zenoss/ZenPacks.zenoss.PythonCollector.git
- Next, perform the installation:
$ zenpack --link --install ZenPacks.zenoss.PythonCollector
- Finally, restart these serivices:
$ zenoss restart
Discuss
New: Don't forget to add yourself to the Zenoss User Map!