Debugging ZenModeler and ZenHub
Note: These are my personal notes from a conversation with Eric Edgar -- they are very rough and need editing...
For starting zenhub for debugging, just start a new one on a different port:
zenhub run -v10 --pbport=8790 --xmlrpc=8082 --workers=0
"workers=0" command prevents background zenhub processes during pdb debugging.
Now we have a new stand-alone representation of all the devices and components. It connects to relstorage and has a direct connection to the database. No modeling is happening so the state will not change.
No other daemons are connected to it.
This way, we don't have a lot of extra stuff going on to confuse us -- model updates, or performance data coming in.
Go to another window with ZenHub running in background in another window
$ zenmodeler run v10 --device=test-rhe6.zenoss.loc --collect="zenoss.snmp.NewDeviceMap" --hubport=8790
Just get debug for that modeler.
If we are going to debug the remote collector daemon, look at zenmodeler.py
vim $ZENHOME/bin/zenmodeler # (and it will say near the top, zenmodeler.py)
Zenmodeler will then request a twisted service that has been registered with zenhub. This exists in
Products/ZenHub/services/ModelerService.py. This is the remote side that it will send proxy objects back to.
ModelerService/CreateDeviceProxy (this is executed on the ZenHub side)
Put a pdb in the front of
remote_getDeviceConfig. So this a method is executed within ZenHub and it sends the results back to the the zenmodeler daemon.
log.info("BLAH") will print out where you are. Then restart ZenHub so the pdb takes effect. Do n, n, n, (no good info, async crap, because it jumps into twisted async land). So you might want to put the pdb after the remote call and examine the results.
self.config().callRemote('getDeviceConfig'…) ----> twisted --->
remote_getDeviceConfig (on hub side) and remote_getDeviceConfig returns the data.
ON ZENHUB SIDE, CREATE A LIST OF DEVICES TO MODEL
On the zenhub side, we skip sending back any data if it is ping down, or if it is below a production state threshold.
createDeviceProxy() - representing the current state -- happens on the ZenHub side. It gets sent back.
Next step is that zenhub will import the modeler plugin on the zenhub side…. if you have a typo in your modeler plugin, zenhub won't even load it. You will get a traceback on the hub side. This happens with the
loadPlugins() method. ALL modeler plugins will be loaded on the ZenHub side.
in pdb, you can inspect self.plugins to see all the plugins being imported on the ZenHub side. For any traceback on a modeler plugin at load time, *zenhub will just skip it and continue* -- without alerting you….
To test a modeler plugin for syntax errors, it may be possible to use zendmd to import it (ask Chet)
It can be tricky to detect syntax errors in modeler plugins for this reason. Zenoss (newer versions) *myay* fire off an event. In zenhub log you would see an exception logged for a syntax error.
then zenhub runs a method inside each modeler plugin, to see if it should even run against a device. This is done by calling the condition() method of the Plugin object. And the default True, to run it.
ON THE MODELER SIDE,
it gets a list of: (device proxy and a list of all plugins that apply to that device)
A MODELER DAEMON WILL CACHE THE DEVICE PROXY UNTIL IT HAS BEEN INFORMED THAT IT SHOULD REQUEST A NEW DEVICE PROXY OBJECT FROM ZENHUB BECAUSE ITS STATE HAS CHANGED (invalidation of existing proxy)
So what this means is that a persistent modeler daemon will contain potentially hundreds of device proxy objects that are cached. The reason for this is to avoid the overhead of regenerating the device proxy objects (and doing all the imports) on the zenhub side, to figure out what modeler plugins should run for each device.
the modeler will import all modeler plugins and then run the appropriate ones against each device proxy. You can use the --ignore or --collect flags to tweak this behavior.
when a device is deleted from the UI, there needs to be a twisted call to all modeler daemons to flush the device from the modeler daemon.
ON MODELER DAEMON STARTUP, AND EVERY 12 HOURS AFTER STARTUP AND/OR A COLLECTOR-SPECFIC SETTING… (modeler cycle interval from the collectors setting in the UI) There is a configure() method in the modeler plugin and it will use these parameters when it cycles and remodels. There is no easy way to see when zenmodel will happen. One workaround is to turn off zenmodel as a daemon and run it from cron…. just do zenmodeler run. This puts more load on zenhub (recreating all the device proxies) but you can schedule it in off-peak hours.
-- files for services/daemons --
services folder in ZenPack, which contains services files loaded by ZenHub, and these files allow it to connect to the new daemon. Then there is /opt/zenoss/Products/ZenHub/services … this one contains all the code to connect to the core daemons.