[op5-users] Check lantency with merlin

[Andreas Ericsson]

Joerg Linge wrote:
> Joerg Linge schrieb:
>> Andreas Ericsson schrieb:
>>> Joerg Linge wrote:
>>>> Hi List,
>>>>
>>>> just tested merlin for about 4 days.
>>>> The check latency rises from 0.2s to 2.3s while merlin is running.
>>>>
>>>> http://www.pnp4nagios.org/merlin_latency.png
>>>>
>>>> Is this a known behavior? 
>>>>
>>> Yes and no. Latency is expected to rise slightly to slightly higher
>>> but equally steady level. This is true as soon as you're loading a
>>> module of any kind. I hadn't expected such a high increase though,
>>> and I would have expected 0.2 to become roughly 0.4 or 0.5, which
>>> seems to hold true for the service check latency. The Merlin module
>>> isn't really doing very much at all, so it's strange that latency
>>> increases so much, and it's *very* strange that it increases such a
>>> lot for host checks but not for service-checks, since the two are
>>> handled in exactly the same way.
>>>
>>> I wonder how much of the overhead is in Nagios. Could you try just
>>> inserting a "return 0" at the top of hook_host_result() and the
>>> same at the top of hook_service_result() in hooks.c and see if the
>>> pattern still holds true? If it does, I'll have a peak at Nagios
>>> and see if it does something strange for hostcheck callbacks that
>>> can be optimized. If it doesn't, I'm quite frankly stumped.
>> OK, done.
>>
>> I will send a new graph soon.
>> BTW: can´t see this behavior while running ndomod on this machine.
> 
> OK, next graph
> 
> http://www.pnp4nagios.org/merlin_latency-2.png
> 
> 11:40 - 13:30 merlin.so running with "return 0;" after hook_service_result() an hook_host_result()
> 
> 13:30 - 15:10 orignal merlin.so
> 

Thanks for the info. It's utterly bizarre that the service check latency
follows the expected pattern and only results in a very small increase
in latency while host checks have such huge spikes.

Since the spiky pattern doesn't appear with the nearly-noop host hook,
I'm forced to conclude that *something* inside Merlin is causing this
but I can't for the life of me figure out why :-/

Has anyone else noticed a similar behaviour?

/Andreas