[op5-users] Merlin crashed on me?
Frater, Greg J
GJFRATER at bechtel.com
Tue Jun 30 19:03:41 CEST 2009
Hello again,
I have been occupied for a bit but now have some time to get back to
Merlin. I grabbed the latest Merlin snapshot from git. After compiling
it and having it recreate the Merlin database I ran the ulimit -c
unlimited command and started it, no core dumps yet. I am seeing a SQL
error though, and what appears to be a configuration problem, I'm only
getting data in a few tables.
SQL problem (from daemon.log):
[1246379421] 6: dbi_conn_query_null(): Failed to run [UPDATE merlin.host
SET scheduled_downtime_depth = scheduled_downtime_depth + 1 WHERE
host_name = 'host52' AND service_description = '']: 1054: Unknown column
'service_description' in 'where clause'
I'm also getting these, they don't seem to indicate a problem (the data
is in the database) but may be duplicate/unnecessary SQL calls? (also
from daemon.log):
[1246379416] 6: dbi_conn_query_null(): Failed to run [INSERT INTO
merlin.comment(comment_type, host_name, service_description, entry_time,
author_name, comment_data, persistent, source, entry_type, expires,
expire_time, comment_id) VALUES(1, 'host0052', '', 1246379410, '(Nagios
Process)', 'This host has been scheduled for fixed downtime from
06-30-2009 09:29:56 to 06-30-2009 11:29:56. Notifications for the host
will not be sent out during that time period.', 0, 0, 2, 0, 0, 12093)]:
1062: Duplicate entry '12093' for key 2
Plus Merlin does not appear to be writing any host or service data, the
only tables that are showing any data are the comment, program_status,
and scheduled_downtime tables. Here is what I see in the daemon.log,
this pattern is repeated over and over again, I get this for each
host/service.
...
[1246380200] 6: inbound data available on ipc socket
[1246380200] 7: Successfully read 1 NEBCALLBACK_SERVICE_CHECK_DATA event
(555 bytes; 491 bytes body) from socket 7
[1246380200] 3: Failed to get stored state for service 'CPU:
Utilization' on host 'host0052'
[1246380200] 7: sel_val: 7; ipc_listen_sock: 5; ipc_sock: 7; net_sock:
6/usr/local/nagios/merlin/logs/
[1246380200] 7: select() returned 1 (errno = 0: Success)
[1246380200] 6: inbound data available on ipc socket
...
I never get neb.log file, should I? When I start nagios I see a console
message that says 'Starting nagios:Logging to
'/usr/local/nagios/merlin/logs/neb.log' but the log file never appears.
For that matter I don't see the binary log files 'daemon.ipc.read.bin'
and 'daemon.ipc.write.bin' either.
Ah, there's my crash, it dumped while I was writing this message. I
still did not find any core dump files, I checked the places you
suggested. I'm also not totally confident I'm doing this right, I'm
primarily a Windows admin, Linux wannabe at best :-). I did get the
following on the console, I think this is the backtrace, it says it is
anyways. Hope this is useful. By the way were you able to get your 64
bit system up and running?
*** glibc detected *** /usr/local/nagios/merlin/merlind: free(): invalid
size: 0x00007fffa3361ef0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3613a71ce2]
/lib64/libc.so.6(cfree+0x8c)[0x3613a7590c]
/usr/local/nagios/merlin/merlind[0x406262]
/usr/local/nagios/merlin/merlind[0x403a71]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3613a1d974]
/usr/local/nagios/merlin/merlind[0x402289]
======= Memory map: ========
00400000-0040d000 r-xp 00000000 fd:00 6096222
/usr/local/nagios/merlin/merlind
0060d000-0060e000 rw-p 0000d000 fd:00 6096222
/usr/local/nagios/merlin/merlind
052cd000-0530b000 rw-p 052cd000 00:00 0
[heap]
31f4e00000-31f4f2a000 r-xp 00000000 fd:00 4949220
/usr/lib64/mysql/libmysqlclient.so.15.0.0
31f4f2a000-31f512a000 ---p 0012a000 fd:00 4949220
/usr/lib64/mysql/libmysqlclient.so.15.0.0
31f512a000-31f5171000 rw-p 0012a000 fd:00 4949220
/usr/lib64/mysql/libmysqlclient.so.15.0.0
31f5171000-31f5172000 rw-p 31f5171000 00:00 0
31f5a00000-31f5a08000 r-xp 00000000 fd:00 4228515
/usr/lib64/libkrb5support.so.0.1
31f5a08000-31f5c07000 ---p 00008000 fd:00 4228515
/usr/lib64/libkrb5support.so.0.1
31f5c07000-31f5c08000 rw-p 00007000 fd:00 4228515
/usr/lib64/libkrb5support.so.0.1
31f5e00000-31f5e2c000 r-xp 00000000 fd:00 4228518
/usr/lib64/libgssapi_krb5.so.2.2
31f5e2c000-31f602c000 ---p 0002c000 fd:00 4228518
/usr/lib64/libgssapi_krb5.so.2.2
31f602c000-31f602e000 rw-p 0002c000 fd:00 4228518
/usr/lib64/libgssapi_krb5.so.2.2
31f6200000-31f6224000 r-xp 00000000 fd:00 4228516
/usr/lib64/libk5crypto.so.3.1
31f6224000-31f6423000 ---p 00024000 fd:00 4228516
/usr/lib64/libk5crypto.so.3.1
31f6423000-31f6425000 rw-p 00023000 fd:00 4228516
/usr/lib64/libk5crypto.so.3.1
31f6600000-31f6691000 r-xp 00000000 fd:00 4228517
/usr/lib64/libkrb5.so.3.3
31f6691000-31f6891000 ---p 00091000 fd:00 4228517
/usr/lib64/libkrb5.so.3.3
31f6891000-31f6895000 rw-p 00091000 fd:00 4228517
/usr/lib64/libkrb5.so.3.3
31f6a00000-31f6a44000 r-xp 00000000 fd:00 61210656
/lib64/libssl.so.0.9.8e
31f6a44000-31f6c44000 ---p 00044000 fd:00 61210656
/lib64/libssl.so.0.9.8e
31f6c44000-31f6c4a000 rw-p 00044000 fd:00 61210656
/lib64/libssl.so.0.9.8e
3613600000-361361c000 r-xp 00000000 fd:00 61210669
/lib64/ld-2.5.so
361381b000-361381c000 r--p 0001b000 fd:00 61210669
/lib64/ld-2.5.so
361381c000-361381d000 rw-p 0001c000 fd:00 61210669
/lib64/ld-2.5.so
3613a00000-3613b4c000 r-xp 00000000 fd:00 61210815
/lib64/libc-2.5.so
3613b4c000-3613d4c000 ---p 0014c000 fd:00 61210815
/lib64/libc-2.5.so
3613d4c000-3613d50000 r--p 0014c000 fd:00 61210815
/lib64/libc-2.5.so
3613d50000-3613d51000 rw-p 00150000 fd:00 61210815
/lib64/libc-2.5.so
3613d51000-3613d56000 rw-p 3613d51000 00:00 0
3613e00000-3613e82000 r-xp 00000000 fd:00 61210969
/lib64/libm-2.5.so
3613e82000-3614081000 ---p 00082000 fd:00 61210969
/lib64/libm-2.5.so
3614081000-3614082000 r--p 00081000 fd:00 61210969
/lib64/libm-2.5.so
3614082000-3614083000 rw-p 00082000 fd:00 61210969
/lib64/libm-2.5.so
3614200000-3614202000 r-xp 00000000 fd:00 61210898
/lib64/libdl-2.5.so
3614202000-3614402000 ---p 00002000 fd:00 61210898
/lib64/libdl-2.5.so
3614402000-3614403000 r--p 00002000 fd:00 61210898
/lib64/libdl-2.5.so
3614403000-3614404000 rw-p 00003000 fd:00 61210898
/lib64/libdl-2.5.so
3614a00000-3614a14000 r-xp 00000000 fd:00 4234122
/usr/lib64/libz.so.1.2.3
3614a14000-3614c13000 ---p 00014000 fd:00 4234122
/usr/lib64/libz.so.1.2.3
3614c13000-3614c14000 rw-p 00013000 fd:00 4234122
/usr/lib64/libz.so.1.2.3
3615200000-361523b000 r-xp 00000000 fd:00 61210977
/lib64/libsepol.so.1
361523b000-361543b000 ---p 0003b000 fd:00 61210977
/lib64/libsepol.so.1
361543b000-361543c000 rw-p 0003b000 fd:00 61210977
/lib64/libsepol.so.1
361543c000-3615446000 rw-p 361543c000 00:00 0
3615600000-3615615000 r-xp 00000000 fd:00 61210978
/lib64/libselinux.so.1
3615615000-3615815000 ---p 00015000 fd:00 61210978
/lib64/libselinux.so.1
3615815000-3615817000 rw-p 00015000 fd:00 61210978
/lib64/libselinux.so.1
3615817000-3615818000 rw-p 3615817000 00:00 0
3616e00000-3616e15000 r-xp 00000000 fd:00 61210974
/lib64/libnsl-2.5.so
3616e15000-3617014000 ---p 00015000 fd:00 61210974
/lib64/libnsl-2.5.so
3617014000-3617015000 r--p 00014000 fd:00 61210974
/lib64/libnsl-2.5.so
3617015000-3617016000 rw-p 00015000 fd:00 61210974
/lib64/libnsl-2.5.so
3617016000-3617018000 rw-p 3617016000 00:00 0
3617a00000-3617b2d000 r-xp 00000000 fd:00 61210980
/lib64/libcrypto.so.0.9.8e
3617b2d000-3617d2c000 ---p 0012d000 fd:00 61210980
/lib64/libcrypto.so.0.9.8e
3617d2c000-3617d4d000 rw-p 0012c000 fd:00 61210980
/lib64/libcrypto.so.0.9.8e
3617d4d000-3617d51000 rw-p 3617d4d000 00:00 0
3618a00000-3618a02000 r-xp 00000000 fd:00 61210979
/lib64/libcom_err.so.2.1
3618a02000-3618c01000 ---p 00002000 fd:00 61210979
/lib64/libcom_err.so.2.1
3618c01000-3618c02000 rw-p 00001000 fd:00 61210979
/lib64/libcom_err.so.2.1
3618e00000-3618e11000 r-xp 00000000 fd:00 61210976
/lib64/libresolv-2.5.so
3618e11000-3619011000 ---p 00011000 fd:00 61210976
/lib64/libresolv-2.5.so
3619011000-3619012000 r--p 00011000 fd:00 61210976
/lib64/libresolv-2.5.so
3619012000-3619013000 rw-p 00012000 fd:00 61210976
/lib64/libresolv-2.5.so
3619013000-3619015000 rw-p 3619013000 00:00 0
361a600000-361a602000 r-xp 00000000 fd:00 61210975
/lib64/libkeyutils-1.2.so
361a602000-361a801000 ---p 00002000 fd:00 61210975
/lib64/libkeyutils-1.2.so
361a801000-361a802000 rw-p 00001000 fd:00 61210975
/lib64/libkeyutils-1.2.so
3621c00000-3621c0d000 r-xp 00000000 fd:00 61210818
/lib64/libgcc_s-4.1.2-20080825.so.1
3621c0d000-3621e0d000 ---p 0000d000 fd:00 61210818
/lib64/libgcc_s-4.1.2-20080825.so.1
3621e0d000-3621e0e000 rw-p 0000d000 fd:00 61210818
/lib64/libgcc_s-4.1.2-20080825.so.1
3626000000-3626009000 r-xp 00000000 fd:00 61210986
/lib64/libcrypt-2.5.so
3626009000-3626208000 ---p 00009000 fd:00 61210986
/lib64/libcrypt-2.5.so
3626208000-3626209000 r--p 00008000 fd:00 61210986
/lib64/libcrypt-2.5.so
3626209000-362620a000 rw-p 00009000 fd:00 61210986
/lib64/libcrypt-2.5.so
362620a000-3626238000 rw-p 362620a000 00:00 0
2acd0773f000-2acd07740000 rw-p 2acd0773f000 00:00 0
2acd0775a000-2acd0775b000 rw-p 2acd0775a000 00:00 0
2acd0775b000-2acd07767000 r-xp 00000000 fd:00 63471618
/usr/lib64/libdbi.so.0.0.5
2acd07767000-2acd07966000 ---p 0000c000 fd:00 63471618
/usr/lib64/libdbi.so.0.0.5
2acd07966000-2acd07967000 rw-p 0000b000 fd:00 63471618
/usr/lib64/libdbi.so.0.0.5
2acd07967000-2acd07969000 rw-p 2acd07967000 00:00 0
2acd07969000-2acd0796f000 r-xp 00000000 fd:00 5505279
/usr/lib64/dbd/libdbdmysql.so
2acd0796f000-2acd07b6e000 ---p 00006000 fd:00 5505279
/usr/lib64/dbd/libdbdmysql.so
2acd07b6e000-2acd07b6f000 rw-p 00005000 fd:00 5505279
/usr/lib64/dbd/libdbdmysql.so
2acd07b6f000-2acd07b70000 rw-p 2acd07b6f000 00:00 0
2acd07b89000-2acd07b93000 r-xp 00000000 fd:00 61210676
/lib64/libnss_files-2.5.so
2acd07b93000-2acd07d92000 ---p 0000a000 fd:00 61210676
/lib64/libnss_files-2.5.so
2acd07d92000-2acd07d93000 r--p 00009000 fd:00 61210676
/lib64/libnss_files-2.5.so
2acd07d93000-2acd07d94000 rw-p 0000a000 fd:00 61210676
/lib64/libnss_files-2.5.so
2acd08000000-2acd08021000 rw-p 2acd08000000 00:00 0
2acd08021000-2acd0c000000 ---p 2acd08021000 00:00 0
7fffa3356000-7fffa336b000 rw-p 7ffffffea000 00:00 0
[stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0
[vdso]
Thanks,
-greg
-----Original Message-----
From: op5-users-bounces at lists.op5.com
[mailto:op5-users-bounces at lists.op5.com] On Behalf Of Andreas Ericsson
Sent: Thursday, June 25, 2009 2:37 AM
To: Mailinglist for op5's products
Subject: Re: [op5-users] Merlin crashed on me?
Frater, Greg J wrote:
> Andreas,
>
> When you have time.
>
> I'm at a stand still with Merlin until this can get resolved. I don't
> get any kind of core dump when it crashes (or I don't know where to
> look).
Try "ulimit -c unlimited" before starting the merlin daemon. If/when you
get one, I'll need to see the backtrace from it as well. You can get
that by simply running
gdb merlind corefile
(gdb prompt) bt
copy-paste the output to an email and send it to me. Without the
backtrace it's impossible for me to see what's going wrong I'm afraid
:-/
The core-dump *should* end up in the directory you're starting Merlin
from, although if it's the module producing the coredumps, they will end
up in the nagios-user's homedirectory, and you'll have to enable
core-dumping in nagios.cfg (daemon_dumps_core=1) and also make sure the
nagios user's home directory is writable by the nagios user.
> I'm not sure how else to help, I'm certainly willing to, I just don't
> know what else to do. Please advise. If you can't get to this for a
> while that's okay to, I will stop spending any cycles on it until you
> have time. I know there's been a large surge in Nagios
> development/activity recently, I understand you guys have a lot going
> on at the moment.
>
I've requested a 64-bit system now, but our sysadmin dude is home sick
today so I'll have to figure out how to set up a new vmware system
myself. This involves some arcane trickery with rdp-ing into a windows
machine, so I'm having lots of fun although making very little progress.
But really, don't waste your time on this until I've made some fixes to
Merlin. Something's wrong with it on 64-bit systems, and I need to
figure out what it is.
--
Andreas Ericsson andreas.ericsson at op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war on
peace.
_______________________________________________
op5-users mailing list
op5-users at lists.op5.com
http://lists.op5.com/mailman/listinfo/op5-users
More information about the op5-users
mailing list