Member Avatar for KMPS

Hi,

Do any one know how to successfully run mapreduce process of chukwa in hadoop.Also I couldn't read my adaptor from initial adaptors file.So I dynamically added adaptors for collecting files.I have collected some files dynamically but now that also causing some problem as follows:

2013-01-09 01:20:01,533 INFO listen thread for /127.0.0.1:40808 ChukwaAgent - started a new adaptor, id = adaptor_79516953ebabf5bc7807cd48f42af798 function=[org.apache.hadoop.chukwa.datacollection.adaptor.DirTailingAdaptor@7c3fab7a]

2013-01-09 01:20:01,534 INFO Thread-34 FileTailingAdaptor - chukwaAgent.fileTailingAdaptor.maxReadSize: 131072

2013-01-09 01:20:01,535 INFO Thread-34 FileTailingAdaptor - started file tailer adaptor_fc5d33edb18b66f930d97dc988df4df9 on file /var/log/syslog with first byte at offset 0

2013-01-09 01:20:01,535 INFO Thread-34 ChukwaAgent - started a new adaptor, id = adaptor_fc5d33edb18b66f930d97dc988df4df9 function=[Lightweight Tailer on /var/log/syslog]

2013-01-09 01:20:01,535 INFO Thread-34 DirTailingAdaptor - DirTailingAdaptor adaptor_79516953ebabf5bc7807cd48f42af798 started new adaptor adaptor_fc5d33edb18b66f930d97dc988df4df9

2013-01-09 01:20:01,715 WARN Thread-10 FileTailingAdaptor - failed to stream data for: /var/log/syslog, graceful period will Expire at now:1357674601715 + 180000 secs, i.e:1357674781715

2013-01-09 01:20:01,953 INFO Timer-0 ChukwaAgent - writing checkpoint 301
...
...
**2013-01-09 01:23:01,745 WARN Thread-10 FileTailingAdaptor - Adaptor|adaptor_fc5d33edb18b66f930d97dc988df4df9|attempts=90| File cannot be read: /var/log/syslog, streaming policy expired. File removed from streaming.
**
2013-01-09 01:23:01,745 INFO Thread-10 FileTailingAdaptor - Enter Shutdown:HARD_STOP - ObjectId:Lightweight Tailer on /var/log/syslog

2013-01-09 01:23:01,745 INFO Thread-10 FileTailingAdaptor - Exit Shutdown:HARD_STOP - ObjectId:Lightweight Tailer on /var/log/syslog
**
**2013-01-09 01:23:01,745 INFO Thread-10 ChukwaAgent - shutdown [Abruptly] on adaptor_fc5d33edb18b66f930d97dc988df4df9, logs 0 /var/log/syslog
...
...

I tried to do demux processing on the collected log data but I am getting this error:

java.lang.NoClassDefFoundError: org/codehaus/jackson/map/JsonMappingException

at org.apache.hadoop.chukwa.extraction.demux.Demux.run(Demux.java:213)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

at org.apache.hadoop.chukwa.extraction.demux.DemuxManager.runDemux(DemuxManager.java:331)

at org.apache.hadoop.chukwa.extraction.demux.DemuxManager.processData(DemuxManager.java:287)

at org.apache.hadoop.chukwa.extraction.demux.DemuxManager.start(DemuxManager.java:200)

at org.apache.hadoop.chukwa.extraction.demux.DemuxManager.main(DemuxManager.java:73)

Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.JsonMappingException

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:423)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:356)

... 6 more

And also I have a great confusion on using what type of log files to be used for system management.If any one knows please kindly guide me.

--Thanks in Advance.

Edited by KMPS
  • 2 Contributors
  • 1 Reply
  • 365 Views
  • 14 Hours Discussion Span
  • Latest Post Latest Post by stormal1

Dani AI

Generated

Two separate issues are being mixed in the thread and should be tackled independently: the Chukwa agent’s file-tailers losing access to system logs, and the Demux MapReduce job failing because a JSON-mapping class isn’t on the job classpath. is seeing the agent remove a stream after repeated read failures; that behaviour is almost always I/O/permission/rotation-related rather than a mapreduce bug. ’s note about updating Hadoop logging/metrics is relevant later, but first confirm the agent process can actually open the files it’s configured to follow.

Quick diagnostics and remedies for the tailer problem:

  • Confirm the agent user can read the log file (permissions, group membership) and that SELinux/AppArmor isn’t blocking access. Example checks:

    sudo -u chukwa cat /var/log/syslog | head -n 5
sudo getenforce   # SELinux status
sudo ausearch -m avc -ts recent

  • If logs are owned by root, either add the chukwa user to the appropriate group (e.g. adm) or configure your syslog daemon to duplicate important streams to a chukwa-readable path and point the adaptor there.

  • Ensure the adaptor’s configuration matches how files are rotated. Many tailers follow inodes and will stop when a file is rotated/renamed; use an adaptor/config that supports rotation or write rotated copies to a stable filename for Chukwa.

Fixing the Demux NoClassDefFoundError (JsonMappingException)

  • That exception means Jackson 1.x (org.codehaus.jackson) isn’t on the Hadoop/MapReduce classpath. Place the Jackson 1.x jars (jackson-core-asl and jackson-mapper-asl 1.x) into CHUKWA_HOME/lib or make them available to the MR job (HADOOP_CLASSPATH or -libjars/distributed cache). Example:

    cp jackson-core-asl-1.9.13.jar jackson-mapper-asl-1.9.13.jar $CHUKWA_HOME/lib/
export HADOOP_CLASSPATH=$CHUKWA_HOME/lib/jackson-core-asl-1.9.13.jar:$CHUKWA_HOME/lib/jackson-mapper-asl-1.9.13.jar:$HADOOP_CLASSPATH

    Important: Chukwa code from the 0.x line expects the old org.codehaus Jackson (1.x). Adding Jackson 2.x (com.fasterxml) will not satisfy that dependency.

What logs to collect (brief guidance)

  • Prioritise logs that matter for ops and security: syslog/daemon/auth (security events), application logs (Hadoop/YARN, HBase, webapps), kernel/dmesg, auditd/selinux, and metrics collectors (collectd/Ganglia). Prefer stable, parseable formats (timestamp, host, component, level, pid, message). Keep raw copies for short-term troubleshooting and only store parsed metrics long-term. Finally, enable agent debug logs while you validate tailing and demux so you get clear failure messages for any remaining classpath or permission issues.
Member Avatar for stormal1

Demux mapreduce jobs can be started by rnning:
CHUKWA_HOME/bin/chukwa demux

The Hadoop configuration files are located in HADOOP_HOME/etc/hadoop. To setup Chukwa to collect logs from Hadoop, you need to change some of the Hadoop configuration files.

Copy CHUKWA_HOME/etc/chukwa/hadoop-log4j.properties file to HADOOP_CONF_DIR/log4j.properties
Copy CHUKWA_HOME/etc/chukwa/hadoop-metrics2.properties file to HADOOP_CONF_DIR/hadoop-metrics2.properties
Edit HADOOP_HOME/etc/hadoop/hadoop-metrics2.properties file and change $CHUKWA_LOG_DIR to your actual CHUKWA log dirctory (ie, CHUKWA_HOME/var/log)

http://incubator.apache.org/chukwa/docs/r0.5.0/admin.html

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.