Ошибка Hive: не удастся: ошибка выполнения, код возврата 2 из org.apache.hadoop.hive.ql.exec.mr.mapredtask ⇐ JAVA

[Anonymous]

У меня есть данные в Твиттере, используя Flume на HDFS. Иметь 3 узла кластера и MySQL Metastore для Hive.

Код: Выделить всё

select user_name.screen_name, user_name.followers_count from BBMP_election_tweets limit 10;
< /code>

Это дает правильные результаты. < /p>

Но если я запускаю ниже запрос < /p>

select user_name.screen_name, user_name.followers_count c from BBMP_election_tweets order by c desc;

It throws below error


Query ID = hduser_20150827102628_15a405ae-59a4-4c90-aa47-045d58c62f02
Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks
determined at compile time: 1 In order to change the average load for
a reducer (in bytes): set
hive.exec.reducers.bytes.per.reducer= In order to limit the
maximum number of reducers: set hive.exec.reducers.max= In
order to set a constant number of reducers: set
mapreduce.job.reduces= Starting Job = job_1439458788229_0016,
Tracking URL =
http://hdmaster:8088/proxy/application_ ... 8229_0016/ Kill
Command = /usr/local/hadoop/bin/hadoop job -kill
job_1439458788229_0016 Hadoop job information for Stage-1: number of
mappers: 2; number of reducers: 1 2015-08-27 10:27:33,567 Stage-1 map
= 0%, reduce = 0% 2015-08-27 10:27:43,908 Stage-1 map = 50%, reduce = 0% 2015-08-27 10:27:45,015 Stage-1 map = 0%, reduce = 0% 2015-08-27 10:28:31,011 Stage-1 map = 100%, reduce = 100% Ended Job =
job_1439458788229_0016 with errors Error during job, obtaining
debugging information... Examining task ID:
task_1439458788229_0016_m_000001 (and more) from job
job_1439458788229_0016

Task with the most failures(4):
----- Task ID: task_1439458788229_0016_m_000001

URL:

http://hdmaster:8088/taskdetails.jsp?jo ... 6_m_000001
----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140)
... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassNotFoundException: Class
com.cloudera.hive.serde.JSONSerDe not found
at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328)
at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109)
... 22 more Caused by: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980)
at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141)
at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
... 24 more

FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched:
Stage-Stage-1: Map: 2 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec


When checked the logs created. It gives below message


2015-08-27 10:28:48,852 INFO [main]
org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service:
job_1439458788229_0016, Ident:
(org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@988145f)
2015-08-27 10:28:49,023 INFO [main]
org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying
again. Got null now. 2015-08-27 10:28:49,627 INFO [main]
org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for
child:
/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1439458788229_0016
2015-08-27 10:28:50,363 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: session.id is
deprecated. Instead, use dfs.metrics.session-id 2015-08-27
10:28:51,186 INFO [main] org.apache.hadoop.mapred.Task: Using
ResourceCalculatorProcessTree : [ ] 2015-08-27 10:28:51,534 INFO
[main] org.apache.hadoop.mapred.MapTask: Processing split:
Paths:/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685:0+52974,/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484188899:0+20119InputFormatClass:
org.apache.hadoop.mapred.TextInputFormat

2015-08-27 10:28:51,701 INFO [main]
org.apache.hadoop.hive.ql.exec.Utilities: PLAN PATH =
hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml
2015-08-27 10:28:51,701 INFO [main]
org.apache.hadoop.hive.ql.exec.Utilities: ***************non-local
mode*************** 2015-08-27 10:28:51,701 INFO [main]
org.apache.hadoop.hive.ql.exec.Utilities: local path =
hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml
2015-08-27 10:28:51,701 INFO [main]
org.apache.hadoop.hive.ql.exec.Utilities: Open file to read in plan:
hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml
2015-08-27 10:28:51,719 INFO [main]
org.apache.hadoop.hive.ql.log.PerfLogger:
2015-08-27 10:28:51,720 INFO [main]
org.apache.hadoop.hive.ql.exec.Utilities: Deserializing MapWork via
kryo 2015-08-27 10:28:52,043 INFO [main]
org.apache.hadoop.hive.ql.log.PerfLogger: 2015-08-27
10:28:52,080 INFO [main]
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: Processing
file
hdfs://hdmaster:8020/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685
2015-08-27 10:28:52,080 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: map.input.file is
deprecated. Instead, use mapreduce.map.input.file 2015-08-27
10:28:52,080 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: map.input.start is
deprecated. Instead, use mapreduce.map.input.start 2015-08-27
10:28:52,080 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: map.input.length is
deprecated. Instead, use mapreduce.map.input.length 2015-08-27
10:28:52,080 INFO [main] org.apache.hadoop.mapred.MapTask:
numReduceTasks: 1 2015-08-27 10:28:52,228 INFO [main]
org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask:
mapreduce.task.io.sort.mb: 100 2015-08-27 10:28:52,228 INFO [main]
org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2015-08-27
10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart =
0; bufvoid = 104857600 2015-08-27 10:28:52,228 INFO [main]
org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600

2015-08-27 10:28:52,243 INFO [main]
org.apache.hadoop.hive.ql.exec.MapOperator: Initializing operator
MAP[3] 2015-08-27 10:28:52,243 INFO [main]
org.apache.hadoop.hive.ql.exec.MapOperator: Initialization Done 3 MAP
2015-08-27 10:28:52,243 INFO [main]
org.apache.hadoop.hive.ql.exec.MapOperator: Operator 3 MAP initialized
2015-08-27 10:28:52,262 WARN [main]
org.apache.hadoop.mapred.YarnChild: Exception running child :
java.lang.RuntimeException: Error in configuring object at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused
by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more Caused by: java.lang.RuntimeException: Error in
configuring object at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 17 more Caused by: java.lang.RuntimeException: Map operator
initialization failed at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140)
... 22 more Caused by:
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassNotFoundException: Class
com.cloudera.hive.serde.JSONSerDe not found at
org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328)
at
org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109)
... 22 more Caused by: java.lang.ClassNotFoundException: Class
com.cloudera.hive.serde.JSONSerDe not found at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980)
at
org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141)
at
org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
... 24 more

2015-08-27 10:28:52,269 INFO [main] org.apache.hadoop.mapred.Task:
Runnning cleanup for the task 2015-08-27 10:28:52,278 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask
metrics system... 2015-08-27 10:28:52,278 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics
system stopped. 2015-08-27 10:28:52,279 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics
system shutdown complete.


Please suggest. How can this issue be resolved. and why is it occuring.

Thanks

Подробнее здесь: https://stackoverflow.com/questions/322 ... adoop-hive