在提交作业的时候出现了以下的异常信息:
2015-05-05 11:09:28,071 INFO [Driver] - Attempting to load checkpoint from file hdfs://iteblogcluster/user/iteblog/checkpoint2/checkpoint-1430794986000 2015-05-05 11:09:28,076 WARN [Driver] - Error reading checkpoint from file hdfs://iteblogcluster/user/iteblog/checkpoint2/checkpoint-1430794986000 java.io.InvalidClassException: org.apache.spark.streaming.Checkpoint; local class incompatible: stream classdesc serialVersionUID = 4260288410130070970, local class serialVersionUID = -1544620567275648823 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.streaming.CheckpointReader$$anonfun$read$2.apply(Checkpoint.scala:258) at org.apache.spark.streaming.CheckpointReader$$anonfun$read$2.apply(Checkpoint.scala:246) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34) at org.apache.spark.streaming.CheckpointReader$.read(Checkpoint.scala:246) at org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:624) at logstatstreaming.LogSplitStreamingKafka$.main(LogSplitStreamingKafka.scala:46) at logstatstreaming.LogSplitStreamingKafka.main(LogSplitStreamingKafka.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480) 2015-05-05 11:09:28,077 ERROR [Driver] - User class threw exception: Failed to read checkpoint from directory /user/iteblog/checkpoint2 org.apache.spark.sparkException: Failed to read checkpoint from directory /user/iteblog/checkpoint2 at org.apache.spark.streaming.CheckpointReader$.read(Checkpoint.scala:272) at org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:624) at logstatstreaming.LogSplitStreamingKafka$.main(LogSplitStreamingKafka.scala:46) at logstatstreaming.LogSplitStreamingKafka.main(LogSplitStreamingKafka.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
出现这个异常的原因是两次提交作业的Spark版本不一致导致的,将Spark版本弄成一致即可从Checkpoint目录中恢复。
本博客文章除特别声明,全部都是原创!原创文章版权归过往记忆大数据(过往记忆)所有,未经许可不得转载。
本文链接: 【Spark提交作业的一些异常】(https://www.iteblog.com/archives/1348.html)