【FAQ】Spark报错: java.lang.UnsupportedOperationException: Unimplemented type:DoubleType

ゝ一纸荒年。 2022-10-05 14:46 221阅读 0赞

更详细说明可以参考[《【FAQ】Impala常见异常incompatible Parquet schema for column》][FAQ_Impala_incompatible Parquet schema for column]

## 异常回顾1 ##

2019-01-01 11:24:32,035 WARN org.apache.spark.scheduler.TaskSetManager: Lost task xx.0 in stage x.0 (xxx 12, xxx, executor 3): org.apache.spark.SparkException: Task failed while writing rows
    	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:398)
    	at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$apply$mcV$sp$1.apply(FileFormatWriter.scala:197)
    	at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$apply$mcV$sp$1.apply(FileFormatWriter.scala:196)
    	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    	at org.apache.spark.scheduler.Task.run(Task.scala:108)
    	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    	at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.UnsupportedOperationException: Unimplemented type: DoubleType
    	at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBinaryBatch(VectorizedColumnReader.java:431)
    	at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:203)
    	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:230)
    	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:137)
    	at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
    	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:105)
    	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
    	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:105)
    	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.scan_nextBatch$(Unknown Source)
    	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
    	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
    	at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:441)
    	at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:384)
    	at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:382)
    	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1380)
    	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:387)
    	... 8 more

### 单独查表 ###

File 'hdfs://nameservice/loongshawn/it/edw/fact/edw_test123456789/p_date=20190301/part-1234567890xxxx' has an incompatible Parquet schema for column 'test.xxxxx_table.filed'. Column type: DOUBLE, Parquet schema: optional byte_array filed [i:xxx d:x r:x] ; nested exception is java.sql.SQLException: File 'hdfs://nameservice/loongshawn/it/edw/fact/edw_test123456789/p_date=20190301/part-1234567890xxxx' has an incompatible Parquet schema for column 'test.xxxxx_table.filed'. Column type: DOUBLE, Parquet schema: optional byte_array filed [i:xxx d:x r:x]

### 原因分析 ###

原因是目录里面的parquet文件的schema信息不一致导致的，目录下面的多个parquet文件中的字段个数和字段的名称是一致的。但是有一个parquet文件的一个字段名字虽然一样，但是字段的类型不一致，导致程序报错了。

### 处理方式 ###

1、drop掉对应分区，重新生成分区文件；  
2、登录服务器修改分区文件，与新文件元数据保持一致。  
需自行评估哪种方式合规，代价低。

## 异常回顾2 ##

错误信息：

INFO org.apache.spark.scheduler.DAGScheduler: ResultStage 11 (sql at SqlAction.scala:411) failed in 42.851 s due to Job aborted due to stage failure: Serialized task 11:0 was 261012415 bytes, which exceeds max allowed: spark.rpc.message.maxSize (134217728 bytes). Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values.

> INFO org.apache.spark.scheduler.DAGScheduler: ResultStage 11 (sql at SqlAction.scala:411) failed in 42.851 s due to Job aborted due to stage failure: Serialized task 11:0 was 261012415 bytes, which exceeds max allowed: spark.rpc.message.maxSize (134217728 bytes). Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values.

### 处理方式 ###

任务提交时，添加如下参数

--conf 'spark.rpc.message.maxSize=1024'

[FAQ_Impala_incompatible Parquet schema for column]: https://blog.csdn.net/loongshawn/article/details/118310695?utm_source=app&app_version=4.10.0&code=app_1562916241&uLinkId=usr1mkqgl919blen