ConvertAvroToParquet
描述:
Converts Avro records into Parquet file format. The incoming FlowFile should be a valid avro file. If an incoming FlowFile does not contain any records, an empty parquet file is the output. NOTE: Many Avro datatypes (collections, primitives, and unions of primitives, e.g.) can be converted to parquet, but unions of collections and other complex datatypes may not be able to be converted to Parquet.
标签:
avro, parquet, convert
参数:
如下列表中，必填参数则标识为加粗. 其他未加粗参数，则表示为可选参数。表中同时提到参数默认值设置, 并且 参数还支持 表达式语言.
| 名字 | 默认值 | 允许值列表 | 描述 |
|---|---|---|---|
| Compression Type | UNCOMPRESSED | - UNCOMPRESSED - SNAPPY - GZIP - LZO - BROTLI - LZ4 - ZSTD | The type of compression for the file being written. |
| Row Group Size | The row group size used by the Parquet writer. The value is specified in the format of <Data Size> <Data Unit> where Data Unit is one of B, KB, MB, GB, TB.支持表达式语言 (支持流属性和变量) | ||
| Page Size | The page size used by the Parquet writer. The value is specified in the format of <Data Size> <Data Unit> where Data Unit is one of B, KB, MB, GB, TB.支持表达式语言 (支持流属性和变量) | ||
| Dictionary Page Size | The dictionary page size used by the Parquet writer. The value is specified in the format of <Data Size> <Data Unit> where Data Unit is one of B, KB, MB, GB, TB.支持表达式语言 (支持流属性和变量) | ||
| Max Padding Size | The maximum amount of padding that will be used to align row groups with blocks in the underlying filesystem. If the underlying filesystem is not a block filesystem like HDFS, this has no effect. The value is specified in the format of <Data Size> <Data Unit> where Data Unit is one of B, KB, MB, GB, TB.支持表达式语言 (支持流属性和变量) | ||
| Enable Dictionary Encoding | - true - false | Specifies whether dictionary encoding should be enabled for the Parquet writer | |
| Enable Validation | - true - false | Specifies whether validation should be enabled for the Parquet writer | |
| Writer Version | - PARQUET_1_0 - PARQUET_2_0 | Specifies the version used by Parquet writer |
连线:
| 名字 | 描述 |
|---|---|
| success | Parquet file that was converted successfully from Avro |
| failure | Avro content that could not be processed |
读取属性:
未提供。
写入属性:
| 名字 | 描述 |
|---|---|
| filename | Sets the filename to the existing filename with the extension replaced by / added to by .parquet |
| record.count | Sets the number of records in the parquet file. |
状态管理:
该组件不保存状态。
限制:
该组件没有限制
输入流要求:
组件必须提供输入流。
系统资源考量:
未提供。