Skip to main content

ParquetRecordSetWriter

描述:​

Writes the contents of a RecordSet in Parquet format.

标签:​

parquet, result, set, writer, serializer, record, recordset, row

参数:​

如下列表中，必填参数则标识为加粗. 其他未加粗参数，则表示为可选参数。表中同时提到参数默认值设置, 并且 参数还支持 表达式语言.

名字默认值允许值列表描述
Schema Write StrategyDo Not Write Schema
- Set 'schema.name' AttributeThe FlowFile will be given an attribute named 'schema.name' and this attribute will indicate the name of the schema in the Schema Registry. Note that ifthe schema for a record is not obtained from a Schema Registry, then no attribute will be added.
- Set 'avro.schema' AttributeThe FlowFile will be given an attribute named 'avro.schema' and this attribute will contain the Avro Schema that describes the records in the FlowFile. The contents of the FlowFile need not be Avro, but the text of the schema will be used.
- HWX Schema Reference AttributesThe FlowFile will be given a set of 3 attributes to describe the schema: 'schema.identifier', 'schema.version', and 'schema.protocol.version'. Note that if the schema for a record does not contain the necessary identifier and version, an Exception will be thrown when attempting to write the data.
- HWX Content-Encoded Schema ReferenceThe content of the FlowFile will contain a reference to a schema in the Schema Registry service. The reference is encoded as a single byte indicating the 'protocol version', followed by 8 bytes indicating the schema identifier, and finally 4 bytes indicating the schema version, as per the Hortonworks Schema Registry serializers and deserializers, as found at https://github.com/hortonworks/registry. This will be prepended to each FlowFile. Note that if the schema for a record does not contain the necessary identifier and version, an Exception will be thrown when attempting to write the data.
- Confluent Schema Registry ReferenceThe content of the FlowFile will contain a reference to a schema in the Schema Registry service. The reference is encoded as a single 'Magic Byte' followed by 4 bytes representing the identifier of the schema, as outlined at http://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html. This will be prepended to each FlowFile. Note that if the schema for a record does not contain the necessary identifier and version, an Exception will be thrown when attempting to write the data. This is based on the encoding used by version 3.2.x of the Confluent Schema Registry.
- Do Not Write SchemaDo not add any schema-related information to the FlowFile.
Specifies how the schema for a Record should be added to the data.
Schema Cache控制器服务API:
RecordSchemaCacheService
实现:
VolatileSchemaCache		Specifies a Schema Cache to add the Record Schema to so that Record Readers can quickly lookup the schema.
Schema Access StrategyInherit Record Schema
- Use 'Schema Name' PropertyThe name of the Schema to use is specified by the 'Schema Name' Property. The value of this property is used to lookup the Schema in the configured Schema Registry service.
- Inherit Record SchemaThe schema used to write records will be the same schema that was given to the Record when the Record was created.
- Use 'Schema Text' PropertyThe text of the Schema itself is specified by the 'Schema Text' Property. The value of this property must be a valid Avro Schema. If Expression Language is used, the value of the 'Schema Text' property must be valid after substituting the expressions.
Specifies how to obtain the schema that is to be used for interpreting the data.
Schema Registry控制器服务API:
SchemaRegistry
实现:
AvroSchemaRegistry
ConfluentSchemaRegistry
HortonworksSchemaRegistry		Specifies the Controller Service to use for the Schema Registry
Schema Name${schema.name}Specifies the name of the schema to lookup in the Schema Registry property

支持表达式语言 (支持流属性和变量)
Schema VersionSpecifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved.

支持表达式语言 (支持流属性和变量)
Schema BranchSpecifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored.

支持表达式语言 (支持流属性和变量)
Schema Text${avro.schema}The text of an Avro-formatted Schema

支持表达式语言 (支持流属性和变量)
Cache Size1000Specifies how many Schemas should be cached
Compression TypeUNCOMPRESSED
- UNCOMPRESSED
- SNAPPY
- GZIP
- LZO
- BROTLI
- LZ4
- ZSTD
The type of compression for the file being written.
Row Group SizeThe row group size used by the Parquet writer. The value is specified in the format of <Data Size> <Data Unit> where Data Unit is one of B, KB, MB, GB, TB.

支持表达式语言 (支持流属性和变量)
Page SizeThe page size used by the Parquet writer. The value is specified in the format of <Data Size> <Data Unit> where Data Unit is one of B, KB, MB, GB, TB.

支持表达式语言 (支持流属性和变量)
Dictionary Page SizeThe dictionary page size used by the Parquet writer. The value is specified in the format of <Data Size> <Data Unit> where Data Unit is one of B, KB, MB, GB, TB.

支持表达式语言 (支持流属性和变量)
Max Padding SizeThe maximum amount of padding that will be used to align row groups with blocks in the underlying filesystem. If the underlying filesystem is not a block filesystem like HDFS, this has no effect. The value is specified in the format of <Data Size> <Data Unit> where Data Unit is one of B, KB, MB, GB, TB.

支持表达式语言 (支持流属性和变量)
Enable Dictionary Encoding
- true
- false
Specifies whether dictionary encoding should be enabled for the Parquet writer
Enable Validation
- true
- false
Specifies whether validation should be enabled for the Parquet writer
Writer Version
- PARQUET_1_0
- PARQUET_2_0
Specifies the version used by Parquet writer
Avro Write Old List Structuretrue
- true
- false
Specifies the value for 'parquet.avro.write-old-list-structure' in the underlying Parquet library
Avro Add List Element Recordstrue
- true
- false
Specifies the value for 'parquet.avro.add-list-element-records' in the underlying Parquet library

状态管理:​

该组件不保存状态。

限制:​

该组件没有限制

系统资源考量:​

未提供。