Data type mapping: SPYT, Spark, and YTsaurus

This article describes the mapping between YTsaurus data types and Apache Spark data types for read and write.

Notes

1. Type hints: Some mappings can be overridden by type hints in field metadata. This enables precise control of type conversion.

2. Arrow support: Some date/time related types do not support the Arrow format and use alternative serialization.

3. Decimal precision: YTsaurus limits decimal precision to 35. Learn more about Decimal

4. String types: The configuration option stringToUtf8 controls whether strings map to string or utf8 in YTsaurus. Example:

   import spyt
   from pyspark.sql import Row
   from pyspark.sql.types import StructType, StructField, StringType, BinaryType
   
   spark.createDataFrame(
       spark.sparkContext.parallelize([Row("string", b"binary")]),
       StructType([
           StructField("string", StringType(), nullable=False),
           StructField("binary", BinaryType(), nullable=False),
       ])
   ).write \
   .option("string_to_utf8", "true") \
   .yt("//path/to/table")
   

5. Composite types: Complex types such as structs, arrays, and maps are mapped recursively using the same rules for their element types.

For special techniques related to data types, see Read options and Write options.

Type mapping when reading from YTsaurus

Composite types

Type mapping when writing to YTsaurus