Read options

schema_hint
transaction

schema_hint

Hard-coded column type. Useful when a column is of type any (a composite data type serialized as yson).
The value will be deserialized as the specified type.

Python example:

spark.read.schema_hint({"value": MapType(StringType(), LongType())}).yt("//sys/spark/examples/example_yson")

Scala example:

df.write
    .schemaHint(Map("a" ->
        YtLogicalType.VariantOverTuple(Seq(
          (YtLogicalType.String, Metadata.empty), (YtLogicalType.Double, Metadata.empty)))))
    .yt(tmpPath)

transaction

Reading from a transaction. For more details, see Reading and writing within a transaction.

Scala example:

val transaction = YtWrapper.createTransaction(None, 10 minute)
df.write.transaction(transaction.getId.toString).yt(tmpPath)
transaction.commit().get(10, TimeUnit.SECONDS)