External State Join in YTsaurus Flow (Python)

This pipeline enriches queue events from a prebuilt dynamic reference table through a read-only external state joiner.

Source code

Pipeline structure

event_reader publishes the event stream. The lookup_join companion computation is grouped by key, reads /reference through external_state_joiners, and sends successful matches to enriched. The queue sink stores the output. The joiner's path may point to a Cypress link, allowing the reference table to be switched without restarting the pipeline.

See the pipeline spec.

Read the joined state

ctx.joined_external_state("/reference", message) selects the joined row by the message key. The returned accessor is read-only. A missing row is represented by an empty payload, so get("name") returns None and the example emits nothing.

class LookupJoin(RowFunction):
    """Looks up each event's name in the reference state and emits the joined row."""

    def on_message(self, message, output, ctx):
        name = ctx.joined_external_state("/reference", message).get("name")
        if name is None:
            return

        builder = ctx.message_builder("enriched")
        builder.set("key", message.payload["key"])
        builder.set("name", name)
        output.add_message(builder.finish())

See also

• External State (Python)
• Static Table Join example