Skill

databricks-spark-structured-streaming

Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.

v0.2.31Databricks License3/3 watchmen clearedinstructions only

Take it home

Then run, where you saved it
mkdir -p ~/.marktplatz/claude-code/.claude-plugin
mv marketplace.json ~/.marktplatz/claude-code/.claude-plugin/
claude plugin marketplace add ~/.marktplatz/claude-code
claude plugin marketplace update marktplatz
claude plugin install databricks-spark-structured-streaming@marktplatz
claude plugin update databricks-spark-structured-streaming@marktplatz

Read the recipe below before you install; it is what your agent will follow.

Put it in thy scroll with others →

The recipe SKILL.md

Spark Structured Streaming

Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.

Quick Start

from pyspark.sql.functions import col, from_json

# Basic Kafka to Delta streaming
df = (spark
    .readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "topic")
    .load()
    .select(from_json(col("value").cast("string"), schema).alias("data"))
    .select("data.*")
)

df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \
    .trigger(processingTime="30 seconds") \
    .start("/delta/target_table")

Core Patterns

Pattern Description Reference
Kafka Streaming Kafka to Delta, Kafka to Kafka, Real-Time Mode See references/kafka-streaming.md
Real-Time Mode (RTM) Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), transformWithState, observability, error classes, delivery semantics See references/real-time-mode.md
Lakebase Sink Write streaming records into Lakebase Postgres with transactional upserts. Native format("postgresql") sink (DBR 18.3+) and manual foreach sink as a fallback See references/lakebase-sink-python.md
Stream Joins Stream-stream joins, stream-static joins See references/stream-stream-joins.md, references/stream-static-joins.md
Multi-Sink Writes Write to multiple tables, parallel merges See references/multi-sink-writes.md
Merge Operations MERGE performance, parallel merges, optimizations See references/merge-operations.md

Configuration

Topic Description Reference
Checkpoints Checkpoint management and best practices See references/checkpoint-best-practices.md
Stateful Operations Watermarks, state stores, RocksDB configuration See references/stateful-operations.md
Trigger & Cost Trigger selection, cost optimization, RTM See references/trigger-and-cost-optimization.md

Best Practices

Topic Description Reference
Production Checklist Comprehensive best practices See references/streaming-best-practices.md

Production Checklist

  • Checkpoint location is persistent (UC volumes, not DBFS)
  • Unique checkpoint per stream
  • Fixed-size cluster (no autoscaling for streaming)
  • Monitoring configured (input rate, lag, batch duration)
  • Exactly-once verified (txnVersion/txnAppId)
  • Watermark configured for stateful operations
  • Left joins for stream-static (not inner)

What's inside

Every turn it costs ~115 tokens of context (its name and description); when it fires, ~808 more. It is instructions only, no scripts.

The watch

SpectorCiscoATR Cleared by all 3 watchmen.
WatchmanVersionNotesReport
SkillSpector NVIDIA2.9.6–raw
Skill Scanner Cisco AI Defense2.1.01raw
Agent Threat Rules the ATR project4.0.0–raw
Read the 1 note
  • info Skill Scanner MANIFEST_MISSING_LICENSE: Skill does not specify a license

Scanned offline at admission. Each watchman's raw report is linked above, and the summary the market reads is kept beside them; see also what the watch does and does not prove.

Provenance

Taken from databricks/databricks-agent-skills at tag v0.2.31, commit aea6a1b9bd88, folder skills/databricks-spark-structured-streaming, under the Databricks License charter. The market serves this exact copy from commit 8f3e8e3e01bb of marktplatz-dev/marktplatz-dev; its files hash to sha256-be141709adab784f155345b20125fa25c24a6f143a2548d1dcfa8b74c865733c.

Versions

  • v0.2.31admitted 2026-10-18cleared
  • v0.2.21admitted 2026-09-25cleared