Materialized views (MV) are a core concept in databases. In streaming databases like KsqlDB and RisingWave, MVs are maintained through continuous incremental stream processing engines. Users can define cascading MVs, or more specifically, MVs on top of other MVs, to express complex stream processing logic. However, the management of cascading MVs can introduce substantial technical hurdles for the database system. To illustrate, consider the scenario where an MV within the stack is unable to promptly process events from its upstream sources. This not only results in immediate spikes in latency for downstream MVs but also creates backpressure, potentially causing a system crash. Additionally, if an MV experiences a crash, it can trigger a pause in the entire MV stack's processing. Overcoming these challenges to recover the MV and its downstream MVs while preserving data consistency is a formidable task. In this presentation, I will begin by exploring the critical considerations when it comes to maintaining cascading materialized views: namely, consistency, elasticity, and fault tolerance. Subsequently, I will delve into the potential advantages and disadvantages of various approaches, along with strategies for efficient logging and checkpointing to minimize system downtime. Finally, I will share insights gained from our experiences in managing hundreds of cascading materialized views in real-world production environments.
