更新时间:2023-08-03 GMT+08:00
分享

Scala样例代码

功能介绍

假定用户需要每隔1秒钟需要统计4秒中窗口中数据的量,并做到状态严格一致性。

代码样例

  1. 发送数据形式。

    下面代码片段仅为演示,完整代码参见FlinkCheckpointScalaExample样例工程下的com.huawei.bigdata.flink.examples.SEvent:

    case class SEvent(id: Long, name: String, info: String, count: Int)
  2. 快照数据

    该数据在算子制作快照时用于保存到目前为止算子记录的数据条数。

    下面代码片段仅为演示,完整代码参见FlinkCheckpointScalaExample样例工程下的com.huawei.bigdata.flink.examples.UDFState:

    // 用户自定义状态
    class UDFState extends Serializable{
        private var count = 0L
        
        // 设置用户自定义状态
        def setState(s: Long) = count = s
    
        // 获取用户自定状态
        def getState = count
    }
  3. 带checkpoint的数据源

    source算子的代码,该段代码每发送10000条数据休息1秒钟,制作快照时将到目前为止已经发送的数据的条数保存在UDFState中;从快照中状态恢复时,读取UDFState中的数据条数并重新赋值给count变量。

    下面代码片段仅为演示,完整代码参见FlinkCheckpointScalaExample样例工程下的com.huawei.bigdata.flink.examples.SEventSourceWithChk:

    import java.util
    import org.apache.flink.streaming.api.checkpoint.ListCheckpointed
    import org.apache.flink.streaming.api.functions.source.RichSourceFunction
    import org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext
    
    // 该类是带有checkpoint的source算子
    class SEventSourceWithChk extends RichSourceFunction[SEvent] with ListCheckpointed[UDFState]{
        private var count = 0L
        private var isRunning = true
        private val alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWZYX0987654321"
    
        // source算子的逻辑,即:每秒钟向流图中注入10000个元组
        override def run(sourceContext: SourceContext[SEvent]): Unit = {
            while(isRunning) {
                for (i <- 0 until 10000) {
                    sourceContext.collect(SEvent(1, "hello-"+count, alphabet,1))
                    count += 1L
                 }
                 Thread.sleep(1000)
            }
        }
    
        // 任务取消时调用
        override def cancel(): Unit = {
            isRunning = false;
        }
    
         override def close(): Unit = super.close()
    
         // 制作快照
         override def snapshotState(l: Long, l1: Long): util.List[UDFState] = {
             val udfList: util.ArrayList[UDFState] = new util.ArrayList[UDFState]
             val udfState = new UDFState
             udfState.setState(count)
             udfList.add(udfState)
             udfList
         }
    
         // 从快照中获取状态
         override def restoreState(list: util.List[UDFState]): Unit = {
             val udfState = list.get(0)
             count = udfState.getState
         }
    }
  4. 带checkpoint的窗口定义

    该段代码是window算子的代码,每当触发计算时统计窗口中元组数量。

    下面代码片段仅为演示,完整代码参见FlinkCheckpointScalaExample样例工程下的com.huawei.bigdata.flink.examples.WindowStatisticWithChk:

    import java.util
    import org.apache.flink.api.java.tuple.Tuple
    import org.apache.flink.streaming.api.checkpoint.ListCheckpointed
    import org.apache.flink.streaming.api.scala.function.WindowFunction
    import org.apache.flink.streaming.api.windowing.windows.TimeWindow
    import org.apache.flink.util.Collector
    
    // 该类是带checkpoint的window算子
    class WindowStatisticWithChk extends WindowFunction[SEvent, Long, Tuple, TimeWindow] with ListCheckpointed[UDFState]{
        private var total = 0L
        
        // window算子的实现逻辑,即:统计window中元组的数量
        override def apply(key: Tuple, window: TimeWindow, input: Iterable[SEvent], out: Collector[Long]): Unit = {
            var count = 0L
            for (event <- input) {
                count += 1L
            }
            total += count
            out.collect(count)
         }
    
         // 制作自定义状态快照
         override def snapshotState(l: Long, l1: Long): util.List[UDFState] = {
             val udfList: util.ArrayList[UDFState] = new util.ArrayList[UDFState]
             val udfState = new UDFState
             udfState.setState(total)
             udfList.add(udfState)
             udfList
         }
    
        // 从自定义快照中恢复状态
        override def restoreState(list: util.List[UDFState]): Unit = {
            val udfState = list.get(0)
            total = udfState.getState
        }
    }
  5. 应用代码

    该段代码是流图定义代码,具体实现业务流程,另外,代码中窗口的触发时间使用了event time。

    下面代码片段仅为演示,完整代码参见FlinkCheckpointScalaExample样例工程下的com.huawei.bigdata.flink.examples.FlinkEventTimeAPIChkMain:

    import com.huawei.rt.flink.core.{SEvent, SEventSourceWithChk, WindowStatisticWithChk}
    import org.apache.flink.contrib.streaming.state.RocksDBStateBackend
    import org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks
    import org.apache.flink.streaming.api.{CheckpointingMode, TimeCharacteristic}
    import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
    import org.apache.flink.streaming.api.watermark.Watermark
    import org.apache.flink.streaming.api.windowing.assigners.SlidingEventTimeWindows
    import org.apache.flink.streaming.api.windowing.time.Time
    import org.apache.flink.api.scala._
    import org.apache.flink.runtime.state.filesystem.FsStateBackend
    import org.apache.flink.streaming.api.environment.CheckpointConfig.ExternalizedCheckpointCleanup
    
    object FlinkEventTimeAPIChkMain {
        def main(args: Array[String]): Unit ={
            val env = StreamExecutionEnvironment.getExecutionEnvironment
            env.setStateBackend(new FsStateBackend("hdfs://hacluster/flink/checkpoint/"))
            env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
            env.getConfig.setAutoWatermarkInterval(2000)
            env.getCheckpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE)
            env.getCheckpointConfig.setCheckpointInterval(6000)
    
            // 应用逻辑
            env.addSource(new SEventSourceWithChk)
               .assignTimestampsAndWatermarks(new AssignerWithPeriodicWatermarks[SEvent] {
                   // 设置watermark
                    override def getCurrentWatermark: Watermark = {
                        new Watermark(System.currentTimeMillis())
                    }
                   // 给每个元组打上时间戳
                    override def extractTimestamp(t: SEvent, l: Long): Long = {
                         System.currentTimeMillis()
                    }
               })
              .keyBy(0)
              .window(SlidingEventTimeWindows.of(Time.seconds(4), Time.seconds(1)))
              .apply(new WindowStatisticWithChk)
              .print()
           env.execute()
        }
    }
分享:

    相关文档

    相关产品