Sunday, October 27, 2019

Event time and process time in the flink


  • Window
    • Sliding time window
      • Can be overlapping
      • timeWindowAll(Time size, Time slide)
    • Tumbling time window
      • No overlapping
      • timeWindowAll(Time size)
  • Event Time based system
    env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
  • Process Time base system
    env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime)

  •  Watermark
    • Work for event time.
    • Define when to stop waiting for the events.
    • Events are in order, then watermarks are just periodic markers in the stream.
    • In out of order stream, define all events up to certain timestamp should have arrived.

      w(17) : point of timestamp if incoming elements smaller than 17, then ignore them.
      w(11) : point of timestamp if incoming elements smaller than 11, then ignore them.

  •  Late Events
    • Event arrived after watermark timestamp.
    • Dropped by the default.
    • Allowed lateness. Keep window state, until expired.

See more detail.


No comments: