Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Kafka timestamp offset_final

Kafka Timestamp offset Final

  • Inicia sesión para ver los comentarios

Kafka timestamp offset_final

  1. 1. Spark, and Kafka timestamp offset charsyam@naver.com
  2. 2. Common Way: Kafka and Spark Streaming Server #1 Server #2 Server ... Server ... Log Aggregation Server Syslogd Kafka Logstash Spark Streaming
  3. 3. Kafka Log
  4. 4. Kafka Log Segments Segment is a file If you set prealloc flag, segment will be 1GB.
  5. 5. What is timestamp Index? Fetching Kafka Logs by Timestamp.(From Kafka-0.10.0.2) You can query from specific timestamp range of message.(very useful) ex) From 2018-09-08 00:00:00 To 2018-09-08-23:59:59
  6. 6. Related KIPs KIP-32 - Add timestamps to Kafka message : 0.10.0.0 KIP-33 - Add a time based log index : 0.10.1.0
  7. 7. Kafka Log Files 00000000000155593652.log Kafka Message Data File 00000000000155593652.index OffsetIndex File 00000000000155593652.timeindex TimeIndex File There are many files for 1 segment. .txnindex, .snapshot, .deleted, .cleaned, .swap, etc
  8. 8. Kafka Index Files OffsetIndex.scala TimeIndex.scala There are different Index Files
  9. 9. When Kafka append Logs. @nonthreadsafe def append(largestOffset: Long, largestTimestamp: Long, shallowOffsetOfMaxTimestamp: Long, records: MemoryRecords): Unit = { …… val appendedBytes = log.append(records) …… offsetIndex.append(largestOffset, physicalPosition) timeIndex.maybeAppend(maxTimestampSoFar, offsetOfMaxTimestamp) …… }
  10. 10. When Kafka writes Logs #1 ● Using MMAP(so OS Page Cache is very important for performance) ● Offset is stored as relative offset ○ Current Offset - Base Offset ● Offset will be returned Absolute Offset
  11. 11. When Kafka writes Logs #2 OffsetIndex Append Int(4 bytes) Offset Int(4 bytes) Position
  12. 12. When Kafka writes Logs #3 TimeIndex Append Long(8 bytes) Timestamp Int(4 bytes) Position
  13. 13. How to fetch by timestamp #1 def fetchOffsetsByTimestamp(targetTimestamp: Long): Option[TimestampOffset] = { …… val targetSeg = { val earlierSegs = segmentsCopy.takeWhile(_.largestTimestamp < targetTimestamp) if (earlierSegs.length < segmentsCopy.length) Some(segmentsCopy(earlierSegs.length)) else None } targetSeg.flatMap(_.findOffsetByTimestamp(targetTimestamp, logStartOffset)) } }
  14. 14. How to fetch by timestamp #2 def findOffsetByTimestamp(timestamp: Long, startingOffset: Long = baseOffset): Option[TimestampOffset] = { // Get the index entry with a timestamp less than or equal to the target timestamp val timestampOffset = timeIndex.lookup(timestamp) val position = offsetIndex.lookup(math.max(timestampOffset.offset, startingOffset)).position // Search the timestamp Option(log.searchForTimestamp(timestamp, position, startingOffset)).map { timestampAndOffset => TimestampOffset(timestampAndOffset.timestamp, timestampAndOffset.offset) } } Using BinarySearch For Search
  15. 15. Simpe Question!!! ● Why offset index is needed? ● How to use binary search in Index File?
  16. 16. How to query by timestamp in spark Convert timestamp to OffsetRange Just Create KafkaRDD with OffsetRange
  17. 17. Create KafkaRDD with OffsetRange KafkaUtils.createRDD[K, V](spark.sparkContext, kafkaParamsMap, offsetRanges, PreferConsistent)
  18. 18. Convert timestamp to OffsetRange val consumer = createKafkaConsumer(props) val startOffset = consumer.offsetsForTimes(topicMap) val endOffset = consumer.offsetsForTimes(topicMap)
  19. 19. KafkaConsumer.offsetsForTimes Client API offsetsForTimes KafkaConsumer.java getOffsetsByTimes Fetcher.java retrieveOffsetsByTimes Fetcher.java Broker API handleListOffsetRequest KafkaApis.scala
  20. 20. Log Append Flows Broker API handleProduceRequest KafkaApis.scala appendRecords ReplicaManager.scala appendToLocalLog ReplicaManager.scala appendRecordsToLeade r Partition.scala appendAsLeader Log.scala append Log.scala append LogSegment.scala
  21. 21. When Segmnet is rolled? def shouldRoll(messagesSize: Int, maxTimestampInMessages: Long, maxOffsetInMessages: Long, now: Long): Boolean = { val reachedRollMs = timeWaitedForRoll(now, maxTimestampInMessages) > maxSegmentMs - rollJitterMs size > maxSegmentBytes - messagesSize || (size > 0 && reachedRollMs) || offsetIndex.isFull || timeIndex.isFull || !canConvertToRelativeOffset(maxOffsetInMessages) }
  22. 22. When Segmnet is rolled? def shouldRoll(messagesSize: Int, maxTimestampInMessages: Long, maxOffsetInMessages: Long, now: Long): Boolean = { val reachedRollMs = timeWaitedForRoll(now, maxTimestampInMessages) > maxSegmentMs - rollJitterMs size > maxSegmentBytes - messagesSize || (size > 0 && reachedRollMs) || offsetIndex.isFull || timeIndex.isFull || !canConvertToRelativeOffset(maxOffsetInMessages) } 1] size > maxSegmentBytes - messageSize 2] size > 0 && reachedRollMs 3] offsetIndex.isFull 4] timeIndex.isFull 5] canCovertToRelativeOffset is false
  23. 23. One Cent for Using Kafka Timestamp offset ● As a default, timestamp is set as sending time by client. ● So it is not a time that log is created. ○ You should specify to use timestamp as created time of log.
  24. 24. Thank you!
  25. 25. Quiz ● If timestamp is older than last timeIndex ○ How Kafka handles this?
  26. 26. Original 00000000317.log TimeIndex Log1 Offset: 317 Timestamp: 10000 …... Log100 Offset: 416 Timestamp: 20000 10000, 317 20000, 416
  27. 27. Append Log with old timestamp 00000000317.log TimeIndex Log1 Offset: 317 Timestamp: 10000 …... Log100 Offset: 416 Timestamp: 20000 10000, 317 20000, 416 Log101 Offset: 417 Timestamp: 20 Log200 Offset: 516 Timestamp: 30000 …... 30000, 516
  28. 28. Please remember!!! ● If you use too early timestamp. ○ It can remove your all kafka-logs. ● If you use too later timestamp. ○ It is never removed.

×