ID,Title,Description,Acceptance criteria,Clarification questions,Source reference,Severity,Probability,Kafka tickets that later did this work
SR-1,Implement System-Wide Configuration for Remote Log Storage Enable Flag,Add a new system-wide boolean configuration property `remote.log.storage.enable` to control whether tiered storage is activated. This property provides backward compatibility for existing Kafka deployments that do not use tiered storage. The default value must be `false` to ensure no impact on clusters not opting into the tiered storage feature. The configuration must be readable at broker startup and govern all remote log manager initialization logic.,"A new configuration property `remote.log.storage.enable` is added to the broker configuration schema
Valid values are strictly `true` or `false`
The default value is `false`
When set to `false`, no remote log manager components are initialized
When set to `true`, the broker proceeds to validate and initialize remote storage manager components
The configuration value is loaded at broker startup and cannot be changed without restart
The property is documented in the configuration reference with its backward compatibility purpose stated",,"KIP-405 Section ""Public Interfaces > Configs > System-Wide"", statement ""remote.log.storage.enable - Whether to enable remote log storage or not. Valid values are `true` or `false` and the default value is false. This property gives backward compatibility.""",MODERATE,OCCASIONAL,KAFKA-12816
SR-2,Implement Mandatory Configuration for Remote Storage Manager Class Name,"Add a mandatory system-wide configuration property `remote.log.storage.manager.class.name` that specifies the fully qualified class name of the RemoteStorageManager implementation to be used. This configuration is required only when `remote.log.storage.enable` is set to `true`. If `remote.log.storage.enable` is `true` and this property is not configured, the broker must fail startup with a clear error message indicating the missing mandatory configuration.","A new configuration property `remote.log.storage.manager.class.name` is added to the broker configuration schema
The property accepts a fully qualified Java class name as a string value
The property is marked as mandatory when `remote.log.storage.enable` is `true`
If `remote.log.storage.enable` is `true` and `remote.log.storage.manager.class.name` is not set, broker startup fails with error message stating the property is mandatory
The specified class is validated at startup to ensure it exists on the classpath and implements the RemoteStorageManager interface
If the class does not exist or does not implement RemoteStorageManager, broker startup fails with a descriptive error",,"KIP-405 Section ""Public Interfaces > Configs > System-Wide"", statement ""remote.log.storage.manager.class.name - This is mandatory if the remote.log.storage.enable is set as true.""",MODERATE,OCCASIONAL,KAFKA-12816
SR-3,Implement Optional Configuration for Remote Log Metadata Manager Class Name,"Add an optional system-wide configuration property `remote.log.metadata.manager.class.name` that specifies the fully qualified class name of a custom RemoteLogMetadataManager implementation. If this property is not configured, the broker must use a default inbuilt metadata manager backed by an internal Kafka topic. The configuration must be validated at startup if provided.","A new configuration property `remote.log.metadata.manager.class.name` is added to the broker configuration schema
The property is optional
The property accepts a fully qualified Java class name as a string value
If the property is not set, the broker initializes the default RemoteLogMetadataManager implementation backed by an internal topic
If the property is set, the specified class is validated at startup to ensure it exists on the classpath and implements the RemoteLogMetadataManager interface
If the class does not exist or does not implement RemoteLogMetadataManager, broker startup fails with a descriptive error
The default internal-topic-based metadata manager is documented as the fallback behavior",,"KIP-405 Section ""Public Interfaces > Configs > System-Wide"", statement ""remote.log.metadata.manager.class.name(optional) - This is an optional property. If this is not configured, Kafka uses an inbuilt metadata manager backed by an internal topic.""",MODERATE,REMOTE,KAFKA-9555 KAFKA-12816 KAFKA-15329
SR-4,Implement Remote Storage Manager-Specific Configuration Namespace,"Define a configuration namespace prefix `remote.log.storage.*` for RemoteStorageManager implementation-specific configuration properties. These configurations are dependent on the specific remote storage backend implementation (e.g., HDFS, S3) and must be passed through to the configured RemoteStorageManager instance during initialization. The broker must not validate the contents of these properties but must collect and provide them to the RemoteStorageManager.","The broker configuration parser recognizes all properties with prefix `remote.log.storage.*`
All properties matching the prefix are collected into a configuration map
The configuration map is passed to the RemoteStorageManager instance during initialization via its configure() method
The broker does not attempt to validate the values of these properties
Properties with this prefix are documented as implementation-dependent",,"KIP-405 Section ""Public Interfaces > Configs > RemoteStorageManager"", statement ""remote.log.storage.*""",MINOR,OCCASIONAL,KAFKA-12816 KAFKA-9565 KAFKA-9569 KAFKA-12458
SR-5,Implement Remote Log Metadata Manager-Specific Configuration Namespace,Define a configuration namespace prefix `remote.log.metadata.*` for RemoteLogMetadataManager implementation-specific configuration properties. These configurations are dependent on the specific metadata storage backend implementation and must be passed through to the configured RemoteLogMetadataManager instance during initialization. The broker must not validate the contents of these properties but must collect and provide them to the RemoteLogMetadataManager.,"The broker configuration parser recognizes all properties with prefix `remote.log.metadata.*`
All properties matching the prefix are collected into a configuration map
The configuration map is passed to the RemoteLogMetadataManager instance during initialization via its configure() method
The broker does not attempt to validate the values of these properties
Properties with this prefix are documented as implementation-dependent",,"KIP-405 Section ""Public Interfaces > Configs > RemoteLogMetadataManager"", statement ""remote.log.metadata.*""",MINOR,OCCASIONAL,KAFKA-12429 KAFKA-12816 KAFKA-15083
SR-6,Implement Remote Log Manager Thread Pool Size Configuration,"Add a system-wide configuration property `remote.log.manager.thread.pool.size` that specifies the number of threads in the remote log manager thread pool. This thread pool is used for scheduling tasks to copy log segments to remote storage, fetch remote log indexes, and clean up remote log segments. The configuration must accept a positive integer value.","A new configuration property `remote.log.manager.thread.pool.size` is added to the broker configuration schema
The property accepts a positive integer value
The property is validated at startup to reject zero, negative, or non-integer values
The thread pool is initialized with the configured number of threads
Each thread is capable of processing topic-partition tasks for copy, fetch, and cleanup operations","CLARIFICATION NEEDED: Default value for remote.log.manager.thread.pool.size not specified
CLARIFICATION NEEDED: Minimum and maximum acceptable values not specified","KIP-405 Section ""Public Interfaces > Configs > Thread pools"", statement ""remote.log.manager.thread.pool.size Remote log thread pool size, which is used in scheduling tasks to copy segments, fetch remote log indexes and clean up remote log segments.""",MODERATE,OCCASIONAL,KAFKA-12816
SR-7,Implement Remote Log Manager Task Interval Configuration,"Add a system-wide configuration property `remote.log.manager.task.interval.ms` that specifies the interval in milliseconds at which the remote log manager runs scheduled tasks such as copying segments to remote storage, fetching remote log indexes, and cleaning up remote log segments. The configuration must accept a positive integer value representing milliseconds.","A new configuration property `remote.log.manager.task.interval.ms` is added to the broker configuration schema
The property accepts a positive integer value in milliseconds
The property is validated at startup to reject zero, negative, or non-integer values
The remote log manager scheduler uses this interval to schedule topic-partition processing tasks
After a topic-partition is successfully processed, its next scheduled processing time is set to (current_time + remote.log.manager.task.interval.ms)","CLARIFICATION NEEDED: Default value for remote.log.manager.task.interval.ms not specified
CLARIFICATION NEEDED: Minimum and maximum acceptable values not specified","KIP-405 Section ""Public Interfaces > Configs > Thread pools"", statement ""remote.log.manager.task.interval.ms The interval at which remote log manager runs the scheduled tasks like copy segments, fetch remote log indexes and clean up remote log segments.""",MODERATE,OCCASIONAL,KAFKA-12816
SR-8,Implement Remote Log Reader Thread Pool Size Configuration,Add a system-wide configuration property `remote.log.reader.threads` that specifies the number of threads in the remote log reader thread pool. This thread pool handles consumer fetch requests that require reading data from remote storage. The configuration must accept a positive integer value.,"A new configuration property `remote.log.reader.threads` is added to the broker configuration schema
The property accepts a positive integer value
The property is validated at startup to reject zero, negative, or non-integer values
The remote storage fetcher thread pool is initialized with the configured number of threads
Each thread processes one remote fetch request at a time","CLARIFICATION NEEDED: Default value for remote.log.reader.threads not specified
CLARIFICATION NEEDED: Minimum and maximum acceptable values not specified","KIP-405 Section ""Public Interfaces > Configs > Thread pools"", statement ""remote.log.reader.threads Remote log reader thread pool size""",MODERATE,OCCASIONAL,KAFKA-12816
SR-9,Implement Remote Log Reader Maximum Pending Tasks Configuration,"Add a system-wide configuration property `remote.log.reader.max.pending.tasks` that specifies the maximum size of the remote log reader thread pool task queue. If the task queue is full, the broker must stop accepting new remote log read requests. The configuration must accept a positive integer value.","A new configuration property `remote.log.reader.max.pending.tasks` is added to the broker configuration schema
The property accepts a positive integer value
The property is validated at startup to reject zero, negative, or non-integer values
The remote storage fetcher thread pool task queue is initialized with the configured maximum capacity
When the task queue is full, the broker rejects new remote fetch requests or applies appropriate backpressure
Rejected requests are handled with a defined error response to the client","CLARIFICATION NEEDED: Default value for remote.log.reader.max.pending.tasks not specified
CLARIFICATION NEEDED: Minimum and maximum acceptable values not specified
CLARIFICATION NEEDED: Exact error response or backpressure mechanism when queue is full not specified","KIP-405 Section ""Public Interfaces > Configs > Thread pools"", statement ""remote.log.reader.max.pending.tasks Maximum remote log reader thread pool task queue size. If the task queue is full, broker will stop reading remote log segments.""",MODERATE,PROBABLE,KAFKA-12816
SR-10,Implement Per-Topic Remote Log Retention Time Configuration,Add a per-topic configuration property `remote.log.retention.minutes` that specifies the retention period in minutes for remote log segments. This value defines how long log segments are retained in remote storage before being eligible for deletion. The configuration must accept a positive integer value representing minutes.,"A new topic-level configuration property `remote.log.retention.minutes` is added to the topic configuration schema
The property accepts a positive integer value in minutes
The property can be set at topic creation or via alter-config operations
The retention calculation for remote log segments uses this value to determine eligibility for deletion
If not set, the broker must have a defined default behavior or require the value to be set explicitly when tiered storage is enabled","CLARIFICATION NEEDED: Default value for remote.log.retention.minutes not specified
CLARIFICATION NEEDED: Minimum and maximum acceptable values not specified
CLARIFICATION NEEDED: Interaction or relationship with local retention (log.retention) not fully specified","KIP-405 Section ""Public Interfaces > Configs > Per Topic Configuration"", statement ""remote.log.retention.minutes""",SERIOUS,OCCASIONAL,KAFKA-12816
SR-11,Implement Per-Topic Remote Log Retention Bytes Configuration,Add a per-topic configuration property `remote.log.retention.bytes` that specifies the retention limit in bytes for remote log segments. This value defines the maximum total size of log segments retained in remote storage for a topic-partition before older segments are eligible for deletion. The configuration must accept a positive integer value representing bytes.,"A new topic-level configuration property `remote.log.retention.bytes` is added to the topic configuration schema
The property accepts a positive integer value in bytes
The property can be set at topic creation or via alter-config operations
The retention calculation for remote log segments uses this value to determine eligibility for deletion based on cumulative size
If not set, the broker must have a defined default behavior or require the value to be set explicitly when tiered storage is enabled","CLARIFICATION NEEDED: Default value for remote.log.retention.bytes not specified
CLARIFICATION NEEDED: Minimum and maximum acceptable values not specified
CLARIFICATION NEEDED: Interaction or relationship with remote.log.retention.minutes not specified (e.g., OR vs AND logic)","KIP-405 Section ""Public Interfaces > Configs > Per Topic Configuration"", statement ""remote.log.retention.bytes""",SERIOUS,OCCASIONAL,KAFKA-12816
SR-12,Enforce Compacted Topics Exclusion from Remote Storage,"Implement logic to prevent compacted topics from using remote log storage. The system must detect at topic creation, configuration change, or broker startup whether a topic has cleanup.policy=compact and, if remote storage is configured or enabled for that topic, reject the configuration or disable remote storage for that topic.","During topic creation, if both compaction is enabled (cleanup.policy includes ""compact"") and remote storage is requested, the request is rejected with a clear error message
During topic configuration update (alter-config), if compaction is enabled and remote storage is already active, or if remote storage is enabled and compaction is active, the request is rejected with a clear error message
At broker startup, topics with both compaction and remote storage enabled are logged with a warning and remote storage is disabled for those topics
The error message explicitly states that compacted topics are not supported with remote storage",CLARIFICATION NEEDED: Exact rejection behavior at topic creation vs runtime not fully specified (reject creation vs disable feature),"KIP-405 Section ""Public Interfaces"", statement ""Compacted topics will not have remote storage support."" and Section ""Non-Goals"", statement ""It does not support compact topics.""",SERIOUS,OCCASIONAL,
SR-13,Define RemoteStorageManager Interface Specification,"Define the RemoteStorageManager interface as specified in the design document. This interface provides the lifecycle operations for remote log segments and indexes, including copying log segments to remote storage, fetching log segment data, and deleting remote log segments. The interface must be the integration point for pluggable remote storage backend implementations.","A Java interface named RemoteStorageManager is created with all methods specified in the design document
The interface includes a copyLogSegment method that accepts RemoteLogSegmentId and LogSegmentData and returns success/failure indication
The interface includes a fetchLogSegmentData method that accepts RemoteLogSegmentMetadata, startPosition (Long), and optional endPosition (Optional<Long>) and returns a Records instance
The interface includes a deleteLogSegment method that accepts RemoteLogSegmentId
The interface includes a configure method to receive implementation-specific configuration properties
The interface includes lifecycle methods (e.g., close) as specified
All method signatures, parameter types, and return types match the design document exactly","CLARIFICATION NEEDED: Complete method signatures and return types not provided in the input (marked as ""[code omitted]"")","KIP-405 Section ""Public Interfaces > Remote Storage Manager"", statement ""RemoteStorageManager is an interface to provide the lifecycle of remote log segments and indexes."" and ""[code omitted]""",CRITICAL,REMOTE,KAFKA-9548 KAFKA-9554
SR-14,Define RemoteLogMetadataManager Interface Specification,"Define the RemoteLogMetadataManager interface as specified in the design document. This interface provides the lifecycle operations for metadata about remote log segments with strongly consistent semantics. The interface must support storing, fetching, and managing remote log segment metadata, and must allow for pluggable custom implementations while providing a default implementation backed by an internal Kafka topic.","A Java interface named RemoteLogMetadataManager is created with all methods specified in the design document
The interface includes a putRemoteLogSegmentData method that accepts RemoteLogSegmentId and RemoteLogSegmentMetadata
The interface includes methods to fetch remote log segment metadata for a given offset
The interface includes a method to register a topic partition for metadata cache building
The interface includes an earliestLogOffset method that accepts TopicPartition and returns the earliest offset tracked in remote storage
The interface includes a configure method to receive implementation-specific configuration properties
The interface includes lifecycle methods (e.g., close) as specified
All method signatures, parameter types, and return types match the design document exactly","CLARIFICATION NEEDED: Complete method signatures and return types not provided in the input (marked as ""[code omitted]"")","KIP-405 Section ""Public Interfaces > RemoteLogMetadataManager"", statement ""RemoteLogMetadataManager is an interface to provide the lifecycle of metadata about remote log segments with strongly consistent semantics."" and ""[code omitted]""",CRITICAL,REMOTE,KAFKA-9548 KAFKA-9554 KAFKA-9555 KAFKA-12429
SR-15,Implement RemoteLogManager Component Initialization,"Implement the RemoteLogManager (RLM) component that receives callback events for leadership changes and stop/delete events of topic partitions on a broker. RLM must delegate copy, read, and delete operations of topic partition segments to the configured RemoteStorageManager implementation and maintain remote log segment metadata through the RemoteLogMetadataManager. RLM must be initialized only when remote.log.storage.enable is true.","A RemoteLogManager class is created as a core broker component
RLM is initialized during broker startup only if remote.log.storage.enable is true
RLM receives callbacks from ReplicaManager for topic-partition leadership changes (onPartitionLeadershipChange)
RLM receives callbacks for topic-partition stop/delete events
RLM initializes the configured RemoteStorageManager instance using remote.log.storage.manager.class.name
RLM initializes the configured or default RemoteLogMetadataManager instance
RLM maintains a registry of managed topic-partitions
RLM delegates copyLogSegment operations to RemoteStorageManager
RLM delegates metadata storage operations to RemoteLogMetadataManager
RLM handles component lifecycle (start, stop, close) correctly",,"KIP-405 Section ""Proposed Changes > High-level design"", statement ""RemoteLogManager (RLM) is a new component which receives callback events for leadership changes and stop/delete events of topic partitions on a broker. delegates copy, read, and delete of topic partition segments to a pluggable storage manager(viz RemoteStorageManager) implementation and maintains respective remote log segment metadata through RemoteLogMetadataManager.""",CRITICAL,REMOTE,
SR-16,Implement RLM Leader Task for Copying Log Segments to Remote Storage,"Implement the RLM Leader Task that runs for each leader topic partition. This task is responsible for identifying inactive log segment files that meet the criteria for remote upload and copying them to remote storage. The task must process segments one by one from earliest to latest, generate a unique RemoteLogSegmentId for each segment, store metadata via RemoteLogMetadataManager, and invoke copyLogSegment on RemoteStorageManager.","For each topic-partition where the broker is the leader, an RLM Leader Task is created and scheduled
The task identifies inactive log segment files (not the active segment)
The task filters segments where the offset range is not already covered by segments in remote storage
The task filters segments where the last offset is less than the last-stable-offset of the partition
Eligible segments are processed in order from earliest to latest
For each segment, a universally unique RemoteLogSegmentId is generated
The task calls RemoteLogMetadataManager.putRemoteLogSegmentData with the RemoteLogSegmentId and RemoteLogSegmentMetadata
The task calls RemoteStorageManager.copyLogSegment with the RemoteLogSegmentId and LogSegmentData
If copyLogSegment is successful, the metadata is updated with success status
If copyLogSegment fails, the metadata entry is removed
The task handles failures gracefully and schedules retry according to the configured retry interval",,"KIP-405 Section ""Proposed Changes > High-level design"" and ""RLM/RSM tasks and thread pools > 1. Remote Log Manager (RLM) Thread Pool > Copy log segments to remote storage (leader)"", full paragraph starting with ""Copy the log segment files that are inactive...""",CRITICAL,OCCASIONAL,KAFKA-9550
SR-17,Implement RLM Task for Handling Expired Remote Segments (Leader and Follower),"Implement the RLM task that handles expiration and deletion of remote log segments based on remote retention configuration. The leader computes segments to be deleted, updates the earliest offset in RemoteLogMetadataManager, removes segment metadata and data from remote storage, and handles failures by logging them to a dedicated topic. Followers fetch the earliest offset and update their local indexes and start offset accordingly.","For leader replicas, the task computes expired remote log segments based on remote.log.retention.minutes and remote.log.retention.bytes
The task updates the earliest offset for the topic partition in RemoteLogMetadataManager
The task retrieves all remote log segment IDs eligible for deletion
The task calls RemoteStorageManager.deleteLogSegment for each segment ID
The task removes metadata entries via RemoteLogMetadataManager
If deletion from remote storage fails, the failure is logged with the RemoteLogSegmentId to a dedicated topic (default: __remote_segments_to_be_deleted)
For follower replicas, the task calls RemoteLogMetadataManager.earliestLogOffset(TopicPartition) to fetch the earliest offset
Followers clean up local indexes up to the received earliest offset
Followers update their local log start offset with the received value","CLARIFICATION NEEDED: Exact name and schema of the __remote_segments_to_be_deleted topic not fully specified
CLARIFICATION NEEDED: Cleanup mechanism for entries in __remote_segments_to_be_deleted topic not specified","KIP-405 Section ""RLM/RSM tasks and thread pools > 1. Remote Log Manager (RLM) Thread Pool > Handle expired remote segments (leader and follower)"", full paragraph starting with ""RLM leader computes the log segments to be deleted...""",SERIOUS,OCCASIONAL,KAFKA-14888
SR-18,Implement RLM Topic-Partition Scheduled Processing Logic,"Implement the scheduling logic where RemoteLogManager maintains a list of managed topic-partitions, each with a scheduled processing time. Topic-partitions are processed by the RLM thread pool when their scheduled time is less than or equal to the current time. After successful processing, the scheduled time is updated to (now + rlm_process_interval_ms). After failure, the scheduled time is updated to (now + rlm_retry_interval_ms).","RLM maintains a thread-safe data structure (list or priority queue) of managed topic-partitions with scheduled processing times
The list is updated in Kafka I/O threads when topic-partitions are added or removed
When a new topic-partition is assigned, it is added to the list with scheduled processing time = 0 (immediate processing)
The RLM thread pool continuously selects topic-partitions where scheduled_time <= current_time for processing
After a topic-partition is successfully processed, its scheduled time is set to (current_time + remote.log.manager.task.interval.ms)
After a topic-partition processing fails due to remote storage error, its scheduled time is set to (current_time + rlm_retry_interval_ms)
When a topic-partition is unassigned and not currently being processed, it is immediately removed from the list
When a topic-partition is unassigned and currently being processed, it is marked as ""deleted"" and removed after the current processing completes","CLARIFICATION NEEDED: The configuration property name ""rlm_retry_interval_ms"" is not defined in the Configs section; only ""remote.log.manager.task.interval.ms"" is defined. Are these the same or is rlm_retry_interval_ms a separate config?","KIP-405 Section ""RLM/RSM tasks and thread pools > 1. Remote Log Manager (RLM) Thread Pool"", paragraph starting with ""RLM maintains a list of the topic-partitions it manages...""",MODERATE,OCCASIONAL,
SR-19,Implement Remote Storage Fetcher Thread Pool and RemoteFetchPurgatory,"Implement the remote storage fetcher thread pool that processes consumer fetch requests requiring data from remote storage. When a fetch request targets an offset in remote storage, the request is added to a RemoteFetchPurgatory (an instance of DelayedOperationPurgatory) to handle timeout, and the request is queued in the remote storage fetcher thread pool task queue. Each thread processes one remote fetch request by finding the RemoteLogSegmentId from RemoteLogMetadataManager, retrieving data from RemoteStorageManager, and building a Records instance to return to the client.","A RemoteFetchPurgatory instance is created as a DelayedOperationPurgatory for remote fetch requests
A remote storage fetcher thread pool is created with size configured by remote.log.reader.threads
The thread pool task queue has maximum capacity configured by remote.log.reader.max.pending.tasks
When a fetch request targets an offset in remote storage, it is added to RemoteFetchPurgatory and queued in the thread pool
Each thread processes one remote fetch request at a time
The thread finds the RemoteLogSegmentId for the requested offset from RemoteLogMetadataManager
The thread determines startPosition and endPosition from the offset index
The thread calls RemoteStorageManager.fetchLogSegmentData(RemoteLogSegmentMetadata, startPosition, endPosition) to retrieve data
The thread builds a Records instance from the fetched data
If the remote segment file is deleted, RemoteFetchPurgatory is notified to return an error to the client
If the remote storage operation fails, the operation is retried with Exponential Back-Off until the original consumer fetch request timeout
The thread pool enforces the maximum pending tasks limit and rejects or applies backpressure when the queue is full","CLARIFICATION NEEDED: Exact Exponential Back-Off parameters (initial delay, multiplier, max delay) not specified
CLARIFICATION NEEDED: Error response type when remote segment is deleted not specified","KIP-405 Section ""RLM/RSM tasks and thread pools > 2. Remote Storage Fetcher Thread Pool"", full paragraph starting with ""When handling consumer fetch request...""",SERIOUS,PROBABLE,KAFKA-9579 KAFKA-14889
SR-20,Implement Local and Remote Log Offset Constraints Enforcement,"Implement logic to maintain and enforce the offset constraints for leader topic partitions with tiered storage enabled. The constraints define the relationships between local log start offset (Lx), local log end offset (Lz), last stable offset (Ly), remote log end offset (Ry), and remote log start offset (Rx). These constraints must be validated and maintained during segment upload, deletion, and offset updates.","The system maintains the following offset values for each leader topic partition: Lx (local log start offset), Lz (local log end offset), Ly (last stable offset), Ry (remote log end offset), Rx (remote log start offset)
The system enforces the constraint: Lz >= Ly
The system enforces the constraint: Ly >= Lx
The system enforces the constraint: Ly >= Ry
The system enforces the constraint: Ry >= Rx
These constraints are validated before and after segment copy operations
These constraints are validated before and after remote segment deletion operations
Any operation that would violate a constraint is rejected or deferred until the constraint can be satisfied",,"KIP-405 Section ""Proposed Changes > Local and Remote log offset constraints"", full paragraph listing offset variables and constraints",CRITICAL,REMOTE,
SR-21,Implement Follower Local Segment Retention Logic for Tiered Storage,"Implement logic for follower replicas to defer deletion of local segment files until the corresponding segment data has been retrieved from remote storage and local indexes have been updated. Local segment files can only be deleted when: (1) the follower has received the corresponding segment data info from remote storage and updated its index files, AND (2) the local files are older than the configured remote retention time.","For follower replicas, segment deletion checks whether the segment metadata is present in RemoteLogMetadataManager
For follower replicas, segment deletion checks whether the segment has exceeded the remote retention time
A segment file is deleted only if both conditions are satisfied
If the segment metadata is not yet available, deletion is deferred even if the local retention time has expired
Follower replicas consume remote log metadata topic partitions to receive segment metadata updates
Upon receiving metadata, followers update their local remote log indexes",,"KIP-405 Section ""Proposed Changes > Manage Remote Log Segments"", paragraph starting with ""The leader may fail to ship segment data to remote storage on time...""",SERIOUS,OCCASIONAL,
SR-22,Implement ReplicaManager Integration for RLM Topic-Partition Lifecycle,"Integrate RemoteLogManager with ReplicaManager to handle topic-partition assignment and removal events. When tiered storage is configured, ReplicaManager must call RLM to assign or remove topic-partitions. When a broker transitions from Leader to Follower, RLM must complete any in-progress segment copy operation before relinquishing control, which may result in duplicate segments that are cleaned up later based on remote retention.","ReplicaManager checks if RemoteLogManager is configured (remote.log.storage.enable is true)
When a topic-partition is assigned to the broker, ReplicaManager calls RemoteLogManager to register the topic-partition
When a topic-partition is removed from the broker, ReplicaManager calls RemoteLogManager to unregister the topic-partition
When a broker transitions from Leader to Follower for a topic-partition, RemoteLogManager is notified
If a segment copy operation is in progress during Leader to Follower transition, the copy completes before the transition is finalized
Duplicate segments resulting from the transition are handled during subsequent remote retention cleanup",,"KIP-405 Section ""Proposed Changes > Replica Manager"", paragraph starting with ""If RLM is configured, ReplicaManager will call RLM...""",MODERATE,OCCASIONAL,
SR-23,Implement Consumer Fetch Request Delegation to RemoteLogManager,"Modify consumer fetch request handling in ReplicaManager to delegate reads to RemoteLogManager when the requested offset is not in local storage. For any fetch request, ReplicaManager calls readFromLocalLog; if this method returns an OffsetOutOfRangeException, ReplicaManager delegates the read call to RemoteLogManager to serve the data from remote storage.","ReplicaManager fetch request handling first attempts to read from local log via readFromLocalLog
If readFromLocalLog succeeds, the data is returned to the client as normal
If readFromLocalLog throws OffsetOutOfRangeException, ReplicaManager checks if the offset is within the remote log offset range
If the offset is in the remote log range, ReplicaManager delegates the read to RemoteLogManager
RemoteLogManager handles the remote fetch as per the remote storage fetcher thread pool logic
If the offset is not in the remote log range (i.e., before remote log start offset or after log end offset), an appropriate error is returned to the client",,"KIP-405 Section ""Proposed Changes > Consumer Fetch Requests"", paragraph starting with ""For any fetch requests, ReplicaManager will proceed...""",SERIOUS,OCCASIONAL,KAFKA-14889
SR-24,Implement Follower Replication Fetch Request Handling for Tiered Storage,"Modify follower fetch request handling so that leaders only return data that is still in the leader's local storage. Log segments that exist only in remote storage are not replicated to followers, as they are already present in remote storage. Followers retrieve information about remote segments from RemoteLogMetadataManager instead of replicating the data.","For follower fetch requests, the leader identifies the requested offset range
The leader serves data only from local log segments
If the requested offset range includes offsets that exist only in remote storage, those offsets are not included in the fetch response
Followers are expected to retrieve remote segment metadata from RemoteLogMetadataManager to learn about remote segments
If a follower becomes a leader, it can locate and serve data from remote storage using the metadata it has retrieved",CLARIFICATION NEEDED: Exact mechanism for followers to know they should not request remote-only offsets not fully specified (is it implicit in the offset range or explicit signaling?),"KIP-405 Section ""Proposed Changes > Follower Requests/Replication"", paragraph starting with ""For follower fetch, the leader only returns the data...""",MODERATE,OCCASIONAL,KAFKA-13369
SR-25,Implement DeleteRecords API Semantics for Tiered Storage,"Ensure the DeleteRecords API semantics are unchanged when tiered storage is enabled. The API deletes records until the given offset if possible, which is equivalent to updating logStartOffset of the partition log. If the new logStartOffset extends into remote storage, the system must asynchronously clean remote logs after updating the log-start-offset.","DeleteRecords API accepts an offset value
The API updates the logStartOffset of the partition log with the given offset if it is greater than the current log-start-offset and less than or equal to high-watermark
The API operation succeeds and returns to the client without waiting for remote log cleanup
After updating log-start-offset, if the new offset extends beyond the previous remote log start offset, the system schedules an asynchronous task to clean remote logs
Remote log cleanup updates the earliest offset in RemoteLogMetadataManager and deletes expired segments from RemoteStorageManager",,"KIP-405 Section ""Proposed Changes > Other APIs > DeleteRecords"", paragraph starting with ""There is no change in the semantics of this API...""",MODERATE,OCCASIONAL,
SR-26,Implement ListOffsets API Enhancement for Tiered Storage,"Enhance the ListOffsets API to support timestamps >= 0 by checking both remote log time indexes and local log time indexes. For timestamps >= 0, return the first message offset whose timestamp is >= the given timestamp, checking remote log time indexes first, then local log time indexes. The API must also support existing timestamp values: EARLIEST_TIMESTAMP (-2) returns logStartOffset, LATEST_TIMESTAMP (-1) returns log-stable-offset or log-end-offset based on isolation level.","ListOffsets API accepts a timestamp value
For timestamp = -2 (EARLIEST_TIMESTAMP), return the logStartOffset of the log (local or remote, whichever is earliest)
For timestamp = -1 (LATEST_TIMESTAMP), return log-stable-offset or log-end-offset based on the isolation level in the request
For timestamp >= 0, first search remote log time indexes for the first offset with timestamp >= requested timestamp
If found in remote log, return that offset
If not found in remote log, search local log time indexes for the first offset with timestamp >= requested timestamp
If found in local log, return that offset
If not found in either, return an appropriate error or end-of-log indication",,"KIP-405 Section ""Proposed Changes > Other APIs > ListOffsets"", paragraph starting with ""ListOffsets API gives the offset(s) for the given timestamp..."" up to ""For timestamps >= 0, it returns the first message offset...""",MODERATE,OCCASIONAL,
SR-27,Implement ListOffsets API New Target Timestamp NEXT_LOCAL_TIMESTAMP,"Add a new target timestamp value NEXT_LOCAL_TIMESTAMP (value -3) to the ListOffsets API. This value is used by follower replicas to request the offset that they should start fetching to replicate local logs, where all records earlier than this offset can be considered as copied to remote storage. The request and response schemas do not add new fields, but the version must be bumped to indicate support for the new timestamp value.","ListOffsets API supports a new target timestamp value: NEXT_LOCAL_TIMESTAMP with value -3
The ListOffsets request and response schemas do not add new fields
The ListOffsets request and response version is incremented to indicate support for NEXT_LOCAL_TIMESTAMP
When a follower sends a request with target timestamp = -3 (NEXT_LOCAL_TIMESTAMP), the leader returns the first offset in the local log (i.e., the earliest offset not yet copied to remote storage or the first offset that followers must replicate)
The returned offset allows followers to avoid fetching records that are already in remote storage
The implementation documents the use case: followers use this to determine the starting offset for replication","CLARIFICATION NEEDED: Exact offset calculation for NEXT_LOCAL_TIMESTAMP not fully specified (is it remote_log_end_offset + 1, or local_log_start_offset, or something else?)","KIP-405 Section ""Proposed Changes > Other APIs > ListOffsets"", paragraph starting with ""This API is enhanced with supporting new target timestamp value as -3...""",MODERATE,OCCASIONAL,KAFKA-13369
SR-28,Implement JBOD Alter Log Dirs Handling for Tiered Storage,Ensure that JBOD (Just a Bunch Of Disks) alter log dirs operations are compatible with tiered storage. JBOD supports altering log directories to a different directory on the local broker or to a directory on a remote broker. The tiered storage implementation must not interfere with these operations.,"Alter log dirs to a different directory on the local broker continues to work when tiered storage is enabled
Alter log dirs to a directory on a remote broker continues to work when tiered storage is enabled
Remote log metadata and indexes are updated or moved as necessary when log directories are altered
The remote storage backend is not affected by local log directory changes",CLARIFICATION NEEDED: Detailed interaction between JBOD alter log dirs and remote log metadata/indexes not fully specified,"KIP-405 Section ""Proposed Changes > Other APIs > JBOD related changes"", paragraph starting with ""Currently, JBOD is supported by altering log dirs...""",MINOR,REMOTE,
SR-29,Define Default Remote Log Metadata Topic Name and Configuration,"Define the default name for the internal remote log metadata topic as `__remote_log_metadata`. This topic stores metadata about remote log segments and is consumed by RemoteLogMetadataManager. Users must be able to configure the topic name, the number of partitions (default 50), and the replication factor.","The default remote log metadata topic name is `__remote_log_metadata`
The default number of partitions is 50
Users can configure the topic name via a configuration property
Users can configure the number of partitions via a configuration property
Users can configure the replication factor via a configuration property
The topic is created automatically by the broker at startup if it does not exist and tiered storage is enabled
The topic creation respects the configured name, partitions, and replication factor","CLARIFICATION NEEDED: Exact configuration property names for topic name, partition count, and replication factor not specified
CLARIFICATION NEEDED: Default replication factor not specified","KIP-405 Section ""Proposed Changes > RemoteLogMetadataManager implemented with an internal topic"", paragraph starting with ""Metadata of remote log segments are stored in an internal topic called `__remote_log_metadata`...""",MODERATE,REMOTE,
SR-30,Implement Remote Log Metadata Topic Partition Assignment Logic,Implement the logic to determine which remote log metadata topic partition is used for a given user topic-partition. The partition is computed as: user-topic-partition.toString().hashCode() % no_of_remote_log_metadata_topic_partitions. This ensures consistent assignment of metadata for a user topic-partition to a specific metadata topic partition.,"For each user topic-partition, the remote log metadata topic partition is computed using the formula: user-topic-partition.toString().hashCode() % number_of_remote_log_metadata_topic_partitions
The hashCode() method is called on the string representation of the TopicPartition object
The modulo operation uses the configured number of remote log metadata topic partitions
The computed partition is used consistently for all metadata operations for that user topic-partition
The logic is documented and unit-tested with known inputs and outputs",,"KIP-405 Section ""Proposed Changes > RemoteLogMetadataManager implemented with an internal topic"", statement ""Remote log metadata topic partition for a given user topic-partition is: user-topic-partition.toString().hashCode() % no_of_remote_log_metadata_topic_partitions""",MODERATE,REMOTE,
SR-31,Implement RemoteLogMetadataManager Cache for Leader and Follower Replicas,"Implement cache management logic in RemoteLogMetadataManager where leader replicas update the cache by producing metadata to the remote log metadata topic, and follower replicas build and maintain the cache by consuming from the respective remote log metadata topic partitions. The cache must be disk-backed (e.g., RocksDB) to handle potentially large metadata sets without excessive memory usage.","RemoteLogMetadataManager maintains an in-memory or disk-backed cache of remote log segment metadata
For leader replicas, after copying a segment to remote storage, RemoteLogManager calls RemoteLogMetadataManager.putRemoteLogSegmentData, which writes the metadata to the remote log metadata topic and updates the local cache
For follower replicas, RemoteLogMetadataManager subscribes to the relevant remote log metadata topic partitions
For follower replicas, RemoteLogMetadataManager consumes metadata messages and adds them to the local cache
When a topic partition is reassigned to a new broker, RemoteLogMetadataManager subscribes to the corresponding remote log metadata topic partition if not already subscribed
The cache is backed by a persistent store (e.g., RocksDB) to minimize memory footprint
Committed offsets for consumed metadata topic partitions are stored in a local file to avoid re-reading messages after broker restart","CLARIFICATION NEEDED: Exact disk-backed cache implementation (RocksDB confirmed as example, but not mandated)
CLARIFICATION NEEDED: Local file format and location for storing committed offsets not specified","KIP-405 Section ""Proposed Changes > RemoteLogMetadataManager implemented with an internal topic"", paragraphs starting with ""RLMM registers the topic partitions..."" and ""This requires the cache to be based on disk storage like RocksDB...""",SERIOUS,OCCASIONAL,KAFKA-12802
SR-32,"Define Remote Log Index File Formats (remoteLogIndex, remoteOffsetIndex, remoteTimeIndex)","Define the file formats for three remote log index file types: remoteLogIndex, remoteOffsetIndex, and remoteTimeIndex. The remoteLogIndex file contains entries mapping offset ranges to Remote Data Identifiers (RDI) in remote storage. The remoteOffsetIndex and remoteTimeIndex files are similar to existing Kafka .index and .timeindex files but point to entries in the remoteLogIndex file instead of log segment files. Index files are rotated at a configurable time interval or size.","The remoteLogIndex file format is defined with the following fields per entry: magic (int16, current value 0), length (int16), crc (int32), firstOffset (int64), lastOffset (int64), firstTimestamp (int64), lastTimestamp (int64), dataLength (int32), rdiLength (int16), rdi (byte[])
The remoteOffsetIndex file format is defined, with entries pointing to remoteLogIndex entries instead of log segment positions
The remoteTimeIndex file format is defined, with entries pointing to remoteLogIndex entries instead of log segment positions
Index files are named with the base offset of the first remote segment they index: {base_offset}.remoteLogIndex, {base_offset}.remoteOffsetIndex, {base_offset}.remoteTimeIndex
Index files are rotated at a configurable time interval or size threshold
RDI (Remote Data Identifier) format is implementation-dependent and encodes the location (e.g., HDFS path, S3 key) and offset of the data in remote storage","CLARIFICATION NEEDED: Configurable time interval and size threshold for index file rotation not specified
CLARIFICATION NEEDED: Exact RDI format for different remote storage implementations not specified (noted as implementation-dependent)","KIP-405 Section ""Proposed Changes > Remote Log Indexes"", paragraph starting with ""For each topic partition that has RLM configured..."" and format specification starting with ""magic: int16...""",CRITICAL,REMOTE,
SR-33,Implement Remote Log Index File Generation and Upload for Leader Replicas,"Implement the logic for leader replicas to generate remoteLogIndex, remoteOffsetIndex, and remoteTimeIndex entries after copying a log segment to remote storage. The remoteLogIndex entries must be appended locally and also shipped to remote storage along with the segment data. Followers retrieve these index entries from remote storage to build their own indices.","After a log segment is copied to remote storage, the leader generates one or more remoteLogIndex entries for that segment
Each remoteLogIndex entry contains the offset range, timestamp range, data length, and RDI (Remote Data Identifier) for a portion of the remote segment
The remoteLogIndex entries are appended to the local remoteLogIndex file
Corresponding remoteOffsetIndex and remoteTimeIndex entries are generated and appended to the respective local index files
The remoteLogIndex entries are uploaded to remote storage along with the segment data
Followers retrieve the remoteLogIndex entries from remote storage
Followers use the retrieved entries to build their own local remoteLogIndex, remoteOffsetIndex, and remoteTimeIndex files",CLARIFICATION NEEDED: Number of remoteLogIndex entries per segment (one or more) and granularity decision criteria not specified,"KIP-405 Section ""Proposed Changes > Remote Log Indexes"", paragraph starting with ""Depends on the implementation, RLM may append 1 or more entries..."" and ""The RemoteLogIndex entries are shipped to remote storage along with the segment data...""",SERIOUS,OCCASIONAL,
SR-34,Implement Bounded LRU Cache for Remote Log Index Files,"Implement a bounded LRU (Least Recently Used) cache in RemoteLogManager to store remote log index files (remoteLogIndex, remoteOffsetIndex, remoteTimeIndex) fetched from remote storage. This cache avoids repeated fetches of the same index files and allows them to be used in the same way as local segment indexes for data retrieval.","RemoteLogManager maintains an LRU cache for remote log index files
The cache is bounded by a configurable maximum size (number of index files or total bytes)
When an index file is needed for a remote fetch, RemoteLogManager checks the cache first
If the index file is in the cache, it is used directly without fetching from remote storage
If the index file is not in the cache, it is fetched from remote storage and added to the cache
The LRU eviction policy removes the least recently used index file when the cache is full
The cache is thread-safe and supports concurrent access by multiple remote fetch threads",CLARIFICATION NEEDED: Configurable maximum cache size (number of files or bytes) not specified,"KIP-405 Section ""Proposed Changes > High-level design"", statement ""RLM maintains a bounded cache(possibly LRU) of the index files of remote log segments to avoid multiple index fetches from the remote storage.""",MODERATE,OCCASIONAL,
SR-35,Define Default Topic Name for Failed Remote Segment Deletion Tracking,"Define the default name for the internal topic used to track remote log segments that failed to be deleted from remote storage. The default topic name is `__remote_segments_to_be_deleted`. This topic stores RemoteLogSegmentId values for segments that could not be removed from remote storage during the cleanup process, allowing users or automated processes to consume these events and retry deletion.","The default topic name for failed deletion tracking is `__remote_segments_to_be_deleted`
Users can configure the topic name via a configuration property
When deletion of a remote log segment via RemoteStorageManager fails, the RemoteLogSegmentId is written to this topic
The topic schema includes at minimum the RemoteLogSegmentId
Users or automated processes can consume events from this topic and retry deletion manually or programmatically
The topic is created automatically by the broker if it does not exist and tiered storage is enabled","CLARIFICATION NEEDED: Configuration property name for topic name not specified
CLARIFICATION NEEDED: Complete topic schema (key, value format) not specified
CLARIFICATION NEEDED: Automated retry mechanism for failed deletions not specified","KIP-405 Section ""RLM/RSM tasks and thread pools > 1. Remote Log Manager (RLM) Thread Pool > Handle expired remote segments"", statement ""If there are any failures in removing remote log segments then those are stored in a specific topic (default as __remote_segments_to_be_deleted)...""",MODERATE,OCCASIONAL,
SR-36,Implement Remote Storage Availability Isolation to Prevent Impact on Core Kafka Operations,"Implement architectural isolation to ensure that when remote storage becomes temporarily unavailable (up to several hours) or experiences high latency (up to minutes), core Kafka operations that do not rely on remote storage continue to function normally. All operations such as produce, consume local data, create/expand topics, and replica management must not be impacted by remote storage unavailability. Consumers attempting to consume remote data should receive reasonable errors when remote storage is unavailable or requests timeout.","All remote storage operations (copy, fetch, delete) are executed in dedicated thread pools separate from Kafka I/O threads and fetcher threads
Kafka I/O threads and fetcher threads do not block or wait on remote storage operations
When remote storage is unavailable, produce operations to local log continue without delay
When remote storage is unavailable, consumer fetch operations for local data continue without delay
When remote storage is unavailable, topic creation and expansion operations continue without delay
When remote storage is unavailable, replica management and leader election operations continue without delay
Consumers attempting to fetch data from remote storage during unavailability receive a clear error response (e.g., timeout or unavailable error)
Remote storage operation failures are logged with appropriate severity and detail
The system retries failed remote storage operations according to the configured retry interval without blocking other operations",,"KIP-405 Section ""RLM/RSM tasks and thread pools"", paragraph starting with ""Remote storage (e.g. HDFS/S3/GCP) is likely to have higher I/O latency..."" and ""When the remote storage becoming temporarily unavailable... Kafka should still be able to operate normally...""",CRITICAL,PROBABLE,
SR-37,Implement Goal: Extend Kafka Storage Beyond Local Disks with Minimal Internal Impact,Implement the core goal of tiered storage: extend Kafka's storage beyond the local storage available on the Kafka cluster by retaining older data in an external store (such as HDFS or S3) with minimal impact on the internals of Kafka. Kafka behavior and operational complexity must not change for existing users that do not have tiered storage feature configured.,"Older log segments are copied to remote storage (HDFS, S3, or other implementations of RemoteStorageManager)
The tiered storage feature is disabled by default (remote.log.storage.enable = false)
When tiered storage is disabled, Kafka behavior is identical to existing Kafka behavior with no performance or operational impact
When tiered storage is enabled, local storage requirements are significantly reduced compared to storing all data locally
The total retention period for data can be extended beyond local disk capacity by using remote storage
The implementation does not require changes to core Kafka protocols or data formats visible to clients
Existing producers and consumers operate without modification when tiered storage is enabled",,"KIP-405 Section ""Goals"", paragraph starting with ""Extend Kafka's storage beyond the local storage available on the Kafka cluster...""",CRITICAL,REMOTE,
SR-38,Implement Non-Goal: Tiered Storage Does Not Replace ETL Pipelines,"Document and implement the non-goal that tiered storage does not replace ETL pipelines and jobs. Existing ETL pipelines must continue to consume data from Kafka as usual, with the benefit that data in Kafka now has a much longer retention period due to tiered storage.","Tiered storage implementation does not alter or replace any existing ETL pipeline functionality
ETL pipelines can continue to consume data from Kafka using existing consumer APIs
The extended retention period provided by tiered storage is available to ETL consumers
Documentation clearly states that tiered storage is not a replacement for ETL and that existing ETL workflows are unaffected",,"KIP-405 Section ""Non-Goals"", statement ""Tiered storage does not replace ETL pipelines and jobs. Existing ETL pipelines continue to consume data from Kafka as is, albeit with data in Kafka having a much longer retention period.""",NEGLIGIBLE,IMPROBABLE,