在《Kafka集群扩展以及重新分布分区》文章中我们介绍了如何重新分布分区,在那里面我们基本上把所有的分区全部移动了,其实我们完全没必要移动所有的分区,而移动其中部分的分区。比如我们想把Broker 1与Broker 7上面的分区数据互换,如下图所示:
可以看出,只有Broker 1与Broker 7上面的分区做了移动。来看看移动分区之前各个分区以及Leader分布情况:
[iteblog@www.iteblog.com kafka]$ ./bin/kafka-topics.sh --topic iteblog --describe \ --zookeeper www.iteblog.com Topic:iteblog PartitionCount:7 ReplicationFactor:2 Configs: Topic: iteblog Partition: 0 Leader: 1 Replicas: 1,4 Isr: 1,4 Topic: iteblog Partition: 1 Leader: 7 Replicas: 7,3 Isr: 3,7 Topic: iteblog Partition: 2 Leader: 5 Replicas: 5,7 Isr: 5,7 Topic: iteblog Partition: 3 Leader: 6 Replicas: 6,1 Isr: 1,6 Topic: iteblog Partition: 4 Leader: 4 Replicas: 4,2 Isr: 4,2 Topic: iteblog Partition: 5 Leader: 2 Replicas: 2,5 Isr: 5,2 Topic: iteblog Partition: 6 Leader: 3 Replicas: 3,6 Isr: 3,6
将Broker 1与Broker 7上面的分区互换,我们配置了以下的replica计划:
{ "version": 1, "partitions": [ { "topic": "iteblog", "partition": 0, "replicas": [ 7, 4 ] }, { "topic": "iteblog", "partition": 1, "replicas": [ 1, 3 ] }, { "topic": "iteblog", "partition": 3, "replicas": [ 6, 7 ] }, { "topic": "iteblog", "partition": 2, "replicas": [ 5, 1 ] } ] }
可以看到只涉及到Broker 1与Broker 7上面的分区,将上面的Json数据保存到iteblog.json文件里面,然后执行它:
[iteblog@www.iteblog.com kafka]$ bin/kafka-reassign-partitions.sh \ --zookeeper www.iteblog.com \ --reassignment-json-file iteblog.json --execute Current partition replica assignment {"version":1,"partitions":[{"topic":"iteblog","partition":3,"replicas":[6,1]}, {"topic":"iteblog","partition":5,"replicas":[2,5]}, {"topic":"iteblog","partition":4,"replicas":[4,2]}, {"topic":"iteblog","partition":0,"replicas":[1,4]}, {"topic":"iteblog","partition":6,"replicas":[3,6]}, {"topic":"iteblog","partition":1,"replicas":[7,3]}, {"topic":"iteblog","partition":2,"replicas":[5,7]}]} Save this to use as the --reassignment-json-file option during rollback Successfully started reassignment of partitions {"version":1,"partitions":[ {"topic":"iteblog","partition":0,"replicas":[7,4]}, {"topic":"iteblog","partition":1,"replicas":[1,3]}, {"topic":"iteblog","partition":3,"replicas":[6,7]}, {"topic":"iteblog","partition":2,"replicas":[5,1]}]}
在运行的时候可以使用verify参数来看看reassign计划是否执行完成:
[iteblog@www.iteblog.com kafka]$ bin/kafka-reassign-partitions.sh \ --zookeeper www.iteblog.com \ --reassignment-json-file iteblog.json --verify Status of partition reassignment: Reassignment of partition [iteblog,0] is still in progress Reassignment of partition [iteblog,1] is still in progress Reassignment of partition [iteblog,3] is still in progress Reassignment of partition [iteblog,2] is still in progress [iteblog@www.iteblog.com kafka]$ bin/kafka-reassign-partitions.sh \ --zookeeper www.iteblog.com \ --reassignment-json-file iteblog.json --verify Status of partition reassignment: Reassignment of partition [iteblog,0] completed successfully Reassignment of partition [iteblog,1] completed successfully Reassignment of partition [iteblog,3] completed successfully Reassignment of partition [iteblog,2] completed successfully
当上面分区全部成功移动完成,然后可以我们再来看看各个分区的分布情况:
[iteblog@www.iteblog.com kafka]$ bin/kafka-topics.sh --topic iteblog --describe \ --zookeeper www.iteblog.com Topic:iteblog PartitionCount:7 ReplicationFactor:2 Configs: Topic: iteblog Partition: 0 Leader: 7 Replicas: 7,4 Isr: 4,7 Topic: iteblog Partition: 1 Leader: 1 Replicas: 1,3 Isr: 3,1 Topic: iteblog Partition: 2 Leader: 5 Replicas: 5,1 Isr: 5,1 Topic: iteblog Partition: 3 Leader: 6 Replicas: 6,7 Isr: 6,7 Topic: iteblog Partition: 4 Leader: 4 Replicas: 4,2 Isr: 4,2 Topic: iteblog Partition: 5 Leader: 2 Replicas: 2,5 Isr: 5,2 Topic: iteblog Partition: 6 Leader: 3 Replicas: 3,6 Isr: 3,6
可以看出已经将Broker 1与Broker 7上面的分区互换了。
本博客文章除特别声明,全部都是原创!原创文章版权归过往记忆大数据(过往记忆)所有,未经许可不得转载。
本文链接: 【如何移动Kafka部分分区的数据】(https://www.iteblog.com/archives/1613.html)