Log Replication in Raft vs Kafka
Abstract
The implementation of a fault-tolerant system requires some type of consensus algorithm for correct operation. From Paxos to View-stamped Replication and Raft multiple algorithms have been developed to handle this problem. This paper presents and compares the Raft algorithm and Apache Kafka, a distributed messaging system which, although at a higher level, implements many concepts present in Raft (strong leadership, append-only log, log compaction, etc.).
This shows that mechanisms conceived to handle one class of problems (consensus algorithms) are very useful to handle a larger category in the context of distributed systems.
References
[2] C. McCabe, Kafka improvements proposals” (having status accepted 2020, Retrieved from https://cwiki.apache.org/confluence/display/KAFKA/KIP-500
[3] D. Ongaro, J. Ousterhout (2013), In Search of an Understandable Consensus Algorithm, USENIX ATC 14, Proceedings of the 2014 USENIX Annual Technical Conference, June 2014, Pages 305-320, Retrieved from https://web.stanford.edu/ ouster/cgi-bin/papers/raft-atc14
[4] J. Kreps, Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines), Retrieved from https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines, Retrieved
[5] J. Ousterhout, Lecture for the Raft User Study, March 2013, Retrieved from https://raft.github.io/slides/raftuserstudy2013.pdf
[6] L. Lamport, The Part-Time Parliament, ACM Transactions on Computer Systems, 16 (2): 133-169. DOI:10.1145/279227.279229.
[7] T. D Chandra, R. Griesemer, J.Redstone, Paxos made live: an engineering perspective. In Proc. PODC’07, ACM Symposium on Principles of Distributed Computing (2007), ACM, pp. 398-407.
[8] W. Guozhang, J. Koshy, S. Subramanian, K. Paramasivam, M. Zadeh, N. Narkhede, J. Rao, J. Kreps, J. Stei, Building a Replicated Logging System with Apache Kafka, Proceedings of the VLDB Endowment, Retrieved from http://www.vldb.org/pvldb/vol8/p1654-wang.pdf, DOI:10.14778/2824032.2824063
[9] W. Han Z. Shang, K. Wolter, Performance Prediction for the Apache Kafka Messaging System, 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City, DOI: 10.1109/HPCC/SmartCity/DSS.2019.00036
[10] B. Reed, F. Junqueira, A simple totally ordered broadcast protocol, Proceedings of the 2nd Workshop on Large-Scale Distributed Systems and Middleware, September 2008, Article No 2, Pages 1-6, doi:10.1145/1529974.1529978,
[11] A. Medeiros, ZooKeeper’s atomic broadcast protocol: Theory and practice, Helsinki University of Technology, 2012, Retrieved from https://www.semanticscholar.org/paper/ZooKeeper-
[12] F. P. Junqueira, B. C. Reed and M. Serafini, Zab: High-performance broadcast for primary-backup systems, 2011 IEEE/IFIP 41st International Conference on Dependable Systems and Networks (DSN), Hong Kong, 2011, pp. 245-256, DOI:10.1109/DSN.2011.5958223
[13] P. Hunt, M. Konar, F. Junqueira, B. Reed, ZooKeeper: wait-free coordination for internet-scale systems, USENIXATC’10: Proceedings of the 2010 USENIX conference on USENIX annual technical conference, June 2010, Pages 11
[14] H. Yusuf1, I Surjandari. (2020). Comparison of Performance Between Kafka and Raft as Ordering Service Nodes Implementation in Hyperledger Fabric. International Journal of Advanced Science and Technology, 29(7s), 3549-3554. Retrieved from http://sersc.org/journals/index.php/IJAST/article/view/17652
[15] H. Yusuf, I Surjandari, Comparison of Performance Between Kafka and Raft as Ordering Service Nodes Implementation in Hyperledger Fabric, International Journal of Advanced Science and Technology Vol. 29, No. 7s, (2020), pp. 3549-3554, ISSN: 2005-4238 IJAST
[16] C. Wang, X. Chu, Performance Characterization and Bottleneck Analysis of Hyper-ledger Fabric, arXiv:2008.05946v1 [cs.DC]
[17] C. McCabe, Kafka improvements proposals (having status accepted), 2020, https://cwiki.apache.org/confluence/display/KAFKA/KIP500
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
When the article is accepted for publication, I, as the author and representative of the coauthors, hereby agree to transfer to Studia Universitatis Babes-Bolyai, Series Informatica, all rights, including those pertaining to electronic forms and transmissions, under existing copyright laws, except for the following, which the author specifically retain: the right to make further copies of all or part of the published article for my use in classroom teaching; the right to reuse all or part of this material in a review or in a textbook of which I am the author; the right to make copies of the published work for internal distribution within the institution that employs me.
