Using Latency Metrics in NoSQL Database Performance Benchmarking
Abstract
This paper presents an experimental study evaluating the performance of NoSQL database management systems. The study compares two NoSQL database management systems (Cassandra and MongoDB) and considers the following parameters/factors: workload and degree of parallelism. Two different workloads (update heavy and mostly read) were used, and different numbers of threads. The measured results are related to average latency: update latency and read latency. Our study shows that with the only exception of 1000 operations, both latency indicators have a quasi-parabolic behavior, where the minimum (i.e. the best performance) depends mainly on the number of threads and slightly varies with the increase in the number of operations. In the case of 1000 operations, there is also a maximum point (i.e. worst performance) case, after which the latency decreases.
References
[2] Apache Cassandra. http://cassandra.apache.org/. Accessed: 2017-09-25.
[3] F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A Distributed Storage System for Structured Data. OSDI ’06 Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation, 7, 2006.
[4] B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Benchmarking Cloud Serving Systems with YCSB. Proceedings of the 1st ACM Symposium on Cloud Computing, pages 143–154, 2010.
[5] CouchDB. http://couchdb.apache.org/. Accessed: 2017-09-25.
[6] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: Amazon’s Highly Available Key-value Store. Proceedings of 21st ACM SIGOPS Symposium on Operating Systems Principles, oct 2007.
[7] Fixstars. GridDB and Cassandra Performance and Scalability. A YCSB Performance Comparison on Microsoft Azure. Technical report, Fixstars Solutions, 2016.
[8] A. Gandini, M. Gribaudo, W. J. Knottenbelt, R. Osman, and P. Piazzolla. Performance Evaluation of NoSQL Databases. EPEW 2014: Computer Performance Engineering, Lecture Notes in Computer Science, 8721:16–29, 2014.
[9] A. Ghazal, T. Rabl, M. Hu, F. Raab, M. Poess, A. Crolotte, and H.-A. Jacobsen. Big-Bench: Towards an Industry Standard Benchmark for Big Data Analytics. Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pages 1197–1208, 2013.
[10] HBase. https://hbase.apache.org/. Accessed: 2017-09-25.
[11] JSON. https://www.json.org/. Accessed: 2018-03-16.
[12] J. Klein, I. Gorton, N. Ernst, P. Donohoe, K. Pham, and C. Matser. Performance Evaluation of NoSQL Databases: A Case Study. Proceedings of the 1st Workshop on Performance Analysis of Big Data Systems, pages 5–10, 2015.
[13] A. Lakshman and P. Malik. Cassandra: A Decentralized Structured Storage System. ACM SIGOPS Operating Systems Review, 44:35–40, 2010.
[14] MongoDB. https://www.mongodb.com/. Accessed: 2017-09-25.
[15] R. O. Nambiar and M. Poess. The Making of TPC-DS. VLDB ’06 Proceedings of the 32nd International Conference on Very Large Data Bases, pages 1049–1058, 2006.
[16] OrientDB. http://orientdb.com/. Accessed: 2017-09-25.
[17] R Statistics Package. https://www.r-project.org/. Accessed: 2017-09-25.
[18] Stress Test for Couchbase Client and Cluster. http://docs.couchbase.com/sdk-api/couchbase-c-client-2.4.8/md_doc_cbc-pillowfight.html. Accessed: 2019-01-03.
[19] The cassandra-stress tool. https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsCStress.html. Accessed: 2019-01-03.
[20] The YCSB Core Workloads. https://github.com/brianfrankcooper/YCSB/wiki/Core-Workloads. Accessed: 2017-09-25.
[21] XML. https://www.w3.org/TR/2008/REC-xml-20081126/. Accessed: 2018-03-16.
[22] YAML. http://yaml.org/. Accessed: 2018-03-16.
[23] YCSB Github Wiki. https://github.com/brianfrankcooper/YCSB/wiki. Accessed: 2017-09-25.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Transfer of copyright agreement: When the article is accepted for publication, I, as the author and the representative of the coauthors, hereby agree to transfer to Studia Universitatis Babes-Bolyai, Series Informatica, all rights, including those pertaining to electronic forms and transmissions, under existing copyright laws, except for the following, which the authors specifically retain: the authors can use the material however they want as long as it fits the NC ND terms of the license. The authors have all rights for reuse according to the below license.
