MERAL Myanmar Education Research and Learning Portal
Item
{"_buckets": {"deposit": "b826e53f-1b2a-45ec-8832-878d3e7b2617"}, "_deposit": {"created_by": 45, "id": "6300", "owner": "45", "owners": [45], "owners_ext": {"displayname": "", "username": ""}, "pid": {"revision_id": 0, "type": "recid", "value": "6300"}, "status": "published"}, "_oai": {"id": "oai:meral.edu.mm:recid/6300", "sets": ["user-uit"]}, "communities": ["uit"], "item_1583103067471": {"attribute_name": "Title", "attribute_value_mlt": [{"subitem_1551255647225": "Evaluation of Apache Kafka in Real-Time Big Data Pipeline Architecture", "subitem_1551255648112": "en"}]}, "item_1583103085720": {"attribute_name": "Description", "attribute_value_mlt": [{"interim": "Today, many applications based on real-time analytics\nneed to enable time-critical decision with real-time\nrequirements and process with reliability requirements.\nModern distributed systems are growing exponentially as\nfar as performance and scale. The main purpose of Big\nData real-time processing is to realize an entire system\nthat can process such mesh data in a short time. And the\nperformance of processing time can be guaranteed in a\nsatisfactory range. To develop a distributed data pipeline,\nthe system proposes real-time big data pipeline by using\nApache Kafka and Apache Storm. Apache Kafka is\ncurrently the most popular framework used to ingest the\ndata streams into the processing platforms. However,\nthere are many challenges how to send reliable messages\non many servers. This paper focuses on the comparison\nof the processing time between successful processes and\nfailed processes on many servers. The experimental\nresults show the performance impact on both producer\nand consumer of the Apache Kafka framework."}]}, "item_1583103108160": {"attribute_name": "Keywords", "attribute_value_mlt": [{"interim": "Apache Kafka"}, {"interim": "Apache storm"}, {"interim": "Asynchronous replication"}, {"interim": "Real time processing"}]}, "item_1583103120197": {"attribute_name": "Files", "attribute_type": "file", "attribute_value_mlt": [{"accessrole": "open_access", "date": [{"dateType": "Available", "dateValue": "2020-11-19"}], "displaytype": "preview", "download_preview_message": "", "file_order": 0, "filename": "Evaluation of Apache Kafka in Real-Time Big Data Pipeline Architecture.pdf", "filesize": [{"value": "1.4 Mb"}], "format": "application/pdf", "future_date_message": "", "is_thumbnail": false, "licensefree": "© 2018 ICAIT", "licensetype": "license_free", "mimetype": "application/pdf", "size": 1400000.0, "url": {"url": "https://meral.edu.mm/record/6300/files/Evaluation of Apache Kafka in Real-Time Big Data Pipeline Architecture.pdf"}, "version_id": "b55ec1a3-4d88-4acd-a1a7-eff8a1f2139a"}]}, "item_1583103147082": {"attribute_name": "Conference papers", "attribute_value_mlt": [{"subitem_acronym": "ICAIT-2018", "subitem_c_date": "1-2 November, 2018", "subitem_conference_title": "2nd International Conference on Advanced Information Technologies", "subitem_place": "Yangon, Myanmar", "subitem_session": "Database and Big Data Analytics", "subitem_website": "https://www.uit.edu.mm/icait-2018/"}]}, "item_1583105942107": {"attribute_name": "Authors", "attribute_value_mlt": [{"subitem_authors": [{"subitem_authors_fullname": "Thandar Aung"}, {"subitem_authors_fullname": "Hla Yin Min"}, {"subitem_authors_fullname": "Aung Htein Maw"}]}]}, "item_1583108359239": {"attribute_name": "Upload type", "attribute_value_mlt": [{"interim": "Publication"}]}, "item_1583108428133": {"attribute_name": "Publication type", "attribute_value_mlt": [{"interim": "Conference paper"}]}, "item_1583159729339": {"attribute_name": "Publication date", "attribute_value": "2018-11-02"}, "item_title": "Evaluation of Apache Kafka in Real-Time Big Data Pipeline Architecture", "item_type_id": "21", "owner": "45", "path": ["1605779935331"], "permalink_uri": "http://hdl.handle.net/20.500.12678/0000006300", "pubdate": {"attribute_name": "Deposited date", "attribute_value": "2020-11-19"}, "publish_date": "2020-11-19", "publish_status": "0", "recid": "6300", "relation": {}, "relation_version_is_last": true, "title": ["Evaluation of Apache Kafka in Real-Time Big Data Pipeline Architecture"], "weko_shared_id": -1}
Evaluation of Apache Kafka in Real-Time Big Data Pipeline Architecture
http://hdl.handle.net/20.500.12678/0000006300
http://hdl.handle.net/20.500.12678/0000006300b430ea6e-c289-4a62-a5cb-3a185d7978ba
b826e53f-1b2a-45ec-8832-878d3e7b2617
Name / File | License | Actions |
---|---|---|
![]() |
© 2018 ICAIT
|
Publication type | ||||||
---|---|---|---|---|---|---|
Conference paper | ||||||
Upload type | ||||||
Publication | ||||||
Title | ||||||
Title | Evaluation of Apache Kafka in Real-Time Big Data Pipeline Architecture | |||||
Language | en | |||||
Publication date | 2018-11-02 | |||||
Authors | ||||||
Thandar Aung | ||||||
Hla Yin Min | ||||||
Aung Htein Maw | ||||||
Description | ||||||
Today, many applications based on real-time analytics need to enable time-critical decision with real-time requirements and process with reliability requirements. Modern distributed systems are growing exponentially as far as performance and scale. The main purpose of Big Data real-time processing is to realize an entire system that can process such mesh data in a short time. And the performance of processing time can be guaranteed in a satisfactory range. To develop a distributed data pipeline, the system proposes real-time big data pipeline by using Apache Kafka and Apache Storm. Apache Kafka is currently the most popular framework used to ingest the data streams into the processing platforms. However, there are many challenges how to send reliable messages on many servers. This paper focuses on the comparison of the processing time between successful processes and failed processes on many servers. The experimental results show the performance impact on both producer and consumer of the Apache Kafka framework. |
||||||
Keywords | ||||||
Apache Kafka, Apache storm, Asynchronous replication, Real time processing | ||||||
Conference papers | ||||||
ICAIT-2018 | ||||||
1-2 November, 2018 | ||||||
2nd International Conference on Advanced Information Technologies | ||||||
Yangon, Myanmar | ||||||
Database and Big Data Analytics | ||||||
https://www.uit.edu.mm/icait-2018/ |