Research Paper | Computer Science & Engineering | India | Volume 4 Issue 7, July 2015
Parallel Data Shuffling for Hadoop Acceleration with Network Levitated Merge and RDMA for Interconnectivity
Kishorkumar Shinde, Venkatesan N.
Performance is measure issue in todays hadoop framework. The execution time required for Map reduce model is depends on multiple factors. Shuffling and merging in map reduce requires much amount of time. Proper implementation of shuffling and merging improves the performance of overall system. With this Serialization, multiple interconnect issues are also covered in this paper. Serialization keeps reduce phase to wait, repetitive merges requires multiple disk access and lack of portability for different interconnections. Repetitive merges can be reduced by network levitated merge algorithm, Serialization issue is overcome by parallelization. RDMA is used to for multiple interconnects. A non Hadoop and non java machine can also use the hadoop features. If we use pipelining to avoid serialization some sort of serialization is there in shuffle and merge phase. In pipelining output file is shuffled and merged before providing it to reduce task. Instead of pipelined shuffling, parallel shuffling is proposed. This reduces the number of disk accesses resulting in improved performance.
Keywords: Hadoop, Network levitated merge, MapReduce, Big- data, RDMA
Edition: Volume 4 Issue 7, July 2015
Pages: 1096 - 1101
How to Cite this Article?
Kishorkumar Shinde, Venkatesan N., "Parallel Data Shuffling for Hadoop Acceleration with Network Levitated Merge and RDMA for Interconnectivity", International Journal of Science and Research (IJSR), https://www.ijsr.net/search_index_results_paperid.php?id=SUB156536, Volume 4 Issue 7, July 2015, 1096 - 1101
111 PDF Views | 80 PDF Downloads
Similar Articles with Keyword 'Hadoop'
M.Tech / M.E / PhD Thesis, Computer Science & Engineering, India, Volume 3 Issue 12, December 2014
Pages: 1103 - 1108Design of a High Performing Cloud Using Load Rebalancing Technique in Distributed File System
Y. Steeven, C. Prakasha Rao
Dissertation Chapters, Computer Science & Engineering, India, Volume 4 Issue 7, July 2015
Pages: 1721 - 1725Secured Load Rebalancing for Distributed Files System in Cloud
Jayesh D. Kamble, Y. B. Gurav
Research Paper, Computer Science & Engineering, India, Volume 4 Issue 7, July 2015
Pages: 1096 - 1101Parallel Data Shuffling for Hadoop Acceleration with Network Levitated Merge and RDMA for Interconnectivity
Kishorkumar Shinde, Venkatesan N.
Research Paper, Computer Science & Engineering, India, Volume 4 Issue 12, December 2015
Pages: 1661 - 1667Performance Enhancement of MapReduce Framework in Big Data Application Using Load Balancing with Cache
Sushant Shirish Nagavkar, Ashishkumar
Survey Paper, Computer Science & Engineering, India, Volume 5 Issue 1, January 2016
Pages: 1967 - 1970Survey on Recommender System using Distributed Framework
Sonali B. Ghodake, R. S. Paswan
Similar Articles with Keyword 'MapReduce'
Survey Paper, Computer Science & Engineering, India, Volume 4 Issue 5, May 2015
Pages: 1164 - 1169A Survey on Scalable Big Data Analytics Platform
Ravindra Phule, Madhav Ingle
Research Paper, Computer Science & Engineering, India, Volume 4 Issue 9, September 2015
Pages: 1598 - 1602Information Acquisition Utilizing Parallel Rough Set and MapReduce from Big Information
Sachin Jadhav, Shubhangi Suryawanshi
M.Tech / M.E / PhD Thesis, Computer Science & Engineering, India, Volume 3 Issue 12, December 2014
Pages: 1103 - 1108Design of a High Performing Cloud Using Load Rebalancing Technique in Distributed File System
Y. Steeven, C. Prakasha Rao
Dissertation Chapters, Computer Science & Engineering, India, Volume 4 Issue 7, July 2015
Pages: 1721 - 1725Secured Load Rebalancing for Distributed Files System in Cloud
Jayesh D. Kamble, Y. B. Gurav
Survey Paper, Computer Science & Engineering, India, Volume 3 Issue 12, December 2014
Pages: 1510 - 1513A Survey on Optimal Data Storage of Cache Manager for Big Data Using Map Reduce Framework
Rupali Pashte, Ritesh Thakur