Parallel Data Shuffling for Hadoop Acceleration with Network Levitated Merge and RDMA for Interconnectivity
International Journal of Science and Research (IJSR)

International Journal of Science and Research (IJSR)
www.ijsr.net | Open Access | Fully Refereed | Peer Reviewed International Journal

ISSN: 2319-7064

Research Paper | Computer Science & Engineering | India | Volume 4 Issue 7, July 2015

Parallel Data Shuffling for Hadoop Acceleration with Network Levitated Merge and RDMA for Interconnectivity

Kishorkumar Shinde, Venkatesan N.

Performance is measure issue in todays hadoop framework. The execution time required for Map reduce model is depends on multiple factors. Shuffling and merging in map reduce requires much amount of time. Proper implementation of shuffling and merging improves the performance of overall system. With this Serialization, multiple interconnect issues are also covered in this paper. Serialization keeps reduce phase to wait, repetitive merges requires multiple disk access and lack of portability for different interconnections. Repetitive merges can be reduced by network levitated merge algorithm, Serialization issue is overcome by parallelization. RDMA is used to for multiple interconnects. A non Hadoop and non java machine can also use the hadoop features. If we use pipelining to avoid serialization some sort of serialization is there in shuffle and merge phase. In pipelining output file is shuffled and merged before providing it to reduce task. Instead of pipelined shuffling, parallel shuffling is proposed. This reduces the number of disk accesses resulting in improved performance.

Keywords: Hadoop, Network levitated merge, MapReduce, Big- data, RDMA

Edition: Volume 4 Issue 7, July 2015

Pages: 1096 - 1101

Share this Article

How to Cite this Article?

Kishorkumar Shinde, Venkatesan N., "Parallel Data Shuffling for Hadoop Acceleration with Network Levitated Merge and RDMA for Interconnectivity", International Journal of Science and Research (IJSR), https://www.ijsr.net/search_index_results_paperid.php?id=SUB156536, Volume 4 Issue 7, July 2015, 1096 - 1101

144 PDF Views | 106 PDF Downloads

Download Article PDF



Similar Articles with Keyword 'Hadoop'

Dissertation Chapters, Computer Science & Engineering, India, Volume 4 Issue 7, July 2015

Pages: 1721 - 1725

Secured Load Rebalancing for Distributed Files System in Cloud

Jayesh D. Kamble, Y. B. Gurav

Share this Article

| Weekly Popularity: ⮙1

Research Paper, Computer Science & Engineering, India, Volume 4 Issue 7, July 2015

Pages: 2241 - 2244

Backup Anomaly Identification with R and Hadoop

Ravindra Phule, Madhav Ingle

Share this Article

| Weekly Popularity: ⮙3

M.Tech / M.E / PhD Thesis, Computer Science & Engineering, India, Volume 4 Issue 3, March 2015

Pages: 2133 - 2136

One Class Clustering Tree for Implementing Many to Many Data Linkage

Ravi R, Michael G

Share this Article

| Weekly Popularity: ⮙4

Informative Article, Computer Science & Engineering, India, Volume 5 Issue 11, November 2016

Pages: 1482 - 1485

Overview of Big Data

Nivedita Manohar

Share this Article

| Weekly Popularity: ⮙2

Survey Paper, Computer Science & Engineering, India, Volume 4 Issue 11, November 2015

Pages: 1249 - 1251

Survey on Resource Allocation in Phase-Level using MapReduce in Hadoop

Suryakant S. Bhalke

Share this Article

Similar Articles with Keyword 'MapReduce'

Dissertation Chapters, Computer Science & Engineering, India, Volume 4 Issue 7, July 2015

Pages: 1721 - 1725

Secured Load Rebalancing for Distributed Files System in Cloud

Jayesh D. Kamble, Y. B. Gurav

Share this Article

| Weekly Popularity: ⮙3

M.Tech / M.E / PhD Thesis, Computer Science & Engineering, India, Volume 4 Issue 3, March 2015

Pages: 2133 - 2136

One Class Clustering Tree for Implementing Many to Many Data Linkage

Ravi R, Michael G

Share this Article

| Weekly Popularity: ⮙1

Survey Paper, Computer Science & Engineering, India, Volume 4 Issue 5, May 2015

Pages: 1164 - 1169

A Survey on Scalable Big Data Analytics Platform

Ravindra Phule, Madhav Ingle

Share this Article

| Weekly Popularity: ⮙2

Survey Paper, Computer Science & Engineering, India, Volume 4 Issue 11, November 2015

Pages: 1249 - 1251

Survey on Resource Allocation in Phase-Level using MapReduce in Hadoop

Suryakant S. Bhalke

Share this Article

| Weekly Popularity: ⮙3

M.Tech / M.E / PhD Thesis, Computer Science & Engineering, India, Volume 3 Issue 12, December 2014

Pages: 1103 - 1108

Design of a High Performing Cloud Using Load Rebalancing Technique in Distributed File System

Y. Steeven, C. Prakasha Rao

Share this Article
Top