What is the Difference Between HDFS and MapReduce

The main difference between HDFS and MapReduce is that HDFS is a distributed file system that provides high throughput access to application data while MapReduce is a software framework that processes big data on large clusters reliably.

Big data is a collection of a large data set. It has three main properties: volume, velocity, and variety. Hadoop is a software that allows storing and managing big data. It is an open source framework written in Java. Moreover, it supports distributed processing of large data sets across clusters of computers. HDFS and MapReduce are two modules in Hadoop architecture.

Key Areas Covered

1. What is HDFS
     – Definition, Functionality
2. What is MapReduce
     – Definition, Functionality
3. What is the Difference Between HDFS and MapReduce
     – Comparison of Key Differences

Key Terms

Big Data, HDFS, MapReduce

What is HDFS

HDFS stands for Hadoop Distributed File System. It is a distributed file system of Hadoop to run on large clusters reliably and efficiently. Also, it is based on the Google File System (GFS). Moreover, it also has a list of commands to interact with the file system.

Furthermore, the HDFS works according to the master, slave architecture. The master node or name node manages the file system metadata while the slave nodes or the data notes store actual data.

Figure 1: HDFS Architecture

Besides, a file in an HDFS namespace is split into several blocks. Data nodes stores these blocks. And, the name node maps the blocks to the data nodes, which handle the reading and writing operations with the file system. Furthermore, they perform tasks such as block creation, deletion etc. as instructed by the name node.

What is MapReduce

MapReduce is a software framework that allows writing applications to process big data simultaneously on large clusters of commodity hardware. This framework consists of a single master job tracker and one slave task tracker per cluster node. The master performs resource management, scheduling jobs on slaves, monitoring and re-executing the failed tasks. On the other hand, the slave task tracker executes the tasks instructed by the master and sends the tasks status information back to the mater constantly.

Figure 2: MapReduce Overview

Also, there are two tasks associated with MapReduce. They are the map task and the reduce task. The map task takes input data and divides them into tuples of key, value pairs while the Reduce task takes the output from a map task as input and connects those data tuples into smaller tuples. Furthermore, the map task is performed before the reduce task.

Difference Between HDFS and MapReduce

Definition

HDFS is a Distributed File System that reliably stores large files across machines in a large cluster. In contrast, MapReduce is a software framework for easily writing applications which process vast amounts of data in parallel on large clusters of commodity hardware in a reliable, fault-tolerant manner. These definitions explain the main difference between HDFS and MapReduce.

Main Functionality

Another difference between HDFS and MapReduce is that the HDFS provides high-performance access to data across highly scalable Hadoop clusters while MapReduce performs the processing of big data.

Conclusion

In brief, HDFS and MapReduce are two modules in Hadoop architecture. The main difference between HDFS and MapReduce is that HDFS is a distributed file system that provides high throughput access to application data while MapReduce is a software framework that processes big data on large clusters reliably.

Reference:

1. “HDFS Architecture Guide”, Apache Hadoop, Available here. 
2. “MapReduce Tutorial”, Apache Hadoop, Available here.
3. “What Is Hadoop Distributed File System (HDFS)? – Definition from WhatIs.com.” SearchDataManagement, Available here.

Image Courtesy:

1. “Hdfsarchitecture” By Magnai17 – Own work (CC BY-SA 4.0) via Commons Wikimedia
2. “Mapreduce Overview” By Poposhka – SVG-Edit (CC BY-SA 3.0) via Commons Wikimedia