Hadoop
A comprehensive Big Data cheatsheet covering
1. What is Big Data?
Big Data refers to extremely large and complex datasets that cannot be processed, stored, or analyzed using traditional database systems.
5 V’s of Big Data:
- Volume – Huge data (TB, PB)
- Velocity – Fast data generation
- Variety – Structured, semi, unstructured
- Veracity – Data uncertainty
- Value – Useful insights
2. What is Hadoop?
Hadoop = Distributed Storage (HDFS) + Distributed Processing (MapReduce)
Created by: Doug Cutting Named after his son’s toy elephant 🐘
3. Hadoop Architecture
A) HDFS Architecture
1. NameNode
- Master node
- Stores metadata (file name, block location)
- Does NOT store actual data
2. DataNode
- Slave node
- Stores actual blocks
- Sends heartbeat to NameNode
3. Secondary NameNode
- Takes checkpoint
- Not backup (important for exam!)
B) MapReduce Architecture
Components:
- JobTracker (Hadoop 1)
- TaskTracker (Hadoop 1)
- ResourceManager (Hadoop 2 – YARN)
- NodeManager
4. HDFS Important Concepts
Block Size
Default:
- 128 MB (Hadoop 2+)
Replication Factor
Default:
- 3
5. Important Hadoop Configuration Files
Location:
$HADOOP_HOME/etc/hadoop/
| File | Purpose |
|---|---|
| core-site.xml | Core configuration |
| hdfs-site.xml | HDFS settings |
| mapred-site.xml | MapReduce settings |
| yarn-site.xml | YARN configuration |
6. Hadoop Startup Commands
Format HDFS
hdfs namenode -format
Start Hadoop
start-dfs.sh
start-yarn.sh
Stop Hadoop
stop-dfs.sh
stop-yarn.sh
Check running daemons
jps
Expected output:
- NameNode
- DataNode
- SecondaryNameNode
- ResourceManager
- NodeManager
7. HDFS Commands (Very Important 🔥)
Basic Commands
Check HDFS
hdfs dfs -ls /
Create directory
hdfs dfs -mkdir /data
Upload file
hdfs dfs -put file.txt /data
Download file
hdfs dfs -get /data/file.txt
View file
hdfs dfs -cat /data/file.txt
Delete file
hdfs dfs -rm /data/file.txt
Delete directory
hdfs dfs -rm -r /data
🗑 Trash Policy (Exam Important)
Enable in core-site.xml:
<property>
<name>fs.trash.interval</name>
<value>60</value>
</property>
Access trash:
hdfs dfs -ls /user/username/.Trash
Restore:
hdfs dfs -mv /user/username/.Trash/Current/data/file.txt /data/
8. HDFS File System Check
Check file block details
hdfs fsck /data/file.txt -files -blocks
Check replication
hdfs dfs -stat %r /data/file.txt
9. MapReduce Example (WordCount – Practical)
Step 1: Create input file
echo "hello hadoop hello big data" > input.txt
Step 2: Put in HDFS
hdfs dfs -mkdir /input
hdfs dfs -put input.txt /input
Step 3: Run WordCount
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount /input /output
Step 4: View output
hdfs dfs -cat /output/part-r-00000
10. YARN Architecture
YARN = Yet Another Resource Negotiator
Components:
- ResourceManager
- NodeManager
- ApplicationMaster
- Container
11. Hadoop Web UI
| Service | URL |
|---|---|
| NameNode | http://localhost:9870 |
| ResourceManager | http://localhost:8088 |
12. Important Hadoop Ports
| Service | Port |
|---|---|
| NameNode | 9870 |
| DataNode | 9864 |
| ResourceManager | 8088 |
13. Common Hadoop Ecosystem Tools
Apache Hive :- SQL on Hadoop
Apache Pig :- Data flow scripting
Apache HBase :- NoSQL DB on HDFS
Apache Spark :- Fast processing engine
14. Important HDFS Commands (Exam Special)
stat formats
hdfs dfs -stat %b file
| Format | Meaning |
|---|---|
| %b | file size in bytes |
| %g | group name |
| %n | file name |
| %o | block size |
15. Checksum Concept
- Each block divided into chunks (512 bytes default)
- Checksum created per chunk
- Stored in .meta file
- Detects corruption
16. Hadoop Modes
| Mode | Description |
|---|---|
| Standalone | Default |
| Pseudo-distributed | Single machine cluster |
| Fully distributed | Real cluster |
17. Interview Important Points
- HDFS write once read many
- Data locality
- Rack awareness
- Fault tolerance
- Heartbeat mechanism
- Checkpointing
18. Directory Management Commands
List files
hdfs dfs -ls /
Detailed listing
hdfs dfs -ls -h /
Recursive listing
hdfs dfs -ls -R /
Create directory
hdfs dfs -mkdir /data
Create multiple directories
hdfs dfs -mkdir -p /data/input/2026
Remove empty directory
hdfs dfs -rmdir /data
Remove non-empty directory
hdfs dfs -rm -r /data
Force delete (skip trash)
hdfs dfs -rm -r -skipTrash /data
19. File Management Commands
Upload local → HDFS
hdfs dfs -put file.txt /data
Alternative upload
hdfs dfs -copyFromLocal file.txt /data
Download HDFS → local
hdfs dfs -get /data/file.txt
Alternative download
hdfs dfs -copyToLocal /data/file.txt
Move file
hdfs dfs -mv /data/file.txt /input/
Copy file inside HDFS
hdfs dfs -cp /data/file.txt /backup/
Append data (Important 🔥)
hdfs dfs -appendToFile newdata.txt /data/file.txt
(HDFS normally write-once-read-many, but append is supported.)
20. Viewing File Content
View full file
hdfs dfs -cat /data/file.txt
View first lines
hdfs dfs -head /data/file.txt
View last lines
hdfs dfs -tail /data/file.txt
Count lines/words/chars
hdfs dfs -cat /data/file.txt | wc
21 File Information Commands (Exam Special 🔥)
Check disk usage
hdfs dfs -du /data
Human readable size
hdfs dfs -du -h /data
Check total disk usage summary
hdfs dfs -df -h
File statistics
hdfs dfs -stat %b /data/file.txt
| Format | Meaning |
|---|---|
| %b | file size |
| %n | file name |
| %g | group |
| %o | block size |
| %r | replication factor |
| %y | modification date |
Check blocks & replication
hdfs fsck /data/file.txt -files -blocks -locations
22 Permission & Ownership Commands
Change permission
hdfs dfs -chmod 755 /data/file.txt
Change owner
hdfs dfs -chown user:group /data/file.txt
Change group
hdfs dfs -chgrp groupname /data/file.txt
Permission Format:
rwx r-x r-x
- 7 = rwx
- 6 = rw-
- 5 = r-x
23. Replication Management
Check replication
hdfs dfs -stat %r /data/file.txt
Change replication
hdfs dfs -setrep 2 /data/file.txt
Force replication immediately
hdfs dfs -setrep -w 2 /data/file.txt
24. Trash Management (Very Important for Exam)
Enable Trash (core-site.xml)
<property> <name>fs.trash.interval</name> <value>60</value> </property>
View trash
hdfs dfs -ls /user/username/.Trash
Restore file
hdfs dfs -mv /user/username/.Trash/Current/data/file.txt /data/
25. Advanced Commands (Admin Level)
Check safe mode
hdfs dfsadmin -safemode get
Enter safe mode
hdfs dfsadmin -safemode enter
Leave safe mode
hdfs dfsadmin -safemode leave
Report cluster info
hdfs dfsadmin -report
Shows:
- Live DataNodes
- Dead DataNodes
- Used space
- Free space
Balancer (balance data across nodes)
hdfs balancer
26. Practical Practice Task (Do This 🔥)
Try this sequence:
hdfs dfs -mkdir /practice echo "big data hadoop practice" > test.txt hdfs dfs -put test.txt /practice hdfs dfs -ls /practice hdfs dfs -cat /practice/test.txt hdfs dfs -setrep 2 /practice/test.txt hdfs fsck /practice/test.txt -blocks hdfs dfs -rm /practice/test.txt
If this works without error → your Hadoop is properly configured ✅
27 Difference: hdfs dfs vs hadoop fs
| Command | Meaning |
|---|---|
| hdfs dfs | HDFS specific |
| hadoop fs | Works for all Hadoop-supported file systems |
Example:
hadoop fs -ls /