Data Analysis
This comprehensive Data Analysis tutorial
1. Introduction to Data Analysis
Data Analysis is the process of collecting, organizing, cleaning, and interpreting data to find meaningful insights or support decision-making.
Types of Data:
- Quantitative: Numerical (e.g., age, income)
- Qualitative: Categorical (e.g., gender, city)
Types of Data Analysis:
| Type | Description | Example |
|---|---|---|
| Descriptive | What happened? | Average sales last month |
| Diagnostic | Why did it happen? | Drop in revenue due to holiday |
| Predictive | What will happen? | Forecast sales using ML |
| Prescriptive | What should we do? | Recommend stock to reorder |
2. Steps in the Data Analysis Process
- Ask a Question — What are we trying to find?
- Collect Data — From files, APIs, databases
- Clean Data — Fix missing, duplicate, or incorrect values
- Analyze Data — Use statistics and visualizations
- Interpret Results — Find insights or patterns
- Communicate Findings — Using reports, dashboards, etc.
3. Tools for Data Analysis
| Tool | Use |
|---|---|
| Python (pandas, matplotlib, seaborn) | Popular for scripting |
| R | Preferred in academia & statistics |
| Excel/Google Sheets | Basic analysis & dashboards |
| Power BI / Tableau | Visual dashboards |
| SQL | Querying databases |
We will use Python + Pandas in this tutorial.
4. Importing and Exploring Data
import pandas as pd # Load data df = pd.read_csv("https://raw.githubusercontent.com/mwaskom/seaborn-data/master/tips.csv") # Show top rows print(df.head()) # Shape of data print(df.shape) # Data types print(df.dtypes) # Summary stats print(df.describe())
5. Data Cleaning
Common Tasks:
- Handle missing values
- Fix data types
- Remove duplicates
# Check missing values print(df.isnull().sum()) # Drop missing values df = df.dropna() # Change data type df['total_bill'] = df['total_bill'].astype(float) # Remove duplicates df = df.drop_duplicates()
6. Data Wrangling / Transformation
Transforming, filtering, and creating new columns.
# Add new column df['tip_percentage'] = (df['tip'] / df['total_bill']) * 100 # Filter rows high_tips = df[df['tip'] > 5] # Grouping avg_tips = df.groupby('day')['tip'].mean() print(avg_tips)
7. Exploratory Data Analysis (EDA)
Identify patterns, trends, outliers.
# Distribution of total bills import seaborn as sns import matplotlib.pyplot as plt sns.histplot(df['total_bill'], bins=20) plt.title("Total Bill Distribution") plt.show() # Boxplot for outliers sns.boxplot(x='day', y='total_bill', data=df) plt.show() # Pair plot sns.pairplot(df) plt.show()
8. Data Visualization
Visualizations help in pattern discovery and reporting.
# Correlation heatmap sns.heatmap(df.corr(), annot=True) plt.show() # Bar plot sns.barplot(x='day', y='total_bill', data=df) plt.show() # Pie chart of gender df['sex'].value_counts().plot.pie(autopct='%1.1f%%') plt.show()
9. Statistical Analysis
Basics:
- Mean, median, mode
- Standard deviation
- Hypothesis testing
# Summary stats print("Mean Tip:", df['tip'].mean()) print("Std Dev of Tip:", df['tip'].std()) # Hypothesis test (t-test) from scipy.stats import ttest_ind male = df[df['sex'] == 'Male']['tip'] female = df[df['sex'] == 'Female']['tip'] t_stat, p_val = ttest_ind(male, female) print("P-Value:", p_val)
10. Correlation & Causation
# Check correlation print(df[['total_bill', 'tip', 'tip_percentage']].corr())
Note: Correlation ≠ Causation. Just because tip and total_bill are correlated doesn’t mean one causes the other.
11. Predictive Data Analysis (Intro to ML)
from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression # Predict tip from total_bill X = df[['total_bill']] y = df['tip'] # Split data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) # Train model model = LinearRegression() model.fit(X_train, y_train) # Predict predictions = model.predict(X_test) # Evaluate from sklearn.metrics import r2_score print("R2 Score:", r2_score(y_test, predictions))
12. Reporting & Communication
Use:
- Jupyter Notebooks for storytelling
- Dashboards (Tableau, Power BI)
- PDF/Reports with visualizations
- Presentations to stakeholders
13. Mini Project Example: Restaurant Tips
Question: Does gender and time (Lunch/Dinner) affect tipping behavior?
Steps:
- Import data tips.csv
- Clean data
- Add new column tip_percentage
- Visualize average tip % by gender and time
- Test hypothesis
- Present insights
# Visualize tip % by gender and time sns.barplot(x='sex', y='tip_percentage', hue='time', data=df) plt.title("Tip % by Gender and Time") plt.show()
Conclusion: You might find that females tip higher during dinner, etc.
14. Practice Datasets & Resources
Datasets:
Learn More:
- "Python for Data Analysis" by Wes McKinney
- Coursera: Data Analysis with Python
- freeCodeCamp YouTube Tutorials
Summary
| Topic | Key Tool |
|---|---|
| Importing & Cleaning | pandas |
| Wrangling | pandas |
| Visualization | seaborn, matplotlib |
| Statistics | scipy, pandas |
| Machine Learning | scikit-learn |
| Reporting | Jupyter, dashboards |