Data Analysis

This comprehensive Data Analysis tutorial

1. Introduction to Data Analysis

Data Analysis is the process of collecting, organizing, cleaning, and interpreting data to find meaningful insights or support decision-making.

Types of Data:

Types of Data Analysis:

TypeDescriptionExample
DescriptiveWhat happened?Average sales last month
DiagnosticWhy did it happen?Drop in revenue due to holiday
PredictiveWhat will happen?Forecast sales using ML
PrescriptiveWhat should we do?Recommend stock to reorder

2. Steps in the Data Analysis Process

  1. Ask a Question — What are we trying to find?
  2. Collect Data — From files, APIs, databases
  3. Clean Data — Fix missing, duplicate, or incorrect values
  4. Analyze Data — Use statistics and visualizations
  5. Interpret Results — Find insights or patterns
  6. Communicate Findings — Using reports, dashboards, etc.

3. Tools for Data Analysis

ToolUse
Python (pandas, matplotlib, seaborn)Popular for scripting
RPreferred in academia & statistics
Excel/Google SheetsBasic analysis & dashboards
Power BI / TableauVisual dashboards
SQLQuerying databases

We will use Python + Pandas in this tutorial.

4. Importing and Exploring Data

import pandas as pd # Load data df = pd.read_csv("https://raw.githubusercontent.com/mwaskom/seaborn-data/master/tips.csv") # Show top rows print(df.head()) # Shape of data print(df.shape) # Data types print(df.dtypes) # Summary stats print(df.describe())

5. Data Cleaning

Common Tasks:

# Check missing values print(df.isnull().sum()) # Drop missing values df = df.dropna() # Change data type df['total_bill'] = df['total_bill'].astype(float) # Remove duplicates df = df.drop_duplicates()

6. Data Wrangling / Transformation

Transforming, filtering, and creating new columns.

# Add new column df['tip_percentage'] = (df['tip'] / df['total_bill']) * 100 # Filter rows high_tips = df[df['tip'] > 5] # Grouping avg_tips = df.groupby('day')['tip'].mean() print(avg_tips)

7. Exploratory Data Analysis (EDA)

Identify patterns, trends, outliers.

# Distribution of total bills import seaborn as sns import matplotlib.pyplot as plt sns.histplot(df['total_bill'], bins=20) plt.title("Total Bill Distribution") plt.show() # Boxplot for outliers sns.boxplot(x='day', y='total_bill', data=df) plt.show() # Pair plot sns.pairplot(df) plt.show()

8. Data Visualization

Visualizations help in pattern discovery and reporting.

# Correlation heatmap sns.heatmap(df.corr(), annot=True) plt.show() # Bar plot sns.barplot(x='day', y='total_bill', data=df) plt.show() # Pie chart of gender df['sex'].value_counts().plot.pie(autopct='%1.1f%%') plt.show()

9. Statistical Analysis

Basics:

# Summary stats print("Mean Tip:", df['tip'].mean()) print("Std Dev of Tip:", df['tip'].std()) # Hypothesis test (t-test) from scipy.stats import ttest_ind male = df[df['sex'] == 'Male']['tip'] female = df[df['sex'] == 'Female']['tip'] t_stat, p_val = ttest_ind(male, female) print("P-Value:", p_val)

10. Correlation & Causation

# Check correlation print(df[['total_bill', 'tip', 'tip_percentage']].corr())

Note: Correlation ≠ Causation. Just because tip and total_bill are correlated doesn’t mean one causes the other.

11. Predictive Data Analysis (Intro to ML)

from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression # Predict tip from total_bill X = df[['total_bill']] y = df['tip'] # Split data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) # Train model model = LinearRegression() model.fit(X_train, y_train) # Predict predictions = model.predict(X_test) # Evaluate from sklearn.metrics import r2_score print("R2 Score:", r2_score(y_test, predictions))

12. Reporting & Communication

Use:

13. Mini Project Example: Restaurant Tips

Question: Does gender and time (Lunch/Dinner) affect tipping behavior?

Steps:

  1. Import data tips.csv
  2. Clean data
  3. Add new column tip_percentage
  4. Visualize average tip % by gender and time
  5. Test hypothesis
  6. Present insights
# Visualize tip % by gender and time sns.barplot(x='sex', y='tip_percentage', hue='time', data=df) plt.title("Tip % by Gender and Time") plt.show()

Conclusion: You might find that females tip higher during dinner, etc.

14. Practice Datasets & Resources

Datasets:

Learn More:

Summary

TopicKey Tool
Importing & Cleaningpandas
Wranglingpandas
Visualizationseaborn, matplotlib
Statisticsscipy, pandas
Machine Learningscikit-learn
ReportingJupyter, dashboards