YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
- ๐ Uber & Lyft Boston Rideshare Data Analysis
- Boston Ride Fare Predictor ๐
- ๐ฎ Live Interactive Demo
- ๐ Dataset Overview
- ๐ Features & Variables
- โ The Research Question
- ๐ ๏ธ Predictive Features & Target Analysis
- ๐ Part 2: Exploratory Data Analysis (EDA)
- Price vs. Distance by Service Tier
- Part 3: Price Prediction Modeling
- ๐Part 4: Model Development: Linear Regression Baseline
- ๐ Model Evaluation: Baseline Performance
- ๐ Model Diagnostic Visualizations
- โ๏ธ Feature Importance: Coefficient Analysis
- ๐ ๏ธ Advanced Feature Engineering & Scaling
- ๐ Engineering Validation & Visual Analysis
- โ๏ธ Environmental Impact Analysis: Rain & Temporal Factors
- ๐งฌ Advanced Segmentation & Dimensionality Reduction (PCA)
- ๐Part 5: Model Optimization: Enhanced Linear Regression
- ๐ฒ Advanced Modeling: Ensemble Methods
- ๐ Feature Importance: The Impact of Engineering
- ๐Part 6: The Definitive Winner: Random Forest Regressor
- ๐ท๏ธPart 7: Price Categorization: Quantile Binning
- โ๏ธ Class Balance Verification
- ๐ Training Set Distribution Summary
- Part 8: Multi-Class Classification Modeling
- ๐ Model Diagnostics: Confusion Matrix & Classification Reports
- ๐ฌ Advanced Model Interpretability & Performance Validation
- ๐ The Definitive Winner: Random Forest Classifier
- ๐ Brand Pricing Dynamics: Uber vs. Lyft
- ๐ Explainable AI: SHAP Feature Importance
- โ๏ธ Model Optimization: Hyperparameter Tuning
- ๐ Model Deployment: Boston Ride Fare Predictor
- ๐ฎ Live Interactive Demo
๐ Uber & Lyft Boston Rideshare Data Analysis
Boston Ride Fare Predictor ๐
An advanced AI system for predicting ride-sharing fares and classifying them into service tiers (Budget, Standard, or Premium). The model is based on a Random Forest Regressor trained on Boston's Uber and Lyft datasets.
๐ฎ Live Interactive Demo
You can test the model in real-time using the interface below:
- Live App: Boston Fare Predictor Space
๐ Dataset Overview
Background
This project explores the dynamic landscape of rideshare services in Boston, MA. The dataset provides a unique opportunity to analyze the interaction between urban mobility and environmental factors by integrating real-world rideshare transactions with high-resolution local weather data.
Data Source
The dataset is sourced from Kaggle ("Uber and Lyft Dataset Boston, MA"). It encompasses a period of rideshare activity in Boston, meticulously merged with weather metrics corresponding to the exact time and location of each trip.
Dataset Scale
- Total Observations: ~693,000 rows.
- Analysis Sample: For computational efficiency and balanced processing, a representative sample of ~50,000 rows is utilized in this project.
- Total Features: 57 initial variables, providing a comprehensive view of each journey.
๐ Features & Variables
The dataset offers a rich mix of data types, making it ideal for advanced feature engineering and predictive modeling:
๐ข Numerical Features
- Trip Metrics:
distance(miles),surge_multiplier. - Weather Metrics:
temp(Fahrenheit),humidity,precipIntensity,windSpeed.
๐๏ธ Categorical Features
- Service Providers:
cab_type(Uber vs. Lyft). - Service Levels:
name(e.g., Black, XL, Shared, Lux, UberPool). - Geography:
sourceanddestination(Specific Boston neighborhoods). - Environment:
short_summary(Weather descriptions like "Cloudy", "Rain", or "Clear").
๐ฏ Target Variable (Label)
price: A continuous numerical value representing the cost of the ride.
โ The Research Question
"To what extent can we accurately predict rideshare prices in Boston based on trip characteristics, service levels, and real-time weather conditions?"
๐ ๏ธ Predictive Features & Target Analysis
The following table summarizes the key variables and the central target variable used in the models:
| Category | Specific Features | Role & Description |
|---|---|---|
| Target Variable | price |
The Core Objective. Represents the transaction cost of the ride in USD ($). This is the continuous value we aim to predict. |
| Service Level | name |
Highest Impact. The primary differentiator for price (e.g., Shared vs. Lux/Black). |
| Trip Metrics | distance |
Primary Baseline. Maintains a strong linear correlation with the final price. |
| Dynamic Pricing | surge_multiplier |
Demand Factor. Captures real-time price increases during peak demand or extreme weather. |
| Environment | temp, precipIntensity |
External Context. Accounts for weather conditions that influence rideshare demand and pricing. |
| Geography | source, destination |
Localized Trends. Enables the model to learn pricing behaviors specific to Boston neighborhoods. |
๐ Part 2: Exploratory Data Analysis (EDA)
Before diving into deep analysis, my first priority was ensuring the dataset's integrity. This phase focused on filtering noise and verifying that my "Bright Orange" visualizations represent a clean, reliable data source.
Key Actions I Took:
- Dimensionality Reduction: I pruned the dataset from 57 initial features down to a streamlined set of high-impact variables, removing redundant IDs and timestamps.
- Missing Value Handling: I successfully addressed null values in the
pricecolumn. The heatmap below confirms a 100% clean dataset ready for processing. - Outlier Detection: By plotting Price vs. Distance, I identified and neutralized anomaliesโsuch as ultra-short trips with unexplained high costsโensuring my models learn from realistic transit patterns.
๐ง๏ธ The "Weather Factor" โ Impact of Rain on Pricing
One of the most interesting parts of this research was investigating whether adverse weather in Boston directly translates to higher fares for passengers. I analyzed the relationship between precipitation and pricing dynamics.
- Direct Impact vs. Price: I used a Regression Plot (Regplot) to examine if higher rain intensity (
precipIntensity) leads to an immediate increase in trip costs.- Finding: While there is a slight positive trend, rain alone isn't the primary driver of the base fare.
โฐ Hourly Trends
In this section, I analyzed how the time of day affects rideshare pricing in Boston, focusing on whether certain hours command higher fares on average.
- Hourly Price Stability: I generated a time-series line plot to track the average trip price across a 24-hour cycle.
- Insight: My analysis shows that the average price remains remarkably stable throughout the day, suggesting that the time of day alone (without considering specific surge events) does not cause massive fluctuations in the base fare.
๐ก๏ธ Feature Correlation Analysis
In this critical stage of the EDA, I conducted a deep statistical dive into the mathematical relationships between all featuresโincluding the encoded service typesโand the final trip price.
- Comprehensive Mapping: I generated an expanded Correlation Heatmap to visualize how each individual variable (from trip distance to specific vehicle tiers) influences the price.
- Key Insights: * The "Price Drivers": I identified that specific luxury service levels (like
Black SUVandLux Black XL) and the tripdistancehold the highest positive correlation with the target variable.- Feature Pruning: By examining the inter-correlations between variables, I was able to identify and remove redundant features (multi-collinearity), ensuring a more efficient and stable machine learning model.
- Strategic Outcome: This map confirmed that while location and weather play a role, the Service Tier and Distance are the dominant mathematical anchors for pricing in the Boston market.
Brand Comparison
In this phase, I conducted a direct comparative analysis between Uber and Lyft to determine if one brand consistently outprices the other.
- Brand Parity: I utilized a Density Plot to compare the price distributions of Uber and Lyft.
- Insight: I discovered a near-perfect overlap between the two providers, indicating a highly competitive market where brand choice has a minimal impact on the base fare.
Price vs. Distance by Service Tier
This section analyzes the correlation between travel distance and fare pricing across various service categories from Uber and Lyft.
Key Insights
- Linear Correlation: There is a clear positive linear relationship between distance and price across all tiers.
- Price Premium: The steepness and starting point (intercept) of the regression lines visually demonstrate the price gap between economy services (like UberX) and luxury tiers (like Lux Black).
- Consistency: The relatively narrow confidence intervals (shaded areas) suggest consistent pricing algorithms based on distance within each specific service tier.
Part 3: Price Prediction Modeling
Project Objective
The goal of this phase is to predict rideshare trip prices in Boston as a continuous numerical value. Primary Metric: Our objective is to minimize prediction errors, specifically focusing on RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error) to ensure the most reliable price estimations.
Feature Selection & Engineering
Based on the Exploratory Data Analysis (EDA), I identified the following features as the most significant predictors for price:
| Category | Features |
|---|---|
| Trip Characteristics | distance, surge_multiplier |
| Service Levels | name, cab_type |
| Location | source, destination |
| Temporal Factors | hour |
| Environmental Factors | temp, precipIntensity |
Justification for Feature Selection
To optimize model performance and maintain interpretability, I performed a strategic reduction of the dataset:
- Multicollinearity Mitigation: Redundant weather variables and technical timestamps were excluded.
- Heatmap Insights: Correlation analysis revealed high multicollinearity among several environmental factors; removing these reduced noise and prevented overfitting.
- Dimensionality Reduction: By keeping only high-impact features, we ensure the model remains efficient without sacrificing predictive power.
Data Preprocessing & Validation
Before training the predictive model, the dataset underwent a rigorous preprocessing pipeline to ensure data integrity and optimal model performance.
Categorical Encoding**
Applied One-Hot Encoding (via pd.get_dummies) to transform categorical variables such as name, source, and destination into numerical formats, enabling the regression algorithms to process categorical data effectively.
Data Splitting & Reproducibility**
- Train-Test Split: Partitioned the dataset into Training (80%) and Testing (20%) sets.
- Random Seed (Reproducibility): I set a fixed
random_state=42to ensure that the data split remains identical every time the code is executed. This ensures that any improvements in model performance are due to tuning and not random variations in the data distribution.
Distribution Validation**
I conducted a comparative analysis of the target variable (price) across both sets:
- Sample Consistency: The split maintained a high volume of data, providing over 500k samples for robust training.
- Distribution Alignment: The KDE (Kernel Density Estimate) plot confirms that the price distribution in the test set mirrors the training set, ensuring a fair and consistent evaluation environment.
๐Part 4: Model Development: Linear Regression Baseline
Baseline Establishment
I initiated the modeling phase by implementing a Linear Regression model using the scikit-learn framework. The primary objective of this step is to establish a Baseline Performance Benchmark.
By starting with a simpler, interpretable model, I can evaluate the inherent complexity of the prediction task and set a standard for comparison. Linear Regression serves as an ideal baseline as it efficiently models the direct relationships between our features (e.g., distance, service tier) and the target variable (price).
Training & Performance
- Approach: The model was trained using default parameters to capture the fundamental linear trends within the dataset.
- Results: The baseline achieved a Training $R^2$ Score of 0.9345.
- Interpretation: This indicates that approximately 93.45% of the variance in ride prices can be explained by the selected features, confirming a strong linear signal in the data.
This benchmark will be used to measure the "value-add" of more sophisticated algorithms (such as Ensemble methods) in subsequent stages of the project.
๐ Model Evaluation: Baseline Performance
After training, the Linear Regression model was evaluated on the unseen test set (20% of the data) to measure its real-world predictive accuracy.
Performance Metrics
| Metric | Value | Interpretation |
|---|---|---|
| MAE (Mean Absolute Error) | $1.65 | On average, the model's price prediction deviates by only $1.65. |
| RMSE (Root Mean Squared Error) | $2.24 | Indicates a low frequency of large prediction errors. |
| R-squared ($R^2$) | 0.9345 | 93.45% of the price variance is accurately captured by the model. |
Analysis of Results
The evaluation reveals a high level of precision for a baseline model. Notably, the $R^2$ score on the test set is identical to the training score, demonstrating that the model generalizes exceptionally well and is not suffering from overfitting. These results establish a very strong benchmark for any future, more complex algorithms.
๐ Model Diagnostic Visualizations
To further validate the baseline's performance, I implemented diagnostic plots to visualize the relationship between predicted values and actual market prices.
Actual vs. Predicted Prices**
This scatter plot compares the model's estimations against the ground truth.
- Interpretation: The high density of data points along the 45-degree reference line (dashed red) indicates a very high predictive accuracy. The model consistently captures price trends across the entire range of trip values.
Residuals Distribution (Error Analysis)**
Analysis of the residuals (Errors = Actual - Predicted) is critical for assessing model bias.
- Normal Distribution: The error distribution follows a near-perfect normal curve centered at zero.
- Insight: This symmetry confirms that the modelโs predictions are unbiased, meaning it doesn't systematically overprice or underprice rides. The lack of significant skewness suggests that the linear functional form is well-suited for this dataset.
โ๏ธ Feature Importance: Coefficient Analysis
To understand the underlying logic of the pricing model, I extracted and visualized the Linear Regression Coefficients. This analysis reveals how much each feature contributes to the final fare estimation.
Key Insights from the Model
- Primary Value Drivers: The
surge_multiplierand premium service tiers likeLux Black XLhave the strongest positive impact on price. This aligns with rideshare pricing strategies where vehicle class and real-time demand significantly inflate costs. - Discount Factors: Economy and shared services (e.g.,
UberPool,Shared,UberX) show strong negative coefficients. These represent the price reduction users receive when opting for high-capacity or standard vehicle types compared to the baseline luxury options. - Distance Impact: The
distancefeature maintains a consistent positive coefficient, quantifying the marginal cost increase for every additional mile traveled. - Geographic Influence: Location-based features (e.g.,
source_North End) show a minimal but measurable impact, suggesting that while the starting point matters, it is secondary to the chosen service tier and distance.
Conclusion
The model effectively prioritizes Service Level and Demand (Surge) as the dominant predictors of price, proving that the user's choice of "how to ride" is often more influential than "where they go."
๐ ๏ธ Advanced Feature Engineering & Scaling
To capture complex, non-linear patterns in the data and enhance model accuracy, I implemented several advanced feature engineering techniques:
Interaction Features**
- Distance-Surge Synergy: Created a
distance_surgeinteraction term. This variable accounts for the compounded effect of surge pricing over long distances, providing a more nuanced predictor than evaluating each feature in isolation.
Unsupervised Clustering (K-Means)**
- Deal Categorization: Implemented a K-Means Clustering algorithm ($k=4$) based on the relationship between distance and price.
- Insight: This allows the model to categorize rides into distinct "transaction profiles" (e.g., premium short trips vs. standard long-haul trips), adding a high-level categorical feature that captures latent market segments.
Feature Scaling**
- Standardization: Utilized
StandardScalerto transform numerical features (distance,temperature,distance_surge). By ensuring all features have a mean of 0 and a standard deviation of 1, I mitigated potential bias toward features with larger numerical magnitudes, ensuring stable and balanced model training.
๐ Engineering Validation & Visual Analysis
After generating the new features, I performed a visual validation to ensure they provide meaningful signals for the predictive model.
Cluster Discovery (Unsupervised Learning)**
The K-Means Clustering visualization (left) illustrates the distinct "Trip Profiles" identified by the algorithm. By grouping rides based on price-distance coordinates, we successfully segmented the data into four density tiers (represented by the Viridis color scale). This adds a high-level context that helps the model differentiate between economy-short trips and luxury-long hauls.
Interaction Power**
The Interaction Plot (center) validates the logic behind the distance_surge feature. The strong positive slope of the red regression line confirms that the interaction between distance and the surge multiplier is a potent predictor of fare price, capturing price escalations that individual features might miss.
Numerical Uniformity (Scaling)**
The Boxplot of Scaled Features (right) demonstrates the result of our standardization process.
- Outcome: All primary numerical predictors now share a consistent mean and variance.
- Benefit: This prevents "Feature Dominance," ensuring that the regression weights are strictly determined by the feature's predictive importance rather than its original unit scale.
โ๏ธ Environmental Impact Analysis: Rain & Temporal Factors
In this section, I analyzed how external environmental and temporal factors influence pricing dynamics in the Boston rideshare market.
Weather Impact (Rain vs. Clear Skies)**
- Visualization: Boxplot analysis of price distribution across weather conditions.
- Findings: Surprisingly, the presence of rain shows a negligible impact on the median fare. The price distribution remains remarkably stable, suggesting that weather conditions alone are not a primary driver of price volatility in this dataset.
Temporal Factors (Peak vs. Off-Peak)**
- Visualization: Comparative bar chart of average fares during peak and off-peak hours.
- Findings: The data reveals that average prices during peak hours are nearly identical to off-peak periods. While demand may increase, the average fare remains consistent, indicating that surge pricing (captured by our
surge_multiplier) might be localized to specific high-intensity events rather than broad time windows.
Strategic Insight
These visualizations justify the model's reliance on Service Tiers and Distance as the dominant predictors. By debunking the assumption that rain or hour-of-day universally inflate prices, we can focus our predictive power on the features that truly drive market variance.
๐งฌ Advanced Segmentation & Dimensionality Reduction (PCA)
To gain deeper insights into market structures, I performed a multi-stage cluster analysis combining K-Means Clustering with Principal Component Analysis (PCA).
1. Cluster Interpretation & Mapping
By analyzing the centroids of the generated clusters, I identified and labeled four distinct business segments within the Boston rideshare market:
- Standard Short: High-frequency, baseline trips.
- Long Distance: Trips characterized by higher mileage and consistent pricing.
- Premium/Luxury: High-end vehicle tiers regardless of distance.
- Surge/High Demand: Trips significantly impacted by real-time price multipliers.
2. PCA Projection
Since the dataset contains numerous high-dimensional features (encoded locations, service tiers, weather, etc.), I applied PCA to project the data into a 2D space.
- Validation: The resulting visualization (left) demonstrates clear, non-overlapping boundaries between the identified segments.
- Significance: The separation in the PCA plot validates that our clusters are mathematically distinct, providing the predictive model with a powerful engineered feature that encapsulates complex multi-dimensional relationships into a single categorical input.
๐Part 5: Model Optimization: Enhanced Linear Regression
The Leap from Baseline to Optimized Model
In this final modeling phase, I retrained the Linear Regression algorithm using the enriched dataset. By integrating the advanced features engineered in the previous steps, the model transitioned from processing raw inputs to understanding complex market dynamics.
Integrated Enhancements
The optimized model now accounts for:
- Synergistic Effects:
distance_surgeinteraction. - Market Segmentation:
cluster_idlabels anddist_to_centroid. - External Context: Binary indicators for weather (
is_raining) and temporal peaks (is_peak_hour).
Performance Comparison
| Metric | Baseline Model | Optimized Model | Improvement |
|---|---|---|---|
| R-squared ($R^2$) | 0.9345 | 0.9690 | +3.45% |
| MAE | $1.65 | $1.15 | -$0.50 |
๐ฒ Advanced Modeling: Ensemble Methods
To push the boundaries of predictive accuracy, I transitioned from linear modeling to advanced Ensemble Learning algorithms. These models are designed to capture complex, non-linear interactions within the features that a standard regression might overlook.
Implementation Strategy
- Efficient Sampling: To optimize computational performance while maintaining statistical integrity, I utilized a representative sample of 100,000 observations for this phase.
- Algorithmic Diversity: I implemented and compared two industry-standard models:
- Random Forest Regressor: An ensemble of 100 decision trees utilizing bagging to reduce variance.
- Gradient Boosting Regressor: A boosting technique that builds trees sequentially to minimize residual errors.
Performance Breakthrough
The results represent the highest accuracy achieved in this project:
| Model | Mean Absolute Error (MAE) | R-squared ($R^2$) |
|---|---|---|
| Linear Regression (Optimized) | $1.15 | 0.9690 |
| Random Forest | $0.83 | 0.9798 |
Key Takeaway
The Random Forest model emerged as the superior predictor, explaining nearly 98% of the price variance. By effectively leveraging the engineered features (clusters and interactions), the model reduced the average prediction error to just $0.83, providing a highly precise solution for rideshare price estimation.
The core objective of this project was to transition from a general estimation to a high-precision predictive model. The final evaluation demonstrates a dramatic performance breakthrough, achieving an approximately 50% reduction in Mean Absolute Error (MAE).
Performance Comparison
| Model Tier | Complexity Level | Mean Absolute Error (MAE) |
|---|---|---|
| Baseline (Linear) | Simple Linear Relationship | $1.65 |
| Improved (Random Forest) | Advanced Ensemble Learning | $0.83 |
๐ Baseline Comparison Analysis
The significant leap in accuracy is attributed to two primary strategic factors:
1. Model Complexity & Interaction Capture
- The Limitation: The Baseline model assumed a simple linear relationship between features and price.
- The Reality: Rideshare pricing is driven by complex, non-linear interactions. For instance, environmental factors like rain impact premium "Lux" tiers differently than "Shared" economy tiers.
- The Solution: While Linear Regression struggled to capture these subtle nuances, the Random Forest model thrived by effectively mapping these multi-dimensional dependencies.
2. Strategic Feature Engineering
- Contextual Data: The introduction of engineered features (such as Trip Clusters and Distance-Surge interactions) provided the model with "pre-digested" insights.
- Breaking the Ceiling: The original Baseline model lacked this contextual depth, which created an inherent predictive ceiling. By incorporating these engineered features, we removed those limitations and enabled the final model to reach a new level of predictive precision.
Conclusion: By combining non-linear ensemble modeling with domain-specific feature engineering, the final model provides highly reliable fare estimations with an average error of less than one dollar ($0.83), making it a robust tool for real-world application.
๐ Feature Importance: The Impact of Engineering
To finalize the model analysis, I extracted the Gini Importance scores to identify which features were most influential in the Random Forestโs decision-making process.
Key Discovery: Engineering Over Raw Data
The visualization reveals a critical insight: The top 3 most influential features are the Trip Clusters created during the Feature Engineering phase.
- Strategic Success: The fact that
trip_cluster_2,trip_cluster_1, andtrip_cluster_3dominate the importance ranking proves that segmenting the market into distinct behavioral profiles provided more predictive power than any single raw input. - Compound Effects: The engineered interaction
distance_surgealso ranks higher than standarddistance, confirming that the relationship between demand and mileage is a more potent predictor than mileage alone. - Operational Tiers: Service tiers (e.g.,
name_Shared,name_UberXL) follow as secondary predictors, showing that once the model understands the trip's "cluster context," it uses the specific service level to fine-tune the final price.
Conclusion: This ranking validates the project's core hypothesis: sophisticated feature engineering, rather than just choosing a complex algorithm, is what truly "breaks the predictive ceiling" and drives model excellence.
๐Part 6: The Definitive Winner: Random Forest Regressor
After evaluating multiple models, the Random Forest Regressor emerged as the superior solution for predicting rideshare prices in Boston. It achieved an exceptional balance of precision, stability, and computational efficiency.
Model Performance Metrics
- Accuracy: Reached a peak $R^2$ score of 0.9798, effectively explaining nearly 98% of the price variance.
- Error Rate: Achieved the lowest MAE of only $0.83$, cutting the initial baseline error by 50%.
- Stability: Compared to Gradient Boosting, Random Forest provided a more robust balance between training time and predictive consistency, making it the most reliable choice for real-world deployment.
Key Drivers of Success
The victory of the Random Forest model is attributed to three core pillars:
- Feature Engineering Excellence: Utilizing the engineered
distance_surgeinteraction andtrip_clusterprovided the model with "pre-digested" insights, allowing it to capture complex demand-price dynamics. - Handling Non-Linearity: Unlike Linear Regression, which fits a straight line, Random Forest successfully partitioned data based on complex conditions (e.g., how rain impact varies significantly across different service tiers).
- Optimized Processing: Systematic scaling and precise One-Hot Encoding allowed the model to process all variablesโlocation, weather, and vehicle typeโoptimally without distortion from outliers.
Final Verdict: By leveraging ensemble-based complexity and high-quality engineered features, this model delivers a sub-dollar accuracy rate, setting a new standard for fare estimation in the Boston rideshare ecosystem.
๐ท๏ธPart 7: Price Categorization: Quantile Binning
To expand the analysis beyond continuous regression, I transformed the target variable into discrete categories. This allows for a Classification approach, focusing on identifying trip price segments rather than exact dollar values.
Binning Strategy: Quantile-Based Discrete Classes
I implemented a Quantile Binning strategy to split the price data into three balanced classes: Low (0), Medium (1), and High (2).
- Methodology: * Used
pd.qcuton the training set to ensure an equal distribution of samples across categories.- Calculated the price boundaries: $2.5 โ $10.5 (Low), $10.5 โ $19.0 (Medium), and $19.0 โ $42.5 (High).
- Applied these identical boundaries to the test set using
pd.cutto maintain strict evaluation standards and prevent data leakage.
- Validation: The class distribution printout confirms that the dataset remains well-balanced across all three tiers, providing a solid foundation for training a Classification model.
โ๏ธ Class Balance Verification
Before proceeding to train the classification model, it is essential to ensure that the newly created target categories are well-balanced. An imbalanced dataset can lead to model bias, where the algorithm favors the majority class.
Methodology & Distribution Analysis
I generated a summary table and a corresponding visualization to inspect the distribution of the three price tiers within the training set:
- Classes: 0 (Budget), 1 (Standard), and 2 (Premium).
- Metrics: Calculated both absolute counts and normalized percentages to verify structural integrity.
- Visualization: The Class Balance Check bar plot confirms a nearly uniform distribution across all tiers.
Significance
Having a balanced training set ensures that the classifier will develop equal predictive sensitivity for all price segments. This prevents the model from becoming biased toward any specific tier and ensures that evaluation metrics (like Accuracy and F1-Score) remain reliable indicators of real-world performance.
๐ Training Set Distribution Summary
To ensure a high-quality classification model, I generated a detailed distribution table for the newly defined price tiers within the training dataset.
Data Breakdown
The following table illustrates the balance between the three price segments:
| Price Tier | Count | Percentage (%) | Label |
|---|---|---|---|
| 0 | 30,120 | 37.65% | Budget |
| 1 | 23,332 | 29.17% | Standard |
| 2 | 26,548 | 33.19% | Premium |
Analysis of Balance
- Consistency: The data is distributed fairly evenly across all three tiers, with the Budget class (0) being the largest and the Standard class (1) the smallest.
- Model Readiness: With each class representing roughly one-third of the total data, the dataset is well-conditioned for multi-class classification. This balanced distribution minimizes the risk of the model developing a majority-class bias, ensuring that the resulting accuracy metrics will be representative of true predictive performance.
Part 8: Multi-Class Classification Modeling
In this phase, I transitioned from price estimation (regression) to price tier prediction (classification). The objective is to identify the most accurate algorithm for categorizing rides into the predefined Budget, Standard, and Premium tiers.
Model Selection & Training Strategy
I implemented a competitive evaluation pipeline using three distinct classification algorithms:
- Logistic Regression: Serving as the statistical baseline for classification.
- Random Forest Classifier: An ensemble method designed to capture complex non-linear patterns.
- K-Nearest Neighbors (KNN): A distance-based algorithm that classifies data points based on their spatial proximity to neighbors ($k=5$).
Execution & Initial Performance
Using an automated training loop, each model was fitted on the categorized training set and evaluated using the Accuracy Score metric on the unseen test set.
Initial Accuracy Results:
- Logistic Regression: 90.67%
- Random Forest: 92.28%
- K-Nearest Neighbors: ~91.2% (estimated)
Key Insight: All models demonstrated high predictive reliability, consistently exceeding 90% accuracy. The Random Forest Classifier emerged as the early leader, further confirming the model's suitability for the intricate pricing structures of the Boston rideshare market.
๐ Model Diagnostics: Confusion Matrix & Classification Reports
To gain a granular understanding of the classifier's performance, I implemented a diagnostic pipeline using Confusion Matrices and detailed Classification Reports for each model.
Diagnostic Tools
- Classification Report: Provided per-class metrics, including Precision, Recall, and F1-Score. This ensures that the model performs consistently across all price tiers (Budget, Standard, and Premium) and doesn't just excel in one at the expense of others.
- Confusion Matrix Visualization: Utilizing
ConfusionMatrixDisplay, I mapped the predicted vs. actual labels to visualize where the models were most successful and where they faced "confusion."
Key Evaluation Insights
- Strong Diagonal Alignment: All models exhibit a dominant diagonal line, indicating high predictive accuracy for each specific tier.
- Adjacent Error Logic: Most misclassifications occur between adjacent categories (e.g., Standard being classified as Budget). Notably, there are near-zero errors between extreme tiers (Budget vs. Premium), confirming that the models have successfully captured the fundamental price boundaries of the market.
- Random Forest Superiority: The Confusion Matrix for the Random Forest model displays the highest "True Positive" counts, particularly within the Premium segment, solidifying its position as the most robust classifier for this project.
๐ฌ Advanced Model Interpretability & Performance Validation
To ensure the highest level of transparency and diagnostic rigor, I implemented two industry-standard evaluation techniques: ROC/AUC Analysis and SHAP Explanations.
ROC Curve Comparison (Class Separation)**
I evaluated the models' ability to distinguish the Premium tier from other categories using the Receiver Operating Characteristic (ROC) curve.
- AUC Score: This metric quantifies the "Separation Power" of each algorithm. A higher Area Under the Curve (AUC) indicates that the model is exceptionally skilled at correctly identifying luxury rides while minimizing false positives.
- Observation: The models demonstrate high sensitivity and specificity, with curves hugging the top-left corner, confirming robust classification thresholds.
SHAP Value Analysis (Model Interpretability)**
To move beyond "Black Box" modeling, I utilized SHAP (SHapley Additive exPlanations) to visualize the global impact of each feature on the model's output.
- Feature Push/Pull: The SHAP summary plot reveals how specific values (e.g., high surge or specific trip clusters) "push" the predicted price higher or lower.
- Validation of Engineering: The analysis clearly shows that our engineered featuresโspecifically the Trip Clustersโcarry the most significant predictive weight. This visual evidence confirms that our feature engineering phase was the primary driver of the model's high performance.
๐ The Definitive Winner: Random Forest Classifier
After a rigorous evaluation involving Accuracy metrics, Confusion Matrices, and Interpretability tools, the Random Forest Classifier has been selected as the optimal model for this project.
Why Random Forest Won:
Superior Class Separation (AUC/ROC): The model demonstrated the highest AUC score, proving its exceptional ability to distinguish between Budget, Standard, and Premium tiers with minimal false positives. Its decision threshold is the most robust among all tested candidates.
Logical Interpretability (SHAP): Through SHAP analysis, we validated that the model's predictions are grounded in meaningful data relationships. It successfully prioritized our engineered featuresโspecifically the Trip Clusters and Distance-Surge interactionsโproving that it "understands" the economic drivers of the Boston rideshare market.
High-Precision Reliability: With an Accuracy score exceeding 92%, the Random Forest model provides the most stable performance across all segments. Its confusion matrix showed nearly zero errors between extreme classes (Budget vs. Premium), ensuring that high-stakes pricing decisions remain accurate.
Final Verdict
The Random Forest Classifier isn't just a "Black Box"; it is a highly calibrated engine that leverages complex feature interactions to provide actionable, sub-dollar-precise classifications. It stands as the most reliable model for real-world dynamic pricing applications.
๐ Brand Pricing Dynamics: Uber vs. Lyft
To further validate the economic relevance of the price tiers, I performed a comparative analysis between the two primary market players in the dataset.
Cross-Tabulation Analysis
Using pd.crosstab with index normalization, I mapped the percentage distribution of each price tierโ**Budget (0), Standard (1), and Premium (2)**โacross Uber and Lyft. This approach reveals the pricing strategy and market positioning of each brand.
Key Visual Insights
The Stacked Bar Chart (above) illustrates the pricing composition of each fleet:
- Market Presence: Both brands maintain a presence across all three tiers, confirming that the classification system captures broad market segments rather than brand-specific anomalies.
- Pricing Nuances: While the distributions are relatively similar, the data suggests that Uber holds a slightly higher proportion of "Budget" tier rides within this sample, whereas Lyft shows a marginally higher density in the "Premium" segment.
- Model Validation: This consistency ensures that the subsequent classification models will learn to differentiate prices based on trip characteristics (distance, surge, vehicle type) rather than relying solely on brand identity.
๐ Explainable AI: SHAP Feature Importance
To ensure the classification model's transparency and understand the drivers behind its predictions, I implemented SHAP (SHapley Additive exPlanations) analysis. This method quantifies the contribution of each feature to the final classification across all three price tiers.
Methodology & Computational Efficiency
- Targeted Sampling: I extracted a representative sample of 100 random observations from the test set. This allowed for an efficient calculation of SHAP values without compromising the interpretability of the results.
- Tree-Based Interpretation: Utilizing
shap.TreeExplaineroptimized for Random Forest, I calculated the impact of each variable on the model's output magnitude.
Visualizing Multi-Class Influence
The SHAP Summary Bar Plot provides a clear breakdown of feature importance, categorized by class (Budget, Standard, and Premium):
- Cluster Dominance: Engineered features, specifically the Trip Clusters, stand out as the most influential variables. Their impact is consistent across all classes, proving they are the primary "anchors" for the model's pricing logic.
- Granular Impact: We can observe how features like
name_Sharedordistance_surgeinfluence specific classes more than others, providing a nuanced view of what drives a trip to be classified as "Budget" versus "Premium."
Conclusion: This analysis validates that the model relies on logically sound economic indicators, with engineered features providing the most significant predictive signal.
โ๏ธ Model Optimization: Hyperparameter Tuning
To squeeze every bit of predictive power out of the Random Forest Classifier, I moved beyond default settings and performed a systematic Hyperparameter Optimization phase.
Strategy: Randomized Search Cross-Validation
I utilized RandomizedSearchCV to explore the parameter space efficiently. This approach balances computational cost with high-quality results by sampling a fixed number of parameter combinations.
- Parameters Explored:
n_estimators: The number of decision trees in the forest.max_depth: The maximum depth of each tree (controlling for potential over-fitting).min_samples_split: The minimum number of samples required to split an internal node.
- Validation: Implemented a 3-fold Cross-Validation (CV=3) to ensure that the chosen parameters generalize well to unseen data and are not just "memorizing" the training set.
Optimal Configuration Found
The search identified the following "Best Parameters" for the final model:
- n_estimators: 200
- max_depth: 20
- min_samples_split: 5
Outcome: By applying these optimized settings, the model is now fine-tuned to the specific nuances of the Boston rideshare market, ensuring higher stability and more precise classification across all price tiers.
๐ Model Deployment: Boston Ride Fare Predictor
The final stage of this project involved transitioning from static analysis to a functional AI-powered application. I developed an interactive web interface that allows users to experience the model's predictive power in real-time.
The Interactive Interface
The Boston Ride Fare Predictor enables users to input specific trip parameters:
- Distance (km): Adjustable slider for trip length.
- Surge Multiplier: Real-time demand intensity.
- Vehicle Type: Selection from the available fleet (e.g., Black, Shared, UberXL).
Behind the Scenes: Real-Time Pipeline
To ensure accurate predictions, the application runs a streamlined version of our data pipeline on every user input:
- Dynamic Feature Engineering: The system automatically injects baseline environmental data (e.g., average Boston temperature) to prevent model errors.
- One-Hot Encoding & Scaling: Categorical inputs are transformed, and numerical features are normalized using the project's original
StandardScalerto maintain consistency with the training environment. - Dual-Model Inference: The app simultaneously runs the Random Forest Regressor and Classifier, providing the user with both the estimated dollar fare and the corresponding price tier category.
Outcome: This deployment demonstrates the practical utility of the project, turning complex machine learning logic into a user-friendly tool for strategic ride planning.























