Propensity-to-Purchase Modeling Framework

The Walt Disney Company • 2015-2017

Python XGBoost Logistic Regression Random Forest Clickstream Analytics Propensity Modeling

Business Context

Disney’s parks and experiences divisions (Walt Disney World, Disneyland, Disney Cruise Line, Aulani, Adventures by Disney) needed to enable personalized marketing experiments through A/B testing. The central analytics team required propensity scores to segment audiences by likelihood to purchase, but lacked the modeling capacity to build and maintain models across five distinct business units with unique customer journeys.

Challenge: Enable a partner team’s personalization experiments by delivering production-ready propensity models while maintaining consistency and quality across diverse product lines.

Technical Approach

Architecture Decisions

I defined the modeling framework and supervised an offshore development team in building five division-specific propensity models:

Model Specifications:

  • Algorithms tested: Logistic regression (baseline), XGBoost, Random Forest
  • Target: Binary purchase conversion within 30 days
  • Performance: Final models achieved 0.85-0.90 AUC across divisions

Feature Engineering Strategy: Given data availability and implementation constraints, I specified a pragmatic feature set using pre-aggregated session data:

  • Last-touch attribution: Final channel/source before session
  • Engagement metrics: Time on site, pages viewed per session
  • Session frequency: Return visit patterns

Scope Decisions:

  • Single-device only: Explicitly excluded cross-device tracking to reduce complexity and data quality risks
  • Session-level aggregation: Accepted pre-aggregated data rather than raw clickstream to align with existing data infrastructure

Quality Assurance & Validation

My primary technical contribution was establishing validation protocols and reviewing deliverables:

Data Leakage Detection:

  • Reviewed feature lists for temporal leakage (e.g., features that wouldn’t be available at prediction time)
  • Verified train/test splits respected temporal ordering
  • Ensured no target-derived features contaminated the input space

Model Appropriateness Review:

  • Evaluated whether included variables were operationally actionable for the marketing team
  • Removed features that were proxies for the target or reflected post-purchase behavior
  • Validated that model complexity matched data size and business interpretability needs

Performance Validation:

  • Cross-validated AUC scores across multiple time periods to ensure stability
  • Compared performance across divisions to identify data quality issues
  • Verified calibration for propensity score binning in downstream A/B testing

Implementation & Handoff

Models were validated and transitioned to the partner marketing team for integration into their experimentation platform. The modular design (one model per division) allowed independent deployment and iteration.

Key Technical Decisions

  1. Algorithm Selection via Benchmarking: Required testing of logistic regression, XGBoost, and Random Forest rather than defaulting to complex methods. XGBoost ultimately provided best performance, but the benchmarking process established reproducible model selection standards for the team.

  2. Pragmatic Feature Scope: Explicitly excluded cross-device tracking and raw clickstream sequence modeling despite their theoretical value. The decision prioritized delivery speed and data reliability over model sophistication — appropriate for enabling a downstream team’s experiments rather than pursuing research-grade accuracy.

  3. Rigorous Validation Protocol: Focused my direct effort on quality assurance (leakage detection, feature appropriateness) rather than model training. This leveraged my statistical expertise where it mattered most while scaling execution through contractor resources.

  4. Division-Specific Models vs. Universal Model: Chose separate models per division rather than a single multi-task model. While less theoretically elegant, this simplified debugging, allowed division-specific feature engineering, and matched the organizational structure of the client teams.

Reflections

This project exemplifies effective technical leadership through quality assurance. The offshore team could train models; my value was in establishing validation standards that caught data leakage and inappropriate features before they reached production. The 0.85-0.90 AUC scores were acceptable, but the rigorous review process was the critical deliverable — it ensured the models were not just accurate, but operationally valid for A/B testing.