📈 Stock Market Prediction

June 2024 · Python · Scikit-learn · Random Forest · Matplotlib

---

🔍 Project Overview

This project focused on predicting the direction (up or down) of the S&P 500 index using engineered time-series features and a machine learning classification model. I aimed to test whether simple lag-based technical features could provide a reliable prediction signal for short-term market trends.

⚙️ Methodology

• Data Source: Used Yahoo Finance to download historical S&P 500 index data.
• Feature Engineering: Created lag features (previous close, returns, 5-day/10-day moving averages, volatility windows).
• Label Creation: Binary classification target set to 1 if the next day's close was higher than current day's close.
• Modeling: Trained a RandomForestClassifier from Scikit-learn and evaluated accuracy, precision, recall, and AUC-ROC.

💡 Features

• Lag-based time series features (rolling mean, std deviation)
• Binary classification of next-day market movement
• Train/test split respecting time order
• ROC curve and confusion matrix visualisation

📉 Errors & Fixes

• Initially used regular random split, which leaked future information – fixed with time-aware train/test splitting.
• Low precision due to imbalanced up/down distribution – experimented with threshold tuning and precision-recall analysis.
• Overfitting on early models – mitigated by limiting tree depth and adjusting min_samples_leaf.

📊 Key Takeaways

• Feature engineering from raw financial time series is critical to improve model signal.
• Market prediction is inherently noisy, so robust evaluation and realistic splitting are essential.
• Binary prediction (up/down) is more interpretable and often more useful than point forecasts.

🔗 Repository

GitHub Repo: Stock Market Prediction

---

← Back to Blog