Historical Data Implementation Documentation
===========================================

Date: 2026-05-01
Author: lexPatia Development Team
Version: 1.0

Overview
--------
This document describes the implementation of the historical data management system for lexPatia trading application. The system provides storage, tracking, and retrieval capabilities for both equity and options historical data.

Directory Structure
-------------------

REPOSITORY/
├── equityHist/           # Equity historical data files
│   └── MSFT.JSON        # Example: MSFT historical data
├── optionHist/          # Option historical data files
│   └── MSFT20260618P000150000.JSON  # Example: MSFT option historical data
├── equityProfiles/      # Equity profile files with data range tracking
│   └── MSFT.json        # Equity profile with historicalDataRange field
└── optionProfiles/      # Option profile files with data range tracking
    └── MSFT20260618P000150000.json  # Option profile with historicalDataRange field

File Naming Conventions
-----------------------

Equity Historical Data:
- Format: {TICKER}.JSON
- Example: MSFT.JSON
- Location: /REPOSITORY/equityHist/

Option Historical Data:
- Format: {OCC_SYMBOL}.JSON
- Example: MSFT20260618P000150000.JSON
- Location: /REPOSITORY/optionHist/

Equity Profiles:
- Format: {TICKER}.json
- Example: MSFT.json
- Location: /REPOSITORY/equityProfiles/

Option Profiles:
- Format: {OCC_SYMBOL}.json
- Example: MSFT20260618P000150000.json
- Location: /REPOSITORY/optionProfiles/

Data Structure
--------------

Equity Historical Data Format:
{
  "YYYY-MM-DD": {
    "open": 375.25,
    "high": 378.90,
    "low": 374.50,
    "close": 377.80,
    "last": 377.80
  }
}

Option Historical Data Format:
{
  "YYYY-MM-DD": {
    "bid": 1.28,
    "ask": 1.32,
    "last": 1.30
  }
}

Historical Data Range Tracking:
{
  "historicalDataRange": {
    "startDate": "2025-01-01",
    "endDate": "2026-05-01",
    "lastUpdated": "2026-05-01 12:00:00"
  }
}

Control Functions
-----------------

PHP Class: HistoricalDataManager
Location: /PHP/historicalDataManager.php

Key Methods:
- updateEquityDataRange(ticker, startDate, endDate)
- getEquityDataRange(ticker)
- updateOptionDataRange(optionSymbol, startDate, endDate, optionInfo)
- getOptionDataRange(optionSymbol)
- storeEquityHistoricalData(ticker, data)
- storeOptionHistoricalData(optionSymbol, data, optionInfo)
- getEquityHistoricalData(ticker, startDate, endDate)
- getOptionHistoricalData(optionSymbol, startDate, endDate)
- listEquitiesWithHistoricalData()
- listOptionsWithHistoricalData()
- needsDataUpdate(symbol, isOption, maxAgeDays)

JavaScript Class: HistoricalDataManagerJS
Location: /JS/historicalDataManager.js

Key Methods:
- getEquityDataRange(ticker)
- getOptionDataRange(optionSymbol)
- getEquityHistoricalData(ticker, startDate, endDate)
- getOptionHistoricalData(optionSymbol, startDate, endDate)
- listEquitiesWithHistoricalData()
- listOptionsWithHistoricalData()
- needsDataUpdate(symbol, isOption, maxAgeDays)
- calculateStatistics(data)
- filterDataByDate(data, startDate, endDate)
- getDataForMovingAverage(data, period)
- createDataSummary(ticker, equityData, optionData)

Usage Examples
--------------

PHP Usage:
```php
$manager = new HistoricalDataManager();

// Store equity historical data
$equityData = [
  "2026-05-01" => ["open" => 414.50, "high" => 416.25, "low" => 413.80, "close" => 415.80, "last" => 415.80]
];
$manager->storeEquityHistoricalData("MSFT", $equityData);

// Store option historical data
$optionData = [
  "2026-05-01" => ["bid" => 2.40, "ask" => 2.44, "last" => 2.42]
];
$optionInfo = [
  "underlying" => "MSFT",
  "expiration" => "2026-06-18",
  "strike" => 150.00,
  "type" => "put"
];
$manager->storeOptionHistoricalData("MSFT20260618P000150000", $optionData, $optionInfo);

// Check if data needs updating
$needsUpdate = $manager->needsDataUpdate("MSFT", false, 1); // 1 day max age
```

JavaScript Usage:
```javascript
// Get equity historical data
const equityData = await historicalDataManager.getEquityHistoricalData("MSFT", "2025-01-01", "2026-05-01");

// Calculate statistics
const stats = historicalDataManager.calculateStatistics(equityData);

// Get moving averages
const ma20 = historicalDataManager.getDataForMovingAverage(equityData, 20);

// Create summary for dashboard
const summary = historicalDataManager.createDataSummary("MSFT", equityData);
```

Data Validation
---------------

- Date format must be YYYY-MM-DD
- All numeric values should be validated as numbers
- File permissions should be set to allow read/write operations
- JSON files should be validated for proper format

Error Handling
--------------

- PHP functions return false on error
- JavaScript functions throw exceptions on API failures
- Invalid data formats are logged and rejected
- File system errors are handled gracefully

Integration Points
------------------

1. Data Download Functions: Will use storeEquityHistoricalData() and storeOptionHistoricalData()
2. Dashboard Display: Will use getEquityHistoricalData() and getOptionHistoricalData()
3. Moving Average Functions: Will use getDataForMovingAverage()
4. Data Update Scheduler: Will use needsDataUpdate() to check for stale data

Future Enhancements
-------------------

1. Additional equity data points: volume, adjusted close, mid price
2. Additional option data points: greeks (delta, gamma, theta, vega), implied volatility, open interest
3. Data compression for large historical datasets
4. Incremental data updates (only new dates)
5. Data validation and quality checks
6. Backup and recovery procedures
7. Performance optimization for large datasets
8. Real-time data streaming capabilities

Security Considerations
-----------------------

- File access should be restricted to authorized users
- Input validation for all data inputs
- Sanitization of file names and paths
- Regular backups of historical data

Maintenance
-----------

- Regular monitoring of disk space usage
- Periodic data integrity checks
- Cleanup of old or corrupted data files
- Performance monitoring and optimization

Notes
-----

- The system is designed to handle large datasets efficiently
- File-based storage provides simplicity and reliability
- Date-based indexing allows for fast range queries
- The modular design allows for easy extension and modification