# Data retrieval timeline

This is the retrieval schedule, organized by when each dataset is needed. It
focuses on data to read or fetch; it does not describe implementation steps.

## 1. Morning sign-on: bring archives current first

Do this before using the archives to populate charts, indicators, or the
dashboard.

| Retrieval | Scope and data needed | Timing / purpose |
|---|---|---|
| Equity archive freshness check | Read the ticker list from `REPOSITORY/equityProfiles/` and each ticker's latest date in `REPOSITORY/equityHist/{TICKER}.JSON`. Compare it with the last completed trading session, accounting for weekends and market holidays. | First data-refresh activity at sign-on. Establish which equity archives are stale. |
| Missing daily equity candle(s) | For every stale equity, retrieve the missing daily candle data from MarketData.app: timestamp and close (`t`, `c`). Usually this is one request per equity for the latest completed session; request the gap if more than one session is missing. Add the result to that ticker's `equityHist` archive and bring the corresponding profile's historical range / last-update information current. | Daily, at morning sign-on, before displaying archive-dependent data. Do not request a new candle if the archive already includes the last completed trading session. |
| Longer equity-history backfill | Retrieve up to 400 daily timestamp/close points for an equity with no usable archive, or when the archive does not have enough data for the 200-day average / required indicator values. | Exception path at sign-on or when opening an equity whose archive is missing or incomplete; not the normal daily refresh. |
| Historical P/E series refresh | For each equity, combine its daily closing prices with the latest reported quarterly earnings that were publicly available on each date, then update its two-year daily P/E series in `REPOSITORY/histPE/{TICKER}.json`. | Desired morning sign-on sequence: after equity prices and earnings are current. P/E history is separate from the equity price-history timeline. |
| Account repository | Read all JSON account files in `REPOSITORY/ACCOUNTS/`: account metadata, equity holdings and transactions, open options, and closed-option history. | At sign-on, so the application knows which holdings and option contracts are relevant. |
| Equity watchlist | Read `REPOSITORY/WATCHLISTS/equityWatchList.json`. | At sign-on, so watchlist equities can be included in the day's quote coverage, separately from account holdings. |
| Equity profiles | Read profile records in `REPOSITORY/equityProfiles/`, including ticker identity and locally stored earnings / update metadata. | At sign-on, after the profile ticker list is used for the archive-freshness check. The profile directory is the archive-refresh universe, not just the set of currently owned equities. |

The expected latest archive date is the most recent completed trading day, not
necessarily the previous calendar day. For example, a Monday morning refresh
usually needs Friday's daily candle; after a market holiday, it needs the last
session before the holiday.

## 2. Morning sign-on: retrieve data for today's view

Only after the archives are current, retrieve live and changing data needed to
show the current portfolio and watchlist.

| Retrieval | Scope and data needed | Timing / purpose |
|---|---|---|
| Current equity quotes | Current quote fields used by the app: symbol, mid, last, and percentage change. Include unique tickers held in accounts and tickers on the equity watchlist. | At sign-on, to populate current tables, valuations, and equity context. This is live quote data, separate from the daily `equityHist` candle. |
| Current option quotes | Current bid, mid, ask, and last for each open option contract found in account records. | At sign-on, to value and display open option positions. Closed options do not need a current quote for the portfolio view. |
| Equity earnings calendar | Earnings data for profile equities: upcoming report date/time and quarterly report details, plus the last-synchronized timestamp. | Refresh at sign-on when stale, so profile earnings information is current for dashboard and calendar displays. |

The daily price candle and the live quote serve different needs: the candle
closes the prior completed trading session and belongs in the archive; the quote
is a current snapshot for the present view.

## Historical P/E series: current state and data requirement

The chart displays the daily price/moving-average series and P/E series in
synchronized panels with a shared date axis and separate actual-value scales.
Store calculated P/E in a separate file for each equity:
`REPOSITORY/histPE/{TICKER}.json`. To generate or refresh this file from the
data already on disk, run `php PHP/buildHistoricalPE.php TICKER`. The output
reports `partial` when the available price archive does not span the full
requested period or is not recent; the calculation does not fetch or backfill
source data. Missing P/E dates are shown as gaps in the chart.

The earnings backfill requests three years of provider earnings data using the
`from` parameter and merges returned quarters into
`REPOSITORY/equityProfiles/{TICKER}.json` without discarding previously stored
quarters when a response is partial. `PHP/syncAllEquityEarnings.php` runs that
sync for matching equity profile files and reports whether at least eight
reported quarters are available in the most recent two years and four earlier
quarters precede that window. This is a manually run initial backfill, not yet
a scheduled daily-open job. The longer three-year lookback is intentional: a
two-year P/E series needs four reported quarters before the first chart date to
calculate trailing EPS.

The inputs are:

- Daily close prices and trading dates from that equity's `equityHist` archive.
- Reported quarterly EPS, including report dates/times, from the equity
  profile's earnings data. The all-profile backfill requests a three-year
  history and prints actual-quarter coverage for each profile. Provider
  availability may still leave a profile short of the required history; verify
  the backfill output before calculating its P/E series. At least eight
  reported quarters cover the two-year chart window, and four additional
  earlier reported quarters are needed to calculate trailing EPS at the first
  date.

**Recommended ownership:** keep reported earnings as source facts in the equity
profile's earnings history and keep calculated daily P/E only in `histPE`.
Keep enough inputs in each daily P/E record to explain and reproduce its value.

**Calculation rule:** daily close divided by the sum of the latest four
reported quarterly EPS values available to investors as of that trading date.
Use the earnings report availability date/time, not the fiscal quarter end, to
avoid using information before it was published. After-close or unspecified
report times become usable on the next trading date, using an adjacent archived
trading date when present and otherwise the next weekday; before-open reports
are usable that date. If trailing EPS is zero, negative, or unavailable,
represent P/E as null for that day rather than an ordinary positive multiple.
The weekday fallback does not account for exchange holidays.

The generated archive records the reported EPS values and dates used for each
point, along with its EPS and split-basis assumptions. It uses the profile's
`reportedEPS` values as stored and assumes historical prices and EPS are on a
consistent per-share split basis. The provider's GAAP-versus-adjusted EPS basis
and split-adjustment behavior have not been independently verified, so the
generated values are a trial series, not a confirmed valuation dataset.

The initial history requires earnings-history data covering the two-year chart
window plus at least four earlier quarters to establish trailing four-quarter
EPS at the first date (roughly twelve quarterly actuals total). The backfill
requests three years and reports observed coverage; provider availability may
still leave an equity with fewer actual quarters, which must be resolved before
its full P/E chart can be populated. The normal daily refresh should reuse
stored earnings, incorporate newly reported quarters when available, and
add/recalculate P/E points whose price or trailing-EPS inputs changed.
Existing `equityHist` daily bars are the source of daily prices, so P/E should
not require a separate daily price request when the equity archive is current.

Each `histPE` record should contain the trading date, close used, trailing EPS
used, earnings availability date, and calculated P/E. This keeps the chart
series auditable and separates reported earnings facts from derived valuation
history.

## 3. During market hours: keep current quote data current

| Retrieval | Scope and data needed | Timing / purpose |
|---|---|---|
| Current equity quotes | Mid, last, and percentage change for held and watched equities. | Refresh while the market is open at the selected quote-refresh cadence. This updates the current view; it is not a replacement for the daily candle archive. |
| Current option quotes | Bid, mid, ask, and last for open option contracts. | Refresh while the market is open at the selected quote-refresh cadence for position valuation and option displays. |

The application has a separate scheduler definition with a three-minute default,
but it is not currently loaded or started by the page. The active startup path
fetches current quotes once at sign-on. A continuous market-hours cadence is
therefore a retrieval requirement, not an active refresh in the current page.

## 4. Earnings refresh while the application remains open

| Retrieval | Scope and data needed | Timing / purpose |
|---|---|---|
| Equity earnings calendar | Upcoming earnings and quarterly results for equities in the profile / tracked universe; update the earnings fields and `lastSyncedAt` in each equity profile. | Recheck periodically because upcoming dates and reported results can change. The current page repeats this retrieval every six hours for equities loaded into account objects; newly added equities are synchronized immediately. |

## 5. When an equity is added or first opened

| Retrieval | Scope and data needed | Timing / purpose |
|---|---|---|
| Equity symbol validation | Confirm that the ticker returns a valid equity quote from MarketData.app. | Once when the user adds an equity. |
| Equity profile / history lookup | Check whether the equity profile and daily history archive already exist. | When adding or opening an equity, before showing profile- or history-dependent views. |
| Equity earnings calendar | Earnings information for the new ticker. | Immediately after adding the equity; refresh again according to the earnings cadence above. |
| Daily equity history backfill | Up to 400 daily timestamps and closes if no valid archive exists or the archive is inadequate for long moving averages. | On first open / archive initialization, instead of the one-candle daily refresh. |

The equity universe in `equityProfiles` is broader than account holdings: an
equity may be watchlisted or newly added without being held in an account.
Account positions and the watchlist remain separate inputs for deciding which
current quotes are needed.

## 6. When an option is added, viewed, or charted

| Retrieval | Scope and data needed | Timing / purpose |
|---|---|---|
| Option contract validation | Validate the OCC option symbol against MarketData.app's option lookup. | Once when the user adds an option contract. |
| Current option quote | Bid, mid, ask, and last for the contract. | At sign-on and during market-hours quote refresh while the contract is open. |
| Option historical series | Recent option price series (normally 30 days at five-minute intervals), with timestamps and bid/mid/ask/last values. | On demand when an option historical chart or analysis is opened; not part of the morning equity archive refresh. |
| Option history archive | Read the matching `REPOSITORY/optionHist/{OCC_SYMBOL}.JSON` before charting; retrieve the provider series when needed and update the archive. | When the option-history view needs data. |
| Option profile metadata | Read `REPOSITORY/optionProfiles/{OCC_SYMBOL}.json` when a feature needs the contract profile or historical-range metadata. | On demand; this is separate from the account's open/closed position data. |

## 7. When account records change

These are local repository reads, not market-data-provider retrievals.

| Retrieval | Scope and data needed | Timing / purpose |
|---|---|---|
| Account record | The selected account's holdings, open options, and transaction history. | Read from the in-memory account state when forms are opened or positions are changed; reload from `REPOSITORY/ACCOUNTS/` on the next sign-on. |
| Equity / option profile and archive | Relevant profile or history record if the user opens that symbol's dashboard or chart. | On demand; account holdings do not contain the market-price history. |

Adding or closing a position changes the account record; it does not itself
require another historical equity candle. Closing an option moves its closed
position details into account history; it no longer needs recurring current
option quotes.

## Retrieval boundaries

- **Morning archive refresh:** daily equity candles for the profile ticker
  universe, limited to missing completed trading sessions.
- **Current market view:** live equity and open-option quotes for the holdings
  and watchlist that need valuation or display.
- **Profile updates:** earnings calendar retrievals update profile earnings
  fields; other profile fields are locally stored unless separately sourced.
- **On-demand chart data:** longer equity history for initialization / repair
  and option historical series for option charts.
- **Account records:** local holding and transaction data, distinct from
  provider quotes and historical price archives.