Time and provenance

Time fields answer different questions. Persistra keeps them separate so a retrieval time is not mistaken for a market event, and a calendar period is not forced into an arbitrary instant.

Calendar labels and instants

Daily and lower-frequency bars use the timezone-naive date column. Intraday bars use the UTC-aware timestamp column. Exactly one applies to each bar row.

from persistra.data import synthetic

daily = synthetic.bars("DAILY", interval="daily")
intraday = synthetic.bars("INTRADAY", interval="5min")

assert daily.frame["date"].notna().all()
assert daily.frame["timestamp"].isna().all()
assert intraday.frame["timestamp"].notna().all()
assert intraday.frame["date"].isna().all()

For intraday bars, source_timezone retains the provider's timezone and timestamp_position records what the provider label means. Normalized timestamps are UTC instants.

Scalar period labels

Commodity and economic series retain the provider's period_label. period_start and period_end remain missing unless the source gives enough information to populate them without inference.

This matters for labels such as a month, quarter, or semiannual period: choosing a start, end, publication date, or midpoint would change the meaning.

from persistra.data import synthetic

series = synthetic.series("CPI", frequency="monthly")
print(series.frame[["period_label", "period_start", "period_end"]].tail())

Vintage availability dates

VintageSeriesSet keeps every supplied version of a scalar observation. available_from is the first calendar date on which a version applies. available_through is its inclusive last applicable date. A missing end means that the version remains open-ended.

from persistra.data import synthetic

history = synthetic.vintage_series("CPI")
print(
    history.frame[
        ["period_label", "available_from", "available_through", "value", "is_deleted"]
    ].head()
)

Availability dates have daily resolution. They are not publication timestamps. Intervals for one observation cannot overlap, including at a shared boundary. Gaps are allowed and mean that the result contains no applicable version for those dates.

The nullable value field preserves source missingness. A missing value with is_deleted=False is an explicitly missing numeric observation. A missing value with is_deleted=True is a source deletion. In a complete history, the earliest version identifies a newly published observation; later versions preserve revisions without replacing it.

Retrieval time records when Persistra obtained the revision history. It never fills a missing availability date.

FRED and ALFRED return inclusive daily real-time periods. Persistra maps those periods directly and converts the provider's 9999-12-31 end sentinel to an open interval. It does not infer publication timestamps from those dates.

Observation time

observed_at is the event or snapshot time attached to an observation when one applies. A latest daily quote may instead have a calendar latest_trading_day. These fields are part of the result schema and can be missing when the provider does not supply them.

Provider as-of time

provider_as_of records the provider's stated as-of time. It appears in result metadata and, where row-level meaning is needed, in normalized frames. It must be timezone-aware when present.

Persistra does not substitute retrieval time when provider as-of time is absent.

Retrieval time

retrieved_at records when Persistra obtained the source response or created deterministic synthetic provenance. It answers, "When did this system observe the data?"

result = synthetic.bars("DEMO")

print(result.metadata.retrieved_at)
print(result.frame["retrieved_at"].head())

Retrieval time supports reproducibility, cache diagnostics, and DuckDB revisions. It does not claim that the observation occurred at that instant.

Within one normalized acquisition result, every row-level retrieved_at value equals ResultMetadata.retrieved_at. Row-level providers likewise equal the metadata provider. This keeps each result's source and observation boundary unambiguous.

Entitlement and cache provenance

ResultMetadata.entitlement records historical, delayed, realtime, or nonapplicable access. cache_status records whether the raw response was a hit, miss, refresh, offline read, or not used.

These fields belong in analysis inputs and reports when freshness or access mode can affect interpretation.

Retrieval-time revisions

DuckDBStore identifies snapshots by normalized content within a family and scope. Identical content updates its last-seen time. Changed content creates another first-seen revision.

Typed load methods select one exact acquisition snapshot. Bar and scalar-series query methods instead assemble the latest eligible revision of every retained normalized row. Nonoverlapping partial acquisitions accumulate; an overlapping row is replaced only by a later acquisition that contains the same normalized identity.

A retrieved_before query asks for the latest content Persistra had first observed by a timezone-aware cutoff:

from datetime import UTC, datetime

from persistra.data import DuckDBStore

cutoff = datetime(2025, 2, 1, tzinfo=UTC)

with DuckDBStore.open("research.duckdb", read_only=True) as store:
    result = store.query_bars(
        instrument_id,
        retrieved_before=cutoff,
    )

This reconstructs the cumulative rows Persistra had observed by the cutoff. It is not the same as provider-native vintage data and does not recover a revision that Persistra never observed.

As-of joins require a staleness limit

Backward as-of matching can make an old value look current unless its age is visible. asof_align therefore requires maximum_staleness and returns matched_label plus matched_age:

import pandas as pd

from persistra.data import asof_align

aligned = asof_align(
    left_frame,
    right_frame,
    maximum_staleness=pd.Timedelta(days=3),
)

Choose the limit from the meaning and expected cadence of the right-hand data, not merely to maximize matched rows.

Point-in-time feature dates

Point-in-time feature construction applies three separate calendar concepts:

1. The decision date states when the feature will be used.
2. Source availability selects the observation version that was public under the chosen publication lag.
3. The configured period_start or period_end field measures observation staleness.

All three are timezone-naive calendar dates because normalized vintage availability has daily resolution. Retrieval time identifies the acquired source history but never selects a feature version. The feature provenance table retains the exact period label, availability interval, retrieval time, lag, age, and staleness limit used for each decision.

Forward-return labels store their actual horizon end dates separately. A temporal training row is safe only when its label ends strictly before the evaluation period begins.

Questions to ask before combining data

Before a join, resample, or return calculation, identify:

1. Is each label a calendar date, period label, or instant?
2. Which timezone defines an intraday bucket?
3. Does the provider label the start or end of an interval?
4. How old may a matched observation be?
5. Is a missing value absent, not applicable, or explicitly reported as missing?
6. Does retrieval time constrain what the system knew, or only when it downloaded a payload?
7. Is an availability boundary inclusive, exclusive, or open-ended?

Persistra exposes the fields needed to answer these questions but leaves the research policy to the caller.