Skip to content

Data access and storage

The data package provides normalized acquisition capabilities, deterministic synthetic data, raw response caching, explicit DuckDB storage, and pandas transforms.

Public data namespace

persistra.data

Acquisition capabilities and offline synthetic data.

__all__ = ['AcquisitionCachePolicy', 'AcquisitionFailure', 'AcquisitionFamily', 'AcquisitionPlan', 'AcquisitionReport', 'AcquisitionRequest', 'AcquisitionRunner', 'AcquisitionSuccess', 'AlphaVantageClient', 'BarSource', 'ColumnarExport', 'ColumnarExportFile', 'ColumnarFormat', 'CumulativeDatasetSelection', 'DuckDBStore', 'ExactSnapshotSelection', 'FredCategory', 'FredClient', 'FredRelease', 'FredSeriesCategoriesResult', 'FredSeriesReleaseResult', 'FredSeriesSearchResult', 'FredSeriesSummary', 'FredSeriesTagsResult', 'FredTag', 'LocalDataAdapter', 'LocalFamily', 'LocalImportSpec', 'LocalSourceIdentity', 'LocalValidation', 'LocalValidationFinding', 'OptionChainSource', 'QuoteSource', 'RawCacheEntry', 'RawResponseCache', 'ReferenceSource', 'ScalarSeriesSource', 'SnapshotDiff', 'SnapshotRow', 'SnapshotValueChange', 'StoreExportSelection', 'StoreVerification', 'StoredDataset', 'StoredOptionSnapshot', 'StoredPage', 'StoredResult', 'StoredSnapshot', 'acquisition_plan_from_json', 'acquisition_plan_to_json', 'align', 'asof_align', 'export_store', 'pivot_bars', 'pivot_series', 'resample_bars', 'synthetic', 'verify_store'] module-attribute

StoreExportSelection = ExactSnapshotSelection | CumulativeDatasetSelection

StoredResult = BarSet | QuoteSet | TopOfBookSet | OptionChain | SeriesSet | VintageSeriesSet | VintageDatesResult | ExchangeRateQuote | CommoditySpotQuote | InstrumentSearchResult | MarketStatusResult | IndexCatalogResult

AcquisitionCachePolicy

Bases: StrEnum

Caller-declared network and raw-cache policy for one request.

AcquisitionFailure dataclass

One request failure retained in the explicit run report.

AcquisitionFamily

Bases: StrEnum

Normalized result families that an acquisition request can require.

AcquisitionPlan dataclass

A versioned ordered collection of portable acquisition requests.

AcquisitionReport dataclass

Completeness report for one plan execution or resume attempt.

is_complete: bool property

Return whether every declared request has a durable success checkpoint.

AcquisitionRequest dataclass

One portable provider operation with explicit scope and output contract.

call_parameters: Mapping[str, Any] property

Return provider parameters with the declared cache flags applied.

AcquisitionRunner

Execute a portable plan sequentially with durable success checkpoints.

run(plan: AcquisitionPlan) -> AcquisitionReport

Run pending requests in plan order and return explicit completeness.

AcquisitionSuccess dataclass

One successfully completed and durably checkpointed request.

AlphaVantageClient

A synchronous client for supported Alpha Vantage primary datasets.

close() -> None

Close the client and its Persistra-owned HTTP session.

from_env(*, base_url: str = 'https://www.alphavantage.co/query', cache_directory: str | Path | None = None, requests_per_minute: float = 150, timeout: float = 30, strict_schema: bool = False, cache_ages: Mapping[str, timedelta | None] | None = None, session: SessionLike | None = None, limiter: TokenRateLimiter | None = None) -> Self classmethod

Create a client from the Persistra Alpha Vantage API-key variable.

RawCacheEntry dataclass

One cached provider response and its raw provenance.

RawResponseCache

A versioned, atomic cache of raw provider responses.

Request parameters use the portable JSON value contract. Cache identity and stored provenance share one sanitized representation with api_key and apikey fields removed at every nesting depth.

get(provider: str, operation: str, parameters: dict[str, Any], *, now: datetime, max_age: timedelta | None, offline: bool = False) -> RawCacheEntry | None

Return a matching fresh entry, or the newest entry offline.

put(entry: RawCacheEntry) -> None

Publish one cache entry atomically.

ColumnarExport dataclass

Published export files and their explicit provenance sidecar.

ColumnarExportFile dataclass

One complete columnar file produced by an export.

ColumnarFormat

Bases: StrEnum

Supported interoperable columnar file formats.

CumulativeDatasetSelection dataclass

Select the latest observed rows in one cumulative dataset.

ExactSnapshotSelection dataclass

Select one immutable store snapshot by identity.

FredCategory dataclass

One FRED category assigned to a series.

FredClient

A synchronous client for source-level FRED discovery and ALFRED series data.

close() -> None

Close the client and its Persistra-owned HTTP session.

from_env(*, base_url: str = 'https://api.stlouisfed.org/fred', cache_directory: str | Path | None = None, timeout: float = 30, strict_schema: bool = False, cache_ages: Mapping[str, timedelta | None] | None = None, session: SessionLike | None = None) -> Self classmethod

Create a client from PERSISTRA_FRED_API_KEY.

FredRelease dataclass

The FRED release that owns one series.

FredSeriesCategoriesResult dataclass

Categories assigned to one provider series.

FredSeriesReleaseResult dataclass

Release context for one provider series.

FredSeriesSearchResult dataclass

Ordered series matches and their acquisition provenance.

FredSeriesSummary dataclass

One source-level series match without canonical identity inference.

FredSeriesTagsResult dataclass

Tags assigned to one provider series.

FredTag dataclass

One source-level tag assigned to a FRED series.

LocalDataAdapter

Read caller-owned CSV, Arrow IPC, or Parquet files through public contracts.

validate(path: str | Path, spec: LocalImportSpec) -> LocalValidation

Dry-run one import and return structured diagnostics instead of a result.

import_file(path: str | Path, spec: LocalImportSpec) -> StoredResult

Import one complete local file or raise its normalized contract error.

LocalFamily

Bases: StrEnum

Normalized result families accepted by the local adapter.

LocalImportSpec dataclass

Explicit target, source-column mapping, and caller-declared semantics.

LocalSourceIdentity dataclass

Stable identity of the exact local file bytes that were read.

LocalValidation dataclass

Dry local-import validation outcome without a published result.

is_valid: bool property

Return whether the file can construct the requested normalized result.

LocalValidationFinding dataclass

One structured local-import validation failure.

BarSource

Bases: Protocol

A source of normalized bars.

bars(symbol: str, *, interval: str) -> BarSet

Return bars for one provider symbol.

OptionChainSource

Bases: Protocol

A source of normalized historical option chains.

historical_chain(symbol: str, *, date: date | None = None) -> OptionChain

Return one historical option chain.

QuoteSource

Bases: Protocol

A source of normalized latest quotes.

latest(symbol: str) -> QuoteSet

Return one latest quote.

ReferenceSource

Bases: Protocol

A source of normalized provider reference data.

search(keywords: str) -> InstrumentSearchResult

Search provider symbols.

market_status() -> MarketStatusResult

Return provider market status.

ScalarSeriesSource

Bases: Protocol

A source of normalized scalar series.

series(key: str, *, frequency: str) -> SeriesSet

Return one scalar series.

DuckDBStore

A one-process DuckDB store with snapshots and cumulative research datasets.

schema_version: int property

Return the validated store schema version.

create(path: str | Path) -> Self classmethod

Create a new store with the current supported schema at an absent path.

open(path: str | Path, *, read_only: bool = False) -> Self classmethod

Open an existing store after validating its schema without migrating it.

close() -> None

Close the explicit DuckDB connection.

load_catalog() -> Catalog

Load the complete persistent instrument catalog into an isolated value.

save_catalog(catalog: Catalog) -> None

Merge one explicit catalog into persistent storage atomically.

save(result: object) -> str

Validate and save one supported normalized result.

load_bars(instrument_id: str, *, retrieved_before: datetime | None = None) -> BarSet | None

Load the latest stored bars for one instrument scope.

load_options(underlying_instrument_id: str, chain_date: date, *, retrieved_before: datetime | None = None) -> OptionChain | None

Load the latest stored historical chain for one date.

load_quotes(symbols: tuple[str, ...], *, retrieved_before: datetime | None = None) -> QuoteSet | None

Load the latest stored quote batch for an exact symbol scope.

load_top_of_book(symbols: tuple[str, ...], *, retrieved_before: datetime | None = None) -> TopOfBookSet | None

Load the latest stored book batch for an exact symbol scope.

load_series(series_id: str, *, retrieved_before: datetime | None = None) -> SeriesSet | None

Load the latest stored scalar series for one identity.

load_vintage_series(series_id: str, *, retrieved_before: datetime | None = None) -> VintageSeriesSet | None

Load the latest stored revision history for one series identity.

load_vintage_dates(provider_series: str, *, retrieved_before: datetime | None = None) -> VintageDatesResult | None

Load the latest stored FRED vintage-date result for one provider series.

Load the latest stored provider search result.

load_market_status() -> MarketStatusResult | None

Load the latest stored provider market status.

load_index_catalog() -> IndexCatalogResult | None

Load the latest stored provider index catalog.

load_exchange_rate(instrument_id: str) -> ExchangeRateQuote | None

Load the latest stored exchange-rate quote.

load_commodity_spot(series_id: str) -> CommoditySpotQuote | None

Load the latest stored commodity spot quote.

latest_payload(family: str, scope_key: str) -> dict[str, Any] | None

Return a copy of the latest stored payload for research diagnostics.

list_datasets() -> tuple[StoredDataset, ...]

List stored family scopes in deterministic order.

list_snapshots(family: str, scope_key: str) -> tuple[StoredSnapshot, ...]

List exact snapshots for one dataset, newest first.

load_snapshot(snapshot_id: str) -> StoredResult | None

Load and validate one exact acquisition snapshot by identity.

query_bars(instrument_id: str, *, interval: str | None = None, start: date | datetime | None = None, end: date | datetime | None = None, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative bars with latest-observed row revisions and inclusive filters.

query_bars_page(instrument_id: str, *, interval: str | None = None, start: date | datetime | None = None, end: date | datetime | None = None, retrieved_before: datetime | None = None, limit: int = 100, offset: int = 0, sort_by: str | None = None, descending: bool = False) -> StoredPage

Query one bounded cumulative bar page with an exact filtered total.

query_series(series_id: str, *, start_label: str | None = None, end_label: str | None = None, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative scalar observations with latest-observed row revisions.

query_series_page(series_id: str, *, start_label: str | None = None, end_label: str | None = None, retrieved_before: datetime | None = None, limit: int = 100, offset: int = 0, sort_by: str | None = None, descending: bool = False) -> StoredPage

Query one bounded cumulative scalar-series page with an exact filtered total.

query_vintage_series(series_id: str, *, start_label: str | None = None, end_label: str | None = None, available_on: date | None = None, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative provider vintages with latest-observed row revisions.

query_vintage_series_page(series_id: str, *, start_label: str | None = None, end_label: str | None = None, available_on: date | None = None, retrieved_before: datetime | None = None, limit: int = 100, offset: int = 0, sort_by: str | None = None, descending: bool = False) -> StoredPage

Query one bounded cumulative vintage page with an exact filtered total.

query_vintage_dates(provider_series: str, *, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative release dates through one retrieval-time cutoff.

query_quote_history(*, provider: str | None = None, symbol: str | None = None, observed_start: datetime | None = None, observed_end: datetime | None = None, retrieved_start: datetime | None = None, retrieved_end: datetime | None = None) -> pd.DataFrame

Return chronological retained quote revisions and recurrence counts.

query_top_of_book_history(*, provider: str | None = None, symbol: str | None = None, observed_start: datetime | None = None, observed_end: datetime | None = None, retrieved_start: datetime | None = None, retrieved_end: datetime | None = None) -> pd.DataFrame

Return chronological retained top-of-book revisions and recurrence counts.

query_option_snapshots(underlying_instrument_id: str, *, provider: str | None = None, chain_date: date | None = None, expiration: date | None = None, strike: float | None = None, option_type: str | None = None, retrieved_before: datetime | None = None) -> tuple[StoredOptionSnapshot, ...]

Return filtered option-chain occurrences in retrieval order.

diff_snapshots(before_snapshot_id: str, after_snapshot_id: str) -> SnapshotDiff

Compare two exact snapshots without exposing serialized payloads.

SnapshotDiff dataclass

Stable source-content and acquisition-provenance snapshot differences.

source_changed: bool property

Return whether normalized source content differs.

provenance_changed: bool property

Return whether acquisition provenance or diagnostics differ.

SnapshotRow dataclass

One immutable normalized row in a snapshot diff.

SnapshotValueChange dataclass

One changed normalized value or provenance field.

StoredDataset dataclass

Summary of one stored result family and scope.

StoredOptionSnapshot dataclass

One retained option-chain occurrence after public query filters.

StoredPage dataclass

One bounded, stably ordered page from a cumulative dataset query.

has_previous: bool property

Return whether an earlier page exists.

has_next: bool property

Return whether a later page exists.

StoredSnapshot dataclass

Identity and observation bounds for one exact acquisition snapshot.

StoreVerification dataclass

Structured result of one read-only store integrity audit.

is_valid: bool property

Return whether the audit found no integrity errors.

to_dict() -> dict[str, object]

Return the versioned JSON representation of this audit.

acquisition_plan_from_json(document: str) -> AcquisitionPlan

Parse and strictly validate one versioned acquisition plan.

acquisition_plan_to_json(plan: AcquisitionPlan, *, indent: int | None = 2) -> str

Serialize a complete acquisition plan as stable portable JSON.

export_store(store: DuckDBStore, selection: StoreExportSelection, destination: str | Path, *, format: ColumnarFormat, overwrite: bool = False) -> ColumnarExport

Export an exact snapshot or cumulative dataset with provenance.

destination names the data file for a single-table export. Multi-table option snapshots insert each table name before the suffix. The provenance sidecar replaces the destination suffix with .provenance.json.

align(values: Mapping[str, pd.Series | pd.DataFrame], *, how: str = 'intersection') -> dict[str, pd.Series | pd.DataFrame]

Align labeled objects by intersection or union without filling gaps.

asof_align(left: pd.DataFrame, right: pd.DataFrame, *, maximum_staleness: pd.Timedelta) -> pd.DataFrame

Backward-align observations and report each matched source age.

pivot_bars(results: Iterable[BarSet], *, field: str) -> pd.DataFrame

Pivot one explicit normalized bar field into a wide frame.

pivot_series(results: Iterable[SeriesSet]) -> pd.DataFrame

Pivot compatible normalized scalar series into a wide frame.

resample_bars(bars: BarSet, *, frequency: str, timezone: str, sessions: set[str]) -> BarSet

Derive OHLCV bars under explicit timezone and session rules.

verify_store(path: str | Path) -> StoreVerification

Audit one existing store without changing its schema, rows, or files.

Resumable acquisition

persistra.data.acquisition

Portable, resumable plans for sequential normalized-data acquisition.

StoredResult = BarSet | QuoteSet | TopOfBookSet | OptionChain | SeriesSet | VintageSeriesSet | VintageDatesResult | ExchangeRateQuote | CommoditySpotQuote | InstrumentSearchResult | MarketStatusResult | IndexCatalogResult

AcquisitionHandler = Callable[[AcquisitionRequest], StoredResult]

DataValidationError

Bases: PersistraError, ValueError

Raised when normalized data violates its contract.

BarSet dataclass

Validated bars and their acquisition provenance.

CommoditySpotQuote dataclass

One provider commodity spot observation.

ExchangeRateQuote dataclass

One provider exchange-rate observation.

Missing and one-sided bid-ask quotes are valid. Locked and crossed quotes are retained with a bid_ask diagnostic.

IndexCatalogResult dataclass

A normalized provider index catalog.

InstrumentSearchResult dataclass

Provider search matches without inferred canonical identity.

MarketStatusResult dataclass

Provider market-status observations.

OptionChain dataclass

Contracts and observations for one historical chain.

Missing and one-sided quotes are valid. Locked and crossed quotes are retained with bid_ask diagnostics. A size without its corresponding price is invalid.

QuoteSet dataclass

Validated latest quotes and acquisition provenance.

SeriesSet dataclass

One validated scalar series and its provenance.

TopOfBookSet dataclass

Validated top-of-book snapshots and provenance.

Missing and one-sided quotes are valid. Locked and crossed quotes are retained with bid_ask diagnostics. A size without its corresponding price is invalid.

VintageDatesResult dataclass

Release dates when one provider series changed and their provenance.

VintageSeriesSet dataclass

One validated scalar-series revision history and its provenance.

DuckDBStore

A one-process DuckDB store with snapshots and cumulative research datasets.

schema_version: int property

Return the validated store schema version.

create(path: str | Path) -> Self classmethod

Create a new store with the current supported schema at an absent path.

open(path: str | Path, *, read_only: bool = False) -> Self classmethod

Open an existing store after validating its schema without migrating it.

close() -> None

Close the explicit DuckDB connection.

load_catalog() -> Catalog

Load the complete persistent instrument catalog into an isolated value.

save_catalog(catalog: Catalog) -> None

Merge one explicit catalog into persistent storage atomically.

save(result: object) -> str

Validate and save one supported normalized result.

load_bars(instrument_id: str, *, retrieved_before: datetime | None = None) -> BarSet | None

Load the latest stored bars for one instrument scope.

load_options(underlying_instrument_id: str, chain_date: date, *, retrieved_before: datetime | None = None) -> OptionChain | None

Load the latest stored historical chain for one date.

load_quotes(symbols: tuple[str, ...], *, retrieved_before: datetime | None = None) -> QuoteSet | None

Load the latest stored quote batch for an exact symbol scope.

load_top_of_book(symbols: tuple[str, ...], *, retrieved_before: datetime | None = None) -> TopOfBookSet | None

Load the latest stored book batch for an exact symbol scope.

load_series(series_id: str, *, retrieved_before: datetime | None = None) -> SeriesSet | None

Load the latest stored scalar series for one identity.

load_vintage_series(series_id: str, *, retrieved_before: datetime | None = None) -> VintageSeriesSet | None

Load the latest stored revision history for one series identity.

load_vintage_dates(provider_series: str, *, retrieved_before: datetime | None = None) -> VintageDatesResult | None

Load the latest stored FRED vintage-date result for one provider series.

Load the latest stored provider search result.

load_market_status() -> MarketStatusResult | None

Load the latest stored provider market status.

load_index_catalog() -> IndexCatalogResult | None

Load the latest stored provider index catalog.

load_exchange_rate(instrument_id: str) -> ExchangeRateQuote | None

Load the latest stored exchange-rate quote.

load_commodity_spot(series_id: str) -> CommoditySpotQuote | None

Load the latest stored commodity spot quote.

latest_payload(family: str, scope_key: str) -> dict[str, Any] | None

Return a copy of the latest stored payload for research diagnostics.

list_datasets() -> tuple[StoredDataset, ...]

List stored family scopes in deterministic order.

list_snapshots(family: str, scope_key: str) -> tuple[StoredSnapshot, ...]

List exact snapshots for one dataset, newest first.

load_snapshot(snapshot_id: str) -> StoredResult | None

Load and validate one exact acquisition snapshot by identity.

query_bars(instrument_id: str, *, interval: str | None = None, start: date | datetime | None = None, end: date | datetime | None = None, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative bars with latest-observed row revisions and inclusive filters.

query_bars_page(instrument_id: str, *, interval: str | None = None, start: date | datetime | None = None, end: date | datetime | None = None, retrieved_before: datetime | None = None, limit: int = 100, offset: int = 0, sort_by: str | None = None, descending: bool = False) -> StoredPage

Query one bounded cumulative bar page with an exact filtered total.

query_series(series_id: str, *, start_label: str | None = None, end_label: str | None = None, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative scalar observations with latest-observed row revisions.

query_series_page(series_id: str, *, start_label: str | None = None, end_label: str | None = None, retrieved_before: datetime | None = None, limit: int = 100, offset: int = 0, sort_by: str | None = None, descending: bool = False) -> StoredPage

Query one bounded cumulative scalar-series page with an exact filtered total.

query_vintage_series(series_id: str, *, start_label: str | None = None, end_label: str | None = None, available_on: date | None = None, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative provider vintages with latest-observed row revisions.

query_vintage_series_page(series_id: str, *, start_label: str | None = None, end_label: str | None = None, available_on: date | None = None, retrieved_before: datetime | None = None, limit: int = 100, offset: int = 0, sort_by: str | None = None, descending: bool = False) -> StoredPage

Query one bounded cumulative vintage page with an exact filtered total.

query_vintage_dates(provider_series: str, *, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative release dates through one retrieval-time cutoff.

query_quote_history(*, provider: str | None = None, symbol: str | None = None, observed_start: datetime | None = None, observed_end: datetime | None = None, retrieved_start: datetime | None = None, retrieved_end: datetime | None = None) -> pd.DataFrame

Return chronological retained quote revisions and recurrence counts.

query_top_of_book_history(*, provider: str | None = None, symbol: str | None = None, observed_start: datetime | None = None, observed_end: datetime | None = None, retrieved_start: datetime | None = None, retrieved_end: datetime | None = None) -> pd.DataFrame

Return chronological retained top-of-book revisions and recurrence counts.

query_option_snapshots(underlying_instrument_id: str, *, provider: str | None = None, chain_date: date | None = None, expiration: date | None = None, strike: float | None = None, option_type: str | None = None, retrieved_before: datetime | None = None) -> tuple[StoredOptionSnapshot, ...]

Return filtered option-chain occurrences in retrieval order.

diff_snapshots(before_snapshot_id: str, after_snapshot_id: str) -> SnapshotDiff

Compare two exact snapshots without exposing serialized payloads.

AcquisitionCachePolicy

Bases: StrEnum

Caller-declared network and raw-cache policy for one request.

AcquisitionFamily

Bases: StrEnum

Normalized result families that an acquisition request can require.

AcquisitionRequest dataclass

One portable provider operation with explicit scope and output contract.

call_parameters: Mapping[str, Any] property

Return provider parameters with the declared cache flags applied.

AcquisitionPlan dataclass

A versioned ordered collection of portable acquisition requests.

AcquisitionSuccess dataclass

One successfully completed and durably checkpointed request.

AcquisitionFailure dataclass

One request failure retained in the explicit run report.

AcquisitionReport dataclass

Completeness report for one plan execution or resume attempt.

is_complete: bool property

Return whether every declared request has a durable success checkpoint.

AcquisitionRunner

Execute a portable plan sequentially with durable success checkpoints.

run(plan: AcquisitionPlan) -> AcquisitionReport

Run pending requests in plan order and return explicit completeness.

atomic_write_bytes(path: Path, document: bytes, *, overwrite: bool = False) -> None

Publish complete bytes from a private same-directory file.

freeze_portable_mapping(value: Mapping[str, Any], *, name: str, redact_api_keys: bool = False) -> Mapping[str, Any]

Copy and recursively freeze one portable JSON mapping.

thaw_portable_mapping(value: Mapping[str, Any]) -> dict[str, Any]

Return a mutable JSON-compatible copy of one frozen mapping.

acquisition_plan_to_json(plan: AcquisitionPlan, *, indent: int | None = 2) -> str

Serialize a complete acquisition plan as stable portable JSON.

acquisition_plan_from_json(document: str) -> AcquisitionPlan

Parse and strictly validate one versioned acquisition plan.

_plan_dictionary(plan: AcquisitionPlan) -> dict[str, Any]

_document_hash(document: Mapping[str, Any]) -> str

_read_checkpoint(path: Path, plan: AcquisitionPlan, plan_hash: str) -> dict[str, AcquisitionSuccess]

_write_checkpoint(path: Path, plan: AcquisitionPlan, plan_hash: str, successes: Mapping[str, AcquisitionSuccess]) -> None

_write_manifest(path: Path, plan_document: Mapping[str, Any], report: AcquisitionReport) -> None

_success_dictionary(success: AcquisitionSuccess) -> dict[str, Any]

_write_json(path: Path, payload: Mapping[str, Any]) -> None

_result_family(result: StoredResult) -> AcquisitionFamily

_object_mapping(value: object, *, name: str) -> dict[str, object]

_text(value: object, *, name: str) -> str

_optional_text(value: object, *, name: str) -> str | None

Synthetic data

persistra.data.synthetic

Deterministic offline fixtures and examples, not calibrated scenario models.

BAR_DTYPES: dict[str, str] = {'instrument_id': 'string', 'provider': 'string', 'provider_symbol': 'string', 'interval': 'string', 'date': 'datetime64[ns]', 'timestamp': 'datetime64[ns, UTC]', 'timestamp_position': 'string', 'source_timezone': 'string', 'session': 'string', 'price_adjustment': 'string', 'currency': 'string', 'open': 'float64', 'high': 'float64', 'low': 'float64', 'close': 'float64', 'adjusted_close': 'Float64', 'volume': 'Float64', 'dividend_amount': 'Float64', 'split_coefficient': 'Float64', 'provider_as_of': 'datetime64[ns, UTC]', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

OPTION_CONTRACT_DTYPES: dict[str, str] = {'contract_id': 'string', 'provider': 'string', 'underlying_instrument_id': 'string', 'provider_symbol': 'string', 'expiration': 'datetime64[ns]', 'strike': 'float64', 'option_type': 'string'} module-attribute

OPTION_OBSERVATION_DTYPES: dict[str, str] = {'contract_id': 'string', 'provider': 'string', 'chain_date': 'datetime64[ns]', 'last': 'Float64', 'mark': 'Float64', 'bid': 'Float64', 'bid_size': 'Int64', 'ask': 'Float64', 'ask_size': 'Int64', 'volume': 'Int64', 'open_interest': 'Int64', 'implied_volatility': 'Float64', 'delta': 'Float64', 'gamma': 'Float64', 'theta': 'Float64', 'vega': 'Float64', 'rho': 'Float64', 'provider_as_of': 'datetime64[ns, UTC]', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

QUOTE_DTYPES: dict[str, str] = {'instrument_id': 'string', 'provider': 'string', 'provider_symbol': 'string', 'price': 'float64', 'open': 'Float64', 'high': 'Float64', 'low': 'Float64', 'previous_close': 'Float64', 'change': 'Float64', 'change_percent': 'Float64', 'volume': 'Float64', 'latest_trading_day': 'datetime64[ns]', 'observed_at': 'datetime64[ns, UTC]', 'entitlement': 'string', 'provider_as_of': 'datetime64[ns, UTC]', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

SERIES_DTYPES: dict[str, str] = {'series_id': 'string', 'provider': 'string', 'provider_series': 'string', 'series_kind': 'string', 'frequency': 'string', 'period_label': 'string', 'period_start': 'datetime64[ns]', 'period_end': 'datetime64[ns]', 'value': 'float64', 'unit': 'string', 'geography': 'string', 'seasonal_adjustment': 'string', 'maturity': 'string', 'provider_as_of': 'datetime64[ns, UTC]', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

TOP_OF_BOOK_DTYPES: dict[str, str] = {'instrument_id': 'string', 'provider': 'string', 'provider_symbol': 'string', 'bid_price': 'Float64', 'bid_size': 'Int64', 'ask_price': 'Float64', 'ask_size': 'Int64', 'observed_at': 'datetime64[ns, UTC]', 'provider_as_of': 'datetime64[ns, UTC]', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

VINTAGE_SERIES_DTYPES: dict[str, str] = {'series_id': 'string', 'provider': 'string', 'provider_series': 'string', 'series_kind': 'string', 'frequency': 'string', 'period_label': 'string', 'period_start': 'datetime64[ns]', 'period_end': 'datetime64[ns]', 'available_from': 'datetime64[ns]', 'available_through': 'datetime64[ns]', 'value': 'Float64', 'is_deleted': 'bool', 'unit': 'string', 'geography': 'string', 'seasonal_adjustment': 'string', 'maturity': 'string', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

INDEX_CATALOG_DTYPES = cast('dict[str, str]', INDEX_CATALOG_CONTRACT.dtypes) module-attribute

MARKET_STATUS_DTYPES = cast('dict[str, str]', MARKET_STATUS_CONTRACT.dtypes) module-attribute

SEARCH_DTYPES = cast('dict[str, str]', SEARCH_CONTRACT.dtypes) module-attribute

SYNTHETIC_NOW = datetime(2025, 1, 31, 21, tzinfo=UTC) module-attribute

__all__ = ['SYNTHETIC_NOW', 'bars', 'commodity_spot', 'exchange_rate', 'index_catalog', 'market_status', 'metadata', 'option_chain', 'quotes', 'search', 'series', 'top_of_book', 'treasury_curve', 'vintage_dates', 'vintage_series'] module-attribute

BarSet dataclass

Validated bars and their acquisition provenance.

CacheStatus

Bases: StrEnum

Raw response cache outcomes.

CommoditySpotQuote dataclass

One provider commodity spot observation.

EntitlementMode

Bases: StrEnum

Provider freshness and entitlement modes.

ExchangeRateQuote dataclass

One provider exchange-rate observation.

Missing and one-sided bid-ask quotes are valid. Locked and crossed quotes are retained with a bid_ask diagnostic.

IndexCatalogResult dataclass

A normalized provider index catalog.

Instrument dataclass

A provider-neutral financial or economic instrument.

InstrumentKind

Bases: StrEnum

Supported instrument families.

InstrumentSearchResult dataclass

Provider search matches without inferred canonical identity.

MarketStatusResult dataclass

Provider market-status observations.

OptionChain dataclass

Contracts and observations for one historical chain.

Missing and one-sided quotes are valid. Locked and crossed quotes are retained with bid_ask diagnostics. A size without its corresponding price is invalid.

QuoteSet dataclass

Validated latest quotes and acquisition provenance.

ResultMetadata dataclass

Required provenance with deeply immutable portable request parameters.

Request parameters may contain strings, integers, finite floats, booleans, None, string-keyed mappings, lists, and tuples. Persistra copies the complete structure, removes api_key and apikey fields recursively, freezes mappings, and converts sequences to tuples.

SeriesDefinition dataclass

Provider-neutral identity for a scalar series.

SeriesKind

Bases: StrEnum

Supported scalar series families.

SeriesSet dataclass

One validated scalar series and its provenance.

TopOfBookSet dataclass

Validated top-of-book snapshots and provenance.

Missing and one-sided quotes are valid. Locked and crossed quotes are retained with bid_ask diagnostics. A size without its corresponding price is invalid.

VintageDatesResult dataclass

Release dates when one provider series changed and their provenance.

VintageSeriesSet dataclass

One validated scalar-series revision history and its provenance.

require_integer(value: object, *, name: str, minimum: int | None = None) -> int

Return a normalized integer after enforcing an optional inclusive minimum.

provider_instrument_id(provider: str, kind: InstrumentKind, symbol: str) -> str

Create a stable provider-scoped instrument identity.

provider_series_id(provider: str, provider_series: str, frequency: str) -> str

Create a stable provider-scoped series identity.

typed_frame(data: Mapping[str, Any], dtypes: Mapping[str, str]) -> pd.DataFrame

Build a frame and apply exact contract dtypes.

metadata(operation: str, *, retrieved_at: datetime = SYNTHETIC_NOW, entitlement: EntitlementMode = EntitlementMode.NOT_APPLICABLE) -> ResultMetadata

Create deterministic synthetic provenance.

vintage_dates(provider_series: str = 'SYNTH_SERIES', *, dates: tuple[date, ...] = (date(2023, 2, 1), date(2023, 3, 1))) -> VintageDatesResult

Create deterministic provider release dates.

bars(symbol: str = 'SYNTH', *, periods: int = 90, seed: int = 7, interval: str = 'daily', kind: InstrumentKind = InstrumentKind.EQUITY, adjusted: bool = False, session: str | None = None) -> BarSet

Create deterministic fixture bars with illustrative price and volume regimes.

The output is useful for tests and offline examples. It is not fitted to acquired data and does not represent a Monte Carlo scenario or a forecast distribution.

quotes(symbols: tuple[str, ...] = ('AAA', 'BBB')) -> QuoteSet

Create deterministic latest quotes.

top_of_book(symbols: tuple[str, ...] = ('AAA', 'BBB')) -> TopOfBookSet

Create deterministic top-of-book observations.

option_chain(symbol: str = 'SYNTH', *, chain_date: date = date(2025, 1, 17)) -> OptionChain

Create a chain with several strikes, expirations, and both sides.

series(provider_series: str = 'SYNTH_GDP', *, periods: int = 24, frequency: str = 'monthly', kind: SeriesKind = SeriesKind.ECONOMIC, unit: str = 'index', geography: str | None = 'United States', maturity: str | None = None) -> SeriesSet

Create a deterministic scalar series with units and frequency.

vintage_series(provider_series: str = 'SYNTH_GDP', *, periods: int = 6, frequency: str = 'monthly', kind: SeriesKind = SeriesKind.ECONOMIC, unit: str = 'index', geography: str | None = 'United States', maturity: str | None = None) -> VintageSeriesSet

Create deterministic initial and revised scalar observations.

exchange_rate(base_currency: str = 'EUR', quote_currency: str = 'USD', *, crypto: bool = False) -> ExchangeRateQuote

Create one deterministic fiat or crypto exchange-rate quote.

commodity_spot(metal: str = 'gold') -> CommoditySpotQuote

Create one deterministic precious-metal spot quote.

search(query: str = 'DEMO') -> InstrumentSearchResult

Create deterministic provider symbol-search matches.

market_status() -> MarketStatusResult

Create one deterministic market-status result.

index_catalog() -> IndexCatalogResult

Create deterministic provider index-catalog rows.

treasury_curve(maturities: tuple[str, ...] = ('3month', '2year', '5year', '10year', '30year'), *, periods: int = 12) -> tuple[SeriesSet, ...]

Create Treasury series while allowing explicitly missing maturities.

DuckDB storage

persistra.data.store.DuckDBStore

A one-process DuckDB store with snapshots and cumulative research datasets.

path = path instance-attribute

_connection = connection instance-attribute

schema_version: int property

Return the validated store schema version.

__init__(path: Path, connection: duckdb.DuckDBPyConnection) -> None

create(path: str | Path) -> Self classmethod

Create a new store with the current supported schema at an absent path.

open(path: str | Path, *, read_only: bool = False) -> Self classmethod

Open an existing store after validating its schema without migrating it.

close() -> None

Close the explicit DuckDB connection.

__enter__() -> Self

__exit__(*_args: object) -> None

load_catalog() -> Catalog

Load the complete persistent instrument catalog into an isolated value.

save_catalog(catalog: Catalog) -> None

Merge one explicit catalog into persistent storage atomically.

save(result: object) -> str

Validate and save one supported normalized result.

_rollback_after_save_failure(failure: Exception) -> None

load_bars(instrument_id: str, *, retrieved_before: datetime | None = None) -> BarSet | None

Load the latest stored bars for one instrument scope.

load_options(underlying_instrument_id: str, chain_date: date, *, retrieved_before: datetime | None = None) -> OptionChain | None

Load the latest stored historical chain for one date.

load_quotes(symbols: tuple[str, ...], *, retrieved_before: datetime | None = None) -> QuoteSet | None

Load the latest stored quote batch for an exact symbol scope.

load_top_of_book(symbols: tuple[str, ...], *, retrieved_before: datetime | None = None) -> TopOfBookSet | None

Load the latest stored book batch for an exact symbol scope.

load_series(series_id: str, *, retrieved_before: datetime | None = None) -> SeriesSet | None

Load the latest stored scalar series for one identity.

load_vintage_series(series_id: str, *, retrieved_before: datetime | None = None) -> VintageSeriesSet | None

Load the latest stored revision history for one series identity.

load_vintage_dates(provider_series: str, *, retrieved_before: datetime | None = None) -> VintageDatesResult | None

Load the latest stored FRED vintage-date result for one provider series.

Load the latest stored provider search result.

load_market_status() -> MarketStatusResult | None

Load the latest stored provider market status.

load_index_catalog() -> IndexCatalogResult | None

Load the latest stored provider index catalog.

load_exchange_rate(instrument_id: str) -> ExchangeRateQuote | None

Load the latest stored exchange-rate quote.

load_commodity_spot(series_id: str) -> CommoditySpotQuote | None

Load the latest stored commodity spot quote.

latest_payload(family: str, scope_key: str) -> dict[str, Any] | None

Return a copy of the latest stored payload for research diagnostics.

list_datasets() -> tuple[StoredDataset, ...]

List stored family scopes in deterministic order.

list_snapshots(family: str, scope_key: str) -> tuple[StoredSnapshot, ...]

List exact snapshots for one dataset, newest first.

load_snapshot(snapshot_id: str) -> StoredResult | None

Load and validate one exact acquisition snapshot by identity.

query_bars(instrument_id: str, *, interval: str | None = None, start: date | datetime | None = None, end: date | datetime | None = None, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative bars with latest-observed row revisions and inclusive filters.

query_bars_page(instrument_id: str, *, interval: str | None = None, start: date | datetime | None = None, end: date | datetime | None = None, retrieved_before: datetime | None = None, limit: int = 100, offset: int = 0, sort_by: str | None = None, descending: bool = False) -> StoredPage

Query one bounded cumulative bar page with an exact filtered total.

query_series(series_id: str, *, start_label: str | None = None, end_label: str | None = None, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative scalar observations with latest-observed row revisions.

query_series_page(series_id: str, *, start_label: str | None = None, end_label: str | None = None, retrieved_before: datetime | None = None, limit: int = 100, offset: int = 0, sort_by: str | None = None, descending: bool = False) -> StoredPage

Query one bounded cumulative scalar-series page with an exact filtered total.

query_vintage_series(series_id: str, *, start_label: str | None = None, end_label: str | None = None, available_on: date | None = None, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative provider vintages with latest-observed row revisions.

query_vintage_series_page(series_id: str, *, start_label: str | None = None, end_label: str | None = None, available_on: date | None = None, retrieved_before: datetime | None = None, limit: int = 100, offset: int = 0, sort_by: str | None = None, descending: bool = False) -> StoredPage

Query one bounded cumulative vintage page with an exact filtered total.

query_vintage_dates(provider_series: str, *, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative release dates through one retrieval-time cutoff.

query_quote_history(*, provider: str | None = None, symbol: str | None = None, observed_start: datetime | None = None, observed_end: datetime | None = None, retrieved_start: datetime | None = None, retrieved_end: datetime | None = None) -> pd.DataFrame

Return chronological retained quote revisions and recurrence counts.

query_top_of_book_history(*, provider: str | None = None, symbol: str | None = None, observed_start: datetime | None = None, observed_end: datetime | None = None, retrieved_start: datetime | None = None, retrieved_end: datetime | None = None) -> pd.DataFrame

Return chronological retained top-of-book revisions and recurrence counts.

query_option_snapshots(underlying_instrument_id: str, *, provider: str | None = None, chain_date: date | None = None, expiration: date | None = None, strike: float | None = None, option_type: str | None = None, retrieved_before: datetime | None = None) -> tuple[StoredOptionSnapshot, ...]

Return filtered option-chain occurrences in retrieval order.

diff_snapshots(before_snapshot_id: str, after_snapshot_id: str) -> SnapshotDiff

Compare two exact snapshots without exposing serialized payloads.

_query_observation_history(family: str, *, provider: str | None, symbol: str | None, observed_start: datetime | None, observed_end: datetime | None, retrieved_start: datetime | None, retrieved_end: datetime | None) -> pd.DataFrame

_snapshot_for_diff(snapshot_id: str) -> tuple[str, StoredResult] | None

_latest(family: str, scope_key: str, retrieved_before: datetime | None) -> dict[str, Any] | None

_query_records(family: str, scope_key: str, retrieved_before: datetime | None, clauses: list[str], filter_parameters: list[object]) -> list[dict[str, Any]]

_query_page_records(family: str, scope_key: str, retrieved_before: datetime | None, clauses: list[str], filter_parameters: list[object], *, limit: int, offset: int, sort_by: str | None, descending: bool) -> tuple[list[dict[str, Any]], int]

_insert_dataset_rows(snapshot_id: str, family: str, payload: dict[str, Any]) -> None

persistra.data.store.StoredDataset dataclass

Summary of one stored result family and scope.

family: str instance-attribute

scope_key: str instance-attribute

snapshot_count: int instance-attribute

first_seen: datetime instance-attribute

last_seen: datetime instance-attribute

latest_snapshot_id: str instance-attribute

__init__(family: str, scope_key: str, snapshot_count: int, first_seen: datetime, last_seen: datetime, latest_snapshot_id: str) -> None

persistra.data.store.StoredSnapshot dataclass

Identity and observation bounds for one exact acquisition snapshot.

snapshot_id: str instance-attribute

family: str instance-attribute

scope_key: str instance-attribute

content_hash: str instance-attribute

first_seen: datetime instance-attribute

last_seen: datetime instance-attribute

saved_order: int instance-attribute

__init__(snapshot_id: str, family: str, scope_key: str, content_hash: str, first_seen: datetime, last_seen: datetime, saved_order: int) -> None

persistra.data.store.StoredPage dataclass

One bounded, stably ordered page from a cumulative dataset query.

frame: pd.DataFrame instance-attribute

total_count: int instance-attribute

limit: int instance-attribute

offset: int instance-attribute

sort_by: str | None instance-attribute

descending: bool instance-attribute

has_previous: bool property

Return whether an earlier page exists.

has_next: bool property

Return whether a later page exists.

__init__(frame: pd.DataFrame, total_count: int, limit: int, offset: int, sort_by: str | None, descending: bool) -> None

persistra.data.store.StoredOptionSnapshot dataclass

One retained option-chain occurrence after public query filters.

snapshot_id: str instance-attribute

retrieved_at: datetime instance-attribute

chain: OptionChain instance-attribute

__init__(snapshot_id: str, retrieved_at: datetime, chain: OptionChain) -> None

persistra.data.store.SnapshotDiff dataclass

Stable source-content and acquisition-provenance snapshot differences.

family: str instance-attribute

before_snapshot_id: str instance-attribute

after_snapshot_id: str instance-attribute

metadata_changes: tuple[SnapshotValueChange, ...] instance-attribute

schema_diagnostics_before: tuple[SchemaDiagnostic, ...] instance-attribute

schema_diagnostics_after: tuple[SchemaDiagnostic, ...] instance-attribute

added_rows: tuple[SnapshotRow, ...] instance-attribute

removed_rows: tuple[SnapshotRow, ...] instance-attribute

changed_values: tuple[SnapshotValueChange, ...] instance-attribute

source_changed: bool property

Return whether normalized source content differs.

provenance_changed: bool property

Return whether acquisition provenance or diagnostics differ.

__init__(family: str, before_snapshot_id: str, after_snapshot_id: str, metadata_changes: tuple[SnapshotValueChange, ...], schema_diagnostics_before: tuple[SchemaDiagnostic, ...], schema_diagnostics_after: tuple[SchemaDiagnostic, ...], added_rows: tuple[SnapshotRow, ...], removed_rows: tuple[SnapshotRow, ...], changed_values: tuple[SnapshotValueChange, ...]) -> None

persistra.data.store.SnapshotRow dataclass

One immutable normalized row in a snapshot diff.

table: str instance-attribute

identity: tuple[object, ...] instance-attribute

values: tuple[tuple[str, object], ...] instance-attribute

__init__(table: str, identity: tuple[object, ...], values: tuple[tuple[str, object], ...]) -> None

persistra.data.store.SnapshotValueChange dataclass

One changed normalized value or provenance field.

table: str instance-attribute

identity: tuple[object, ...] instance-attribute

field: str instance-attribute

before: object instance-attribute

after: object instance-attribute

__init__(table: str, identity: tuple[object, ...], field: str, before: object, after: object) -> None

Columnar exports

persistra.data.export

Atomic Arrow and Parquet exports for retained store data.

pa: Any = _pyarrow module-attribute

pq: Any = _parquet module-attribute

EXPORT_SCHEMA_VERSION = 1 module-attribute

StoredResult = BarSet | QuoteSet | TopOfBookSet | OptionChain | SeriesSet | VintageSeriesSet | VintageDatesResult | ExchangeRateQuote | CommoditySpotQuote | InstrumentSearchResult | MarketStatusResult | IndexCatalogResult

StoreExportSelection = ExactSnapshotSelection | CumulativeDatasetSelection

StoreError

Bases: PersistraError

Raised when a normalized store operation fails.

BarSet dataclass

Validated bars and their acquisition provenance.

ExchangeRateQuote dataclass

One provider exchange-rate observation.

Missing and one-sided bid-ask quotes are valid. Locked and crossed quotes are retained with a bid_ask diagnostic.

IndexCatalogResult dataclass

A normalized provider index catalog.

InstrumentSearchResult dataclass

Provider search matches without inferred canonical identity.

MarketStatusResult dataclass

Provider market-status observations.

OptionChain dataclass

Contracts and observations for one historical chain.

Missing and one-sided quotes are valid. Locked and crossed quotes are retained with bid_ask diagnostics. A size without its corresponding price is invalid.

QuoteSet dataclass

Validated latest quotes and acquisition provenance.

ResultMetadata dataclass

Required provenance with deeply immutable portable request parameters.

Request parameters may contain strings, integers, finite floats, booleans, None, string-keyed mappings, lists, and tuples. Persistra copies the complete structure, removes api_key and apikey fields recursively, freezes mappings, and converts sequences to tuples.

SeriesSet dataclass

One validated scalar series and its provenance.

TopOfBookSet dataclass

Validated top-of-book snapshots and provenance.

Missing and one-sided quotes are valid. Locked and crossed quotes are retained with bid_ask diagnostics. A size without its corresponding price is invalid.

VintageDatesResult dataclass

Release dates when one provider series changed and their provenance.

VintageSeriesSet dataclass

One validated scalar-series revision history and its provenance.

DuckDBStore

A one-process DuckDB store with snapshots and cumulative research datasets.

schema_version: int property

Return the validated store schema version.

create(path: str | Path) -> Self classmethod

Create a new store with the current supported schema at an absent path.

open(path: str | Path, *, read_only: bool = False) -> Self classmethod

Open an existing store after validating its schema without migrating it.

close() -> None

Close the explicit DuckDB connection.

load_catalog() -> Catalog

Load the complete persistent instrument catalog into an isolated value.

save_catalog(catalog: Catalog) -> None

Merge one explicit catalog into persistent storage atomically.

save(result: object) -> str

Validate and save one supported normalized result.

load_bars(instrument_id: str, *, retrieved_before: datetime | None = None) -> BarSet | None

Load the latest stored bars for one instrument scope.

load_options(underlying_instrument_id: str, chain_date: date, *, retrieved_before: datetime | None = None) -> OptionChain | None

Load the latest stored historical chain for one date.

load_quotes(symbols: tuple[str, ...], *, retrieved_before: datetime | None = None) -> QuoteSet | None

Load the latest stored quote batch for an exact symbol scope.

load_top_of_book(symbols: tuple[str, ...], *, retrieved_before: datetime | None = None) -> TopOfBookSet | None

Load the latest stored book batch for an exact symbol scope.

load_series(series_id: str, *, retrieved_before: datetime | None = None) -> SeriesSet | None

Load the latest stored scalar series for one identity.

load_vintage_series(series_id: str, *, retrieved_before: datetime | None = None) -> VintageSeriesSet | None

Load the latest stored revision history for one series identity.

load_vintage_dates(provider_series: str, *, retrieved_before: datetime | None = None) -> VintageDatesResult | None

Load the latest stored FRED vintage-date result for one provider series.

Load the latest stored provider search result.

load_market_status() -> MarketStatusResult | None

Load the latest stored provider market status.

load_index_catalog() -> IndexCatalogResult | None

Load the latest stored provider index catalog.

load_exchange_rate(instrument_id: str) -> ExchangeRateQuote | None

Load the latest stored exchange-rate quote.

load_commodity_spot(series_id: str) -> CommoditySpotQuote | None

Load the latest stored commodity spot quote.

latest_payload(family: str, scope_key: str) -> dict[str, Any] | None

Return a copy of the latest stored payload for research diagnostics.

list_datasets() -> tuple[StoredDataset, ...]

List stored family scopes in deterministic order.

list_snapshots(family: str, scope_key: str) -> tuple[StoredSnapshot, ...]

List exact snapshots for one dataset, newest first.

load_snapshot(snapshot_id: str) -> StoredResult | None

Load and validate one exact acquisition snapshot by identity.

query_bars(instrument_id: str, *, interval: str | None = None, start: date | datetime | None = None, end: date | datetime | None = None, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative bars with latest-observed row revisions and inclusive filters.

query_bars_page(instrument_id: str, *, interval: str | None = None, start: date | datetime | None = None, end: date | datetime | None = None, retrieved_before: datetime | None = None, limit: int = 100, offset: int = 0, sort_by: str | None = None, descending: bool = False) -> StoredPage

Query one bounded cumulative bar page with an exact filtered total.

query_series(series_id: str, *, start_label: str | None = None, end_label: str | None = None, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative scalar observations with latest-observed row revisions.

query_series_page(series_id: str, *, start_label: str | None = None, end_label: str | None = None, retrieved_before: datetime | None = None, limit: int = 100, offset: int = 0, sort_by: str | None = None, descending: bool = False) -> StoredPage

Query one bounded cumulative scalar-series page with an exact filtered total.

query_vintage_series(series_id: str, *, start_label: str | None = None, end_label: str | None = None, available_on: date | None = None, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative provider vintages with latest-observed row revisions.

query_vintage_series_page(series_id: str, *, start_label: str | None = None, end_label: str | None = None, available_on: date | None = None, retrieved_before: datetime | None = None, limit: int = 100, offset: int = 0, sort_by: str | None = None, descending: bool = False) -> StoredPage

Query one bounded cumulative vintage page with an exact filtered total.

query_vintage_dates(provider_series: str, *, retrieved_before: datetime | None = None) -> pd.DataFrame

Query cumulative release dates through one retrieval-time cutoff.

query_quote_history(*, provider: str | None = None, symbol: str | None = None, observed_start: datetime | None = None, observed_end: datetime | None = None, retrieved_start: datetime | None = None, retrieved_end: datetime | None = None) -> pd.DataFrame

Return chronological retained quote revisions and recurrence counts.

query_top_of_book_history(*, provider: str | None = None, symbol: str | None = None, observed_start: datetime | None = None, observed_end: datetime | None = None, retrieved_start: datetime | None = None, retrieved_end: datetime | None = None) -> pd.DataFrame

Return chronological retained top-of-book revisions and recurrence counts.

query_option_snapshots(underlying_instrument_id: str, *, provider: str | None = None, chain_date: date | None = None, expiration: date | None = None, strike: float | None = None, option_type: str | None = None, retrieved_before: datetime | None = None) -> tuple[StoredOptionSnapshot, ...]

Return filtered option-chain occurrences in retrieval order.

diff_snapshots(before_snapshot_id: str, after_snapshot_id: str) -> SnapshotDiff

Compare two exact snapshots without exposing serialized payloads.

StoredSnapshot dataclass

Identity and observation bounds for one exact acquisition snapshot.

ColumnarFormat

Bases: StrEnum

Supported interoperable columnar file formats.

ExactSnapshotSelection dataclass

Select one immutable store snapshot by identity.

CumulativeDatasetSelection dataclass

Select the latest observed rows in one cumulative dataset.

ColumnarExportFile dataclass

One complete columnar file produced by an export.

ColumnarExport dataclass

Published export files and their explicit provenance sidecar.

_PreparedFile dataclass

file_identity(path: Path) -> FileIdentity

Return the device and inode identity of one path without following symlinks.

Unlink a path only while it still names the expected file.

thaw_portable_mapping(value: Mapping[str, Any]) -> dict[str, Any]

Return a mutable JSON-compatible copy of one frozen mapping.

export_store(store: DuckDBStore, selection: StoreExportSelection, destination: str | Path, *, format: ColumnarFormat, overwrite: bool = False) -> ColumnarExport

Export an exact snapshot or cumulative dataset with provenance.

destination names the data file for a single-table export. Multi-table option snapshots insert each table name before the suffix. The provenance sidecar replaces the destination suffix with .provenance.json.

_validate_destination(path: Path, format: ColumnarFormat) -> None

_select(store: DuckDBStore, selection: StoreExportSelection) -> tuple[str, str, str | None, dict[str, pd.DataFrame], dict[str, Any]]

_find_snapshot(store: DuckDBStore, snapshot_id: str) -> StoredSnapshot

_metadata_document(metadata: ResultMetadata) -> dict[str, Any]

_cumulative_frame(store: DuckDBStore, selection: CumulativeDatasetSelection) -> pd.DataFrame

_result_frames(result: StoredResult) -> dict[str, pd.DataFrame]

_table_targets(destination: Path, tables: tuple[str, ...], format: ColumnarFormat) -> dict[str, Path]

_prepare_columnar(table_name: str, frame: pd.DataFrame, target: Path, format: ColumnarFormat, directory: Path) -> _PreparedFile

_prepare_bytes(table: str, document: bytes, target: Path, directory: Path) -> _PreparedFile

_sync(path: Path) -> None

_file_digest(path: Path) -> str

_publish(files: tuple[_PreparedFile, ...], *, overwrite: bool) -> None

_json_value(value: object) -> object

Local-file imports

persistra.data.local

Explicit local-file imports into normalized result contracts.

BAR_DTYPES: dict[str, str] = {'instrument_id': 'string', 'provider': 'string', 'provider_symbol': 'string', 'interval': 'string', 'date': 'datetime64[ns]', 'timestamp': 'datetime64[ns, UTC]', 'timestamp_position': 'string', 'source_timezone': 'string', 'session': 'string', 'price_adjustment': 'string', 'currency': 'string', 'open': 'float64', 'high': 'float64', 'low': 'float64', 'close': 'float64', 'adjusted_close': 'Float64', 'volume': 'Float64', 'dividend_amount': 'Float64', 'split_coefficient': 'Float64', 'provider_as_of': 'datetime64[ns, UTC]', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

OPTION_CONTRACT_DTYPES: dict[str, str] = {'contract_id': 'string', 'provider': 'string', 'underlying_instrument_id': 'string', 'provider_symbol': 'string', 'expiration': 'datetime64[ns]', 'strike': 'float64', 'option_type': 'string'} module-attribute

OPTION_OBSERVATION_DTYPES: dict[str, str] = {'contract_id': 'string', 'provider': 'string', 'chain_date': 'datetime64[ns]', 'last': 'Float64', 'mark': 'Float64', 'bid': 'Float64', 'bid_size': 'Int64', 'ask': 'Float64', 'ask_size': 'Int64', 'volume': 'Int64', 'open_interest': 'Int64', 'implied_volatility': 'Float64', 'delta': 'Float64', 'gamma': 'Float64', 'theta': 'Float64', 'vega': 'Float64', 'rho': 'Float64', 'provider_as_of': 'datetime64[ns, UTC]', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

QUOTE_DTYPES: dict[str, str] = {'instrument_id': 'string', 'provider': 'string', 'provider_symbol': 'string', 'price': 'float64', 'open': 'Float64', 'high': 'Float64', 'low': 'Float64', 'previous_close': 'Float64', 'change': 'Float64', 'change_percent': 'Float64', 'volume': 'Float64', 'latest_trading_day': 'datetime64[ns]', 'observed_at': 'datetime64[ns, UTC]', 'entitlement': 'string', 'provider_as_of': 'datetime64[ns, UTC]', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

SERIES_DTYPES: dict[str, str] = {'series_id': 'string', 'provider': 'string', 'provider_series': 'string', 'series_kind': 'string', 'frequency': 'string', 'period_label': 'string', 'period_start': 'datetime64[ns]', 'period_end': 'datetime64[ns]', 'value': 'float64', 'unit': 'string', 'geography': 'string', 'seasonal_adjustment': 'string', 'maturity': 'string', 'provider_as_of': 'datetime64[ns, UTC]', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

TOP_OF_BOOK_DTYPES: dict[str, str] = {'instrument_id': 'string', 'provider': 'string', 'provider_symbol': 'string', 'bid_price': 'Float64', 'bid_size': 'Int64', 'ask_price': 'Float64', 'ask_size': 'Int64', 'observed_at': 'datetime64[ns, UTC]', 'provider_as_of': 'datetime64[ns, UTC]', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

VINTAGE_SERIES_DTYPES: dict[str, str] = {'series_id': 'string', 'provider': 'string', 'provider_series': 'string', 'series_kind': 'string', 'frequency': 'string', 'period_label': 'string', 'period_start': 'datetime64[ns]', 'period_end': 'datetime64[ns]', 'available_from': 'datetime64[ns]', 'available_through': 'datetime64[ns]', 'value': 'Float64', 'is_deleted': 'bool', 'unit': 'string', 'geography': 'string', 'seasonal_adjustment': 'string', 'maturity': 'string', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

INDEX_CATALOG_DTYPES = cast('dict[str, str]', INDEX_CATALOG_CONTRACT.dtypes) module-attribute

MARKET_STATUS_DTYPES = cast('dict[str, str]', MARKET_STATUS_CONTRACT.dtypes) module-attribute

SEARCH_DTYPES = cast('dict[str, str]', SEARCH_CONTRACT.dtypes) module-attribute

pa: Any = _pyarrow module-attribute

pq: Any = _parquet module-attribute

_FRAME_DTYPES: dict[LocalFamily, Mapping[str, str]] = {LocalFamily.BARS: BAR_DTYPES, LocalFamily.INDEX_CATALOG: INDEX_CATALOG_DTYPES, LocalFamily.MARKET_STATUS: MARKET_STATUS_DTYPES, LocalFamily.QUOTES: QUOTE_DTYPES, LocalFamily.SEARCH: SEARCH_DTYPES, LocalFamily.SERIES: SERIES_DTYPES, LocalFamily.TOP_OF_BOOK: TOP_OF_BOOK_DTYPES, LocalFamily.VINTAGE_SERIES: VINTAGE_SERIES_DTYPES} module-attribute

_SCALAR_DTYPES: dict[LocalFamily, Mapping[str, str]] = {LocalFamily.EXCHANGE_RATE: {'instrument_id': 'string', 'provider': 'string', 'base_currency': 'string', 'quote_currency': 'string', 'exchange_rate': 'float64', 'bid': 'Float64', 'ask': 'Float64', 'provider_timestamp': 'datetime64[ns, UTC]', 'provider_timezone': 'string', 'retrieved_at': 'datetime64[ns, UTC]'}, LocalFamily.COMMODITY_SPOT: {'series_id': 'string', 'provider': 'string', 'metal': 'string', 'value': 'float64', 'unit': 'string', 'provider_timestamp': 'datetime64[ns, UTC]', 'retrieved_at': 'datetime64[ns, UTC]'}} module-attribute

StoredResult = BarSet | QuoteSet | TopOfBookSet | OptionChain | SeriesSet | VintageSeriesSet | VintageDatesResult | ExchangeRateQuote | CommoditySpotQuote | InstrumentSearchResult | MarketStatusResult | IndexCatalogResult

DataValidationError

Bases: PersistraError, ValueError

Raised when normalized data violates its contract.

BarSet dataclass

Validated bars and their acquisition provenance.

CacheStatus

Bases: StrEnum

Raw response cache outcomes.

CommoditySpotQuote dataclass

One provider commodity spot observation.

EntitlementMode

Bases: StrEnum

Provider freshness and entitlement modes.

ExchangeRateQuote dataclass

One provider exchange-rate observation.

Missing and one-sided bid-ask quotes are valid. Locked and crossed quotes are retained with a bid_ask diagnostic.

IndexCatalogResult dataclass

A normalized provider index catalog.

Instrument dataclass

A provider-neutral financial or economic instrument.

InstrumentKind

Bases: StrEnum

Supported instrument families.

InstrumentSearchResult dataclass

Provider search matches without inferred canonical identity.

MarketStatusResult dataclass

Provider market-status observations.

OptionChain dataclass

Contracts and observations for one historical chain.

Missing and one-sided quotes are valid. Locked and crossed quotes are retained with bid_ask diagnostics. A size without its corresponding price is invalid.

QuoteSet dataclass

Validated latest quotes and acquisition provenance.

ResultMetadata dataclass

Required provenance with deeply immutable portable request parameters.

Request parameters may contain strings, integers, finite floats, booleans, None, string-keyed mappings, lists, and tuples. Persistra copies the complete structure, removes api_key and apikey fields recursively, freezes mappings, and converts sequences to tuples.

SeriesDefinition dataclass

Provider-neutral identity for a scalar series.

SeriesKind

Bases: StrEnum

Supported scalar series families.

SeriesSet dataclass

One validated scalar series and its provenance.

TopOfBookSet dataclass

Validated top-of-book snapshots and provenance.

Missing and one-sided quotes are valid. Locked and crossed quotes are retained with bid_ask diagnostics. A size without its corresponding price is invalid.

VintageDatesResult dataclass

Release dates when one provider series changed and their provenance.

VintageSeriesSet dataclass

One validated scalar-series revision history and its provenance.

LocalFamily

Bases: StrEnum

Normalized result families accepted by the local adapter.

LocalImportSpec dataclass

Explicit target, source-column mapping, and caller-declared semantics.

LocalSourceIdentity dataclass

Stable identity of the exact local file bytes that were read.

LocalValidationFinding dataclass

One structured local-import validation failure.

LocalValidation dataclass

Dry local-import validation outcome without a published result.

is_valid: bool property

Return whether the file can construct the requested normalized result.

LocalDataAdapter

Read caller-owned CSV, Arrow IPC, or Parquet files through public contracts.

validate(path: str | Path, spec: LocalImportSpec) -> LocalValidation

Dry-run one import and return structured diagnostics instead of a result.

import_file(path: str | Path, spec: LocalImportSpec) -> StoredResult

Import one complete local file or raise its normalized contract error.

freeze_portable_mapping(value: Mapping[str, Any], *, name: str, redact_api_keys: bool = False) -> Mapping[str, Any]

Copy and recursively freeze one portable JSON mapping.

thaw_portable_mapping(value: Mapping[str, Any]) -> dict[str, Any]

Return a mutable JSON-compatible copy of one frozen mapping.

typed_frame(data: Mapping[str, Any], dtypes: Mapping[str, str]) -> pd.DataFrame

Build a frame and apply exact contract dtypes.

_read_source(path: Path) -> tuple[pd.DataFrame, LocalSourceIdentity]

_expected_columns(family: LocalFamily) -> set[str]

_metadata(spec: LocalImportSpec, source: LocalSourceIdentity, imported_at: datetime) -> ResultMetadata

_construct(source: pd.DataFrame, spec: LocalImportSpec, metadata: ResultMetadata, imported_at: datetime) -> StoredResult

_mapped_frame(source: pd.DataFrame, spec: LocalImportSpec, dtypes: Mapping[str, str], imported_at: datetime) -> pd.DataFrame

_semantic_text(spec: LocalImportSpec, name: str) -> str

_semantic_optional_text(spec: LocalImportSpec, name: str) -> str | None

_semantic_date(spec: LocalImportSpec, name: str) -> date

_declared_column(frame: pd.DataFrame, name: str, spec: LocalImportSpec) -> None

_optional_float(value: object) -> float | None

_optional_string(value: object) -> str | None

_optional_datetime_value(value: object) -> datetime | None

persistra.data.verification.StoreVerification dataclass

Structured result of one read-only store integrity audit.

path: Path instance-attribute

findings: tuple[ValidationFinding, ...] instance-attribute

snapshot_count: int | None instance-attribute

occurrence_count: int | None instance-attribute

is_valid: bool property

Return whether the audit found no integrity errors.

__init__(path: Path, findings: tuple[ValidationFinding, ...], snapshot_count: int | None, occurrence_count: int | None) -> None

to_dict() -> dict[str, object]

Return the versioned JSON representation of this audit.

persistra.data.verification.verify_store(path: str | Path) -> StoreVerification

Audit one existing store without changing its schema, rows, or files.

Raw response cache

persistra.data.cache

Atomic raw provider response cache.

CACHE_FORMAT_VERSION = 1 module-attribute

CacheError

Bases: PersistraError

Raised when a raw cache operation fails.

RawCacheEntry dataclass

One cached provider response and its raw provenance.

RawResponseCache

A versioned, atomic cache of raw provider responses.

Request parameters use the portable JSON value contract. Cache identity and stored provenance share one sanitized representation with api_key and apikey fields removed at every nesting depth.

get(provider: str, operation: str, parameters: dict[str, Any], *, now: datetime, max_age: timedelta | None, offline: bool = False) -> RawCacheEntry | None

Return a matching fresh entry, or the newest entry offline.

put(entry: RawCacheEntry) -> None

Publish one cache entry atomically.

freeze_portable_mapping(value: Mapping[str, Any], *, name: str, redact_api_keys: bool = False) -> Mapping[str, Any]

Copy and recursively freeze one portable JSON mapping.

thaw_portable_mapping(value: Mapping[str, Any]) -> dict[str, Any]

Return a mutable JSON-compatible copy of one frozen mapping.

_redact(parameters: Mapping[str, Any]) -> dict[str, Any]

_safe_component(value: str) -> str

Transforms

persistra.data.utils

Explicit reshaping, alignment, and resampling utilities.

BAR_DTYPES: dict[str, str] = {'instrument_id': 'string', 'provider': 'string', 'provider_symbol': 'string', 'interval': 'string', 'date': 'datetime64[ns]', 'timestamp': 'datetime64[ns, UTC]', 'timestamp_position': 'string', 'source_timezone': 'string', 'session': 'string', 'price_adjustment': 'string', 'currency': 'string', 'open': 'float64', 'high': 'float64', 'low': 'float64', 'close': 'float64', 'adjusted_close': 'Float64', 'volume': 'Float64', 'dividend_amount': 'Float64', 'split_coefficient': 'Float64', 'provider_as_of': 'datetime64[ns, UTC]', 'retrieved_at': 'datetime64[ns, UTC]'} module-attribute

_TIME_UNITS_FINEST_FIRST: tuple[_TimeUnit, ...] = ('ns', 'us', 'ms', 's') module-attribute

_TimeUnit = Literal['s', 'ms', 'us', 'ns']

DataValidationError

Bases: PersistraError, ValueError

Raised when normalized data violates its contract.

BarSet dataclass

Validated bars and their acquisition provenance.

CacheStatus

Bases: StrEnum

Raw response cache outcomes.

ResultMetadata dataclass

Required provenance with deeply immutable portable request parameters.

Request parameters may contain strings, integers, finite floats, booleans, None, string-keyed mappings, lists, and tuples. Persistra copies the complete structure, removes api_key and apikey fields recursively, freezes mappings, and converts sequences to tuples.

SchemaDiagnostic dataclass

A nonfatal provider schema difference.

SeriesSet dataclass

One validated scalar series and its provenance.

typed_frame(data: Mapping[str, Any], dtypes: Mapping[str, str]) -> pd.DataFrame

Build a frame and apply exact contract dtypes.

_finest_time_unit(left: _TimeUnit, right: _TimeUnit) -> _TimeUnit

pivot_bars(results: Iterable[BarSet], *, field: str) -> pd.DataFrame

Pivot one explicit normalized bar field into a wide frame.

pivot_series(results: Iterable[SeriesSet]) -> pd.DataFrame

Pivot compatible normalized scalar series into a wide frame.

align(values: Mapping[str, pd.Series | pd.DataFrame], *, how: str = 'intersection') -> dict[str, pd.Series | pd.DataFrame]

Align labeled objects by intersection or union without filling gaps.

resample_bars(bars: BarSet, *, frequency: str, timezone: str, sessions: set[str]) -> BarSet

Derive OHLCV bars under explicit timezone and session rules.

asof_align(left: pd.DataFrame, right: pd.DataFrame, *, maximum_staleness: pd.Timedelta) -> pd.DataFrame

Backward-align observations and report each matched source age.

_bar_temporal_kind(frame: pd.DataFrame) -> str | None

_validate_asof_columns(left: pd.DataFrame, right: pd.DataFrame) -> None

Capability protocols

persistra.data.protocols

Small provider-neutral acquisition capabilities.

BarSet dataclass

Validated bars and their acquisition provenance.

InstrumentSearchResult dataclass

Provider search matches without inferred canonical identity.

MarketStatusResult dataclass

Provider market-status observations.

OptionChain dataclass

Contracts and observations for one historical chain.

Missing and one-sided quotes are valid. Locked and crossed quotes are retained with bid_ask diagnostics. A size without its corresponding price is invalid.

QuoteSet dataclass

Validated latest quotes and acquisition provenance.

SeriesSet dataclass

One validated scalar series and its provenance.

BarSource

Bases: Protocol

A source of normalized bars.

bars(symbol: str, *, interval: str) -> BarSet

Return bars for one provider symbol.

QuoteSource

Bases: Protocol

A source of normalized latest quotes.

latest(symbol: str) -> QuoteSet

Return one latest quote.

OptionChainSource

Bases: Protocol

A source of normalized historical option chains.

historical_chain(symbol: str, *, date: date | None = None) -> OptionChain

Return one historical option chain.

ScalarSeriesSource

Bases: Protocol

A source of normalized scalar series.

series(key: str, *, frequency: str) -> SeriesSet

Return one scalar series.

ReferenceSource

Bases: Protocol

A source of normalized provider reference data.

search(keywords: str) -> InstrumentSearchResult

Search provider symbols.

market_status() -> MarketStatusResult

Return provider market status.