Power

Price Forward Curves

Shifting

class rivapy.marketdata_tools.PFCShifter(shape: DataFrame, contracts: List[EnergyFutureSpecifications])[source]

Bases: FactoryObject

A shifting methodology for PFC shapes. This class gets a PFC shape as an input and shifts it in such a way, that the resulting PFC contains the future prices defined in the contracts dictionary. We follow the methodology described here: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2706366

Parameters:

  • shape (pd.DataFrame) – PFC shape, where the DataFrame index are datetime objects.

  • contracts (Dict[str, EnergyFutureSpecifications]) – Dictionary containing the future contracts (EnergyFutureSpecifications objects)

Usage:

# iterative usage
pfc_shifter = PFCShifter(shape=shape, contracts=contracts)
transition_matrix = pfc_shifter.generate_transition_matrix()
transition_matrix = pfc_shifter.detect_redundant_contracts(transition_matrix)
transition_matrix = pfc_shifter.generate_synthetic_contracts(transition_matrix)
pfc = pfc_shifter.shift(transition_matrix)

# direct call
pfc_shifter = PFCShifter(shape=shape, contracts=contracts)
pfc = pfc_shifter.compute()
compute() → DataFrame[source]

Compute method to directly call all the individual steps involved for the shifting

Returns:

Shifted PFC shape

Return type:

pd.DataFrame

detect_redundant_contracts(transition_matrix: DataFrame) → DataFrame[source]

In order to obtain an invertable matrix, the matrix must be of full rank. Linear dependent contracts will yield linear dependent row vectors. This is the case if e.g. a Cal Base and all four quarter contracts are provided. This method finds all redundant (linear dependent) contracts and omits the last found linear dependent contract in order to make sure that the row vectors are linearly independent.

Parameters:

transition_matrix (pd.DataFrame) – Transition matrix generated by the generate_transition_matrix method.

Returns:

Transition matrix without linearly dependent row vectors.

Return type:

pd.DataFrame

generate_synthetic_contracts(transition_matrix: DataFrame) → DataFrame[source]

In order to fulfill the requirement of an invertable transition matrix, not only the row vectors but also the column vectors must generate a basis. In cases where m > n, we need to additionally generate synthetic contracts. The delivery period for the synthetic contracts are chosen in such a way that the column vectors become linearly independent. The forward price for each synthetic contract is computed based on the rations of the average shape values over the corresponding delivery period of the synthetic contract and a reference contract. The shape ratio is multiplied with the forward price of the reference contract in order to obtain a forward price for the synthetic contract. The reference contract is implemented to be always the first contract in the contracts dictionary.

Parameters:

transition_matrix (pd.DataFrame) – Transition matrix generated by the detect_redundant_contracts method.

Returns:

Full rank transition matrix

Return type:

pd.DataFrame

generate_transition_matrix() → DataFrame[source]

The transition matrix is the basis of the shifting algorithm. This method generates a (n x m) matrix with zero and one entries, where n is the number of contracts and m are start and end dates for the delivery periods. Hence, the matrix row vectors indicate the delivery periods of each contract. Note that the latest delivery end date is not displayed in the transition matrix.

Returns:

Transition matrix containing zeros and ones indicating delivery periods of individual contracts.

Return type:

pd.DataFrame

shift(transition_matrix: DataFrame) → DataFrame[source]

This method is the final step in the shifting algorithm. The transition matrix is inversed and multiplied with the forward price vector to obtain a non overlapping forward price vector.

\[f^{no} = T^{-1}\cdot f\]

Where:

  • \(f^{no}\) is the Non-overlapping forward price vector

  • \(T\) is the Transition matrix

  • \(f\) is the Forward price vector

Afterwards the PFC \(S(t)\) is obtained from the shape \(s(t)\) by the follwoing formular:

\[S(t) = s(t)\cdot \frac{\sum_{u=T_s}^{T_e} f^{no}(u)}{\sum_{u=T_s}^{T_e} s(u)}\]

with \(T_s\) and \(T_e\) being the start and end dates of the individual delivery periods.

Parameters:

transition_matrix (pd.DataFrame) – Full rank transition matrix generated by the generate_synthetic_contracts method

Returns:

Shifted shape.

Return type:

pd.DataFrame

Shaping

class rivapy.marketdata_tools.PFCShaper(spot_prices: DataFrame, holiday_calendar: HolidayBase, normalization_config: Dict[Literal['D', 'W', 'ME'], int | None] | None = None)[source]

Bases: FactoryObject

PFCShaper interface. Each shaping model for energy price forward curves must inherit from this base class.

Parameters:

  • spot_prices (pd.DataFrame) – Data used to calibrate the shaping model.

  • holiday_calendar (holidays.HolidayBase) – Calendar object to obtain country specific holidays.

  • normalization_config (Optional[Dict[Literal["D", "W", "ME"], Optional[int]]], optional) – A dictionary configurating the shape normalization periods. Here D defines the number of days at the beginning of the shape over which the individual mean is normalized to one. W defines the number of weeks at the beginning of the shape over which the individual mean is normalized to one. ME defines the number of months at the beginning of the shape over which the individual mean is normalized to one. The remaining shape is then normalized over the individual years.Defaults to None.

abstract apply(apply_schedule: List[datetime])[source]

Applies the model on a schedule in order to generate a shape for future dates.

Parameters:

apply_schedule (List[dt.datetime]) – List of datetimes in order to generate a shape for future dates.

abstract calibrate()[source]

Calibration of the shaping model

normalize_shape(shape: DataFrame) → DataFrame[source]

Normalizes the shape based on normalization_config.

D defines the number of days at the beginning of the shape over which the individual mean is normalized to one.

W defines the number of weeks at the beginning of the shape over which the individual mean is normalized to one.

ME defines the number of months at the beginning of the shape over which the individual mean is normalized to one. The remaining shape is then normalized over the individual years.

Example: D is 2, W is 2 and ME is 1. The shape starts at 03.03.2025 (monday). Since D is 2, the shape is normalized for 03.03.2025 and 04.03.2025 individually.

The weeks are normalized from 05.03.2025 to 09.03.2025 and from 10.03.2025 to 16.03.2025.

The month is then normalized from 17.03.2025 to 31.03.2025. The remaining shape (starting from 01.04.2025) is normalized on a yearly level.

Parameters:

shape (pd.DataFrame) – Shape which should be normalized

Returns:

Normalized shape

Return type:

pd.DataFrame

class rivapy.marketdata_tools.CategoricalRegression(spot_prices: DataFrame, holiday_calendar: HolidayBase, normalization_config: Dict[Literal['D', 'W', 'M'], int | None] | None = None, remove_outlier_season: bool = False, remove_outlier_id: bool = False, lower_quantile_season: float = 0.005, upper_quantile_season: float = 0.995, lower_quantile_id: float = 0.005, upper_quantile_id: float = 0.995)[source]

Bases: PFCShaper

Linear regression model using categorical predictor variables to construct a PFC shape. We follow the methodology in:

https://cem-a.org/wp-content/uploads/2019/10/A-Structureal-Model-for-Electricity-Forward-Prices.pdf

https://ieeexplore.ieee.org/document/6607349

https://www.researchgate.net/publication/229051446_Robust_Calculation_and_Parameter_Estimation_of_the_Hourly_Price_Forward_Curve

We create a regression model for bot the seasonality shape and the intra day shape. For the regression model of the seasonality shape, the days are split into weekday, Saturdays and Sundays. Public holidays are considered as Sundays while bridge days are expected to behave like Saturdays. Afterwards, weekdays are split into clusters representing the month they are in, while Saturdays and Sundays are assigned to clusters reaching over three months. For the regression model of the intra day shape we keep the seasonality clusters but add a hourly cluster for each individual hour such that the total number of intra day clusters becomes #Season Clusters * 24.

\[\begin{split}\begin{aligned} y_\text{season}(d) &= \frac{\frac{1}{24}\sum_{i=1}^{24}h_i^d}{\frac{1}{N_y}\sum_{i=1}^{N_y} h_i^y} \\ \hat{y}_\text{season}(d) & =\beta^{0}_\text{season} + \sum_{c \in C^\text{season}}\beta^c_{\text{season}}\cdot\mathbb{I}_{\text{Cluster}(d)=c} \\ y_\text{id}(h_i,d) &= \frac{h_i^d}{\frac{1}{24}\sum_{i=1}^{24}h_i^d} \\ \hat{y}_\text{id}(h,d) & =\beta^{0}_\text{id} + \sum_{c \in C^\text{id}}\beta^c\cdot\mathbb{I}_{\text{Cluster}(h_i^d)=c} \\ s(h_i,d) &= \hat{y}_\text{id}(h_i,d)\cdot\hat{y}_\text{season}(d) \end{aligned}\end{split}\]

where:

\(h_i^d\): i-th hour of d-th day

\(h_i^y\): i-th hour of the year \(y\)

\(N_y\): number of days in year \(y\)

\(C^\text{season}\): set of all clusters for the seasonality shape

\(C^\text{id}\): set of all clusters for the intra day shape

\(\text{Cluster}(X)\): returns the cluster of X

\(\mathbb{I}_x = \begin{cases} 1, & \text{if the } x \text{ expression renders true}\\ 0, & \text{if the } x \text{ expression renders false} \end{cases}\)

Parameters:

  • spot_prices (pd.DataFrame) – Data used to calibrate the shaping model.

  • holiday_calendar (holidays.HolidayBase) – Calendar object to obtain country specific holidays.

  • normalization_config (Optional[Dict[Literal["D", "W", "ME"], Optional[int]]], optional) – A dictionary configurating the shape normalization periods. Here D defines the number of days at the beginning of the shape over which the individual mean is normalized to one. W defines the number of weeks at the beginning of the shape over which the individual mean is normalized to one. ME defines the number of months at the beginning of the shape over which the individual mean is normalized to one. The remaining shape is then normalized over the individual years.Defaults to None.

  • remove_outlier_season (bool) – Wether to remove outliers for the seasonality shape regression. Defaults to False.

  • remove_outlier_id (bool) – Wether to remove outliers for the intra day shape regression. Defaults to False.

  • lower_quantile_season (float) – Lower quantile for outlier detection. Defauls to 0.005.

  • upper_quantile_season (float) – Upper quantile for outlier detection. Defaults to 0.995.

  • lower_quantile_id (float) – Lower quantile for outlier detection. Defauls to 0.005.

  • upper_quantile_id (float) – Upper quantile for outlier detection. Defaults to 0.995.

apply(apply_schedule: List[datetime]) → DataFrame[source]

Applies the model on a schedule in order to generate a shape for future dates.

Parameters:

apply_schedule (List[dt.datetime]) – List of datetimes in order to generate a shape for future dates.

calibrate()[source]

Calibration of the shaping model

class rivapy.marketdata_tools.SimpleCategoricalRegression(spot_prices: DataFrame, holiday_calendar: HolidayBase, normalization_config: Dict[Literal['D', 'W', 'M'], int | None] | None = None, remove_outlier: bool = False, lower_quantile: float = 0.005, upper_quantile: float = 0.995)[source]

Bases: PFCShaper

Linear regression model using categorical predictor variables to construct a PFC shape.

\[s(t) = s_0 + \sum^{23}_{i=1}\beta^h_i\cdot\mathbb{I}_{h(t)=i} + \beta^d\cdot\mathbb{I}_{d(t)=1} + \beta^H\cdot\mathbb{I}_{H(t)=1} + \sum^{12}_{i=2}\beta^m_i\cdot\mathbb{I}_{m(t)=i}\]

where:

\(s_0\): Shape level level

\(\mathbb{I}_x = \begin{cases} 1, & \text{if the } x \text{ expression renders true} \\ 0, & \text{if the } x \text{ expression renders false} \end{cases}\)

\(h(t)\): Hour of t

\(d(t) = \begin{cases} 1, & \text{if t is a weekday} \\ 0, & \text{if t is a day on a weekend} \end{cases}\)

\(H(t) = \begin{cases} 1, & \text{if t public holidy} \\ 0, & \text{if t is not a public holiday} \end{cases}\)

\(m(t)\): Month of t

Parameters:

  • spot_prices (pd.DataFrame) – Data used to calibrate the shaping model.

  • holiday_calendar (holidays.HolidayBase) – Calendar object to obtain country specific holidays.

  • normalization_config (Optional[Dict[Literal["D", "W", "ME"], Optional[int]]], optional) – A dictionary configurating the shape normalization periods. Here D defines the number of days at the beginning of the shape over which the individual mean is normalized to one. W defines the number of weeks at the beginning of the shape over which the individual mean is normalized to one. ME defines the number of months at the beginning of the shape over which the individual mean is normalized to one. The remaining shape is then normalized over the individual years.Defaults to None.

  • remove_outlier (bool) – Wether to remove outliers for the seasonality shape regression. Defaults to False.

  • lower_quantile (float) – Lower quantile for outlier detection. Defauls to 0.005.

  • upper_quantile (float) – Upper quantile for outlier detection. Defaults to 0.995.

apply(apply_schedule: List[datetime]) → DataFrame[source]

Applies the model on a schedule in order to generate a shape for future dates.

Parameters:

apply_schedule (List[dt.datetime]) – List of datetimes in order to generate a shape for future dates.

calibrate()[source]

Calibration of the shaping model

class rivapy.marketdata_tools.CategoricalFourierShaper(spot_prices: DataFrame, holiday_calendar: HolidayBase, normalization_config: Dict[Literal['D', 'W', 'M'], int | None] | None = None, k_fourier: int = 2, remove_outlier_season: bool = True, remove_outlier_id: bool = True, lower_quantile_season: float = 0.005, upper_quantile_season: float = 0.995, lower_quantile_id: float = 0.005, upper_quantile_id: float = 0.995)[source]

Bases: PFCShaper

Linear regression model using categorical predictor variables to construct a PFC shape. We follow the methodology in:

https://cem-a.org/wp-content/uploads/2019/10/A-Structureal-Model-for-Electricity-Forward-Prices.pdf

https://ieeexplore.ieee.org/document/6607349

https://www.researchgate.net/publication/229051446_Robust_Calculation_and_Parameter_Estimation_of_the_Hourly_Price_Forward_Curve

We create a regression model for bot the seasonality shape and the intra day shape. For the regression model of the seasonality shape, the days are split into weekday, Saturdays and Sundays. Public holidays are considered as Sundays while bridge days are expected to behave like Saturdays. Afterwards, weekdays are split into clusters representing the month they are in, while Saturdays and Sundays are assigned to clusters reaching over three months. For the regression model of the intra day shape we use a fourier series to model periodicities over each hour in a year. This way we can model the solar dip over the year more reliably.

\[\begin{split}\begin{aligned} y_\text{season}(d) &= \frac{\frac{1}{24}\sum_{i=1}^{24}h_i^d}{\frac{1}{N_y}\sum_{i=1}^{N_y} h_i^y} \\ \hat{y}_\text{season}(d) & =\beta^{0}_\text{season} + \sum_{c \in C^\text{season}}\beta^c_{\text{season}}\cdot\mathbb{I}_{\text{Cluster}(d)=c} \\ y_\text{id}(h_i,d) &= \frac{h_i^d}{\frac{1}{N_y}\sum_{i=1}^{N_y} h_i^y} \\ \hat{y}_\text{id}(h_i,d) & =\beta^{0,H(h_i^d)}_\text{id} + \sum_{k=1}^{K}\beta^{k,H(h_i^d)}_\text{id}\cdot\left(\sin(2\pi k\cdot t(h_i^d)) + \cos(2\pi k\cdot t(h_i^d))\right)\\ s(h_i,d) &= \hat{y}_\text{id}(h_i,d)\cdot\hat{y}_\text{season}(d) \end{aligned}\end{split}\]

where:

\(h_i^d\): i-th hour of d-th day

\(h_i^y\): i-th hour of the year \(y\)

\(N_y\): number of days in year \(y\)

\(H(h_i^d)\): hour (0-23) of \(h_i^d\) independent of the day

\(t(h_i^d)\): function which returns a number between [0,1] depending on the position of \(h_i^d\) in the respecitve year

\(C^\text{season}\): set of all clusters for the seasonality shape

\(K\): number of fourier partials

\(\text{Cluster}(X)\): returns the cluster of X

\(\mathbb{I}_x = \begin{cases} 1, & \text{if the } x \text{ expression renders true}\\ 0, & \text{if the } x \text{ expression renders false} \end{cases}\)

Parameters:

  • spot_prices (pd.DataFrame) – Data used to calibrate the shaping model.

  • holiday_calendar (holidays.HolidayBase) – Calendar object to obtain country specific holidays.

  • normalization_config (Optional[Dict[Literal["D", "W", "ME"], Optional[int]]], optional) – A dictionary configurating the shape normalization periods. Here D defines the number of days at the beginning of the shape over which the individual mean is normalized to one. W defines the number of weeks at the beginning of the shape over which the individual mean is normalized to one. ME defines the number of months at the beginning of the shape over which the individual mean is normalized to one. The remaining shape is then normalized over the individual years. Defaults to None.

  • k_fourier (int) – Number of partial sums for the fourier series .Defaults to 2.

  • remove_outlier_season (bool) – Wether to remove outliers for the seasonality shape regression. Defaults to False.

  • remove_outlier_id (bool) – Wether to remove outliers for the intra day shape regression. Defaults to False.

  • lower_quantile_season (float) – Lower quantile for outlier detection. Defauls to 0.005.

  • upper_quantile_season (float) – Upper quantile for outlier detection. Defaults to 0.995.

  • lower_quantile_id (float) – Lower quantile for outlier detection. Defauls to 0.005.

  • upper_quantile_id (float) – Upper quantile for outlier detection. Defaults to 0.995.

apply(apply_schedule: List[datetime]) → DataFrame[source]

Applies the model on a schedule in order to generate a shape for future dates.

Parameters:

apply_schedule (List[dt.datetime]) – List of datetimes in order to generate a shape for future dates.

calibrate()[source]

Calibration of the shaping model

times_to_zero_one(h: DatetimeIndex)[source]

Convert any time point into the corresponding fraction w.r.t. the beginning of the year it belongs to. 1st of January are always zeros, while 31th of December can be 1 / 365 or 1/366 according to the year. The idea is to map any year to [0,1), on which the seasonality curve, periodic on [0,1], is fitted.