Effect of a resampling method on the effectiveness of multi-layer neural network models in PV power forecasting

The primary aim of this study was to explore the impact of employing the K-fold Cross Validation resampling method in contrast to the hold-out set validation approach on the efficacy of forecasting models utilizing Multi-layer Neural Networks (MNN) for predicting photovoltaic (PV) output power. Real data sourced from southern Algeria was utilized for this purpose. The performance of various configurations of MNN models, with differing learning rate values, was evaluated using the coefficient of variation of Root Mean Square Error (CV(RMSE)). The findings consistently demonstrate that models developed using K-fold Cross Validation exhibited superior performance across most scenarios. These results underscore the potential advantages of leveraging such resampling techniques in terms of both generalization and robustness of forecasting models.


Introduction
Nowadays, relying on renewable energy is a vital piece within the energy strategy adopted by many countries in order to fulfill CO2 reduction in response to global warming [1], [2].By 2026, global renewable electricity capacity is forecasted to rise more than 60% from the 2020 levels to a value over 4800 GW [3].Renewable energy offers numerous advantages, including sustainability and zero carbon emissions [4].However, its intermittent nature complicates its integration into power systems [5].High penetration of renewable energy sources (RES) significantly impacts power quality, volt-VAR control and protection systems [6].Forecasting tools play a crucial role in integrating RES into smart grids [7].Accurate long-term and short-term renewable generation forecasts are vital for ensuring effective power dispatch and operations [8].Photovoltaic (PV) power forecasting can be direct or indirect.Direct forecasting approaches are employed to directly predict the power production of photovoltaic (PV) systems [9].Numerous studies have centred on direct forecasting [10]- [12].Indirect PV power forecasting is categorized into three distinct areas: solar radiation forecasting, estimation of plane of array irradiance, and the utilization of PV performance models.These three categories represent the sequential stages of the methodology employed for indirectly forecasting PV power [9].Indirect PV power forecasting has been studied extensively [13], [14].
Artificial intelligence (AI) provides a powerful tool for monitoring and controlling the operation of renewable integrated power systems [15].Machine learning, a significant subset of AI, is widely used in many applications, including pattern recognition, spam filtering, classification, data mining problems and forecasting.Multi-layered artificial neural networks (MNNs) are machine learning algorithms commonly used to forecast and predict PV power both directly [16] and indirectly [17], [18].
Resampling approaches are vital in contemporary statistical analysis.This process involves iteratively selecting samples derived from a training set, followed by re-estimating a model of interest for each selected sample.This iterative approach provides additional insights on the fitted model [19].Cross-validation is a widely used resampling technique that assesses the efficacy of a statistical learning approach by estimating the test error associated with it [19], and examining the generalization of a predictive model.K-Fold Cross-Validation involves partitioning the dataset into k subsets.One subset is designated as the validation set, while the remaining subsets are utilized for training purposes.The robustness of the forecasting models may be enhanced by adopting this strategy during the modelbuilding phase [20].K-subsets are commonly employed as training subsets in various applications.Each of the k subsets serves as the validation set in turn, iterating the procedure k times.This ensures that the final outcome is not dependent on a single training dataset.
The classic holdout validation strategy has a significant drawback: the high variability of the validation estimates of the test error rate.This variability arises due to a dependence on the specific observations in the training and validation sets.Since only a subset of the observations, specifically those included in the training set rather than the validation set, are utilized for model fitting, the performance of statistical methods tends to decrease when trained on a smaller number of observations.Consequently, the error rate estimated from the validation set may overstate the error rate when the model is fitted on the complete dataset [19].This paper investigated the impact of adopting the k-fold cross-validation method over the traditional holdout validation method during the training phase on the accuracy of multivariate PV power forecasting models using Multi-layered Neural Networks (MNN).Real data gathered from the southern of Algeria was used for this study, with the performance of models evaluated using the coefficient of variation of the root mean squared error (CVRMSE).

Forecasting methodology
In our study we utilized different configurations of Multi-layered Neural Networks (MNN) to forecast PV power output.We examined the impact of using the k-Fold Cross-Validation resampling method compared to the traditional holdout validation method to evaluate the effect on the accuracy of MNN models.

Multi-layer Neural Network MNN
A multi-layered feedforward artificial neural network consists of at least three layers of nodes: input, hidden and output layers.Fig. 1a illustrates an MNN's general structure.This type of neural network operates under a supervised learning paradigm, requiring the provision of desired output values during the training process.Each layer is comprised of a multitude of interconnected nodes, which use an activation function (see Fig. 1b) to limit the overall output signal amplitude to a finite value, as mathematically formulated below: (

Training of MNN
Training a multi-layer feedforward neural network involves a sophisticated process aimed at enabling the network to learn complex patterns and relationships within the data.At its core, this process is a delicate interplay between mathematical optimization and statistical learning principles.In the context of predictive modeling the training involves the process of adjusting the weights and biases of the network to minimize the error between the predicted output and the actual output.This process is typically done using optimization algorithms, such as the back-propagation algorithm [23]; this algorithm calculates the gradient of the error signals which are propagated backward through the network to adjust the weights and biases efficiently.Adjusting the weights and biases in the direction that minimizes the error (loss function) is typically done using optimization algorithms such as stochastic gradient descent (SGD) or its variants.This process is usually repeated for multiple iterations (epochs) or until a stopping criterion is met, such as reaching a predefined number of epochs or observing minimal improvement in the validation loss.

Validation
Validation aims to periodically evaluate the performance of the network on a separate validation dataset to monitor for overfitting and adjust hyperparameters accordingly.

Hold-out set Validation
The hold-out set validation methodology is a commonly employed technique for estimating the test error of a statistical learning method when applied to a given sample set.The process entails the random partitioning of the given collection of samples into two distinct subsets, namely a training set and a hold-out set.The model is trained on the training set, and the trained model is employed to make predictions for the responses of the observations in the validation set [19].

k-fold cross-validation
The k-Fold Cross-Validation strategy involves the random partitioning of a set of observations into k groups, sometimes referred to as folds, with the aim of achieving roughly equal sizes for each fold.The initial fold is utilized as a validation set, while the technique is trained on the remaining k -1 folds.The statistical error, denoted as SE1, is subsequently calculated based on the data within the held-out fold.This process is repeated k times, with each iteration involving the selection of a distinct collection of observations to serve as the validation set.This procedure yields k estimations of the test error, denoted as SE1, SE2, …, SEk.The k-fold Cross-Validation estimate is calculated by taking the average of these values as shown in equation 2 [19].While the mean squared error is commonly used to compute Cross_Validation estimates, in our case we used the root mean square error (RMSE).

Testing
The testing phase aims to assess the performance of the trained model on a separate testing dataset to evaluate its generalization ability.Figure 2

Performance metric
In fact, predictive models rarely achieve perfect accuracy.
The main objective is to minimize the error associated with the forecast of the time series, as far as feasible.A wide range of metrics can help to determine the performance of predictive models, such as the mean squared error (MSE), the mean absolute error (MAE) and the root mean squared error (RMSE), which can be calculated with Equation 3.
Since MAE highly depends on the series values and RMSE error is more sensitive to occasional large errors, the coefficient of variation (CV), which is determined by the ratio between the standard deviation (σ) and the mean value of the evaluators, can overcome this issue [24], [25].
The lower the CV value, the better the model fit.One of the statistical methods included in ASHRAE Guideline 14 is the coefficient of Variation of Root Mean Squared Error CV(RMSE) [26], [27] which is a statistical metric for determining the overall accuracy of a predictive model which has been applied in PV power generation prediction [28].The CV(RMSE) can be calculated using Equation 5.

Forecasting steps
In this paper, the prediction was performed using MNN models trained with a back-propagation algorithm based on the Levenberg-Marquardt (LM) minimization method.This method adjusts the weights to ensure the network produces the required output for the given input data.
The multivariate time series model pertains to a model that encompasses two or more input variables, with observations being collected sequentially at regular time intervals.The system not only considers its own historical data but also integrates the historical data of other variables [29].
The use of additional meteorological variables as potential inputs to the forecasting model can enhance the forecasting process and the accuracy of the model; in our case we used eight meteorological variables with the measured PV power as inputs, namely: ambient temperature, four types of solar radiation (direct, global, reflect, diffuse), wind speed and direction, and humidity, whilst the output layer of MNN was the PV power output.The inputs and the output of the model are presented in figure 3. The selected data was divided into two parts: 80% for the training, 10% for validation, and the remaining 10% for evaluation of the model's performance.It is important to note that the optimal data split may vary depending on the specific application and dataset [30].Also studies have compared different architectures of ANN models and found that a specific configuration with a 80/10/10 data split achieved the best performance in predicting mortality among COVID-19 patients [31].
Performance of the forecasting involves several steps: 1-Data collection: Gathering relevant historical data of PV power and metrological data to ensure it is clean and consistent.

Location
The solar photovoltaic facility located near Oued Nechou, Ghardaïa, as shown in figure 5

Collected Data
The operational data used in this study was acquired in order to train and evaluate the proposed model.The available database includes measurements for ambient temperature, four types of solar radiation (G), wind speed and direction, humidity, and PV output power.Data were collected every 4 minutes, resulting in a dataset of 11,160 samples.Figure 6 displays the generated PV power by the system for the first 5 days (1,800 samples).The PV power data covers January.The dataset is partitioned into two subsets.Firstly, a training dataset covering January 1-28 (10,080 samples), used to construct the predictive models.Secondly, a testing dataset January 29-31 (1,080 samples), used to evaluate model performance.Despite the small amount of data covering a period of only one month, this dataset was chosen due to the unavailability of data covering a longer period.The primary objective is to compare the impact of different validation methods on the prediction models, rather than to build robust models to predict PV panel output at the study location.

Result and discussion
The neural network toolbox of MATLAB was used to assess the effect of adopting various resampling techniques on the performance of MNN predictive models.The feedforward neural network representation in MATLAB is shown in figure 7. from a location in southern Algeria.These techniques were evaluated and compared using the coefficient of variation (cv) metric described in section 2.3.The simulation results are presented as follows.

Evaluation of MNN PV power forecasting models
This section presents the results of adopting k-fold cross validation technique over the traditional holdout validation technique on the accuracy of MNN models in forevasting PV output power.The results compare the power forecasts to the test data for both developed models, MNN-hov and MNN-kfcv.The MNN models were built with a single hidden layer containing 10 neurons and trained with a learning rate of 0.01.The test data spanned three days, from January 29-31.
Results indicated that the PV power forecasting accuracy of MNN-kfcv significantly outperforms MNN-hov, with significant CV metric values of 0.44912 and 0.60627, respectively.Figure 8 show the forecasting results during the testing days.It can be observed that both MNN-hov and MNN-kfcv have a good fitting performance during these three days.However, the forecasting results of MNN-kfcv are more accurate, demonstrating that the kfold cross validation technique can enhance the predicted results of PV power.s  From the tables, it can be observed that MNN-kfcv generally outperformed MNN-hov across most configurations and learning rates.Figure 9 clearly demonstrates the superiority of MNN-kfcv over MNNhov.It is noteworthy, however, that in configuration 3 with a learning rate 0.005 and configuration 4 with learning a rate 0.01, that MNN-hov showed better performance compared to MNN-kfcv.

Conclusion
Resampling methods play a crucial role in statistical and data-driven modeling, as they help address issues such as overfitting, underfitting, and imbalanced data.Among these methods, K-fold cross validation is a well-known resampling method with the ability to reduce prediction variance, thereby enhancing the generalization of predictive models.The study conducted in this paper investigated whether employing k-fold cross validation resampling significant improves the performance of MNN-based forecasting models trained to predict PV output power.Real data collected from southern Algeria was utilized and the results compared to traditional holdout set validation, using the CV(RMSE) metric across various configurations of MNN models and learning rates.
The result indicates that MNN models with k-fold cross validation generally outperform hold-out set validation in the majority of cases.This underscores the usefulness of adopting k-fold cross-validation in developing predictive models, particularly in applications related to renewable energy, due to the ability to enhance model generalization and result in increased accuracy.

Figure 2 .
Figure 2. Hold-out set validation and k-fold cross validation process.

Figure 3 .
Figure 3.The inputs and output of the MNN model used in the study.

2 -
Data partitioning: splitting the dataset into training, validation, and testing dataset.3-Preparation models for training: preparing the models based on each validation method, including hold-out set validation and k-fold cross-validation.4-Model training: train the selected model using historical data.5-Model evaluation: Evaluate the performance of the trained model using performance metrics.The flowchart in figure 4 describes the forecasting steps and methodology used in this study.
is a key component of the program initiated by the supervisory ministry with the aim of promoting the advancement of renewable energy sources.The most recent enhancements to the facility include the incorporation of an interconnected photovoltaic system, and solar panels positioned at a 30° inclination.The solar panels are composed of various materials, including monocrystalline, polycrystalline, cadmium telluride (Cd-Te), and amorphous materials.

Figure 5 .
Figure 5. Geographical view of the studied location.

Figure 6 .
Figure 6.The first 5 days of PV power dataset.

Figure 7 .
Figure 7. feedforward neural network in MATLAB neural network toolbox.MNN models using holdout validation (MNN-hov) and kfold cross validation (MNN-kfcv) techniques have been developed to forecast PV power using real data collected

Figure 8 .
Figure 8. MNN-kfcv versus MNN-hov forecasting of PV power during the testing dataset period.

Figure 8 .
Figure 8. Performance evaluation of MNN-kfcv and MNN-hov models using CV(RMSE) with different configurations and learning rate levels.

Table I .
Performance evaluation with learning rate 0.01.TableII.Performance evaluation with learning rate 0.015.