Forecasting Chaotic Dynamics with Recurrent Neural Networks (RNNs)
Natural systems, ranging from climate and ocean circulation to organisms and cells, involve complex dynamics extending over multiple spatio-temporal scales. Centuries old efforts to comprehend and forecast the dynamics of such systems have spurred developments in large scale simulations, dimensionality reduction techniques and a multitude of forecasting methods. The goals of understanding and prediction have been complementing each other but have been hindered by the high dimensionality and chaotic behavior of these systems. We propose to utilise state-of-the-art recurrent neural network architectures for forecasting spatio-temporal chaotic systems build surrogate models and analyse dynamical characteristics (e.g. Lyapunov spectrum) of such systems. These methods can be either applied in the original space entailing the full information of the system’s state, or in the reduced order space, in a fully data-driven scheme or complementing equations from first-principles.
We introduced a data-driven forecasting method for high dimensional, chaotic systems using a Long Short-Term Memory (LSTM) recurrent neural network. The proposed LSTM perform inference of high dimensional dynamical systems in their reduced order space and are shown to be an effective set of non-linear approximators of their attractor. We demonstrate the forecasting performance of the LSTM and compare it with Gaussian processes (GPs) in time series obtained from the Lorenz 96 system, the Kuramoto-Sivashinsky equation and a prototype climate model. Initially, we collect data from each application and perform dimensionality reduction keeping the modes with the largest energy. In this way, we construct a reduced order observable from the dynamical system. We train the network to forecast the dynamics of this observable. The history of the observable is exploited (in the memory cells of the LSTM) to account for the missing information of the reduced order state inspired by the Taken’s theorem that states that the dynamics of the original system can be reconstructed in an embedding given by delayed versions of the reduced order observable. The LSTM networks outperform the GPs in short-term forecasting accuracy in all applications considered owing to the scaling of their training procedure to large data-sets. A hybrid architecture, extending the LSTM with a mean stochastic model (MSM-LSTM), is proposed to ensure convergence to the invariant measure in fully turbulent scenarios. This novel hybrid method is fully data-driven and extends the forecasting capabilities of LSTM networks.
Moreover, we analysed the effectiveness of various state-of-the-art RNN cells in forecasting the spatiotemporal dynamics of chaotic systems in both their original space and a reduced order observable. We performed a comparative study of Reservoir Computing (RC) and backpropagation through time (BPTT) algorithms for gated network architectures on a number of benchmark problems. We quantified their relative prediction accuracy on the long-term forecasting of Lorenz-96 and the Kuramoto-Sivashinsky equation and calculation of its Lyapunov spectrum. We discuss their implementation on parallel computers and highlight advantages and limitations of each method. We find that, when the full state dynamics are available for training, RC outperforms BPTT approaches in terms of predictive performance and capturing of the long-term statistics, while at the same time requiring much less time for training. However, in the case of reduced order data, large RC models can be unstable and more likely, than the BPTT algorithms, to diverge in the long term. In contrast, RNNs trained via BPTT capture well the dynamics of these reduced order models. This study confirms that RNNs present a potent computational framework for the forecasting of complex spatio-temporal dynamics.
2018
- P. R. Vlachas, W. Byeon, Z. Y. Wan, T. P. Sapsis, and P. Koumoutsakos, “Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks,” P. Roy. Soc. A-Math. Phy., vol. 474, iss. 2213, p. 20170844, 2018.
[BibTeX] [PDF] [DOI]@article{vlachas2018a, author = {Pantelis R. Vlachas and Wonmin Byeon and Zhong Y. Wan and Themistoklis P. Sapsis and Petros Koumoutsakos}, doi = {10.1098/rspa.2017.0844}, journal = {{P. Roy. Soc. A-Math. Phy.}}, month = {may}, number = {2213}, pages = {20170844}, publisher = {The Royal Society}, title = {Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks}, url = {http://www.cse-lab.ethz.ch/wp-content/papercite-data/pdf/vlachas2018a.pdf}, volume = {474}, year = {2018} } - Z. Y. Wan, P. R. Vlachas, P. Koumoutsakos, and T. P. Sapsis, “Data-assisted reduced-order modeling of extreme events in complex dynamical systems,” PLoS ONE, vol. 13, iss. 5, pp. 1-22, 2018.
[BibTeX] [PDF] [DOI]@article{wan2018a, author = {Zhong Y. Wan and Pantelis R. Vlachas and Petros Koumoutsakos and Themistoklis P. Sapsis}, doi = {10.1371/journal.pone.0197704}, journal = {{PL}o{S} {ONE}}, month = {may}, number = {5}, pages = {1-22}, publisher = {Public Library of Science}, title = {Data-assisted reduced-order modeling of extreme events in complex dynamical systems}, url = {http://www.cse-lab.ethz.ch/wp-content/papercite-data/pdf/wan2018a.pdf}, volume = {13}, year = {2018} }