Throughout recorded time, mankind has attempted to eliminate, or at least circumscribe, uncertainty and its fellow travelers: volatility, randomness, noise, risk. These qualities have often been cast as enemies of order, certainty and peace of mind. The sailor is averse to weather volatility. The investor is averse to economic uncertainty. Human beings generally seem to be genetically wired to try to avoid risk. To minimize risk, we try to hedge our exposure. But because we require some exposure to risk factors to generate returns, in practice we are never immune to unpleasant market surprises. The future is unknown and uncertain. Risk aversion boils down to minimizing uncertainty in the timing of these surprises.
We have learned a few things, however. Today we have become more effective at limiting near-term uncertainty by analyzing available information through a practice known as “nowcasting,” a term that marries “now” with “forecasting.” This does not make us fortune tellers or sages. Instead, nowcasting aims to improve the timing and robustness of real-time analysis and very short-term forecasting through statistical modeling and lots of data.
Nowcasting is the dynamic process of making short-term estimates of lagging target variables — that is, estimates of economic variables that are announced relatively infrequently and with long delays. In fact, some modelers define nowcasting as making predictions about not only the near-term future but also the present and the very recent past. What does this Zen-like statement mean? Many important factors that describe aspects of complex systems — from natural systems such as the weather, climate and tectonics to the macroeconomy to electronic signal processing — take time to gather and analyze. As there is often a significant delay in the information flow, by the time a provisional estimate is made (and often revised), we learn more about the recent past than about the present or future. Paradoxically, we have to “forecast” the recent past and present, as well as the future, to seek accurate results.
Large amounts of data are necessary to help deal with the randomness, or uncertainty, that often characterizes complex, dynamic systems. The flood of data produced by the digital world feeds machine learning algorithms, which can improve forecasting accuracy. There are, however, countervailing forces that could spoil this forecasting future.1 Complexity skyrockets with large amounts of data of different velocities, increasing variety and questionable veracity. Because every bit of data carries information, we aim to make use of all of it. Still, the complexity can sometimes overwhelm us, so there are limits to what we can do. A model cannot reasonably predict a totally random walk or the timing of a black swan event.
Nowcasting in Practice
Weather forecasting is an excellent and venerable example of nowcasting. To be even roughly accurate, weather forecasting must know the current weather in the neighborhood. This observation was first made by vice admiral Robert FitzRoy, who pioneered weather forecasting after the British passenger clipper Royal Charter famously sank in the midst of a vicious storm in 1859.2 (FitzRoy is most famous for serving as captain of the H.M.S. Beagle on a voyage that proved key to the development of a theory of evolution by a young scientist on board, Charles Darwin.) To help prevent such tragedies, FitzRoy came up with the first gale-warning service, a system based on real-time data collected from 15 land stations and rapidly transmitted and shared by the newly invented telegraph. He cleverly surmised that weather conditions in one part of England were indicative of future weather conditions at other locations. FitzRoy founded the British Meteorological Office, now known as the Met Office, which has been developing nowcasting techniques ever since.
Today, thanks to the evolution of meteorological science and data from weather satellites, we are much better at predicting the behavior of Mother Nature, at least over the short to medium term. Understanding future business, economic and market trends is, of course, a different, more difficult matter. For instance, central bank monetary policy decisions are based on a knowledge of current and future economic conditions, and on interest rates, currency rates and GDP growth. Central banks allocate resources to precisely estimate a broad range of economic metrics so they can make informed decisions on issues like monetary policy. These decisions, in turn, are vitally important to investors, who continually update their strategies as new information emerges.
The essence of nowcasting is the combination of a multiplicity of data generated at different times and different frequencies into a single dynamic model. It’s natural to think of a model as some form of a regression, which relates a low-velocity variable of interest — for instance, a target value, like GDP or the jobless rate — to one or a number of high-velocity variables, such as monthly, weekly or daily measures of industrial production, oil prices or business conditions. (Velocity is one of three characteristics of Big Data, along with volume and variability, and refers to the speed at which the data can be processed.) The objective is to read the real-time information inflow and predict the outcome.
The Importance of Big Data
We live in an age when the sheer amount of data is growing exponentially. A discerning observation by Buono et al. is that relationships between target variables and explanatory data can, and are likely to, vary over time.4 It is something that should be considered in modeling and why we refer to nowcasting as a dynamic factor model.
Besides these nontrivial dynamics, nowcasting data has other issues. The “jagged edge” problem arises when data is published at different times. This asynchronicity results in an incomplete set of data for the most recent period, with a jagged edge at the end of the sample. Forecasting methods that incorporate the most recent information have to pay close attention to this because almost all techniques require a complete dataset. For example, many macroeconomic variables are subject to substantial revisions after their first release, presenting a challenge to forecasters. The first release is a timely estimate based on limited information and is subsequently revised as more information becomes available; eventually, the information becomes “final.” This final value is the goal of forecasters, whereas preliminary data is just a noisy indicator.
A number of techniques have been developed to deal with the data issues that arise in nowcasting.
Typically, the target variables that we forecast are released infrequently and lag with respect to other information in the dataset, which could be based on news or surveys. The purpose of nowcasting is to use available data as early as possible to produce an estimate of the current state of the target variable before the actual announcement.
We are using the work done by Domenico Giannone and his colleagues over the past decade to “formalize” a nowcasting model.5, 6 We define the nowcasting of a target variable xt at time t as the orthogonal projection P of xt on the information set ΩT, which is the information available at time T in equation no. 1:
where E[xt lΩT] stands for the conditional expectation, or the expected value.
Data released asynchronously at various frequencies and with different time lags is a significant challenge to any prediction algorithm. The literature offers several solutions to this challenge. The issue was first tackled with the help of so-called partial modeling, in which individual equations are developed to explain the relationship between explanatory, or independent, variables and the observed value of interest. For instance, what relationship does the jobless rate, the value of the dollar or the inflation rate have to the quarterly GDP number? One example of partial modeling is so-called bridge equations, which run regressions among a number of subcomponent values (like the dollar or inflation) to relate, say, GDP to quarterly aggregates of one or a few monthly aggregates. Bridge equations can be mathematically represented by a series of regressions such as
where the value of interest yt is modeled using a linear regression of a predictor variable xt and a residual error term et. The explanatory variables in this case are aggregated to the lower frequency of the target variable — the value of interest. One approach for this low-frequency aggregation is known as “blocking,” in which high-frequency predictor data are split into multiple data series and updated as frequently as the target value. For example, in a system with monthly data and quarterly response frequency, the blocking approach creates three different time series for every monthly variable in the predictor data. To handle missing or lagging data, auxiliary models such as ARMA and VAR are used to close gaps in the predictor variables. ARMA, which stands for autogressive moving average, is a technique for understanding or predicting future values in a time series. VAR stands for vector autoregression and is used to capture interdependencies among multiple time series.
Another commonly used example of partial models in nowcasting is known as mixed-data sampling, or MIDAS. The difference between MIDAS and bridge equations is that MIDAS incorporates high-frequency observations with different frequencies and lags in the modeling equations. This approach relies on empirical estimates of the regression to set a weight for each observation in the model (irrespective of lag and frequency), in contrast to bridge equations, in which contributions of different data realizations are largely predetermined by a time-series aggregation of the data. The more general form of the MIDAS equation is
where B(L;θ) is a lag polynomial that sets different weights to the individual lagged instances of the predictor variable. This approach resolves the jagged-edge problem; the data series naturally realign in the equations, in which the most recent values for predictor variables are aligned in their order of appearance.
A significant breakthrough in nowcasting came with the introduction to the models of what’s known in statistics as state-space representation, a technique for analyzing systems with multiple inputs and outputs. State-space representation expresses variables as vectors — that is, as quantities that have direction and magnitude. In this representation, a vector of observed variables yt is assumed to be explained by a number of unobserved factors in the vector of xt. These “latent” or hidden factors follow an autoregressive relationship, which means that their current values depend on their past values.
The general form of the model is found in two equations:
where the vectors for gt and ht are Gaussian noise terms, a random component in the model that is assumed to be distributed in a statistically normal (Gaussian) way. Equation 4 expresses the relationship between predictor variables and the observed variable (the target value) and can be “realized,” or measurably observed, using a factor model or bridge equations. Equation 5 expresses the internal dynamics of unobserved predictor variables and can be realized using well-known autoregressive techniques such as ARMA and VAR.
The overall structure expressed in Equations 4 and 5 naturally fits into the framework of the so-called Kalman filtering technique, which can incorporate and keep track of an arbitrary number of mixed-interval data with very low complexity to the user. Kalman filtering, named for its inventor, electrical engineer and mathematician Rudolf Kálmán, provides a mechanism for using past observations to constantly improve the model. (For more on Kalman filters, see the appendix in the accompanying PDF.) A Kalman filter is an optimal estimation algorithm that uses a series of measurements observed over time, some of which contain statistical noise and other inaccuracies, to estimate unknown variables. Kalman filters estimate a joint probability distribution over the variables for each time frame and tend to be more accurate than those based on a single measurement. Kalman filtering also handles jagged data by providing model estimates for all predictor variables and automatically filling gaps that occur due to a lack of data.
The model developed by Giannone and his colleagues is able to produce accurate predictions without relying on human judgments. 7 By aggregating information from a large number of data series with different frequencies and different publication lags — for example, using jobless figures and trade figures — their model extracts signals about the direction of change in GDP before official numbers are published. The framework has been extensively tested in producing macroeconomic forecasts across datasets of various origin, such as large and small economies and developed and emerging markets.
This approach employs a dynamic factor model with a Kalman filter at its heart. The model is based on a Big Data framework that is able to capture and digest inflowing data as news. “News” is defined as the difference between released data — observations — and the model’s predictions. Because the Kalman filter itself is a form of weighted average, the forecast revision is the time-weighted average of the news during the week. This news is assimilated in the target variable, which attempts to mimic the way stock markets operate when news items are priced into share prices.
Following the Kalman strategy, we have a model with a state-space representation that is a predictor-corrector system of two types of equations. The measurement equations link observations to a latent-state process, while the transition equations describe the state-process dynamic.8, 9
The latent-state process infers the unobserved state of the economy, which is the higher-frequency counterpart of the target variable. Kalman filtering can then provide estimates for both observed and unobserved variables. This approach allows us to handle such significant nowcasting challenges as missing data and different data frequencies. Empirically, it comes as no surprise that employing high-frequency data is essential for the accuracy of the nowcast.
When data complexity increases and black swans arrive with greater frequency, forecasting becomes more storytelling than science. Nowcasting, however, has developed into an essential tool for real-time analysis based on facts, not stories. It digests high-velocity asynchronous data at arbitrary arrival rates to estimate a low-velocity target variable using statistical techniques. At present, nowcasting is unmatched by any other technique in its ability to handle complexity and make use of vast amounts of data.
Nowcasting has evolved significantly since its inception in meteorology in the 1860s. It began by using current events at one location to make a prediction about forthcoming events at a different location. Since then, scientists have realized that nowcasting can be performed using a wide range of data and different combinations of prediction techniques, such as linear regression, ARMA, VAR and Kalman filtering. Nowcasting is currently employed in a number of fields, including meteorology, air-quality forecasting, seismology and radar tracking. In finance, nowcasting has mainly been applied to predicting macroeconomic factors, such as GDP. The straightforward generalization of this technique to any data-modeling field suggests that very soon it could become the standard in data analytics.
The future is hard to predict, but the world has become so complex that even the present is often unknown to us. If we want to conquer the future, we need the right set of tools to master the present. Nowcasting is one of those tools.
Michael Kozlov is Senior Executive Research Director at WorldQuant, LLC, in Cambridge, Massachusetts, and has a Ph.D. in theoretical particle physics from Tel Aviv University. Svetoslav Karaivanov is a Senior Quantitative Researcher and has an MSc in computer science from ETH Zürich. Radoslav Valkov is a Senior Quantitative Researcher and has a Ph.D. in mathematics from the University of Antwerp. Dobroslav Tsonev is a Senior Quantitative Researcher and has a Ph.D. in digital communications from the University of Edinburgh.
1. Igor Tulchinsky. “The Age of Prediction.” WorldQuant Journal (2017).
2. Peter Moore. “The Birth of the Weather Forecast.” BBC News, April 30, 2015.
3. J.B. Rundle, D.L. Turcotte, A. Donnellan, L. Grant Ludwig, M. Luginbuhl and G. Gong. “Nowcasting Earthquakes.” Earth and Space Science 3, no. 28 (2016): 480-486.
4. Dario Buono, Gian Luigi Mazzi, George Kapetanios, Massimiliano Marcellino and Fotis Papailias. “Big Data Types for Macroeconomic .” Eurostat (2017).
5. Domenico Giannone, Lucrezia Reichlin and David Small. “Nowcasting: The Real-Time Information Content of Macroeconomic Data.” Journal of Monetary Economics 55, no. 4 (2008): 665-676.
6. Marta Banbura, Domenico Giannone and Lucrezia Reichlin. “Nowcasting.” European Central Bank Working Paper Series, no. 1275 (2010).
7. Marta Banbura, Domenico Giannone, Michele Modugno and Lucrezia Reichlin. “Now-casting and the Real-Time Data Flow.” European Central Bank Working Paper Series, no. 1564 (2013).
8. R.E. Kalman. “A New Approach to Linear Filtering and Prediction Problems.” Journal of Basic Engineering 82, no. 1 (1960): 35-45.
9. R.E. Kalman and R.S. Bucy. “New Results in Linear Filtering and Prediction Theory.” Journal of Basic Engineering 83, no. 1 (1961): 95-108.
Thought Leadership articles are prepared by and are the property of WorldQuant, LLC, and are being made available for informational and educational purposes only. This article is not intended to relate to any specific investment strategy or product, nor does this article constitute investment advice or convey an offer to sell, or the solicitation of an offer to buy, any securities or other financial products. In addition, the information contained in any article is not intended to provide, and should not be relied upon for, investment, accounting, legal or tax advice. WorldQuant makes no warranties or representations, express or implied, regarding the accuracy or adequacy of any information, and you accept all risks in relying on such information. The views expressed herein are solely those of WorldQuant as of the date of this article and are subject to change without notice. No assurances can be given that any aims, assumptions, expectations and/or goals described in this article will be realized or that the activities described in the article did or will continue at all or in the same manner as they were conducted during the period covered by this article. WorldQuant does not undertake to advise you of any changes in the views expressed herein. WorldQuant and its affiliates are involved in a wide range of securities trading and investment activities, and may have a significant financial interest in one or more securities or financial products discussed in the articles.