Study on simulated data

Data with no information

It would be very worrying to discover that AGNOSTIC is able to extract information where there is none. This is why we first want to test its behaviour on a random set of 100 simulated assets, all based on a geometric brownian motion with expected return of 0% and a constant annualized volatility of 15%. Although some scenarios may lead to decent realized performance, the algorithm is at a loss to profitably exploit opportunity as the odds are steeped against it.

100 random assets with \(\mu\)=0% and \(\sigma\)=15%

If we generate an independent second set of random assets, unsurprisingly AGNOSTIC produces a random result. Were we to run more such simulations and average all the results, AGNOSTIC would show a realized return of 0%, exactly in line with the specified expectation.

100 random assets with \(\mu\)=0% and \(\sigma\)=15%, different scenarios

Data with information

We keep the setting with 100 random assets, but this time assets numbered 91 to 100 have positive expectations. We choose a Sharpe Ratio of 2 (an average daily return of around 12bps). Immediately visible on the plot, AGNOSTIC is able to concentrate recommended weights on assets 91 to 100 (remember that only the most significant weights are shown in our plots). The performance track of AGNOSTIC is revealing, as it spends most of its time well above the 90th percentile of performance of all underlyings.

random assets 1 to 90 with \(\mu\)=0% and \(\sigma\)=15%, random assets 91-100 with SR=2 and \(\sigma\)=15%

In response to the above result, we typically are asked how AGNOSTIC compares with traditional tools to identify trends. Once you have access to the full dataset of random scenarios presented above, it is an easy task to design a model able to extract performance efficiently. In this particular case, the best strategy is to equal weight the ten assets with the best cumulated performance since inception. Only, it is worth remembering that on the very first day of this dataset, no one knew:

  • how many assets had an expected return above zero,
  • expected returns would stay the same throughout the period,
  • volatilities would be equal and constant throughout the years.

Neither did AGNOSTIC know. Which did not prevent the algorithm from exploiting available performance as of day one. This demonstrates AGNOSTIC is able to start immediately with the available information and adapt as additional information is released, in contrast to the quant approach which first needs complete access to the dataset in order to design a model, which in this example would become usable only after… 14 years! A bit late in the game.

One may object it is possible to develop a model that relies on a (shorter) rolling window of data, to be able to exploit performance sooner. This is true, but then how does the quant choose the length of the said rolling window? By setting another rolling window? Or better by relying on cross-validation? But how then does he choose the parameters related to cross-validation? Parametric models naturally call for parameters. AGNOSTIC is parameter-free.

We have heard clients mention they address this issue by establishing “gold” standards, such that in-house quantitative research could be guided. Some time ago, this had involved careful analysis to select the tools / models of choice and set their parameters. Hence no overfitting was possible.

Unfortunately such methodology does not reduce overfitting. In this context, we put forward that AGNOSTIC is just that: agnostic; i.e. open by nature, uncommitted to any idea in particular and happy to provide recommendations on how to achieve coming close to the best when fed with the returns of selected underlyings. It could therefore consider the performance of your tools / models of choice alongside the performance of other inputs. If it can do better than your tool / model, it will recommend to allocate elsewhere. If it cannot, it will happily recommend to allocate to your tools / models of choice, together with a guarantee that if it comes out as the best in the long run AGNOSTIC will do almost as well. Comes a time in the future when AGNOSTIC thinks it can do better by allocating to others, it will do so with no second thoughts. Also noteworthy is AGNOSTIC’s progressive approach when recommending allocation weights, in contrast to a typical all-in or all-out implementation as a consequence of the typical glorifying and vilifying of investments associated with fads, fashion and conventional wisdom.

Data with changing information

You may actually wonder what happens if non-stationarities are introduced in the above mentionned simulations. What if, on 1 January 1960, assets numbered 91 to 100 experience a sudden drop of their Sharpe Ratios from 2 to 0, whilst assets numbered 1 to 10 experience the opposite?

AGNOSTIC is not perturbed about the change. Although a period of adaptation is required, it is quick to discard what does not live up to its expectations and reconsider its priors to settle on overweighting assets numbered 1 to 10 at the expense of assets numbered 91 to 100.

random assets 1-10 with SR=0 from 1950 to 1959 and SR=2 from 1960 and \(\sigma\)=15%, random assets 11 to 90 with \(\mu\)=0% and \(\sigma\)=15%, random assets 91-100 with SR=2 from 1950 to 1959 and SR=0 from 1960 and \(\sigma\)=15%

Let’s now recall the optimal methodology which we discussed earlier. What if, after analysis of data up to 15 December 1959, the decision was taken to invest in an equally weighted portfolio of the ten assets with the best cumulated performance since inception? The result would have been disappointing because it would take a long time before the cumulated performance of assets 1 to 10 overtake the cumulated performance of assets 91 to 100. The rolling window approach mentioned above would have fared better, though at an expense of a choice of parameters, hard to come by without knowing the return data in anticipation.

Finally it is worth noting that, would you have added one or both equal weighting or rolling window tools to the opportunity set of AGNOSTIC, it would have quickly discarded them with no harm done.