ENSO Proxy Validation

This is a straightforward validation of the ENSO model presented at last December's AGU.

What I did was use the modern instrumental record of ENSO — the NINO34 data set — as a training interval, and then tested across the historical coral proxy record — the UEP data set.

The correlation coefficient in the out-of-band region of 1650 to 1880 is excellent, considering that only two RHS lunar periods (draconic and anomalistic month) are used for forcing. As a matter of fact, trying to get any kind of agreement with the UEP using an arbitrary set of sine waves is problematic as the time-series appears nearly chaotic and thus requires may Fourier components to fit. With the ENSO model in place, the alignment with the data is automatic. It predicts the strong El Nino in 1877-1878 and then nearly everything before that.

Continue reading

ENSO and Tidal SLH - A Biennial Connection

It's becoming abundantly clear that ENSO is driven by lunisolar mechanisms, especially true considering that we use the preferred models describing sloshing of water volumes -- the Mathieu equation and the delay differential equation. What's more, given the fact that the ENSO model works so well, one can guess that a connection between ENSO and lunar tidal forces should carry over to models for historical sea-level height (SLH) tidal data.

From my preliminary work analyzing SLH tidal data in 2014, I found an intriguing pattern relating to the same biennial pattern observed in the ENSO model. An extract from that post is reproduced below:

"The idea, related to delay differential equations, is to determine if it is at all possible to model at least part of ENSO (through the SOI) with data from a point in time in the tidal record with a compensated point from the past. This essentially models the effect of the current wave being cancelled partially by the reflection of a previous wave.

This is essentially suggesting that SOI = k f(t) - k f(t-\Delta t) where f(t) is the tidal record and k is a constant.

After some experimenting, a good fit is obtained when the current tidal data is set to 3 months ago, and the prior data is taken from 26 months in the past. To model the negative of ENSO, the 3-month old data is subtracted from the 26-month old data, Figure 1:"

Fig 1: Model of ENSO uses Tidal Gauge readings from Sydney Harbor.

This observation suggests a biennial pattern whereby high correlations in the time series occur for events observed close to two-years apart, with the difference taken up by the ENSO signal. The latter is at least partly due to the inverted barometer effect on SLH.

The sensitivity of the biennial effect is shown in the following contour diagram, where the peak CC is indicated and a slanted line showing the 2-year differencing points (table of the correlation coefficients here)

Focussing on the mechanisms for the correlation, the predictor can be reformulated as

f(t) = f(t-\Delta t) + SOI / k

This brings up an interesting question as to why this biennial signal apparently can't be detected via conventional means. A likely guess is because it gets obscured by the ENSO signal. And since the ENSO signal has never been adequately modeled, no one has an inkling that a biennial signal coexists in the mix. Yet this unconventional delay differencing approach (incidentally discovered with the help of machine learning) is able to discern it with a strong statistical significance.

The gist of this discovery is that a biennial tidal SLH motion exists and likely contributes (or at least identifies) the nonlinear biennial modulation driving the ENSO model described in previous posts.

Summary:

ENSO has elements of an annual teeter-totter. If you look through the research literature, you will find numerous references to a hypothesized behavior that one annual peak is followed by a lesser peak the next year. Yet, no evidence that this strict biennial cycle is evidence in the data — it's more of a hand-wavy physical argument that this can or should occur.

The way to model this teeter-totter behavior is via Mathieu equations and delay differential equations. Both of these provide a kind of non-linear modulation that can sustain a biennial feedback mechanism.

The other ingredient is a forcing mechanism. The current literature appears to agree that this is due to prevailing wind bursts, which to me seems intuitive but doesn't answer what forces the wind in the first place. As it turns out, only two parameters are needed to force the DiffEq and these align precisely with the primary lunar cycles that govern transverse and longitudinal directional momentum, the draconic and anomalistic months.

The ENSO model turns into a metrology tool

Note the following table and how the draconic and anomalistic values zone in on the true value when correlating model against data:

Fitting the DiffEq values progressively away from the true value for the lunar tidal cycle results in a smaller correlation coefficient.

Having one of these values align may be coincidence, but having both combine with that kind of resolution is telling. The biennial connection to SLH tidal measurements is substantiation that a biennial modulation is intrinsic to the physical process.

The ENSO model turns into a metrology tool

When the model is able to discern values of fundamental physical constants to high precision, it has ceased to become of hypothetical interest and transformed into one of practical significance and of essentially able to determine ground truth.

The ENSO model is a Mathieu differential equation with a biennial modulation and an added delay differential of one year. The DiffEq is this :

f''(t) +\gamma f'(t) + \omega_o^2 \cdot ( \alpha + \beta \cos( \pi t +\theta)) \cdot f(t) + K \cdot f(t-1{year}) = F(t)

This is solved with a straightforward differential expansion

f(t+\Delta) = A \cdot f(t) + B \cdot f(t- \Delta t) + C \cdot f(t- 2 \Delta t) + M \cdot \cos(\pi t + \theta) \cdot f(t) + K f(t- 1{year}) + F_{drac} (t) + F_{anom} (t)

The resultant f(t) is compared to the NINO34 time series by maximizing the correlation coefficient. All the DiffEq parameters are allowed to vary (constrained to the appropriate sign for feedback parameters) and also the two unknown forcing periods corresponding to what we hypothesize as the strongly seasonally-pulsed Draconic and Anomalistic cycles (the Tropical/Synodic/Sidereal cycle is not considered first-order as a global effect). With minimal filtering of the NINO34 signal over the range of 1880 to present day, the correlation coefficient reaches upwards of 0.65 with a clearly obvious peak matching.

The DiffEq iterative solver is only constrained such that the Draconic and Anomalistic periods are close to the known cycles of 27.21222082 and 27.55454988 days. So we constrain them to the following intervals and find out whether the solver homes in on the actual values.

27.2 \leq {Drac} \leq 27.3

27.5 \leq {Anom} \leq 27.6

This is respect to an average calendar year of 365.2422 days. (As an aside, this computation is so sensitive that knowledge of leap years has a real impact)

We also clamp the unknown phases to a known value of the node crossing (for Draconic) and of a known value of a perigee point (for Anomalistic). This does not influence the precise knowledge of the unknown cycle period but supplying this particular ansatz prevents the solver from wandering around too much and giving a strong hint as to where to phase align the two cycles.

Incredibly, the results after about an hour's worth of computation gives

Drac = 27.21178772 days

Anom = 27.55490106 days

This precision amount to predicting the Draconic lunar month period to within 37 seconds and the Anomalistic lunar month to within 30 seconds. This is with respect to a starting search window of 0.1 day or 2.4 hours, which is 8640 seconds, thus winnowing down the initial guess to a much finer resolution. So, if this was a random chance occurrence it would have a probability of (37/8640)*(30/8640) = 0.000015 to occur within that error margin. Remember that both periods match closely and are independent so the likelihoods are multiplied. That gives a 1 in over 60,000 chance of a random draw falling within that margin. Moreover, this value gives a strong peak in the correlation coefficient, falling off quickly away from these values. For a 30 second accumulated error per lunar month, this when propagated over ~1800 lunar months in a ~130 year interval will lead to a 15 hour error, or 15/24 out of 27 days, which is about a 0.15 radian propagated error. For comparison, 1.57 radians is needed to cause the phase to interfere destructively against the true value of the detected lunar month over this long interval. But cosine(0.15) is 0.99, which will give only a weak 1% of the worst case destructive interference.

As it stands the result is deterministically repeatable, as it will hone in on the same periods independent of a seed value. Evidently, the 1800 lunar months in the ENSO series is enough to provide a high resolution determination of its value -- this is the equivalent to taking sea-level height tidal readings over many months to extract the diurnal or semi-diurnal periods. The difference is that this ENSO analysis is detecting the monthly long-period tidal values and not the much more convenient daily tidal data.

The significance of the described model and the associated results should not be undersold. The ENSO time series record obviously tracks the primary lunar cycles to such an extent that the model becomes an extremely sensitive metrology tool for indirectly estimating the lunar periods. (The direct measurement is obviously done though visual or sensor-based lunar tracking techniques, the most rudimentary of which has been known for centuries)

And bottom-line, realistically anybody can do this computation -- as supplemental guide, a spreadsheet was supplied in the previous post . It's all data driven and there really is no hands-on manipulation except a nudge to point the computation in the right direction. Odds are in favor that everyone that tries will find the same result, much like a tidal analysis will generate an identical result for the primary constituent periodic factors.

Unfortunately, the perceptions of a tidal period contributing to a clearly regular and discernible pattern (see this) remains a powerful incentive to question the results.  Yet, no one is under the pretense that a non-linear model will produce an obviously periodic pattern either. It's really a case of a cyclic attractor that is stationary and stable,  and thus conducive to predictive modeling. Contrary to what the climate science deniers such as Judith Curry and Anastasios Tsonis assert, ENSO may not really be that complicated.

 

Canonical Solution of Mathieu Equation for ENSO

From a previous post, we were exploring possible solutions to the Mathieu equation given a pulsed stimulus.  This is a more straightforward decomposition of the differential equation using a spreadsheet.

The Mathieu equation:

f''(t) + \omega_0^2 (1 + \alpha \cos(\nu t)) f(t) = F(t)

can be approximated as a difference equation, where the second derivative f''(t) is ~ (f(t)-2f(t-dt)+f(t-2dt))/dt. But perhaps what we really want is a difference to the previous year and determine if that is enough to reinforce the biennial modulation that we are seeing in the ENSO behavior.

Setting up a spreadsheet with a lag term and a 1-year-prior feedback term, we apply both the biennial impulse-modulated lunar forcing stimulus and a yearly-modulated Mathieu term.

Fig. 1: Training (in shaded blue) and test for different intervals.

I was surprised by how remarkable the approximate fit was in the recent post, but this more canonical analysis is even more telling. The number of degrees of freedom in the dozen lunar amplitude terms apparently has no impact on over-fitting, even on the shortest interval in the third chart. There is noise in the ENSO data no doubt, but that noise seems to be secondary considering how the fit seems to mostly capture the real signal. The first two charts are complementary in that regard — the fit is arguably better in each of the training intervals yet the test interval results aren't really that much different from the direct fit looking at it by eye.

Just like in ocean tidal analysis, the strongest tidal cycles dominate;  in this case the Draconic and Anomalistic monthly, the Draconic and Anomalistic fortnightly, and a Draconic monthly+Anomalistic fortnightly cross term are the strongest (described here). Even though there is much room for weighting these factors differently on orthogonal intervals, the Excel Solver fit hones in on nearly the same weighted set on each interval.  As I said in a previous post, the number of degrees of freedom apparently do not lead to over-fitting issues.

One other feature of this fit was an application of a sin() function applied to the result. This is derived from the Sturm-Liouville solution to Laplace's tidal equation used in the QBO analysis  — which works effectively to normalize the model to the data, since the correlation coefficient optimizing metric does not scale the result automatically.

Pondering for a moment, perhaps the calculus is not so different to work out after all:

C93Q9eVWsAA8N4Q[1]

(from @rabaath on Twitter)

A bottom-line finding is that there is really not much complexity to this unique tidal formulation model of ENSO.  But because of the uniqueness of the seasonal modulation that we apply, it just doesn't look like the more-or-less regular cycles contributing to sea-level height tidal data.  Essentially similar algorithms are applied to find the right weighting of tidal factors, but whereas the SLH tidal data shows up in daily readings, the ENSO data is year-to-year.

Further, the algorithm does not take more than a minute or two to finish fitting the model to the data. Below is a time lapse of one such trial. Although this isn't an optimal fit, one can see how the training interval solver adjustment (in the shaded region) pulls the rest of the modeled time-series into alignment with the out-of-band test interval data.

Shortest Training Fit for ENSO

This is remarkable. Using the spreadsheet linked in the last post, the figure below is a model of ENSO derived completely by a training fit over the interval 1900 to 1920, using the Nino3.4 data series and applying the precisely phased Draconic and Anomalistic long-period tidal cycles.

Fig. 1 : The ENSO model in red. The blue BG region is used for training of the lunar tidal amplitudes against the Nino3.4 data in green. That data is square root compacted to convert it to an equivalent velocity.

Not much more to say. There is a major disturbance starting in the mid-1980's, but that is known from a Takens embedding analysis described in the first paper in this post.

Tidal Model of ENSO

The input forcing to the ENSO model includes combinations of the three major lunar months modulated by the seasonal solar cycle. This makes it conceptually similar to an ocean tidal analysis, but for ENSO we are more concerned about the long-period tides rather than the diurnal and semi-diurnal cycles used in conventional tidal analysis.

The three constituent lunar month factors are:

Month type Length in days
anomalistic 27.554549
tropical 27.321582
draconic 27.212220

So the essential cyclic terms are the following phased sinusoids

Continue reading

Solution to Pulsed Mathieu Equation

Here is a bit of applied math that I have never seen described before.  It considers solving a variant of the Mathieu differential equation, an unwieldy beast that finds application in models of fluid sloshing (among others). Normally the Mathieu equation is only solvable as a stimulus forcing function convolved against the transcendental Mathieu function. Tools such as Mathematica include the Mathieu function in their library, but any other solution would require a full numerical DiffEq integration.

This is the typical Mathieu equation formulation:

f''(t) + ( \omega_0^2 + B cos (\nu_0 t) ) f(t) = 0

But instead of this nonlinear time-varying DiffEq, consider that we replace the sinusoidal modulation with a delta pulse. This is precisely our model of ENSO that we are trying to create from first principles. The pulse represents a forcing impulse from the alignment of a lunar and solar tidal cycle.

f''(t) + ( \omega_0^2 + B \delta (t) ) f(t) = 0

Taking the Laplace transform, we quickly arrive at what looks like an ordinary 2nd-order differential equation solution, albeit with an interesting initial condition:

s^2 F(s) + \omega_0^2 F(s) + B f(0) = 0

After taking the inverse Laplace transform, we have:

f(t) = I(t) \ast B f(0) \delta(t)

where I(t) represents the impulse response of the first two terms, which is then convolved with a delta function evaluated at a value of f(t) for t=0. This is the only remnant of the nonlinear nature of the classical Mathieu function, in that the initial condition has a scaling proportional to the value of the function at that time. On the other hand, the solution to an ordinary DiffEq would not be dependent on the value of the function.

Thereafter we can extend this to a general solution; by creating a pulse train of delta functions we get this final convolution:

f(t) = I(t) \ast B \sum_{n=0}^{\infty} f(t - n T) \delta(t - n T)

where T is the pulse train period.

This is straightforward to evaluate for any pulse train -- all we have to do is keep track of the changing value of f(t) as we come across each pulse.

Fig 1: A pulse train with both annual and biannual contributions.

In the above figure, why does the annual pulse have a two-year periodicity? That's due to the alignment of a non-congruent tidal period with a seasonal pulse in terms of a Fourier series as described here and mathematically refined here.

If that is not intuitive, we can still consider it more of an anzat and see where it takes us.

For an ENSO time series, we an convolve the above delta pulse train with a set of known tidal periods of arbitrary amplitude and phase. For a fit trained up to 1980, this is the extrapolation post-1980:

Fig 2: Fit of pulsed Mathieu equation using a set of tidal periods as a forcing function. The impulse response here is a simple year-long rectangular window, i.e. the response has a memory of only a calendar year. This can be further refined if necessary. Yet, the projected time-series evaluated out-of-band from the fitting interval does a good job of capturing the ENSO profile.

The projected waveform matches the last 16 years very effectively considering how noisy the ENSO series is, and is very close to the fit over the entire interval. This is apart from the possible perturbation around the 1982 El Nino. The fact that the monthly tidal periods differ by slight amounts dictates that long multi-decadal intervals should be used for fitting. (In contrast, the model of QBO is primarily Draconic so the fitting interval can be much shorter)

Fig 3: Fit over the entire interval. The set of tidal periods applied was limited to the 3 major months (draconic, anomalistic, and sidereal/tropical) and the multiplicative combinations creating the fortnightly tides. This may not be enough to get the details right but erred on the side of under-fitting to establish the physical mechanism.

This approach is the logical follow-on to the wave transformed fitting approach that I had been using, most recently here and here. Both of these approaches are equally clever, which is a necessary ingredient when one is dealing with the unwieldy Mathieu equations. Moreover they both pull out the obvious stationary aspects of the ergodic ENSO time series.

The Chandler Wobble Challenge

In the last post I mentioned I was trying to simplify the ENSO model. Right now the forcing is a mix of angular momentum variations related to Chandler wobble and lunisolar tidal pull. This is more complex than I would like to see, as there are a mix of potentially confounding factors. So what happens if the Chandler wobble is directly tied to the draconic/nodal cycles in the lunar tide? There is empirical evidence for this even though it is not outright acknowledged in the consensus geophysics literature. What you will find are many references to the long period nodal cycle of 18.6 years (example), which is clearly a lunar effect. If that is indeed the case, then the behavior of ENSO is purely lunisolar, as the Chandler wobble behavior is subsumed. That simplification would be significant in further behavioral modeling.

The figure below is my fit to the Chandler wobble, seemingly matching the aliased lunar draconic cycle rather precisely, taken from a previous blog post:

cw

The consensus is that it is impossible for the moon to induce a nutation in the earth's rotation to match the Chandler wobble. Yet, the seasonally reinforced draconic pull leads to an aliasing that is precisely the same value as the Chandler wobble period over the span of many years. Is this just coincidence or is there something that the geophysicists are missing?

It's kind of hard to believe that this would be overlooked, and I have avoided discussing the correlation out of deference to the research literature. Yet the simplification to the ENSO model that a uniform lunisolar forcing would result in shouldn't be dismissed. To quote Clinton: "What if it is the moon, stupid?"

Continue reading

The ENSO Challenge

It's been quite a challenge decoding the physics of ENSO. Anything that makes the model more complex and with more degrees of freedom needs to be treated carefully. The period doubling bifurcation properties of wave sloshing has been an eye-opener for me. I experimented with adding a sub-harmonic period of 4 years to the 2-year Mathieu modulation and see if that improves the fit. By simply masking the odd behavior around 1981-1983, I came up with this breakdown of the RHS/LHS comparison.

Continue reading

Lindzen doth protest too much

Incredible that Richard Lindzen was quoted as saying this:

Richard Lindzen, the Alfred P. Sloan Professor of Meteorology at MIT and a member of the National Academy of Sciences who has long questioned climate change orthodoxy, is skeptical that a sunnier outlook is upon us.

“I actually doubt that,” he said. Even if some of the roughly $2.5 billion in taxpayer dollars currently spent on climate research across 13 different federal agencies now shifts to scientists less invested in the calamitous narrative, Lindzen believes groupthink has so corrupted the field that funding should be sharply curtailed rather than redirected.

“They should probably cut the funding by 80 to 90 percent until the field cleans up,” he said. “Climate science has been set back two generations, and they have destroyed its intellectual foundations.”

Consider the psychological projection aspect of what Lindzen is asserting. The particularly galling part is this:

“Climate science has been set back two generations, and they have destroyed its intellectual foundations.”

It may actually be Lindzen that has set back generations of atmospheric science research with his deeply flawed model of the quasi-biennial oscillation of equatorial stratospheric winds — see my recent QBO presentation for this month's AGU meeting.   He missed a very simple derivation that he easily could have derived back in the 1960’s, and that could have set a nice “intellectual foundation” for the next 40+ years. Instead he has essentially "corrupted the field" of atmospheric sciences that could have been solved with the right application of Laplace's tidal equations — equations known since 1776 !

The "groupthink" that Lindzen set in motion on the causes behind QBO is still present in the current research papers, with many scientists trying to explain the main QBO cycle of 28 months via a relationship to an average pressure. See for example this paper I reviewed earlier this year.

To top it all off, he was probably within an eyelash of figuring out the nature of the forcing, given that he actually considered the real physics momentarily:

Alas, all those millions of taxpayer funds that Lindzen presumably received over the years didn't help, and he has been reduced to whining over what other climate scientists may receive in funding as he enters into retirement.

Methinks it's usually the case that the one that "doth protest too much" is the guilty party.

Added: here is a weird graphic of Lindzen I found on the cliscep blog. The guy missed the simple while focussing on the complex.

richardlindzen

From climate scientist Dessler

From climate scientist Dessler