Some technical points about combining sigmas.

In the latest reports ATLAS is claiming 3.5σ local significance for their combined plot and 2.2σ global significance after ‘Look Elsewhere Effect’. For the diphoton channel alone they have 2.8σ local significance and 1.5σ global significance. Meanwhile over at CMS the figures are 3.1σ local significance and 2.1σ global significance for the combined plot, and for the diphoton channel they have also 3.1σ global significance and 2.1σ after LEE.

Now everyone wants to combine these numbers. How can that be done and what is the answer? Concentrating on the combined plots for the moment, a common method is just to add them in quadrature

s = \sqrt{s_1^2 + s_2^2}

giving \sqrt{3.5^2 + 2.8^2} = 4.5 for the combined local significance and \sqrt{2.2^2 + 1.5^2} = 2.7 for the global significance. Is this correct?

No that is wrong. Look elsewhere effect must be applied after combining.

The global significance is wrong because we have combined two results with LEE already applied. We should combine the local sigmas first and then apply LEE again. Well LEE is a subjective observer dependent quantity that nobody agrees about how to apply so lets just look at the local significance and let everyone estimate their own LEE afterwards. So are we correct for the global significance?

That is wrong too. The observed excesses were not in the same place.

It’s a good point. We can only combine the excesses at the same mass and the peaks of the excess differ by 1 or 2 GeV. If we do this we will get a smaller answer, but is that fair? The difference could be due to a systematic calibration error in one or other of the experiments. In fact this is looking increasingly likely as more data is added and the peaks do not get closer. We will have a much better idea about that when the data is doubled by the summer. So let’s be optimistic and just assume that the peaks will nearly coincide after recalibration. In that case we still have 4.5σ. Have we got it right now?

It is still wrong. Combining the numbers in quadrature is not the right formula.

If you think combining sigmas in quadrature is right, or even just approximately right, consider this scenario. Suppose in the first run of data I get a 2 sigma excess at some mass, but it is really just a statistical fluctuation. When the data is double we expect no excess at that mass so combine in quadrature to get \sqrt{2^2+0^2} = 2 so the excess is still two sigma. Double the data again and we might get a 1 sigma excess so the total significance is \sqrt{2^2+1^2} = 2.2  , even if we double the data again and get a deficit of one sigma below expected we add in quadrature to get \sqrt{2.2^2+(-1)^2} = 2.4 So if you believe that sigmas are added in quadrature you must believe that no excess can ever get smaller as more data is added. In fact they will probably grow like a random walk everywhere. This is obviously rubbish.

The reason for this is that these numbers are not error estimates that are normally added in quadrature. Have a look at this signal plot for the CMS and ATLAS signals in the diphoton channel

The excess for CMS is about 2.1 times the standard model signal plus or minus 0.6. For ATLAS it is 2.4 ± 0.7. These are not sigmas.  Those would be given by the size divided by the error so 2.1/0.6 = 3.5 sigma for CMS and 2.4/0.7 = 3.4 sigma for ATLAS (not quite right but I’ll come back to that). If we assume flat normal distributions then figures like these have to be averaged weighting by the errors. It is those errors which are combined in quadrature. For equal size data sets the errors should normally be the same which means that the correct formula for combining the sigmas is actually

s = \frac{s_1+s_2}{\sqrt{2}}

So redoing the calculation we get (3.5 + 2.8) \times 0.707 = 4.5 , the same answer.  In fact if the two sigma levels are similar this formula gives an answer very close to what you would get by adding in quadrature, so why should we care?

The present excesses in the diphoton channel are larger than predicted by the standard model with the Higgs boson at that mass. This excess is not as big as the excess over the standard model with no Higgs boson. It could be a sign that something non-standard is at work, but let’s assume that it is just a statistical fluctuation. In that case when we double the data we expect to get just the standard model signal for the second half of the data. In that case the signal next time will be given by (2.1 + 1.0) \times 0.707 = 2.2 for CMS and (2.4 + 1.0) \times 0.707 = 2.4  for ATLAS. In other words if the excess is due to a standard model Higgs boson then we should not expect much increase when the data is doubled. Don’t get your hopes up for a discovery by the summer. In fact the size of the signal in diphoton could easily go down. Even with quadruple the data it may not grow much bigger. Hopefully the combination with ZZ and WW will fare better because we have not seen the same over-excess in those channels and they will have a discovery by the end of the year, but don’t bank on it unless you think the over-excess is a real non-standard effect.

So do we have the right number of sigmas yet?

It is still wrong. You forgot the systematic correlations and have produced NONSENSE.

OK, but get serious. The previous unofficial combinations have shown that the correlations have a negligible effect on the combination when compared to official results. So the  combined result of 4.5 sigma still stands.

It is still wrong. The distribution is not flat normal. It is log normal.

Again this has been found to be a good approximation for doing the combination but there is a good point to be made here. Should be we read the number of sigmas off the plot when the CLs scale is linear or logarithmic? Have a look at these two plots which are the same thing on log and linear scale.

Remember that the green and yellow bands show one and two sigma deviations so the excess looks like three sigma on the log scale and four sigma on the linear scale. Which is right? If we assume the flat normal distribution is correct we should be using the linear scale but the bands are more equally spaced on the log scale, so presumably that is more correct.  The flat normal approximation is good for generating the plot but we should be careful to read the size of the excess from the log scale. if we do that will the answer be correct.

It’s still wrong. For best results use the combined p-value plot.

Have a look at what ATLAS and CMS are quoting for their local significance for the diphoton channel. CMS say 3.1 and ATLAS say it is 2.8. This does not match what you would get from reading either the linear or logarithmic plots. The numbers come from the p-value plots which are converted to sigma-equivalents. It looks like trying to get these numbers from the exclusion plots or the signal plots will never be that accurate. The bottom line is that we have to wait for the full official combination if we want to know the real answer. Until then adding in quadrature is probably as good as anything else. 🙂

2 Responses to Some technical points about combining sigmas.

  1. Still one proposal. Probably wrong;-).

    Let us start with random variable (x_1+x_2)/2 representing the mean from two series of observations and assuming that sigma_1 and sigma_2 defined the Gaussian distributions for the expectations.

    u=(x_1+x_2)/2 obeys Gaussian distribution which is product of two Gaussians with delta function constraint forcing u to given value.

    One can obtain the probability distribution for u by integrating either over x_2 or x_1 by using for instances x_1=2u-x_2. Doing the integration one obtains distribution for u which is Gaussian with

    sigma= sqrt(sigma_1^2+sigma_2^2)/2.

    For sigma_1=sigma_2 one obtains sigma_1/sqrt(2)

    The number of sigmas for u in combined data would be

    u= (n_1sigma_1+n_2sigma_2)/2 using sigma as unit giving


    For sigma_1=sigma_2 one would have

    n = (n_1+n_2)/sqrt(2)

    This is the formula that also Phil wrote with s denoting n.

    If sigma_1 and sigma_2 are not same, one cannot say anything definite: their ratio should be known. Sigmas characterize to some degree subjective expectations and it would be nice to know how the underlying Gaussian is deduced and can one be sure that the sigmas are same. Misunderstood something? Wise comments welcome;-)!

  2. Luboš Motl says:

    A good list of subtleties, Phil.

%d bloggers like this: