How to calculate the error of fitted parameters

[ptc-6023676]

Hi,

I am using genfit() to fit my experimental data to one function with 3 free parameters. Mathcad could give the best fit parameters very fast.

But how to calculate the error of these fitted parameters.

Thanks a lot.

[Werner_E]

Lookup and evaluate the built-in varaible ERR or calculate the residual sum of squares yourself.

[Werner_E]

As usual Richard is right.

ERR will be available if you determine the fit yourself using a solve block with minerr() but not if you use genfit().

[RichardJ]

That's not really the issue. What Jiao wants is the estimated errors in the parameters, not the error in the fit. Unfortunately, a reliable estimate of the errors in the parameters of a non-linear least squares fit is hard to obtain. In my experience (and many years ago I spent a lot of time going through the exercise of trying to get reliable estimates), so much so that I don't bother even trying any more.

[RichardJ]

If you are using Mathcad 15 look in the Data Analysis Extension Pack, "Quality of fit". Be aware that in some cases the error estimates can be very poor ones (it depends on how well conditioned your fitting problem is).

[ptc-6023676]

Hi Werner and Richard,

Thanks for your quick and nice advice. I have modified my program and figured out the error of the fitted parameters based on the instruction in Data Analysis Extension Pack (which is named as CI in my program). The fitting is very good, however, the error seems to be too big here, which is much larger than my expectation based on their physical meaning. I just start to use Mathcad and not sure about the programming.

Here, I attached my program, could you please help me to check it.

Many thanks.

[RichardJ · Accepted solution]

You are fitting the sum of two functions that only differ slightly in their shape, and your data has no sharp or distinguishing features that forces the fit to a well defined, unique, solution. Based on considerable experience, your fit is probably ill conditioned, and the estimated errors in your parameters are not reliable, but the errors may indeed be large. The real errors might even be worse than the estimates.

You eliminated the last point in the calculations of the residuals, but not in the fit itself. You can't do that, because the last point affects the fit. I would dump the last point altogether.

[ptc-6023676]

Since the function could not give a good fit, I have to assume there are two channels that give rise to the result.You should be right on this case. The fit condition is not so good here.

Also, I am really admire at your meticulous check.

[RichardJ]

Fitting the sum of broad, featureless, functions to broad, featureless, data is prone to a lot of error, especially when the functions are all of the same form. There may be ways to improve the situation, but I would need to know a lot more about your data. For example, if you can obtain multiple sets of data where you know, for physical reasons, that some (but not all; the data sets must be distinct in some way) of the parameters should have the same value in all data sets, you may be able to reduce the errors in the estimated parameters considerably.

[ptc-6023676]

Unfortunately, the data solely depende on the physical properties of the sample, and we do not apply any external condition to change it. So, we could not get multiple sets of data. That is the weakness of our technique.

It seems you have a good intuition at such work, maybe we could find a way to intruduce some extermal conditions to get more data.

[RichardJ]

Unfortunately, the data solely depende on the physical properties of the sample, and we do not apply any external condition to change it. So, we could not get multiple sets of data.

Can you say anything more about your data, and what it physically represents?

[ptc-6023676]

Ok, the data is the temperature dependence of the superfluid density of a superconductor. The theory i am using is a so-called two-gap BCS model, while the classical ona gap BCS model could not reproduce the experimental data well. The fitting parameters are the gap-weight of gap1, the size of gap1 and the size of gap2. You could find the defination of superfluid density and the meaning of gap size from wiki et al., if you are not familar with this topic.

[RichardJ]

My background is physics, but it's a long time since I looked at BCS theory . I see your point about not being able to change the sample though.

I have a question about the errors in your data. Do you expect the absolute errors in the gap measurements to be the same at all temperatures? I ask because of course that certainly does not appear to be the case when I look at the residuals. A related question is what is the estimated error in the temperature? It seems to me that either your errors in the gap measurement are not constant, or you have an appreciable error in the temperatures. Either way, you are violating the assumptions of the least squares fit. Your estimates of the parameters are therefore not the optimal estimates, and the estimates of the errors in te parameters are not reliable. What to do about it depends on the answers.

[ptc-6023676]

It is very nice that you have a physics background and know the BCS theory.

Yeah, you are right again. The error bar of the superfluid density should not be a constant, but the error of temperature should not change too much. A standard commercial thermometer may possess an error of ~1% in our measuring range. But the real superfluid density will be affected by the thermal fluctuations and vortex flux at high temperature, especially in the vicinity of superconducting transition temperature (Tc). That is why we see an increasing residual sum of square with increasing temperature. But we normally ignore these factors in a physicist's stlye.

A more deep thought will related to the theory, as you know the BCS theory works for the simple metals or alloys, while we are research much more complex electron correlated maters nowadays. The temperature dependence of the gap function and its anisotropy should give rise some deviation from the BCS model. But we have to use such classical theory to treat our data in order to get some general information.

It so nice to have such a deep discussion.

[RichardJ]

Yeah, you are right again. The error bar of the superfluid density should not be a constant, but the error of temperature should not change too much. A standard commercial thermometer may possess an error of ~1% in our measuring range. But the real superfluid density will be affected by the thermal fluctuations and vortex flux at high temperature, especially in the vicinity of superconducting transition temperature (Tc). That is why we see an increasing residual sum of square with increasing temperature. But we normally ignore these factors in a physicist's stlye.

In which case a weighted least squares fit will give a better estimate of the parameters. The attached worksheet shows how to do such a fit using minerr. It's also possible to do it using genfit, but for least squares fitting minerr is easier to set up and much more versatile.

A more deep thought will related to the theory, as you know the BCS theory works for the simple metals or alloys, while we are research much more complex electron correlated maters nowadays. The temperature dependence of the gap function and its anisotropy should give rise some deviation from the BCS model.

And that is exactly the problem with estimating the errors in the parameters. A least squares fit will give good estimates (I would even say surprisingly good estimates) of the parameters if the only source of error is random noise, and the fit is suitably weighted if the errors are not constant. The calculation of the estimated errors in the parameters should also be reasonable in such cases, although in my experience that is not always so. But if there is an error in the model, and the fit is ill-conditioned, all bets are off. Not only is it possible for the estimated parameters to have very large errors, but there is no way I know of to calculate any reasonable estimate of those errors. This can be true even if the error in the model is very small. It just depends on how badly conditioned the fit is. In your case the residuals do at least look random, and you are only fitting the sum of two curves, so your estimates may be OK. Depending on how much work it is to repeat the experiment, you might be able to get an estimate of the effects of random noise on the estimated parameters by repeating it several times, and fitting each data set. You could get some idea of the effect of systematic errors (i.e. a deviation of the model from the real physics) by deliberately distorting the data in some small way, and seeing how that affects the estimated parameters. Simple examples would be to add a small sloping line to the data, or to multiply the data by a line with an offset of 1 and a small slope.

[ptc-6023676]

Do you expect that we can use the same function 'confidence()' to get the error of the fitted parameters in the revised programe? In our case, genfit() and minerr() show no big difference, but I believe minerr is right mathematically.

It seems the big error is from the theoritical model other than the measurement, the data could be repeated from difference measurement.

[RichardJ]

The confidence function is applicable to any least squares fit, no matter how it was done. Minerr and genfit both do a least squares fit (in fact, they use the same algorithm: Levenberg-Marquardt). You can't use it for a weighted least squares fit though, because it doesn't take the weights into account. To calculate the confidence limits the long way, as shown in the data analysis extension pack, I tihnk you just use a weighted chisq, but I am not sure of that. what you could do is use the LeastsquaresFit function (also described in the extension pack), which allows weights and returns the confidence limits.

[ptc-6023676]

Thanks again.