cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - Did you get an answer that solved your problem? Please mark it as an Accepted Solution so others with the same problem can find the answer easily. X

non linear regression best fit references?

TyeAldana
2-Explorer

non linear regression best fit references?

I have noisy data that I filtered using the medfilt1d smoothing function. Knowing the physics of the recorded data I know it should behave sinusoidally. If I try to do a sinfit curve fit to the filtered/or non filtered data I get a no convergence error. I have ok results using the linfit function with corr() results in the 0.85 range. I'm now looking into using the LeastSquaresFit function, and have not found much informative quicksheets or ebook examples about using this function. Can anybody provide some resources?
24 REPLIES 24

Post the worksheet in MathCad 11 or higher, and we can select from a lot of smoothing functions that will work OK. A Fourier series seems likely to be best.

sinfit is very limited, as it does not fit the period. It works only with data with period of 2π radians.

I have no heard of a function LeastSquaresFit. If you have seen such, it would be a user defined function and not a part of Mathcad. The usual tool for doing least squares fits is minerr.

Don't smooth before fitting. Fitting is an alternative to the various ad hoc smoothing functions.
__________________
� � � � Tom Gutman

>I have noisy data that I filtered using the medfilt1d smoothing function <<br> ______________________________

A dramatic misconcept to filter data set before fitting. A more dramatic case if the collected data are not raw, raw meaning "raw" ! Curve fitting expertise is about unlimited in this collab. "Save as" at least as low as 11, and just the data table.

jmG

On 8/10/2009 5:44:39 PM, tyealdana wrote:
>I have noisy data that I
...

Not having any real data to test, this naive approach is a blind shot, but maybe you can use something similar.

The equations in the Given/Find block are derived from the condition of minimizing the squares of errors.

Saludos,
Al

There are a few problems with this sheet.

Minerr is defined to do a least squares solution, there is no need to work out the least squares equations for each special case. Just use the model equations as is.
__________________
� � � � Tom Gutman

I have used genetic algorithm for solvung one Reverse Chemical Kinetics task!
Val
http://twt.mpei.ac.ru/ochkov/v_ochkov.htm

On 8/10/2009 11:12:22 PM, Al2000 wrote:
>On 8/10/2009 5:44:39 PM, tyealdana
>wrote:
>>I have noisy data that I
>...
>
>Not having any real data to test, this
>naive approach is a blind shot, but
>maybe you can use something similar.
>
>The equations in the Given/Find block
>are derived from the condition of
>minimizing the squares of errors.
>
>Saludos,
>Al
________________________

More in there.

jmG



Adding noise is a quick gadget. There are several good models for the "white noise", including Stuart "twister" [am I right here ? ]. A classical model is attached.

jmG

Thanks for the input. I should have noted I'm using MCad 14. FYI, I've included function descriptions of what I discussed in my original question. My data file is large, so I'm not sure how to attach the MCad file showing the characteristics I'm dealing with. I should add that the data was recorded at 100Hz sample rate for about 6 minutes. Over the whole 6 minute period the expected sinusoid will have completed ony 60 degrees of the 360 degree period. This was my reasoning to justify using the median filter prior to curve fitting.
---------

The LeastSquaresFit solver gives the most flexibility for solving nonlinear regression problems. It allows you to enter constraint equations for any of the dependent parameters, lower and upper bounds on parameters, standard deviations on the x values, as well as a confidence limit for the calculation. With so many arguments, it's best to explain this function by example.

LeastSquaresFit(vx, vy, F, guess, conf, [Stdy], [LBUB], [Acc]) Takes real vectors vx and vy of identical length, a fitting function F(x,b) of one variable with an arbitrary number of parameters, b, a guess vector with one guess value for each parameter, the desired confidence limit conf, expressed as a number between 0 and 1, inclusive. An additional optional vector of standard deviations in y, Stdy, the same length as vy, an optional two-column matrix of lower and upper bounds on the parameters, LBUB, with the same number of rows as guess; and an optional convergence accuracy Acc, may be specified. Returns a three-column matrix, where the first column contains the values for the fitted parameters, and the second and third columns contain the left and right boundaries, respectively, for the confidence interval on the parameters.

-----------

medfilt1d, however, is not equivalent to a linear filter.

� medfilt1d differs from the Mathcad function medsmooth. It can work on either vectors or matrices, does not smooth the residuals and add the result to the smoothed data, and it handles the beginning and end of the signal by using an effective filter length that guarantees that the filter is completely contained inside signal.

Median filtering replaces each element of the input with the median of the element and n-1 of its neighbors, or fewer near the ends of the signal: a good method for smoothing while maintaining edges, as well as for noise reduction. The function takes some time on large matrices because it performs a sort at each pixel. In addition to removing the noise, median filtering also tends to remove sharp, small transients from a signal. Median filtering is a good way to filter certain types of noise, like random salt-and-pepper noise, because it does not change the intensity levels in the signal; it only uses available levels.

-------

"LeastSquaresFit(vx, vy, F, guess, conf, [Stdy], [LBUB], [Acc]) ....." DAEP
__________________________

Just a gadget of no use, invented by dreamers. If you filter a data set, you can't fit. Filtering a data set belongs to a "Decision making" to best suit an unfittable data set that is too corrupted or wrongly collected. It would indicate plant defectuosity in some parts of the process or control. No collab can help you + w/o the data set itself. Put your data set in a Mathcad input table and attach if not exceeding 2 MB. If too big, decimate by �, by 1/3, � ... etc, up until you have < 2 MB. "Save as" 11 or lower version.

jmg

jmg, you are a bit cryptic, but I find your Dilbert-like attittude humorous. I'll figure out a way to post the data file.

On 8/11/2009 12:25:39 PM, tyealdana wrote:
>jmg, you are a bit cryptic,
>but I find your Dilbert-like
>attittude humorous. I'll
>figure out a way to post the
>data file.
_____________________________

Make it simpler:
put your data set in an Excel sheet, check the size < 2MB. From there, you will have plenty of replies. Yes, maths are cryptic ... numerical maths even more cryptic.

jmG



Your data set is not all that large in terms of posting -- only about 36000 numbers. You should be able to post the file in whatever format it is originally recorded in, usually text or Excel file. Mathcad can read either.

What sort of theory do you have, and what is known and unknown? Is the frequency known xactly from the theory or is that something that needs to be fitted? How well do you know the phase of the part of the data that you sampled? Is your theory in the form of an explicit equation, or is it an ODE, with an expected periodic solution? You can directly fit ODEs to data. And is the expected shape purely periodic, or is there an exponential decay (or growth) in there as well?

Again, don't smooth before fitting. Smoothing is itself a type of fit, and merely removes information (hopefully mostly, but never only, irrelevant) from the data.

Why such a high sampling rate? Usually the sampling rate is chosen to capture features on a meaningful time scale. But here the only relevant feature you've described has a time scale of 36 minutes. The Nyquist criterion requires only one sample every 18 minutes, a sample every minute is better then tenfold oversampling, more than adequate to characterize a signal with a 36 minute period. Are there sub-second features of interest in the signal?
__________________
� � � � Tom Gutman

If your data is periodic wouldn't a Fourier filter be better for smoothing prior to fitting?

On 8/11/2009 12:52:34 PM, mikej1980 wrote:
>If your data is periodic
>wouldn't a Fourier filter be
>better for smoothing prior to
>fitting ?
______________________________

NO ! Fourier filtering and fitting are like cat and onion.

Again: attach the data set.

jmG



On 8/11/2009 1:16:18 PM, jmG wrote:
>On 8/11/2009 12:52:34 PM, mikej1980
>wrote:
>>If your data is periodic
>>wouldn't a Fourier filter be
>>better for smoothing prior to
>>fitting ?
>______________________________
>
>NO ! Fourier filtering and fitting are
>like cat and onion.
>
>Again: attach the data set.
>
>jmG
>
>
>

Why do you say that? He says the recorded data should be periodic. If his noise is random his data should be easily *smoothed* using Fourier transform. Once the data is smoothed he can fit it.

Is this another example of calling black white?

On 8/11/2009 2:38:45 PM, mikej1980 wrote:
>On 8/11/2009 1:16:18 PM, jmG wrote:
>>On 8/11/2009 12:52:34 PM, mikej1980
>>wrote:
>>>If your data is periodic
>>>wouldn't a Fourier filter be
>>>better for smoothing prior to
>>>fitting ?
>>______________________________
>>
>>NO ! Fourier filtering and fitting are
>>like cat and onion.
>>
>>Again: attach the data set.
>>
>>jmG

>Why do you say that? He says the
>recorded data should be periodic. If his
>noise is random his data should be
>easily *smoothed* using Fourier
>transform. Once the data is smoothed he
>can fit it.
>
>Is this another example of calling black
>white?
>________________________

"Once the data is smoothed he can fit it".
3 of the most expert fitters in this collab have immediately said NO, don't smooth before fitting . All smoothing methods are destructive and not preservative of the noise. And the best Fourier filter don't exist in Mathcad, OK, a missing tool. Fitting is an art, expecting the tool box to help.
Please wait for the data set and some immediate comments. Noise and local/temporal perturbations aren't the same. Attached is an example of decision making.

jmG




Regarding decision making: the question I posted was whether a Fourier filter would be better than the median filter, i.e. given that the poster said that it was a two step process, smooth then fit, and that the data was expected to be periodic ...and we now know that there are 36000 points which should be enough for a good power spectrum.

The question of whether or not to smooth before fitting is a different and separate question and I'll defer to those who know more about it.

>>and we now know that there are 36000 points which should be enough for a good power spectrum.<<

Actually not. There are two aspects to the power spectrum -- the frequency range covered and the resolution (steps between frequencies). The resolution depends not on the number of points but on the time interval covered. The number of points (for a fixed time interval) controls the frequency range covers (the Nyquist frequency). With a time interval that is only one sixth of a cycle, the frequency resolution will be about six times the fundamental frequency. Not a good spectrum at all. But with 100Hz sampling it will cover frequencies up to 50Hz -- quite a large range compared to the fundamental frequency of .00046Hz.
__________________
� � � � Tom Gutman

On 8/11/2009 6:30:27 PM, mikej1980 wrote:
>Regarding decision making: the
>question I posted was whether
>a Fourier filter would be
>better than the median filter,

==> the decision making example is ksmooth,
it won't be any other filter

>i.e. given that the poster
>said that it was a two step
>process, smooth then fit,

==> "smooth and fit"
No one can tell that in advance

> and that the data was expected to
>be periodic ...

==> usually, the periodicity is detectable
in the total ignorance of "fit, smooth ..."

>and we now know that there are 36000 points
>which should be enough for a good power spectrum.

==> no mention was made of power spectrum
only "fit" the data set.


>The question of whether or not
>to smooth before fitting is a
>different and separate
>question and I'll defer to
>those who know more about it.

==> The data set is long awaited at least for inspection.

"I have noisy data" [the poster].

What is "noise": the phenomenon ? the sensor ? the digitization ? Instability ? not enough decimals ? insufficient setup ? loop scan de-synchro ? white noise ? ...

jmG

On 8/11/2009 10:21:59 PM, jmG wrote:
>On 8/11/2009 6:30:27 PM, mikej1980
>wrote:
>>Regarding decision making: the
>>question I posted was whether
>>a Fourier filter would be
>>better than the median filter,
>
>==> the decision making example is
>ksmooth,
>it won't be any other filter
>
>>i.e. given that the poster
>>said that it was a two step
>>process, smooth then fit,
>
>==> "smooth and fit"
>No one can tell that in advance


read the original post. The guy filtered then tried to fit. What is it with you?


>
>> and that the data was expected to
>>be periodic ...
>
>==> usually, the periodicity is
>detectable
>in the total ignorance of "fit, smooth
>..."
>
>>and we now know that there are 36000 points
>>which should be enough for a good power spectrum.
>
>==> no mention was made of power
>spectrum
>only "fit" the data set.


actually he mentioned filter and fit. I suggested a Fourier filter and if you used that you'd want to know about the power spectrum.

At least Tom Gutman can post rational thoughtful contributions about this, and I thank him for informing me about other aspects I hadn't considered.


The Maple newsgroup was destroyed by the ramblings of a strange person. Hope that doesn't happen with this forum.

On 8/12/2009 12:02:14 PM, mikej1980 wrote:
>On 8/11/2009 10:21:59 PM, jmG wrote:
>>On 8/11/2009 6:30:27 PM, mikej1980 wrote:
..............
"The guy filtered then tried to fit".
_______________________

The 3 global replies are: fit and don't filter before fitting.
Any eventual need for filtering will appear by inspection. Wait for the data set that may have to be collected again whereas the filtering is a destructive process, i.e: irreversible.

jmG


While I've abandoned usenet, I have spent time in various usenet groups. My observation is that no individual, whether troll or twit, can destroy a newsgroup. What can destroy a group is the reaction of the group.

Both trolls and twits are impotent if ignored. Remember the battle cry "DON'T FEED THE TROLLS" and use your newsreaders twit filter, and there is no real problem. This forum has no twit filter but activity is low enough that on can easily do the equivalent mentally.

jmG has been a part of this forum from the beginning. While he has managed to get at least one person banned, he has certainly not destroyed the forum. There is no reason to think that the future will be any different. There is no point in arguing with jmG. He has summarized his knowledge very well in http://collab.mathsoft.com/read?124709,7 and has proven quite resistant to any sort of learning.
__________________
� � � � Tom Gutman

On 8/12/2009 12:02:14 PM, mikej1980 wrote:
...

>I suggested a Fourier filter and if you used that you'd want to know about the power spectrum.

At least Tom Gutman can post rational thoughtful contributions about this, and I thank him for informing me about other aspects I hadn't considered. <<br> __________________________

You can have more than the power spectrum ! The attached, you can try at leisure. A real data set several years old in this collab. Try "Fourier all the ways", I haven't at the time because the proposed function was found satisfactory. "Fourier all the ways" encompasses more than what you may have and may defeat "rational thoughtful about this". "Curve fitting is an art" [F. B. Hildebrand], that "rational thoughtful" blinds.

So, do your best on the attached.
Deceptively simple or simply deceptive.

jmG



Announcements

Top Tags