Spline2 problems?

lbond-disabled · ‎Sep 03, 2003

Richard and Jean, down in the thread about Savitzky Golay, you were experiencing problems with Spline2. I'm starting a new thread (so I can find it).

you said:
>>------------
You can get the same level of smoothing (sort of), but the values needed to get it do not agree with the original Spline2. First, set w:=y*0 so that you have some weights. Then go to the section where you can set the significancy level, and set it to 0.0004. You get 12 knots, and DWS=1.619. Now change the significancy level to 0.0003, and you get 5 knots and DWS=1.452. WHY THE BIG JUMP? It is only allowing a few possible knot densities. Also, the default significancy level for the original spline2 is 0.05, which gives the spline in your worksheet, with DWS=1.9844. If you set level=0.05 you get DWS=3.084 with the datapack spline2. Something is wrong!

>>---------------

OK, I tried to set up the example as described (why are you using weights if they are all 0?), and I don't get the same results you do. Let's start with an example worksheet. Does the attached show what you are trying to show? I have to reduce the significancy down to about 0.00001 to get the same sort of curve that's in the third column.

RichardJ · ‎Sep 03, 2003

On 9/3/2003 1:32:43 PM, lbond wrote:

>OK, I tried to set up the
>example as described (why are
>you using weights if they are
>all 0?)

Because I was too lazy to change the function definition

>, and I don't get the
>same results you do. Let's
>start with an example
>worksheet. Does the attached
>show what you are trying to
>show? I have to reduce the
>significancy down to about
>0.00001 to get the same sort
>of curve that's in the third
>column.

Sorry. I was off by one order of magnitude. See the attached sheet. There are three problems.

1) Only certain numbers of knots are possible. (3, 5, 12, 15...)

2) The curve never matches that from the original spine2 program (presumably because we need a different number of knots)

3) The data in column 3 was obtained from the original spline2 program using the default significancy level of 0.05, and DWS was 1.9844. This is completely different to the spline2 function, which gives a very different curve with a significancy level of 0.05, and a DWS of 3.084. Something does not match here.

Richard

ptc-1368288 · ‎Sep 04, 2003

We can reproduce very close to the original trace 3. But using quadratic and lowering the level. As per attached sheet.

Attachment: 3 items
1. PDF tutorial.
2. Spline2xV6 as is from my program file.
3. Your sheet Leslie ... some comments.

jmG

lbond-disabled · ‎Sep 05, 2003

Ah yes, now I am finally understanding. We have version 5 of the spline code, and version 6 includes the autocorrelation stuff. This seems to play only in data sets with autocorrelation between the points, which, in this case, seems to be data set 3. The other example sets perform well (including what is described as the "famous Titanium heat data" - extra bonus points to anyone who can tell me what this actually is...)

I think, at this point, that my chances of getting the code upgraded to version 6, adding the autocorrelation control arguments, and getting it all tested in time for release are slim. Perhaps for DataPack version 2, which will have all the multivariate stuff that Xavier wanted 😉

RichardJ · ‎Sep 05, 2003

On 9/5/2003 3:33:15 PM, lbond wrote:
> The other example sets
>perform well (including what
>is described as the "famous
>Titanium heat data" - extra
>bonus points to anyone who can
>tell me what this actually
>is...)
>
>I think, at this point, that
>my chances of getting the code
>upgraded to version 6, adding
>the autocorrelation control
>arguments, and getting it all
>tested in time for release are
>slim. Perhaps for DataPack
>version 2,

Version 2, or volume 2? Or both? "Son of Datapack", or "Bride of Datapack"? If you have strong moral beliefs, perhaps there should be not son without a bride? 😉

>which will have all
>the multivariate stuff that
>Xavier wanted 😉

That sounds interesting. What multivariate stuff? I could ask for more of that too (PLS for a start. Perhaps Evolving factor analysis, various cluster analysis routines .....)

Richard

Richard

ptc-1178444 · ‎Sep 05, 2003

On 9/5/2003 3:47:21 PM, rijackson wrote:
>>which will have all
>>the multivariate stuff that
>>Xavier wanted 😉

Should I hold my breath? ;-]

>That sounds interesting. What
>multivariate stuff? I could ask for more
>of that too (PLS for a start. Perhaps
>Evolving factor analysis, various
>cluster analysis routines .....)

I was interested in something similar to LeastSquaresFit but extended to dimensions higher than 2D - as you've seen I tend to deal a lot with (x,y,z) data clouds. I was hoping also for more flexible multivariate interpolation routines, improved 3D plots, etc. I guess I'll have to wait for the Son of Datapack...

Xavier

lbond-disabled · ‎Sep 07, 2003

Son of Datapack... would make an interesting ad campaign. My guess is that, if this pack does well financially, I'll be able to make a case for a multivariate data pack as a separate entity. I'm told there is nothing that limits the SQP solver from handling the multivariate case, other than more dollars for more programming effort.

The 2D interpolation thing is a little stickier. It's unclear what the best methods are, particularly in the case of holes. If I ever get approval for a multivariate extension pack, we'll have to have some conversations about kriging, and other techniques.

Also, Xavier, I'm wondering if you've tried out the localmax and localmin functions, which do work in 3D, and might help you with some of your nearest-neighbor searches, since that's essentially what these are based on.

ptc-1178444 · ‎Sep 08, 2003

On 9/7/2003 10:30:44 AM, lbond wrote:
>Also, Xavier, I'm wondering if
>you've tried out the localmax
>and localmin functions, which
>do work in 3D, and might help
>you with some of your
>nearest-neighbor searches,
>since that's essentially what
>these are based on.

I haven't tried these functions. I guess one could use localmin to find the data point closest to a given interpolation position. That would work for a nearest-neighbor interpolation approach. I have to think some more about it to see if there is a way of using this to find the 4 closest-neighbors as well.

Xavier

ptc-1178444 · ‎Sep 08, 2003

On 9/7/2003 10:30:44 AM, lbond wrote:
>The 2D interpolation thing is
>a little stickier. It's
>unclear what the best methods
>are, particularly in the case
>of holes. If I ever get
>approval for a multivariate
>extension pack, we'll have to
>have some conversations about
>kriging, and other techniques.

Holes are indeed one of the key problems!

There is a thread going on right now about 3D plots and the interpolations that take place to create surfaces from discrete points (http://collab.mathsoft.com/~mathcad2000/read?42871,11). What's interesting (but also unfortunate for people like me who have to deal with holes) is that clearly 99.99% of 3D plots are created with continuous surfaces in mind.

Xavier

ptc-1368288 · ‎Sep 08, 2003

Holes are indeed one of the key problems!

There is a thread going on right now about 3D plots and the interpolations that take place to create surfaces from discrete points (http://collab.mathsoft.com/~mathcad2000/read?42871,11). What's interesting (but also unfortunate for people like me who have to deal with holes) is that clearly 99.99% of 3D plots are created with continuous surfaces in mind.

Xavier
___________________
Your link above (in your posting) does not open the thread. It makes a full page of the left portion of the collab. New web problem.

� you are saying that data points should be transformed into formulas for a complete 3D interpolation.

Can you post an example, maybe in the general collab as well.

jmG

ptc-1368288 · ‎Sep 08, 2003

...
http://collab.mathsoft.com/~Mathcad2000/read?42871,11

ptc-1368288 · ‎Sep 08, 2003

...
If the link is within brackets: does not open.

ptc-1178444 · ‎Sep 08, 2003

On 9/8/2003 10:56:31 AM, jmG wrote:
>� you are saying that data
>points should be transformed
>into formulas for a complete
>3D interpolation.

Not really.
I think that users may have different goals when creating 3D plots. Most of the time it is implied (in the user's mind or the plotting routine) that the (x,y,z) samples provided belong to a smooth continuous surface. There are arguably also a few cases where the data correspond to discontinuous surfaces or surfaces with holes, in which case interpolating indiscriminately between samples to create a continuous surface is undesirable. Conveying that information is not trivial. Mathcad can't handle holes easily at this point in time. However, the thread mentioned in my previous message shows that there are problems even in the case of an assumed continuous surface.

To finish, I'm sure I've shown similar examples before, but here is a 3D plot of a "holey" surface that I cannot create in Mathcad.

Xavier