filterNaN weirdness

[Raiko]

Hi,

I have a rather large matrix (1100 x 40) from which I tried to filter out the data that have NaN content, but to no avail.

When I searched for the problem in my code I found that the the filterNaN function sometimes didn't filter NaNs although the entite columns was filled with NaN. I tested this by checking whether the value is indeed NaN, which it is. Checking whether the same value is a scalar returned also a positive?!?  (see attache sheet which contains a subset of that data).

Does anybody have an explanation for this contradiction and/or can tell me how to avoid this?

Thanks

Raiko

[MJG]

Two notes:

• IsScalar(NaN) returns 1 (true), so I think this is unrelated to the filterNaN problem you are having.  Though, I agree it's questionable that Mathcad returns true here.
• Please note that filterNaN removes all rows that have NaN in any of their columns.  Therefore, if you have a column of all NaNs, then the result will be empty.

If I create a set of data with a single column of NaNs, then filterNaN returns the error, "The resulting matrix contains no data."

If I create a set of data with a two columns of NaNs, then filterNaN returns the error, "These array dimensions do not match."

In trying to evaluate filterNaN(Q) with your data, I get the second error.  I assume this is only because you have two columns of NaNs.  Though, I believe the first error message would be more helpful.

Edit:

After further investigation, I've found several examples where filterNaN gives the error, "These array dimensions do not match." - even if you have some rows without any NaNs.

[Raiko]

I'm aware that mathcad throws an error when all columns in a matrix are NaN and have dealt programatically with this (on error statement). However, this error also appears matrices where about 40% of the columns are NaNs.

Because I couldn't understand why Mathcad fails to filter for NaN I had a look at the data to check whether some of them are wonky. That's why I checked for NaN and Scalar. However, a matrix element can't be NaN and a scalar although Luc's idea that Mathcad probably only checks whether a variable is neither a string nor an array and from that judges that it must be a scalar is a good one I doubt it.

It seems I need to contact PTC.

Raiko

[LucMeekes]

Maybe Mathcad judges that an item is a scalar when it's 'numeric' (not a string) AND not an array.

Luc

[Werner_E]

I have no problem with the fact that Mathcad considers NaN a scalar. After all NaN is an expression we want to be able to apply mathematical operations on (like your division by 2) without running into an error. We see in the 3Dplot component what happens if NaN's are not considered numerical/scalar - the component throws an error.

On the other hand there sure is a bug in the filterNaN function. If I change your matrix Q so that at least one row does not contain a NaN I still get an error when trying to run it through filterNaN. This "array dimensions does not match" is the very error message we get when every row contains at least a NaN and the result would be empty.

As you can see from the attached screenshot the problem does not occur if I just use the first three columns of your matrix (I even tried addding the last column with success).

Still no idea what could cause that error, but it sure is a bug.

[Raiko]

Hello Werner,

on the perill of being called a stickler for accuracy I beg to differ. Calling a variable content "Not a Number"; i.e. it is neither integer, real nor complex and letting it to be handled as if it is one is a contradiction. even if this helps in 3D plots.

I've opened a call at PTC on the filterNaN issue. Let's see how long it takes.

Raiko

[Werner_E]

I see NaN as "something" which, while not being a number, still is an allowed argument wherever a scalar is allowed. Seen that way "IsScalar(NaN)=1" has some consistency.

I would not have high hopes for a speedy bug fix from PTC, but we'll see.

[LucMeekes]

I would not expect PTC to fix any of this in Mathcad (15, let alone before).

How does Prime fare? (I can't check, it's a Premium feature...)

Luc

[Werner_E]

---

@LucMeekes wrote:

I would not expect PTC to fix any of this in Mathcad (15, let alone before).

How does Prime fare? (I can't check, it's a Premium feature...)

Luc

---

Prime shows the very same error

[Raiko]

Hello All,

FYI. I started to check the type of variables because the numbers were supplied by a transient recorder, converted by Matlab to a CSV format and finally read in by Mathcad. I wanted to rule out that there's something wonky with the numbers and therefore checked for IsNaN and IsScalar.

The fact that also Prime has the same bug is not encouraging.

Raiko

[Werner_E]

---

@Raiko wrote:

The fact that also Prime has the same bug is not encouraging.

---

That depends - it could mean that there is a very slight chance for a bug fix being provided somewhere in the future which then may even be applied to real Mathcad (if its still supported then).

BTW, my simplified example with the 3x3 matrix (and also my test with your matrix Q, where I changed 1 line to avoid that the filtered matrix would be empty) show an error which means that Mathcad filters too much. There should be one row remaining but the error message is the same we get when all the rows would get filtered.

On contrary to that error you wrote in your initial post "I found that the the filterNaN function sometimes didn't filter NaNs" which would mean you ran into the opposite error of Mathcad not filtering some rows containing a NaN. Are we talking about two different bugs here? Can you provide an example where Mathcad misses a row containing an NaN.

[Werner_E]

Here is a very small test case where filterNaN fails.

Chances are that filterNaN internally uses "matchNaN" and "trim" in a similar way my filterNaN_2 does. As you can see in the attached screenshot "trim" fails sometimes, if the vector of indices is too long, containing duplicate indices.

[MJG]

I combined Werner's filterNaN_2 with Luc's RemoveDuplicates and added a few bells & whistles:

The result is a function that does not error no matter how many NaNs are in the input, and runs just as fast as the built-in function (proof & additional documentation attached).

[LucMeekes · Accepted solution]

Raiko,

the following function definitions were in the Data Analysis Extension Pack 'electronic book' for Mathcad 11:

(NOTE: Bad programming because not ORIGIN-aware.)

I added:

Gives these results:

Hope this helps. If yes, you can just override the Mathcad 15 internal definitions...

Attached is a mathcad sheet with ORIGIN-aware versions.

Success!

Luc

[Werner_E]

This is what I get when I recalculate your sheet with the current MC15. Obviously the function returns the unaltered argument.

Will have a look at it as to why later. Found it. Its your IsNaN1. Replace it by IsNaN ant filterNaN1 works OK.

FilterNAN1 shows the same behaviour

[Werner_E]

According your matchNaN:

a=NaN does not give you 1 even though a is assigned NaN

Replace "...if (Vv=NaN)" by "...if IsNaN(Vv)" and your filterNaN works OK, too.

Don't use IsNaN1, as otherwise matchNaN will return 0 all the time. 0 is scalar and so every row is included in the result matrix.

[Raiko]

Salut Luc,

thank you for your help (and your#s as well Werner). This did the trick. I'm still baffled as to why it is so weird but have to live with it.

Raiko

[LucMeekes]

Raiko,

you might want to check out these items:

https://community.ptc.com/t5/PTC-Mathcad-Questions/Inconsistent-NaNs/m-p/299123#M177158
https://community.ptc.com/t5/PTC-Mathcad-Questions/NaN-bug-Has-anyone-seen-this-before/m-p/53427#M21309
https://community.ptc.com/t5/PTC-Mathcad-Questions/NaN-sorting/m-p/550583#M180782

Luc