17-Peridot

Solved

filterNaN weirdness

Forum|Forum|8 years ago
May 14, 2018
4 replies
6603 views

Hi,

I have a rather large matrix (1100 x 40) from which I tried to filter out the data that have NaN content, but to no avail.

When I searched for the problem in my code I found that the the filterNaN function sometimes didn't filter NaNs although the entite columns was filled with NaN. I tested this by checking whether the value is indeed NaN, which it is. Checking whether the same value is a scalar returned also a positive?!? (see attache sheet which contains a subset of that data).

Does anybody have an explanation for this contradiction and/or can tell me how to avoid this?

Thanks

Raiko

1_NaN-weirdness.zip

Best answer by LucMeekes

Raiko,

the following function definitions were in the Data Analysis Extension Pack 'electronic book' for Mathcad 11:

(NOTE: Bad programming because not ORIGIN-aware.)

I added:

Gives these results:

Hope this helps. If yes, you can just override the Mathcad 15 internal definitions...

Attached is a mathcad sheet with ORIGIN-aware versions.

Success!

Luc

M

MJG

1-Visitor

Two notes:

IsScalar(NaN) returns 1 (true), so I think this is unrelated to the filterNaN problem you are having. Though, I agree it's questionable that Mathcad returns true here.
Please note that filterNaN removes all rows that have NaN in any of their columns. Therefore, if you have a column of all NaNs, then the result will be empty.

If I create a set of data with a single column of NaNs, then filterNaN returns the error, "The resulting matrix contains no data."

If I create a set of data with a two columns of NaNs, then filterNaN returns the error, "These array dimensions do not match."

In trying to evaluate filterNaN(Q) with your data, I get the second error. I assume this is only because you have two columns of NaNs. Though, I believe the first error message would be more helpful.

Edit:

After further investigation, I've found several examples where filterNaN gives the error, "These array dimensions do not match." - even if you have some rows without any NaNs.

R

Raiko

Author

17-Peridot

I'm aware that mathcad throws an error when all columns in a matrix are NaN and have dealt programatically with this (on error statement). However, this error also appears matrices where about 40% of the columns are NaNs.

Because I couldn't understand why Mathcad fails to filter for NaN I had a look at the data to check whether some of them are wonky. That's why I checked for NaN and Scalar. However, a matrix element can't be NaN and a scalar although Luc's idea that Mathcad probably only checks whether a variable is neither a string nor an array and from that judges that it must be a scalar is a good one I doubt it.

It seems I need to contact PTC.

Raiko

L

LucMeekes

23-Emerald IV

Maybe Mathcad judges that an item is a scalar when it's 'numeric' (not a string) AND not an array.

Luc

W

Werner_E

25-Diamond I

I have no problem with the fact that Mathcad considers NaN a scalar. After all NaN is an expression we want to be able to apply mathematical operations on (like your division by 2) without running into an error. We see in the 3Dplot component what happens if NaN's are not considered numerical/scalar - the component throws an error.

On the other hand there sure is a bug in the filterNaN function. If I change your matrix Q so that at least one row does not contain a NaN I still get an error when trying to run it through filterNaN. This "array dimensions does not match" is the very error message we get when every row contains at least a NaN and the result would be empty.

As you can see from the attached screenshot the problem does not occur if I just use the first three columns of your matrix (I even tried addding the last column with success).

Still no idea what could cause that error, but it sure is a bug.

R

Raiko

Author

17-Peridot

Hello Werner,

on the perill of being called a stickler for accuracy I beg to differ. Calling a variable content "Not a Number"; i.e. it is neither integer, real nor complex and letting it to be handled as if it is one is a contradiction. even if this helps in 3D plots.

I've opened a call at PTC on the filterNaN issue. Let's see how long it takes.

Raiko

W

Werner_E

25-Diamond I

I see NaN as "something" which, while not being a number, still is an allowed argument wherever a scalar is allowed. Seen that way "IsScalar(NaN)=1" has some consistency.

I would not have high hopes for a speedy bug fix from PTC, but we'll see.