cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - When posting, your subject should be specific and summarize your question. Here are some additional tips on asking a great question. X

Multiplying a variable by "ft" changes the value - MathCAD bug

rspears
1-Visitor

Multiplying a variable by "ft" changes the value - MathCAD bug

I am having an issue with variables changing value from an integer to an integer +/- a very small value like 0.0000000000000002 when I assign the variable a unit. I've changed the default units to U.S. thinking that there was some small conversion being done incorrectly, but the issue is still there. Does anyone have any idea what could be causing this? This is really putting a kink in my work day. I have to subtract these values from other values and then test that result. If that result is < 0 instead of exactly 0, the MathCAD sheet doesn't work correctly. To me, this is a major issue, and I'm not sure why MathCAD would have a bug like this. I've included a screenshot for clarity on what I'm talking about.

mathcadissue.PNG

4 REPLIES 4
RichardJ
19-Tanzanite
(To:rspears)

It's not a bug. It's inherent to numerical computing. All numbers in Mathcad are stored as double precision floats, which gives you about 15-16 digits of precision. What you are looking at are numerical round-off errors, and it's why you should never do an exact comparison of two floating point numbers (a rule that is universal, not just for Mathcad). Go to "Tools", "Worksheet options", and select the "Calculation" tab. Uncheck "Use exact equality for comparisons and truncation". Unfortunately, the default is for this to be checked, which may not be a bug, but it's a very bad choice for the default.

I unchecked the box you described and it doesn't solve my problem. The spreadsheet still says that 0.00000000000000002 is greater than 0 and my sheet still doesn't get the correct answer. I tried rounding the value that is being compared to 0 and apparently you can't round a value with units.

RichardJ
19-Tanzanite
(To:rspears)

Please post the worksheet (Click on "Use advanced editor" at the top right of the edit dialog").

RichardJ
19-Tanzanite
(To:rspears)

Perhaps this will help:

Announcements

Top Tags