Rounding error with trunc() and floor() functions that return exact integers

[mappleby811]

I'm having an issue where I need to do division, then eliminate the decimal portion of the answer and only keep the integer. I've successfully used both trunc() and floor() for this application in the past. But when the quotient results in an exact integer (like 84/12=7 in my example), Mathcad rounds down to 6 instead of returning 7. I'm assuming this is the result of some approximation internal to how Mathcad does division. Like it's really calculating 6.99999999999999... maybe? Are there any work arounds for this? Obviously I can just make the equation 84/12, but this is in a template that I'm trying to make apply to all cases.

[MartinHanak]

Hi,

it seems to me that the problem is related to ft unit somehow (I do not understand why).

In Mathcad Prime 9.0 following notation works well.

[MartinHanak]

[Werner_E · Accepted solution]

Yes, the problem is because of numerical round offs - especially because of the usage of non SI standard unit (Prime stores values with SI units no matter how they are defined)

You may either add a small percentage to the nominator first

or get rid of the units by dividing b.road by the unit feet

[terryhendicott]

Here is another way

[ChrisKaswer]

I think you may have hit the nail on the head - here's what I found with a simple modification to the equation:

Seems there's a conversion limit (bug?) 16 places out that is still observed/obeyed by the engine for the final result. All the suggestions to date will address it. Maybe there'll be a way in the future to adjust this...

Cheers

[Werner_E]

---

@ChrisKaswer wrote:

I think you may have hit the nail on the head - here's what I found with a simple modification to the equation:

Seems there's a conversion limit (bug?) 16 places out that is still observed/obeyed by the engine for the final result. All the suggestions to date will address it. Maybe there'll be a way in the future to adjust this...

Cheers

---

No, I guess as long as the usual IEEE number format is used there cannot be a generic official "solution" as round off errors in one way or the other are inevitable. Symbolic engines use their own way to represent numbers and calculate with them in much higher precision but in Prime the symbolic engine does not know anything about units and unit conversions ...
So using the symbolic evaluation is not only slower but it also fails if the units don't cancel directly

So IMHO the solution provided by Terry seems to be the best - rounding the result to a certain number of decimals before applying functions like trunc or floor. You may (re)define your own functions which do this automatically and if you know for sure that your application does not need a precision higher than the number of decimals you round to.

[mappleby811]

Dividing out the unit looks to be the most logical way for me and my reviewers. Thanks for identifying the unit as the culprit, it really helped!