using Prime 10 and 8 and getting the same result. when using the floor command and getting the incorrect output.
did some checking and found Mathcad is rounding out to 21 decimals places.
is this something where we should add a step to round a previous set of calculations before adding the Floor command?
or am I doing this incorrectly?
See attached file.
You are running into numerical inaccuracies, probably due to conversion from decimal to binary and retour and the limits of the IEEE number format.
As you can see, even though FinNom should be exactly 3.849, its actually a little bit smaller. The 10^-16 is a usual precision for IEEE numbers.
And because FinNom is smaller than 3.849, Prime's Floor function "correctly" gives the result 3.848.
Actually the round-off error already happens when calculating DimAmax
You also found the workaround yourself - apply some kind of rounding first.
You may define your own utility function to do this
or maybe its better to use
Another way to deal with the problem would be to use the symbolic engine which provides higher precision. But I wouldn't recommend that because that would be like shooting sparrows with a cannon. In addition, you would then also have to dispense with the inline evaluations of the calculated expressions, but rather should display the calculated variables in another region.
BTW, its very easy to run into these kinds of numerical inaccuracies when using software which uses IEEE format to store the numbers (which most non-symbolic math software does):
So if this may be a problem, you have to be aware of it and take measures to cope with it.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
