cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - Did you know you can set a signature that will be added to all your posts? Set it here! X

Large Numbers Round On Their Own In Parameters

pmajor
6-Contributor

Large Numbers Round On Their Own In Parameters

When I input a number that is greater than 16 digits, it begins to round. For example, I insert the following:

2222222222222222222222222.00000

and it automatically corrects it to:

2222222222222222184939520.000000

 

I don't understand why it is doing this. I need these numbers to be exact becuase it is a code that drives parameters based on the numbers 3 and 2. 

 

For instance, if I put 232323223, an interchange will grab the part associated with each individual number. It is not being using for a dimensional value.

 

Thanks,

 John

 

1 ACCEPTED SOLUTION

Accepted Solutions
pmajor
6-Contributor
(To:pmajor)

In the end, I used the Extract function.

 

IF EXTRACT(CODEBASESTRING,1,1)== "3"
D10=30
BRIDGE="BRIDGE2"
ELSE
IF EXTRACT(CODEBASESTRING,1,1) == "2"
D10=20
BRIDGE="BRIDGE1"
ELSE
BRIDGE="NA"
D10=0
ENDIF
ENDIF

 


IF EXTRACT(CODEBASESTRING,2,1) == "3"
D40=30
BRIDGE2="BRIDGE2"
ELSE
IF EXTRACT(CODEBASESTRING,2,1) == "2"
D40=20
BRIDGE2="BRIDGE1"
ELSE
BRIDGE2="NA"
D40=0
ENDIF
ENDIF

 

Though I get an "Probable Error" for the digits that do not exist (because the CODEBASESTRING is too short). I would suggest only using this if an empty cell is used for something not existing. For example, I have 75 total pieces. If I only have a code that is STRING_LENGTH(CODEBASESTRING) =  7, then the pieces past 7 are not included in the assembly. You should do the same for the code as well.

 

Cheers,

   Jonathyn

View solution in original post

9 REPLIES 9
BenLoosli
23-Emerald II
(To:pmajor)

Can you use a string parameter to drive your interchange instead of a floating number? That would make it cleaner and there would be no rounding. Possibly an integer may be an alterntive.

pmajor
6-Contributor
(To:BenLoosli)

The code is configured numerically. For instance, 23232223. I take the mod,10 of this number to recieve the last digit. Floor,10 to get the next number. It needs to be a real number. I am assuming from your answer that there isn't a way to achieve this?

 

 

RandyJones
19-Tanzanite
(To:pmajor)


@pmajor wrote:

The code is configured numerically. For instance, 23232223. I take the mod,10 of this number to recieve the last digit. Floor,10 to get the next number. It needs to be a real number. I am assuming from your answer that there isn't a way to achieve this?

 


Real numbers will always have this issue:

http://analytics.ncsu.edu/sesug/2008/PO-082.pdf

http://www.lexjansen.com/phuse/2011/pp/PP12.pdf

http://support.sas.com/resources/papers/proceedings11/275-2011.pdf

 

You could probably use an integer parameter to do what you are trying to do. However it sounds like what you are attempting to do is extract each individual digit. This is what strings are good at. If you can then post the complete code on how you are actually extracting these digits.

pmajor
6-Contributor
(To:RandyJones)

I will try the solution 

 

dschenken
21-Topaz I
(To:pmajor)

I think I misplaced an ")" snuck in my example. I'll edit the related post.

This is disturbing - my earlier post is gone. Lithium problem?

 

It included a link to https://en.wikipedia.org/wiki/Double-precision_floating-point_format which pointed out that the IEEE Double precision format is limited to between 15 and 17 decimal digits because it used 53 bits for the significant value. 53* log(10)/log(2) ~= 15.9

 

It also included the following relations:

(rel_model_name is a string parameter for which each character is extracted and then used to reconstruct a value)

 

/*Without comments 
digitlist = "0123456789" 
value = 0 
digit_found = search(digitlist, extract(rel_model_name, len(rel_model_name)-0,1))-1 
value = value+digit_found 
digit_found = search(digitlist, extract(rel_model_name, len(rel_model_name)-1,1))-1 
value = value+digit_found*10 
digit_found = search(digitlist, extract(rel_model_name, len(rel_model_name)-2,1))-1 
value = value+digit_found*100 
digit_found = search(digitlist, extract(rel_model_name, len(rel_model_name)-3,1))-1 
value = value+digit_found*1000 
d0=value

pmajor
6-Contributor
(To:pmajor)

In the end, I used the Extract function.

 

IF EXTRACT(CODEBASESTRING,1,1)== "3"
D10=30
BRIDGE="BRIDGE2"
ELSE
IF EXTRACT(CODEBASESTRING,1,1) == "2"
D10=20
BRIDGE="BRIDGE1"
ELSE
BRIDGE="NA"
D10=0
ENDIF
ENDIF

 


IF EXTRACT(CODEBASESTRING,2,1) == "3"
D40=30
BRIDGE2="BRIDGE2"
ELSE
IF EXTRACT(CODEBASESTRING,2,1) == "2"
D40=20
BRIDGE2="BRIDGE1"
ELSE
BRIDGE2="NA"
D40=0
ENDIF
ENDIF

 

Though I get an "Probable Error" for the digits that do not exist (because the CODEBASESTRING is too short). I would suggest only using this if an empty cell is used for something not existing. For example, I have 75 total pieces. If I only have a code that is STRING_LENGTH(CODEBASESTRING) =  7, then the pieces past 7 are not included in the assembly. You should do the same for the code as well.

 

Cheers,

   Jonathyn

RandyJones
19-Tanzanite
(To:BenLoosli)


@BenLoosli wrote:

Can you use a string parameter to drive your interchange instead of a floating number? That would make it cleaner and there would be no rounding. Possibly an integer may be an alterntive.


Geez I completely missed this reply...

You must have been typing faster than me.

RandyJones
19-Tanzanite
(To:pmajor)


@pmajor wrote:

When I input a number that is greater than 16 digits, it begins to round. For example, I insert the following:

2222222222222222222222222.00000

and it automatically corrects it to:

2222222222222222184939520.000000

 

I don't understand why it is doing this. I need these numbers to be exact becuase it is a code that drives parameters based on the numbers 3 and 2. 

 

For instance, if I put 232323223, an interchange will grab the part associated with each individual number. It is not being using for a dimensional value.

 

Thanks,

 John

 


I would suggest using a string parameter for this since it is actually a part number and not a numeric value.

Top Tags