cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - You can subscribe to a forum, label or individual post and receive email notifications when someone posts a new topic or reply. Learn more! X

Floating point error problem in programming

vvenkatesan
1-Newbie

Floating point error problem in programming

I have done pressure calculations in normal worksheet format and i got results without any error. but it needs to change a value once for every iterations. So i moved on to programming. I compiled the program using while loop and break conditions. After completing and running the program i got error message showing "encountered a floating point error".Dont know how to fix the bug.. Can anyone help me pls?????????

1 ACCEPTED SOLUTION

Accepted Solutions

Vigneshwar Venkatesan wrote:

I have done pressure calculations in normal worksheet format and i got results without any error. but it needs to change a value once for every iterations. So i moved on to programming. I compiled the program using while loop and break conditions. After completing and running the program i got error message showing "encountered a floating point error".Dont know how to fix the bug.. Can anyone help me pls?????????

Somewhere within your loop the deltaP .dc and .r variables are blowing up to the limits of numeric handling (ie, they're in the order of 10307 )

Stuart

View solution in original post

1 REPLY 1

Vigneshwar Venkatesan wrote:

I have done pressure calculations in normal worksheet format and i got results without any error. but it needs to change a value once for every iterations. So i moved on to programming. I compiled the program using while loop and break conditions. After completing and running the program i got error message showing "encountered a floating point error".Dont know how to fix the bug.. Can anyone help me pls?????????

Somewhere within your loop the deltaP .dc and .r variables are blowing up to the limits of numeric handling (ie, they're in the order of 10307 )

Stuart

Top Tags