0

Hi All,
The problem discussed is accuracy of numeric calculation result,in vb. Namely I compare a numerical mathematic calculation, Line-Plan Intersection, in VB.Net and Matlab 2007 the intersection point result specify in below:
Matlab 2007 ---->
Xint=2.000000000000000
Yint=1.333333333333333
Zint=0.666666666666667
But
Visual Studio 2008 ---->
Xint=2.0000000596046448
Yint=1.3333333730697632
Zint=0.66666668653488159
Hand Calculated ----->
Xint=2.000000000000000
Yint=1.333333333333333
Zint=0.666666666666667
why this error occurs?
Thanks.

2
Contributors
2
Replies
3
Views
8 Years
Discussion Span
Last Post by A.Najafi
0

The error is due to the way floating point values are stored. Single precision, which it looks like you used, is only good to about 7 digits. Double precision will get you 15 digits. It's best to think of floating point values as approximations of the "real" value.

See here for a chart of the VB data types

0

Hi,
This is correct because 3 of my variables were single. moreover when I would changed those to Double this error solved.
Thanks Again.

This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.