Hi All,
The problem discussed is accuracy of numeric calculation result,in vb. Namely I compare a numerical mathematic calculation, Line-Plan Intersection, in VB.Net and Matlab 2007 the intersection point result specify in below:
Matlab 2007 ---->
Xint=2.000000000000000
Yint=1.333333333333333
Zint=0.666666666666667
But
Visual Studio 2008 ---->
Xint=2.0000000596046448
Yint=1.3333333730697632
Zint=0.66666668653488159
Hand Calculated ----->
Xint=2.000000000000000
Yint=1.333333333333333
Zint=0.666666666666667
why this error occurs?
Thanks.
A.Najafi
0
Newbie Poster
Recommended Answers
Jump to PostThe error is due to the way floating point values are stored. Single precision, which it looks like you used, is only good to about 7 digits. Double precision will get you 15 digits. It's best to think of floating point values as approximations of the "real" value.
See …
All 2 Replies
vmanes
1,165
Posting Virtuoso
The error is due to the way floating point values are stored. Single precision, which it looks like you used, is only good to about 7 digits. Double precision will get you 15 digits. It's best to think of floating point values as approximations of the "real" value.
See here for a chart of the VB data types
A.Najafi
0
Newbie Poster
Hi,
This is correct because 3 of my variables were single. moreover when I would changed those to Double this error solved.
Thanks Again.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.