943,783 Members | Top Members by Rank

Ad:
  • C++ Discussion Thread
  • Marked Solved
  • Views: 987
  • C++ RSS
Sep 1st, 2008
0

string.size and accentuated words

Expand Post »
hi,

when I try string.size on accentuated words, the result is bigger than it was "supposed" to be, as accentuated characters count as 2 size units instead of one.

how can I count them as one?

cheers
Reputation Points: 10
Solved Threads: 0
Light Poster
onemanclapping is offline Offline
25 posts
since Feb 2008
Sep 1st, 2008
0

Re: string.size and accentuated words

Please post example code. Are you compiling for UNICODE ?
Sponsor
Team Colleague
Featured Poster
Reputation Points: 5608
Solved Threads: 2282
Retired and Enjoying Life
Ancient Dragon is offline Offline
21,950 posts
since Aug 2005
Sep 1st, 2008
0

Re: string.size and accentuated words

Please post example code.
the code is irrelevant, I think, as my question applies to any code calling this function, but here it is:
C++ Syntax (Toggle Plain Text)
  1. void geraMenu(string titulo,string versao)
  2. {
  3. int numEstrelas = titulo.size() + 40;
  4. string linha = string(numEstrelas,'*');
  5. string meiaLinha = string(19,'*');
  6. cout << linha << endl << linha << endl;
  7. cout << meiaLinha << " " << titulo << " " << meiaLinha << endl;
  8. cout << linha << endl << linha << endl;
  9. cout << string(5,'*') << " " << versao << " " << string(numEstrelas - 7 - versao.size(),'*') << endl;
  10. }

the output with "PROJECTO GESTÃO" as 'titulo':
C++ Syntax (Toggle Plain Text)
  1. ********************************************************
  2. ********************************************************
  3. ******************* PROJECTO GESTÃO *******************
  4. ********************************************************
  5. ********************************************************
  6. ***** beta 1 *******************************************

the output with "PROJECTO GESTAO" as 'titulo':
C++ Syntax (Toggle Plain Text)
  1. *******************************************************
  2. *******************************************************
  3. ******************* PROJECTO GESTAO *******************
  4. *******************************************************
  5. *******************************************************
  6. ***** beta 1 ******************************************

as you see, the '*' aren't aligned in the first case, as I use 'Ã' instead of 'A' in the word "GESTÃO".
in the first case titulo.size() counts as 21 and in the second case counts as 20 (the correct amount of letters).

what I want is to know how can I count the right number of letters, independently of them being accentuated or not.

Are you compiling for UNICODE ?
I'm sorry, but I'm a beginner C++ programmer to such a "noob" level I don't know what you're talking about
all I can do is tell you that I'm using Eclipse SDK under Ubuntu and show you this picture of the properties menu: http://img366.imageshack.us/img366/9537/help1yh9.png
Last edited by onemanclapping; Sep 1st, 2008 at 9:31 am.
Reputation Points: 10
Solved Threads: 0
Light Poster
onemanclapping is offline Offline
25 posts
since Feb 2008
Sep 1st, 2008
0

Re: string.size and accentuated words

Yes I have no issue on windowsxp using dev-cpp, but with ubuntu (under a vmware environment) I do.
C++ Syntax (Toggle Plain Text)
  1. #include <iostream>
  2. #include <string>
  3.  
  4. using namespace std;
  5.  
  6. int main()
  7. {
  8. string a = "Ã";
  9. string b = "A";
  10.  
  11. //cout << a << " " << b << endl;
  12. cout << a.length();
  13. cout << "\n";
  14. cout << b.length();
  15.  
  16.  
  17. cin.get();
  18. }

Output in ubuntu
C++ Syntax (Toggle Plain Text)
  1. user@ubuntu804desktop:~$ g++ -Wall pedantic.cc
  2. user@ubuntu804desktop:~$ ./a.out
  3. 2
  4. 1

Output in windowsxp using dev-cpp
C++ Syntax (Toggle Plain Text)
  1. 1
  2. 1
Last edited by iamthwee; Sep 1st, 2008 at 10:35 am.
Featured Poster
Reputation Points: 1536
Solved Threads: 431
Posting Expert
iamthwee is offline Offline
5,865 posts
since Aug 2005
Sep 1st, 2008
0

Re: string.size and accentuated words

I don't have ubantu, but Microsoft VC++ 2008 Express reports 1 for the program that jamthwee posted. The compiler stored -61 in that byte. crappy *nix

>>I'm sorry, but I'm a beginner C++ programmer to such a "noob" level I don't know what you're talking about

UNICODE is a standard way to use non-English languages in computer programs. The standard UNICODE character is wchar_t, not char. Under MS-Windows wchar_t is defined to be unsigned short while in *nix (the last time I heard) it is unsigned long This is because many languages, such as Chinese, use graphic symbols which can be accommodated by wchar_t. In order to compile for UNICODE you have to set specific flags in the makefile -- I have no clue what those flags are for your compiler.

[edit]Considering jamthwee's test I would not bother with the UNICODE described above. It appears to be a compiler issue.[/edit]
Last edited by Ancient Dragon; Sep 1st, 2008 at 10:51 am.
Sponsor
Team Colleague
Featured Poster
Reputation Points: 5608
Solved Threads: 2282
Retired and Enjoying Life
Ancient Dragon is offline Offline
21,950 posts
since Aug 2005
Sep 1st, 2008
1

Re: string.size and accentuated words

Yes, it is an encoding issue. I suspect that it comes from the way your editor is saving the text file.

There are several ways to 'encode', or store, character data.

There is the old char-sized ASCII encoding, but that is limited to only 7-bit ASCII characters and any system dependant character codes above 127. Microsoft calls this "ANSI" and the exact selection of extended characters depends on your output code page. Obviously, this is not very convenient for languages using anything but straight-up Roman characters.

Then came (eventually) Unicode, which handles all language graphemes. (This doesn't mean it is complete --additions are still being made, but most industrialized nations can express their native language[s] with Unicode.)

There are several ways to store Unicode: three of which are of interest to us.

UTF-8 uses our venerable char. Only those graphemes that need more than one byte use more than one byte.

UTF-16/UCS-2 variable-width characters, like UTF-8, but where the smallest element is a 16-bit word instead of a byte. This format is considered deprecated, but it is still very much in use.

UTF-32/UCS-4 simply stores every character in a 32-bit word. This is how the GCC treats Unicode (wchar_t) values. As such, modern Linux systems in general are moving toward the exclusive use of this encoding.


So, now that you've had the lecture, on to the point: your text editor is using UTF-8, which you will recall is variable-width. I don't have Portugese installed, but I do have Spanish, so I hope you'll forgive the language choice in the examples. The file I've encoded is
C++ Syntax (Toggle Plain Text)
  1. Espanol
  2. Español
"ANSI" (Microsoft's way), produces the following byte sequence
(escapes are either C-style or HEX, and the code page is Notepad's default)
C++ Syntax (Toggle Plain Text)
  1. E s p a n o l \r \n
  2. E s p a \F1 o l \r \n
UTF-16 produces
(Notepad's "Unicode" option; notice the byte-order mark at the beginning)
C++ Syntax (Toggle Plain Text)
  1. \FF \FE
  2. E \0 s \0 p \0 a \0 n \0 o \0 l \0 \r \0 \n \0
  3. E \0 s \0 p \0 a \0 \F1 \0 o \0 l \0 \r \0 \n \0
UTF-8 produces
(I removed Notepads weird BOM prefix)
C++ Syntax (Toggle Plain Text)
  1. E s p a n o l \r \n
  2. E s p a \C3 \B1 o l \r \n
Notice how the second line is a different length than the first, due to the two-byte code for 'ñ'.

You are using UTF-8. And you have found UTF-8's limitation: you can't use any of the standard C or C++ string length functions on a UTF-8 string. You must either roll your own or use a library of some kind. Here is one using the STL:
C++ Syntax (Toggle Plain Text)
  1. #include <algorithm>
  2. #include <functional>
  3. #include <string>
  4.  
  5. std::size_t UTF8_length( const std::string& s )
  6. {
  7. return std::count_if(
  8. s.begin(),
  9. s.end(),
  10. std::bind2nd( std::less <char> (), 0x80 )
  11. );
  12. }
Hope this helps.
Last edited by Duoas; Sep 1st, 2008 at 12:58 pm.
Featured Poster
Reputation Points: 1140
Solved Threads: 229
Postaholic
Duoas is offline Offline
2,039 posts
since Oct 2007
Sep 1st, 2008
0

Re: string.size and accentuated words

Click to Expand / Collapse  Quote originally posted by Duoas ...
Hope this helps.
Thank you very much for your help in explaining me this problem! It's now very clear why it happens.

The only problem is that the code you gave me for helping me count characters does not work

here's my code:
C++ Syntax (Toggle Plain Text)
  1. #include <iostream>
  2. using std::cout;
  3. using std::cin;
  4. using std::endl;
  5.  
  6. #include <string>
  7. using std::string;
  8.  
  9. #include <fstream>
  10. using std::ifstream;
  11.  
  12. #include <algorithm>
  13.  
  14. #include <functional>
  15.  
  16. std::size_t UTF8_length(const string& s )
  17. {
  18. return std::count_if(s.begin(),s.end(),std::bind2nd(std::less <char> (), 0x80));
  19. }
  20.  
  21. void geraMenu(const string& titulo,const string& versao)
  22. {
  23. int numEstrelas = titulo.size() + 40;
  24. string linha = string(numEstrelas,'*');
  25. string meiaLinha = string(19,'*');
  26. cout << linha << endl << linha << endl;
  27. cout << meiaLinha << " " << titulo << " " << meiaLinha << endl;
  28. cout << linha << endl << linha << endl;
  29. cout << string(5,'*') << " " << versao << " " << string(numEstrelas - 7 - versao.size(),'*') << endl;
  30.  
  31. // UTF8_length tests
  32. cout << titulo.size() << endl;
  33. cout << UTF8_length(titulo) << endl;
  34. cout << UTF8_length("coco") << endl;
  35. cout << UTF8_length("cocó") << endl;
  36. }

here's the output when I call geraMenu("PROJECTO GESTÃO", "beta 1"):
C++ Syntax (Toggle Plain Text)
  1. ********************************************************
  2. ********************************************************
  3. ******************* PROJECTO GESTÃO *******************
  4. ********************************************************
  5. ********************************************************
  6. ***** beta 1 *******************************************
  7. 16
  8. 0
  9. 0
  10. 0

I can't see where the problem is as I've no idea what this [ std::bind2nd(std::less <char> (), 0x80) ] means.

thanks!
Reputation Points: 10
Solved Threads: 0
Light Poster
onemanclapping is offline Offline
25 posts
since Feb 2008
Sep 1st, 2008
0

Re: string.size and accentuated words

Argh! I'm so sorry! (Recent med changes have made my brain work worse than usual...)

I forgot a couple of things:
  1. force proper type comparison
  2. non-ASCII characters
This will work. (I tested it to be sure!)
C++ Syntax (Toggle Plain Text)
  1. #include <algorithm>
  2. #include <ciso646>
  3. #include <functional>
  4. #include <string>
  5.  
  6. struct UTF8_ischar
  7. {
  8. bool operator () ( unsigned char c ) const
  9. {
  10. return (c < 0x80) or (c >= 0xC0);
  11. }
  12. };
  13.  
  14. std::size_t UTF8_length( const std::string& s )
  15. {
  16. return std::count_if( s.begin(), s.end(), UTF8_ischar() );
  17. }
The above is an optimized version of
C++ Syntax (Toggle Plain Text)
  1. std::size_t UTF8_length( const std::string& s )
  2. {
  3. return std::count_if(
  4. s.begin(),
  5. s.end(),
  6. std::bind2nd( std::less <unsigned char> (), 0x80 )
  7. )
  8. + std::count_if(
  9. s.begin(),
  10. s.end(),
  11. std::bind2nd( std::greater_equal <unsigned char> (), 0xC0 )
  12. );
  13. }
Don't worry too much about the weird stuff. You'll learn about it soon enough. It is just C++'s way of giving the user simple lambdas.

Essentially it says "count every character that has the msb == 0 or the two msbs == 11", which are the UTF-8 prefix codes for individual character sequences [1].

Sorry again!
Have fun now!
Last edited by Duoas; Sep 1st, 2008 at 6:01 pm. Reason: Fixed another stupid typo
Featured Poster
Reputation Points: 1140
Solved Threads: 229
Postaholic
Duoas is offline Offline
2,039 posts
since Oct 2007
Sep 1st, 2008
0

Re: string.size and accentuated words

Click to Expand / Collapse  Quote originally posted by Duoas ...
Argh! I'm so sorry!
Oh! Don't be sorry at all, you've been very kind for explaining me all this...

The new code works perfectly! Thank you for your precious help!

Best regards,
André
Reputation Points: 10
Solved Threads: 0
Light Poster
onemanclapping is offline Offline
25 posts
since Feb 2008

This thread is solved

Either the thread starter or a moderator has marked this thread as solved. You can most likely trust the responses and answers given. There is most likely no reason for any further responses to be posted here. If you have a related question, please start a new thread in this forum instead.

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.
Message:
Previous Thread in C++ Forum Timeline: Variable set of rules
Next Thread in C++ Forum Timeline: One of the sound commands





About Us | Contact Us | Advertise | Acceptable Use Policy
Forum Index | Build Custom RSS Feed


Follow us on Twitter


© 2011 DaniWeb® LLC