DaniWeb Forum Index > Software Development > C++

C++ RSS

string.size and accentuated words

Please support our C++ advertiser: Intel Parallel Studio Home

Thread Solved

•

Join Date: Feb 2008

Posts: 25

Reputation: onemanclapping is an unknown quantity at this point

Solved Threads: 0

onemanclapping onemanclapping is offline

Offline

Light Poster

string.size and accentuated words

Sep 1st, 2008

hi,

when I try string.size on accentuated words, the result is bigger than it was "supposed" to be, as accentuated characters count as 2 size units instead of one.

how can I count them as one?

cheers

•

Join Date: Aug 2005

Posts: 15,413

Reputation: Ancient Dragon has a reputation beyond repute

Solved Threads: 1470

Ancient Dragon Ancient Dragon is offline

Offline

Still Learning

Re: string.size and accentuated words

Sep 1st, 2008

Please post example code. Are you compiling for UNICODE ?

Don't PM me with questions -- you might get a nasty PM in response. If you have a question then post it in one of the forums.

•

Join Date: Feb 2008

Posts: 25

Reputation: onemanclapping is an unknown quantity at this point

Solved Threads: 0

onemanclapping onemanclapping is offline

Offline

Light Poster

Re: string.size and accentuated words

Sep 1st, 2008

•

Originally Posted by Ancient Dragon

Please post example code.

the code is irrelevant, I think, as my question applies to any code calling this function, but here it is:

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
void geraMenu(string titulo,string versao)
{
	int numEstrelas = titulo.size() + 40;
	string linha = string(numEstrelas,'*');
	string meiaLinha = string(19,'*');
	cout << linha << endl << linha << endl;
	cout << meiaLinha << " " << titulo << " " << meiaLinha << endl;
	cout << linha << endl << linha << endl;
	cout << string(5,'*') << " " <<  versao << " " << string(numEstrelas - 7 - versao.size(),'*') << endl;
}
void geraMenu(string titulo,string versao)
{
	int numEstrelas = titulo.size() + 40;
	string linha = string(numEstrelas,'*');
	string meiaLinha = string(19,'*');
	cout << linha << endl << linha << endl;
	cout << meiaLinha << " " << titulo << " " << meiaLinha << endl;
	cout << linha << endl << linha << endl;
	cout << string(5,'*') << " " <<  versao << " " << string(numEstrelas - 7 - versao.size(),'*') << endl;
}

the output with "PROJECTO GESTÃO" as 'titulo':

Help with Code Tags

C++ Syntax (Toggle Plain Text)

********************************************************
********************************************************
******************* PROJECTO GESTÃO *******************
********************************************************
********************************************************
***** beta 1 *******************************************

the output with "PROJECTO GESTAO" as 'titulo':

Help with Code Tags

C++ Syntax (Toggle Plain Text)

*******************************************************
*******************************************************
******************* PROJECTO GESTAO *******************
*******************************************************
*******************************************************
***** beta 1 ******************************************

as you see, the '*' aren't aligned in the first case, as I use 'Ã' instead of 'A' in the word "GESTÃO".
in the first case titulo.size() counts as 21 and in the second case counts as 20 (the correct amount of letters).

what I want is to know how can I count the right number of letters, independently of them being accentuated or not.

•

Originally Posted by Ancient Dragon

Are you compiling for UNICODE ?

I'm sorry, but I'm a beginner C++ programmer to such a "noob" level I don't know what you're talking about

all I can do is tell you that I'm using Eclipse SDK under Ubuntu and show you this picture of the properties menu: http://img366.imageshack.us/img366/9537/help1yh9.png

Last edited by onemanclapping; Sep 1st, 2008 at 9:31 am.

•

Join Date: Aug 2005

Posts: 5,266

Reputation: iamthwee is a splendid one to behold

Solved Threads: 377

iamthwee

Offline

Posting Expert

Re: string.size and accentuated words

Sep 1st, 2008

Yes I have no issue on windowsxp using dev-cpp, but with ubuntu (under a vmware environment) I do.

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
#include <iostream>
#include <string>
 
using namespace std;
 
int main()
{
    string a = "Ã";
    string b = "A";
 
    //cout << a << " " << b << endl;
   cout << a.length();
   cout << "\n";
   cout << b.length();
 
 
    cin.get();
}
#include <iostream>
#include <string>

using namespace std;

int main()
{
    string a = "Ã";
    string b = "A";
    
    //cout << a << " " << b << endl;
   cout << a.length();
   cout << "\n";
   cout << b.length();

    
    cin.get();
}

Output in ubuntu

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
user@ubuntu804desktop:~$ g++ -Wall pedantic.cc
user@ubuntu804desktop:~$ ./a.out
2
1
user@ubuntu804desktop:~$ g++ -Wall pedantic.cc
user@ubuntu804desktop:~$ ./a.out
2
1

Output in windowsxp using dev-cpp

Help with Code Tags

C++ Syntax (Toggle Plain Text)

1
1

Last edited by iamthwee; Sep 1st, 2008 at 10:35 am.

*Voted best profile in the world*

•

Join Date: Aug 2005

Posts: 15,413

Reputation: Ancient Dragon has a reputation beyond repute

Solved Threads: 1470

Ancient Dragon Ancient Dragon is offline

Offline

Still Learning

Re: string.size and accentuated words

Sep 1st, 2008

I don't have ubantu, but Microsoft VC++ 2008 Express reports 1 for the program that jamthwee posted. The compiler stored -61 in that byte. crappy *nix

>>I'm sorry, but I'm a beginner C++ programmer to such a "noob" level I don't know what you're talking about

UNICODE is a standard way to use non-English languages in computer programs. The standard UNICODE character is wchar_t, not char. Under MS-Windows wchar_t is defined to be unsigned short while in *nix (the last time I heard) it is unsigned long This is because many languages, such as Chinese, use graphic symbols which can be accommodated by wchar_t. In order to compile for UNICODE you have to set specific flags in the makefile -- I have no clue what those flags are for your compiler.

[edit]Considering jamthwee's test I would not bother with the UNICODE described above. It appears to be a compiler issue.[/edit]

Last edited by Ancient Dragon; Sep 1st, 2008 at 10:51 am.

Don't PM me with questions -- you might get a nasty PM in response. If you have a question then post it in one of the forums.

•

Join Date: Oct 2007

Posts: 1,951

Reputation: Duoas has much to be proud of

Solved Threads: 214

Duoas

Offline

Posting Virtuoso

Re: string.size and accentuated words

Sep 1st, 2008

Yes, it is an encoding issue. I suspect that it comes from the way your editor is saving the text file.

There are several ways to 'encode', or store, character data.

There is the old char-sized ASCII encoding, but that is limited to only 7-bit ASCII characters and any system dependant character codes above 127. Microsoft calls this "ANSI" and the exact selection of extended characters depends on your output code page. Obviously, this is not very convenient for languages using anything but straight-up Roman characters.

Then came (eventually) Unicode, which handles all language graphemes. (This doesn't mean it is complete --additions are still being made, but most industrialized nations can express their native language[s] with Unicode.)

There are several ways to store Unicode: three of which are of interest to us.

UTF-8 uses our venerable char. Only those graphemes that need more than one byte use more than one byte.

UTF-16/UCS-2 variable-width characters, like UTF-8, but where the smallest element is a 16-bit word instead of a byte. This format is considered deprecated, but it is still very much in use.

UTF-32/UCS-4 simply stores every character in a 32-bit word. This is how the GCC treats Unicode (wchar_t) values. As such, modern Linux systems in general are moving toward the exclusive use of this encoding.

So, now that you've had the lecture, on to the point: your text editor is using UTF-8, which you will recall is variable-width. I don't have Portugese installed, but I do have Spanish, so I hope you'll forgive the language choice in the examples. The file I've encoded is

Help with Code Tags

C++ Syntax (Toggle Plain Text)

Espanol
Español

"ANSI" (Microsoft's way), produces the following byte sequence
(escapes are either C-style or HEX, and the code page is Notepad's default)

Help with Code Tags

C++ Syntax (Toggle Plain Text)

 E   s   p   a   n   o   l   \r  \n
 E   s   p   a   \F1 o   l   \r  \n

UTF-16 produces
(Notepad's "Unicode" option; notice the byte-order mark at the beginning)

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
 \FF \FE
 E   \0  s   \0  p   \0  a   \0  n   \0  o   \0  l   \0  \r  \0  \n  \0
 E   \0  s   \0  p   \0  a   \0  \F1 \0  o   \0  l   \0  \r  \0  \n  \0
 \FF \FE
 E   \0  s   \0  p   \0  a   \0  n   \0  o   \0  l   \0  \r  \0  \n  \0
 E   \0  s   \0  p   \0  a   \0  \F1 \0  o   \0  l   \0  \r  \0  \n  \0

UTF-8 produces
(I removed Notepads weird BOM prefix)

Help with Code Tags

C++ Syntax (Toggle Plain Text)

 E   s   p   a   n   o   l   \r  \n
 E   s   p   a   \C3 \B1 o   l   \r  \n

Notice how the second line is a different length than the first, due to the two-byte code for 'ñ'.

You are using UTF-8. And you have found UTF-8's limitation: you can't use any of the standard C or C++ string length functions on a UTF-8 string. You must either roll your own or use a library of some kind. Here is one using the STL:

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
#include <algorithm>
#include <functional>
#include <string>
 
std::size_t UTF8_length( const std::string& s )
  {
  return std::count_if(
           s.begin(),
           s.end(),
           std::bind2nd( std::less <char> (), 0x80 )
           );
  }
#include <algorithm>
#include <functional>
#include <string>

std::size_t UTF8_length( const std::string& s )
  {
  return std::count_if(
           s.begin(),
           s.end(),
           std::bind2nd( std::less <char> (), 0x80 )
           );
  }

Hope this helps.

Last edited by Duoas; Sep 1st, 2008 at 12:58 pm.

•

Join Date: Feb 2008

Posts: 25

Reputation: onemanclapping is an unknown quantity at this point

Solved Threads: 0

onemanclapping onemanclapping is offline

Offline

Light Poster

Re: string.size and accentuated words

Sep 1st, 2008

•

Originally Posted by Duoas

Hope this helps.

Thank you very much for your help in explaining me this problem! It's now very clear why it happens.

The only problem is that the code you gave me for helping me count characters does not work

here's my code:

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
#include <iostream>
using std::cout;
using std::cin;
using std::endl;
 
#include <string>
using std::string;
 
#include <fstream>
using std::ifstream;
 
#include <algorithm>
 
#include <functional>
 
std::size_t UTF8_length(const string& s )
{
	return std::count_if(s.begin(),s.end(),std::bind2nd(std::less <char> (), 0x80));
}
 
void geraMenu(const string& titulo,const string& versao)
{
	int numEstrelas = titulo.size() + 40;
	string linha = string(numEstrelas,'*');
	string meiaLinha = string(19,'*');
	cout << linha << endl << linha << endl;
	cout << meiaLinha << " " << titulo << " " << meiaLinha << endl;
	cout << linha << endl << linha << endl;
	cout << string(5,'*') << " " <<  versao << " " << string(numEstrelas - 7 - versao.size(),'*') << endl;
 
	// UTF8_length tests
	cout << titulo.size() << endl;
	cout << UTF8_length(titulo) << endl;
	cout << UTF8_length("coco") << endl;
	cout << UTF8_length("cocó") << endl;
}
#include <iostream>
using std::cout;
using std::cin;
using std::endl;

#include <string>
using std::string;

#include <fstream>
using std::ifstream;

#include <algorithm>

#include <functional>

std::size_t UTF8_length(const string& s )
{
	return std::count_if(s.begin(),s.end(),std::bind2nd(std::less <char> (), 0x80));
}

void geraMenu(const string& titulo,const string& versao)
{
	int numEstrelas = titulo.size() + 40;
	string linha = string(numEstrelas,'*');
	string meiaLinha = string(19,'*');
	cout << linha << endl << linha << endl;
	cout << meiaLinha << " " << titulo << " " << meiaLinha << endl;
	cout << linha << endl << linha << endl;
	cout << string(5,'*') << " " <<  versao << " " << string(numEstrelas - 7 - versao.size(),'*') << endl;
	
	// UTF8_length tests
	cout << titulo.size() << endl;
	cout << UTF8_length(titulo) << endl;
	cout << UTF8_length("coco") << endl;
	cout << UTF8_length("cocó") << endl;
}

here's the output when I call geraMenu("PROJECTO GESTÃO", "beta 1"):

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
********************************************************
********************************************************
******************* PROJECTO GESTÃO *******************
********************************************************
********************************************************
***** beta 1 *******************************************
16
0
0
0
********************************************************
********************************************************
******************* PROJECTO GESTÃO *******************
********************************************************
********************************************************
***** beta 1 *******************************************
16
0
0
0

I can't see where the problem is as I've no idea what this [ std::bind2nd(std::less <char> (), 0x80) ] means.

thanks!

•

Join Date: Oct 2007

Posts: 1,951

Reputation: Duoas has much to be proud of

Solved Threads: 214

Duoas

Offline

Posting Virtuoso

Re: string.size and accentuated words

Sep 1st, 2008

Argh! I'm so sorry! (Recent med changes have made my brain work worse than usual...)

I forgot a couple of things:

force proper type comparison
non-ASCII characters

This will work. (I tested it to be sure!)

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
#include <algorithm>
#include <ciso646>
#include <functional>
#include <string>
 
struct UTF8_ischar
  {
  bool operator () ( unsigned char c ) const 
    {
    return (c < 0x80) or (c >= 0xC0);
    }
  };
 
std::size_t UTF8_length( const std::string& s )
  {
  return std::count_if( s.begin(), s.end(), UTF8_ischar() );
  }
#include <algorithm>
#include <ciso646>
#include <functional>
#include <string>

struct UTF8_ischar
  {
  bool operator () ( unsigned char c ) const 
    {
    return (c < 0x80) or (c >= 0xC0);
    }
  };

std::size_t UTF8_length( const std::string& s )
  {
  return std::count_if( s.begin(), s.end(), UTF8_ischar() );
  }

The above is an optimized version of

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
std::size_t UTF8_length( const std::string& s )
  {
  return std::count_if(
           s.begin(),
           s.end(),
           std::bind2nd( std::less <unsigned char> (), 0x80 )
           )
       + std::count_if(
	   s.begin(),
           s.end(),
           std::bind2nd( std::greater_equal <unsigned char> (), 0xC0 )
           );
  }
std::size_t UTF8_length( const std::string& s )
  {
  return std::count_if(
           s.begin(),
           s.end(),
           std::bind2nd( std::less <unsigned char> (), 0x80 )
           )
       + std::count_if(
	   s.begin(),
           s.end(),
           std::bind2nd( std::greater_equal <unsigned char> (), 0xC0 )
           );
  }

Don't worry too much about the weird stuff. You'll learn about it soon enough. It is just C++'s way of giving the user simple lambdas.

Essentially it says "count every character that has the msb == 0 or the two msbs == 11", which are the UTF-8 prefix codes for individual character sequences [1].

Sorry again!

Have fun now!

Last edited by Duoas; Sep 1st, 2008 at 6:01 pm. Reason: Fixed another stupid typo

•

Join Date: Feb 2008

Posts: 25

Reputation: onemanclapping is an unknown quantity at this point

Solved Threads: 0

onemanclapping onemanclapping is offline

Offline

Light Poster

Re: string.size and accentuated words

Sep 1st, 2008

•

Originally Posted by Duoas

Argh! I'm so sorry!

Oh! Don't be sorry at all, you've been very kind for explaining me all this...

The new code works perfectly! Thank you for your precious help!

Best regards,
André

Quick Reply to Thread

Show Printable Version

Other Threads in the C++ Forum

Previous Thread: Variable set of rules
Next Thread: One of the sound commands

Thread Tools	Search this Thread
Show Printable Version Email this Page

DaniWeb Message

Cancel Changes