Member Avatar for dboltz03

Hi all,

Recently I've run into a problem where a string I am reading from a file is being read in with a nonprinting character appended to the end. The character being appended is U+0020.

I'm just unsure how to get rid of this. I know that I could just chop off the last character of the string, but this wouldn't solve my problem because some strings have this Unicode character on the end and others don't.

Any help would be greatly appreciated.

Thanks,

Dylan

  • 3 Contributors
  • 2 Replies
  • 3K Views
  • 12 Hours Discussion Span
  • Latest Post Latest Post by vijayan121

Dani AI

Generated

What you are really fighting here is a trailing space, not a general nonprintable. In that case you do not need to scan and cleanse the whole string; just trim the right edge when a space is present and leave everything else untouched. That preserves internal spaces and avoids chopping when nothing is there. Building on ’s suggestion and complementing ’s whole-string approach, here is a tiny, safe rtrim you can drop in. It works for both std::string and std::wstring.

// Remove a single class of trailing chars: plain space U+0020.
template<class CharT>
void rtrim_space(std::basic_string<CharT>& s)
{
    while (!s.empty() && s.back() == CharT(' '))
        s.pop_back();
}

// If your input sometimes has tabs or CRLF line endings, widen the set:
template<class CharT>
void rtrim_common_ws(std::basic_string<CharT>& s)
{
    auto is_ws = [](CharT c) {
        return c == CharT(' ') || c == CharT('\t') ||
               c == CharT('\r') || c == CharT('\n');
    };
    while (!s.empty() && is_ws(s.back()))
        s.pop_back();
}

Use rtrim_space(line); right after std::getline(...). If you discover that editors or tools are leaving tabs or a stray \r at the end, switch to rtrim_common_ws(line);.

Two quick checks if results still look odd:

  • Confirm what the last code unit actually is by logging std::hex << int(unsigned char(line.back())) for std::string, or std::hex << unsigned(line.back()) for std::wstring.
  • If you ever encounter a non-breaking space at the end (U+00A0) or a zero-width character, add those code points to the predicate in rtrim_common_ws so they are trimmed only at the end and not from the middle.

Recommended Answers

All 2 Replies

Member Avatar for mrnutty

U+0020 is the code for whitespace. So you have a whitespace at the end of your sentences in your file.
Run through the file and remove any whitespace at the end of the sentence. Alternatively, you can create a trim function that removes whites spaces from the front/end.

Member Avatar for vijayan121

Though the standard does not say anything specific about the representation of a wchar_t, it is either strictly Unicode (UCS-2) or ISO 10646 (UCS-4) on every implementation. These two have an identical character repertoire and code points for the Basic Multilingual Plane. In practice, this will remove non-printable characters from a string.

void remove_non_printable_chars( std::wstring& wstr )
{
    // get the ctype facet for wchar_t (Unicode code points in pactice)
    typedef std::ctype< wchar_t > ctype ;
    const ctype& ct = std::use_facet<ctype>( std::locale() ) ;

    // remove non printable Unicode characters
    wstr.erase( std::remove_if( wstr.begin(), wstr.end(),
                    [&ct]( wchar_t ch ) { return !ct.is( ctype::print, ch ) ; } ),
                wstr.end() ) ;
}

Incidentally, U+0020 is a printable Unicode character.

commented: man you know too much +13
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.