I am facing issue when i try to tokenize a whole column that contains reviews of customer.

my file is stored in readfile
my column that i want to tokenize is 'Customer reviews'

This is my python code:

appling_tokenization= readfile.apply(lambda row: nltk.word_tokenize(row['Customer reviews']), axis=1)

My error is:

TypeError: expected string or bytes-like object

The question not being so clear, i guess i cannot be much help but here are some tips that might help you

  1. As the error suggests , maybe you are passing it a list of strings,in this case i believe you extract the string from the list using a loop .If this isnot working i bet the second one will work
  2. Your data contains Nan values, I suggest you filter them out and again tokenize.
Be a part of the DaniWeb community

We're a friendly, industry-focused community of 1.18 million developers, IT pros, digital marketers, and technology enthusiasts learning and sharing knowledge.