Member Avatar for Vignesh_8

Hey!!
I am facing issue when i try to tokenize a whole column that contains reviews of customer.

my file is stored in readfile
my column that i want to tokenize is 'Customer reviews'

This is my python code:

print("tokenization..\n")
appling_tokenization= readfile.apply(lambda row: nltk.word_tokenize(row['Customer reviews']), axis=1)

My error is:

TypeError: expected string or bytes-like object
Edited by Reverend Jim because: Corrected code formatting
  • 2 Contributors
  • 1 Reply
  • 98 Views
  • 2 Weeks Discussion Span
  • Latest Post Latest Post by hussainmujtaba
Member Avatar for hussainmujtaba

The question not being so clear, i guess i cannot be much help but here are some tips that might help you

  1. As the error suggests , maybe you are passing it a list of strings,in this case i believe you extract the string from the list using a loop .If this isnot working i bet the second one will work
  2. Your data contains Nan values, I suggest you filter them out and again tokenize.
commented: Alright!! Thanks for your response. +0
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.