Effective tokenizing is all about handling sentences that contain features beyond words & punctuations such as e-mails, mentions, emojis, emoticons, hashtags, urls and more! Here is an example:
"wink-sentiment" handles negation intelligently; for example, phrase "good product" will get a positive score whereas "not a good product" gets a negative score. Here is a little more complex example:
Just give any phrase and checkout the sentiment score. A positive score means a positive sentiment, whereas a negative score indicates a negative sentiment. Neutral sentiment is signalled by a near zero score.