Clone and edit this document
Runkit
Runkit
home page
user forum
new notebook
clone notebook
download notebook
support & documentation
log in
sign up
new notebook
help & feedback
clone this notebook
download this notebook
Sign In
Sign Up
Tokenize a Sentence
node v8.17.0
version:
1.0.0
endpoint
share
tweet
"wink-tokenizer" is an advance yet simple to use tokenizer that supports multiple languages. Here is a simple example to get you started:
// Load wink tokenizer. var winkTokenizer = require( 'wink-tokenizer' ); // Instantiate and obtain tokenize api. var tokenize = winkTokenizer().tokenize; // Notice how each token is tagged! var sentence = 'Do you have a meeting at 10 am tomorrow?'; console.log( tokenize( sentence ) ); // -> [ { value: 'Do', tag: 'word' }, // { value: 'you', tag: 'word' }, // { value: 'have', tag: 'word' }, // { value: 'a', tag: 'word' }, // { value: 'meeting', tag: 'word' }, // { value: 'at', tag: 'word' }, // { value: '10 am', tag: 'time' }, // { value: 'tomorrow', tag: 'word' }, // { value: '?', tag: 'punctuation' } ]
Loading…
no comments
sign in
to comment