Tokenize a Sentence

node v8.17.0
version: 1.0.0
endpointsharetweet
"wink-tokenizer" is an advance yet simple to use tokenizer that supports multiple languages. Here is a simple example to get you started:
// Load wink tokenizer. var winkTokenizer = require( 'wink-tokenizer' ); // Instantiate and obtain tokenize api. var tokenize = winkTokenizer().tokenize; // Notice how each token is tagged! var sentence = 'Do you have a meeting at 10 am tomorrow?'; console.log( tokenize( sentence ) ); // -> [ { value: 'Do', tag: 'word' }, // { value: 'you', tag: 'word' }, // { value: 'have', tag: 'word' }, // { value: 'a', tag: 'word' }, // { value: 'meeting', tag: 'word' }, // { value: 'at', tag: 'word' }, // { value: '10 am', tag: 'time' }, // { value: 'tomorrow', tag: 'word' }, // { value: '?', tag: 'punctuation' } ]
Loading…

no comments

    sign in to comment