Next, we will construct the vocabulary. This requires going through the whole training set once, applying our preprocess() function, and using a Counter() to count the number of occurrences of each word.
Note:
Counter().update() : We can add values to the Counter by using update() method.
map(myfunc) of the tensorflow datasets maps the function(or applies the function) myfunc across all the samples of the given dataset. More here.
Make sure to write each block of code below in different code-cells.
Import Counter from collections.
from << your code comes here >> import << your code comes here >>
Get the Counter() object vocabulary.
<< your code comes here >> = Counter()
For each review in every batch of the train data, let us make a vocabulary dictionary containing the words and their counts correspondingly:
for X_batch, y_batch in datasets["train"].batch(2).map(preprocess):
for review in X_batch:
vocabulary.update(list(review.numpy()))
Let’s look at the 5 most common words:
vocabulary.most_common()[:5]
Let us find the length of the vocabulary using len function.
<< your code comes here >>(vocabulary)
Note - Having trouble with the assessment engine? Follow the steps listed here
Loading comments...