Performance – Optimize the application of Python Gensim Search Functions

I have a DataFrame that has a text column. I divide the DataFrame into two parts based on the value in another column. One of these parts is included in a similarity model of Gensim. The other part is then fed into the model to find the indexed text that is most similar. This includes several search functions listed for each item in the indexed part. It's fast with the toy data, but with my real data, it's way too slow apply, Here is the code sample:

Import pandas as pd
Import gensim
import nltk
From nltk.tokenize import word_tokenize (& # 39; point & # 39;)

d = {# number: #: [1,2,3,4,5], & # 39; text & # 39 ;: ['do you like python', 'do you hate python','do you like apples','who is nelson mandela','i am not interested'], & # 39; answers & # 39 ;:['no','yes','no','no','yes']}
df = pd.DataFrame (data = d)

df_yes = df[df['answer']== & # 39; yes & # 39;]df_no = df[df['answer']== & # 39; no & # 39;]df_no = df_no.reset_index ()

docs = df_no['text'].list()
genDocs = [[w.lower() for w in word_tokenize(text)] for text in documents]dictionary = gensim.corpora.Dictionary (genDocs)
Corpus = [dictionary.doc2bow(genDoc) for genDoc in genDocs]
tfidf = gensim.models.TfidfModel (Corpus)
sims = gensim.similarities.MatrixSimilarity (tfidf[corpus], num_features = len (dictionary))

def search (line):
Query = [w.lower() for w in word_tokenize(row)]
    query_bag_of_words = dictionary.doc2bow (query)
query_tfidf = tfidf[query_bag_of_words]
    return query_tfidf

def searchAll (line):
max_similarity = max (sims[search(row)])
Index =[Ifürijaufzählen(sims[Iforijinenumerate(sims[ifürijaufzählen(sims[iforijinenumerate(sims[search(row)]if j == max_similarity]Return max_similarity, index

df_yes = df_yes.copy ()

df_yes['max_similarity']df_yes['index'] = zip (* df_yes['text'].apply (searchAll))

I've tried converting the operations to these dataframes, as well as Python multiprocessing. How can I make these functions more efficient? Is it possible to vectorize some / all functions?