{"id":262370,"date":"2025-05-29T10:06:18","date_gmt":"2025-05-29T10:06:18","guid":{"rendered":"https:\/\/peraltafinancing.com\/business\/marketing\/how-to-automate-seo-keyword-clustering-by-search-intent-with-python\/"},"modified":"2025-05-29T10:06:18","modified_gmt":"2025-05-29T10:06:18","slug":"how-to-automate-seo-keyword-clustering-by-search-intent-with-python","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=262370","title":{"rendered":"How To Automate SEO Keyword Clustering By Search Intent With Python"},"content":{"rendered":"


\n<\/p>\n

\n

There\u2019s a lot to know about search intent, from using deep learning to infer search intent by classifying text and breaking down SERP titles using Natural Language Processing (NLP) techniques, to clustering based on semantic relevance<\/a>, with the benefits explained.<\/p>\n

Not only do we know the benefits of deciphering search intent, but we also have a number of techniques at our disposal for scale and automation.<\/p>\n

So, why do we need another article on automating search intent?<\/p>\n

Search intent is ever more important now that AI search<\/a> has arrived.<\/p>\n

While more was generally in the 10 blue links search era, the opposite is true with AI search technology, as these platforms generally seek to minimize the computing costs (per FLOP) in order to deliver the service.<\/p>\n

SERPs Still Contain The Best Insights For Search Intent<\/h2>\n

The techniques so far involve doing your own AI, that is, getting all of the copy from titles of the ranking content for a given keyword and then feeding it into a neural network model (which you have to then build and test) or using NLP to cluster keywords.<\/p>\n

What if you don\u2019t have time or the knowledge to build your own AI or invoke the Open AI API?<\/p>\n

While cosine similarity has been touted as the answer to helping SEO professionals navigate the demarcation of topics for taxonomy<\/a> and site structures, I still maintain that search clustering by SERP results is a far superior method.<\/p>\n

That\u2019s because AI is very keen to ground its results on SERPs and for good reason \u2013 it\u2019s modelled on user behaviors.<\/p>\n

There is another way that uses Google\u2019s very own AI to do the work for you, without having to scrape all the SERPs content and build an AI model.<\/p>\n

Let\u2019s assume that Google ranks site URLs by the likelihood of the content satisfying the user query in descending order. It follows that if the intent for two keywords is the same, then the SERPs are likely to be similar.<\/p>\n

For years, many SEO professionals compared SERP results for keywords<\/a> to infer shared (or shared) search intent to stay on top of core updates, so this is nothing new.<\/p>\n

The value-add here is the automation and scaling of this comparison, offering both speed and greater precision.<\/p>\n

How To Cluster Keywords By Search Intent At Scale Using Python (With Code)<\/h2>\n

Assuming you have your SERPs results in a CSV download, let\u2019s import it into your Python notebook.<\/p>\n

1. Import The List Into Your Python Notebook<\/h3>\n
\n
import pandas as pd
\nimport numpy as np
\n
\nserps_input = pd.read_csv('data\/sej_serps_input.csv')
\ndel serps_input['Unnamed: 0']
\nserps_input
\n<\/code><\/pre>\n<\/div>\n
Below is the SERPs file now imported into a Pandas dataframe.<\/p>\n
\"\"Image from author, April 2025<\/span><\/div>\n
2. Filter Data For Page 1<\/h3>\n
We want to compare the Page 1 results of each SERP between keywords.<\/p>\n
We\u2019ll split the dataframe into mini keyword dataframes to run the filtering function before recombining into a single dataframe, because we want to filter at the keyword level:<\/p>\n
\n
# Split 
\nserps_grpby_keyword = serps_input.groupby(\"keyword\")
\nk_urls = 15
\n
\n# Apply Combine
\ndef filter_k_urls(group_df):
\n    filtered_df = group_df.loc[group_df['url'].notnull()]
\n    filtered_df = filtered_df.loc[filtered_df['rank'] <\/code><\/pre>\n<\/div>\n
\"SERPsImage from author, April 2025<\/span><\/div>\n\n
3. Convert Ranking URLs To A String<\/h3>\n
Because there are more SERP result URLs than keywords, we need to compress those URLs into a single line to represent the keyword\u2019s SERP.<\/p>\n
Here\u2019s how:<\/p>\n
\n

\n# convert results to strings using Split Apply Combine 
\nfiltserps_grpby_keyword = filtered_serps_df.groupby(\"keyword\")
\n
\ndef string_serps(df): 
\n   df['serp_string'] = ''.join(df['url'])
\n   return df # Combine strung_serps = filtserps_grpby_keyword.apply(string_serps) 
\n
\n# Concatenate with initial data frame and clean 
\nstrung_serps = pd.concat([strung_serps],axis=0) 
\nstrung_serps = strung_serps[['keyword', 'serp_string']]#.head(30) 
\nstrung_serps = strung_serps.drop_duplicates() 
\nstrung_serps
\n<\/code><\/pre>\n<\/div>\n
Below shows the SERP compressed into a single line for each keyword.<\/p>\n
\"SERPImage from author, April 2025<\/span><\/div>\n
4. Compare SERP Distance<\/h3>\n
To perform the comparison, we now need every combination of keyword SERP paired with other pairs:<\/p>\n
\n

\n# align serps
\ndef serps_align(k, df):
\n    prime_df = df.loc[df.keyword == k]
\n    prime_df = prime_df.rename(columns = {\"serp_string\" : \"serp_string_a\", 'keyword': 'keyword_a'})
\n    comp_df = df.loc[df.keyword != k].reset_index(drop=True)
\n    prime_df = prime_df.loc[prime_df.index.repeat(len(comp_df.index))].reset_index(drop=True)
\n    prime_df = pd.concat([prime_df, comp_df], axis=1)
\n    prime_df = prime_df.rename(columns = {\"serp_string\" : \"serp_string_b\", 'keyword': 'keyword_b', \"serp_string_a\" : \"serp_string\", 'keyword_a': 'keyword'})
\n    return prime_df
\n
\ncolumns = ['keyword', 'serp_string', 'keyword_b', 'serp_string_b']
\nmatched_serps = pd.DataFrame(columns=columns)
\nmatched_serps = matched_serps.fillna(0)
\nqueries = strung_serps.keyword.to_list()
\n
\nfor q in queries:
\n    temp_df = serps_align(q, strung_serps)
\n    matched_serps = matched_serps.append(temp_df)
\n
\nmatched_serps
\n<\/code><\/pre>\n<\/div>\n\n
\"Compare<\/p>\n
The above shows all of the keyword SERP pair combinations, making it ready for SERP string comparison.<\/p>\n
There is no open-source library that compares list objects by order, so the function has been written for you below.<\/p>\n
The function \u201cserp_compare\u201d compares the overlap of sites and the order of those sites between SERPs.<\/p>\n
\n

\nimport py_stringmatching as sm
\nws_tok = sm.WhitespaceTokenizer()
\n
\n# Only compare the top k_urls results 
\ndef serps_similarity(serps_str1, serps_str2, k=15):
\n    denom = k+1
\n    norm = sum([2*(1\/i - 1.0\/(denom)) for i in range(1, denom)])
\n    #use to tokenize the URLs
\n    ws_tok = sm.WhitespaceTokenizer()
\n    #keep only first k URLs
\n    serps_1 = ws_tok.tokenize(serps_str1)[:k]
\n    serps_2 = ws_tok.tokenize(serps_str2)[:k]
\n    #get positions of matches 
\n    match = lambda a, b: [b.index(x)+1 if x in b else None for x in a]
\n    #positions intersections of form [(pos_1, pos_2), ...]
\n    pos_intersections = [(i+1,j) for i,j in enumerate(match(serps_1, serps_2)) if j is not None] 
\n    pos_in1_not_in2 = [i+1 for i,j in enumerate(match(serps_1, serps_2)) if j is None]
\n    pos_in2_not_in1 = [i+1 for i,j in enumerate(match(serps_2, serps_1)) if j is None]
\n    
\n    a_sum = sum([abs(1\/i -1\/j) for i,j in pos_intersections])
\n    b_sum = sum([abs(1\/i -1\/denom) for i in pos_in1_not_in2])
\n    c_sum = sum([abs(1\/i -1\/denom) for i in pos_in2_not_in1])
\n
\n    intent_prime = a_sum + b_sum + c_sum
\n    intent_dist = 1 - (intent_prime\/norm)
\n    return intent_dist
\n
\n# Apply the function
\nmatched_serps['si_simi'] = matched_serps.apply(lambda x: serps_similarity(x.serp_string, x.serp_string_b), axis=1)
\n
\n# This is what you get
\nmatched_serps[['keyword', 'keyword_b', 'si_simi']]
\n<\/code><\/pre>\n<\/div>\n\n
\"Overlap<\/p>\n
Now that the comparisons have been executed, we can start clustering keywords.<\/p>\n
We will be treating any keywords that have a weighted similarity of 40% or more.<\/p>\n
\n

\n# group keywords by search intent
\nsimi_lim = 0.4
\n
\n# join search volume
\nkeysv_df = serps_input[['keyword', 'search_volume']].drop_duplicates()
\nkeysv_df.head()
\n
\n# append topic vols
\nkeywords_crossed_vols = serps_compared.merge(keysv_df, on = 'keyword', how = 'left')
\nkeywords_crossed_vols = keywords_crossed_vols.rename(columns = {'keyword': 'topic', 'keyword_b': 'keyword',
\n                                                                'search_volume': 'topic_volume'})
\n
\n# sim si_simi
\nkeywords_crossed_vols.sort_values('topic_volume', ascending = False)
\n
\n# strip NAN
\nkeywords_filtered_nonnan = keywords_crossed_vols.dropna()
\nkeywords_filtered_nonnan
\n<\/code><\/pre>\n<\/div>\n
We now have the potential topic name, keywords SERP similarity, and search volumes of each.
\"Clustering<\/p>\n
You\u2019ll note that keyword and keyword_b have been renamed to topic and keyword, respectively.<\/p>\n
Now we\u2019re going to iterate over the columns in the dataframe using the lambda technique.<\/p>\n
The lambda technique is an efficient way to iterate over rows in a Pandas dataframe because it converts rows to a list as opposed to the .iterrows() function.<\/p>\n
Here goes:<\/p>\n
\n

\nqueries_in_df = list(set(matched_serps['keyword'].to_list()))
\ntopic_groups = {}
\n
\ndef dict_key(dicto, keyo):
\n    return keyo in dicto
\n
\ndef dict_values(dicto, vala):
\n    return any(vala in val for val in dicto.values())
\n
\ndef what_key(dicto, vala):
\n    for k, v in dicto.items():
\n            if vala in v:
\n                return k
\n
\ndef find_topics(si, keyw, topc):
\n    if (si >= simi_lim):
\n
\n        if (not dict_key(sim_topic_groups, keyw)) and (not dict_key(sim_topic_groups, topc)): 
\n
\n            if (not dict_values(sim_topic_groups, keyw)) and (not dict_values(sim_topic_groups, topc)): 
\n                sim_topic_groups[keyw] = [keyw] 
\n                sim_topic_groups[keyw] = [topc] 
\n                if dict_key(non_sim_topic_groups, keyw):
\n                    non_sim_topic_groups.pop(keyw)
\n                if dict_key(non_sim_topic_groups, topc): 
\n                    non_sim_topic_groups.pop(topc)
\n            if (dict_values(sim_topic_groups, keyw)) and (not dict_values(sim_topic_groups, topc)): 
\n                d_key = what_key(sim_topic_groups, keyw)
\n                sim_topic_groups[d_key].append(topc)
\n                if dict_key(non_sim_topic_groups, keyw):
\n                    non_sim_topic_groups.pop(keyw)
\n                if dict_key(non_sim_topic_groups, topc): 
\n                    non_sim_topic_groups.pop(topc)
\n            if (not dict_values(sim_topic_groups, keyw)) and (dict_values(sim_topic_groups, topc)): 
\n                d_key = what_key(sim_topic_groups, topc)
\n                sim_topic_groups[d_key].append(keyw)
\n                if dict_key(non_sim_topic_groups, keyw):
\n                    non_sim_topic_groups.pop(keyw)
\n                if dict_key(non_sim_topic_groups, topc): 
\n                    non_sim_topic_groups.pop(topc) 
\n
\n        elif (keyw in sim_topic_groups) and (not topc in sim_topic_groups): 
\n            sim_topic_groups[keyw].append(topc)
\n            sim_topic_groups[keyw].append(keyw)
\n            if keyw in non_sim_topic_groups:
\n                non_sim_topic_groups.pop(keyw)
\n            if topc in non_sim_topic_groups: 
\n                non_sim_topic_groups.pop(topc)
\n        elif (not keyw in sim_topic_groups) and (topc in sim_topic_groups):
\n            sim_topic_groups[topc].append(keyw)
\n            sim_topic_groups[topc].append(topc)
\n            if keyw in non_sim_topic_groups:
\n                non_sim_topic_groups.pop(keyw)
\n            if topc in non_sim_topic_groups: 
\n                non_sim_topic_groups.pop(topc)
\n        elif (keyw in sim_topic_groups) and (topc in sim_topic_groups):
\n            if len(sim_topic_groups[keyw]) > len(sim_topic_groups[topc]):
\n                sim_topic_groups[keyw].append(topc) 
\n                [sim_topic_groups[keyw].append(x) for x in sim_topic_groups.get(topc)] 
\n                sim_topic_groups.pop(topc)
\n
\n        elif len(sim_topic_groups[keyw]) <\/code><\/pre>\n<\/div>\n
Below shows a dictionary containing all the keywords clustered by search intent into numbered groups:<\/p>\n
\n
{1: ['fixed rate isa',
\n  'isa rates',
\n  'isa interest rates',
\n  'best isa rates',
\n  'cash isa',
\n  'cash isa rates'],
\n 2: ['child savings account', 'kids savings account'],
\n 3: ['savings account',
\n  'savings account interest rate',
\n  'savings rates',
\n  'fixed rate savings',
\n  'easy access savings',
\n  'fixed rate bonds',
\n  'online savings account',
\n  'easy access savings account',
\n  'savings accounts uk'],
\n 4: ['isa account', 'isa', 'isa savings']}<\/code><\/pre>\n<\/div>\n
Let\u2019s stick that into a dataframe:<\/p>\n
\n

\ntopic_groups_lst = []
\n
\nfor k, l in topic_groups_numbered.items():
\n    for v in l:
\n        topic_groups_lst.append([k, v])
\n
\ntopic_groups_dictdf = pd.DataFrame(topic_groups_lst, columns=['topic_group_no', 'keyword'])
\n                                
\ntopic_groups_dictdf
\n<\/code><\/pre>\n<\/div>\n
\"TopicImage from author, April 2025<\/span><\/div>\n
The search intent groups above show a good approximation of the keywords inside them, something that an SEO expert would likely achieve.<\/p>\n
Although we only used a small set of keywords, the method can obviously be scaled to thousands (if not more).<\/p>\n
Activating The Outputs To Make Your Search Better<\/h2>\n
Of course, the above could be taken further using neural networks, processing the ranking content for more accurate clusters and cluster group naming, as some of the commercial products out there already do.<\/p>\n
For now, with this output, you can:<\/p>\n