Internal links are what Google walks through your site and decides which page is important and which is not. An article to which no link from the text of other articles leads is almost invisible for search: no one “recommends” it from within the site.
The problem is that you can’t do this by hand. On a hundred articles, you need to make thousands of decisions: which article is related to whom, where to put the link, what anchor. A person can’t stand it, and the “Related Posts” plugins do the wrong thing.
This guide contains a working method: show the computer the meaning of each article and get a ready-made linking plan. Free, no API keys, in 10 minutes. Everything is verified on a live blog of 125 articles, including the rake that I personally stepped on.
Why “Related Posts” is not a link
Standard plugins link articles by category or tag and place a block of links in the footer. Three troubles:
- The heading is not the point.In one heading “SEO” there may be an article about domains and an article about texts: there is zero connection between them.
- The footer is not read – neither people nor the search engine pay attention it has the same weight as a link inside the text.
- The anchor “Read also”does not tell the search engine anything about what is on the other side.
You need a link in the body of the article, placed where it makes sense, with an anchor that describes the landing page.
What is semantic proximity, in simple terms
The computer does not understand words, but it can turn text into a set of numbers – a vector. Texts about the same thing give close vectors, even if the words are different: “buy a car” and “buy a car” will be nearby. This is called embedding.
Then everything is simple: we turn each article into a vector, compare all the pairs with each other and get a list of who is related to whom. These are the candidates for linking.
The model for this is free and multilingual, it downloads itself and works directly in the browser via Google Colab. No keys, no payment, no own server.
What you need
- Google Colab – free, you only need a Google account.
- The address of your sitemap – usually site.ru/sitemap.xml or site.ru/sitemap_index.xml. If you don’t know, take a look at site.ru/robots.txt, it’s listed there.
- 10 minutes.No need to install anything on your computer.
Step 1. Open Colab and paste the script
Go to colab.research.google.com, create a new notebook (File → New notebook), paste the code below into the cell.
In the settings block, change two lines to suit you:
SITEMAP_URL– the address of your sitemap.URL_FILTER– if the site is multilingual, leave one language, for example “/ru/”. Otherwise, translations of one article will be considered related to each other, and the linking plan will turn into garbage. If the site is in one language, put an empty line “”.
!pip -q install requests beautifulsoup4 lxml pandas scikit-learn sentence-transformers
# ============================================
# SEMANTIC INTERLINK PLANNER (Google Colab)
# Finds the meaning of where to put internal links in each article,
# and shows orphans - pages that no one links to.
# ============================================
import re, time, collections, requests
import numpy as np, pandas as pd
from bs4 import BeautifulSoup
from sklearn.metrics.pairwise import cosine_similarity
from xml.etree import ElementTree as ET
from sentence_transformers import SentenceTransformer
SITEMAP_URL = "https://MY-SITE/sitemap.xml"
URL_FILTER = "/ru/"
MAX_URLS = 500
EXCLUDE = ["/category/", "/tag/", "/author/", "/page/"] # categories and tags are not articles
LINKS_PER_PAGE = 4 # article link ceiling
MIN_SIM = 0.35 # below - relatives are far-fetched
BODY_CHARS = 1500
HEADERS = {"User-Agent": "Mozilla/5.0 (interlink planner)"}
def clean(t):
return re.sub(r"\s+", " ", (t or "").replace("\xa0", " ")).strip()
def fetch(u):
r = requests.get(u, headers=HEADERS, timeout=20); r.raise_for_status(); return r.text
def sitemap(u, acc=None):
if acc is None: acc = []
root = ET.fromstring(fetch(u)); tag = lambda x: x.tag.split("}")[-1]
if tag(root) == "sitemapindex":
for sm in root:
for ch in sm:
if tag(ch) == "loc": sitemap(clean(ch.text), acc)
elif tag(root) == "urlset":
for ue in root:
for ch in ue:
if tag(ch) == "loc": acc.append(clean(ch.text))
return acc
def page(u, host):
try:
html = fetch(u)
except Exception:
return None
s = BeautifulSoup(html, "lxml")
# IMPORTANT: first throw out the menu/footer/sidebar, otherwise their links
# will be counted as “already worth” (we ended up with 32 links to the article instead of 2.5)
for t in s(["script","style","noscript","svg","nav","footer","header","aside","form"]): t.decompose()
body = s.find("article") or s.find("main") or s
out = set()
for a in body.find_all("a", href=True):
h = a["href"]
if host not in h: continue
# only links INSIDE A PARAGRAPH: author block, signatures and widgets - not linking
if not a.find_parent("p"): continue
out.add(h.split("#")[0].rstrip("/"))
title = clean(s.title.get_text(" ")) if s.title else u
h1 = clean(s.h1.get_text(" ")) if s.h1 else ""
return {"url": u.rstrip("/"), "title": title, "h1": h1,
"text": clean(s.get_text(" ", strip=True))[:BODY_CHARS], "outgoing": out}
host = re.sub(r"https?://([^/]+)/.*", r"\1", SITEMAP_URL)
all_urls = [u for u in dict.fromkeys(sitemap(SITEMAP_URL)) if URL_FILTER in u]
before = len(all_urls)
all_urls = [u for u in all_urls if not any(x in u for x in EXCLUDE)]
# the main page and the roots of languages are not articles, they should not be goals
all_urls = [u for u in all_urls if len(u.rstrip("/").split(URL_FILTER.strip("/"))[-1].strip("/")) > 3]
print(f"categories/tags eliminated: 0 (they have short text, they are “similar” to everything in the category and clog up the report)")
urls = all_urls[:MAX_URLS]
partial = len(urls) < len(all_urls)
print(f"pages in progress: 0 of 1")
if partial:
print("ATTENTION: the crawl is incomplete - orphans will not be counted (inboxes are only visible throughout the site).")
print("To find orphans, raise MAX_URLS to the size of the site.")
docs = [d for d in (page(u, host) for u in urls) if d]
print("disassembled:", len(docs))
# we cut off the brand tail, as in a cannibalization audit
tails = collections.Counter()
for d in docs:
for sep in ["|","➣","—","–","-","·","»"]:
if sep in d["title"]: tails[d["title"].rsplit(sep,1)[-1].strip().lower()] += 1
brand = tails.most_common(1)[0][0] if tails and tails.most_common(1)[0][1] >= max(3, len(docs)*0.3) else None
for d in docs:
t = d["title"]
cut = t[:t.lower().rfind(brand)].rstrip(" |➣—–-·»") if brand and brand in t.lower() else t
d["title_clean"] = cut or d["h1"] or t # slice should not nullify header
m = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
emb = m.encode([f'{d["title_clean"]}. {d["h1"]}. {d["text"]}' for d in docs],
normalize_embeddings=True, show_progress_bar=False)
sim = emb @ emb.T
np.fill_diagonal(sim, -1)
rows = []
for i, d in enumerate(docs):
order = np.argsort(-sim[i])
added = 0
for j in order:
if added >= LINKS_PER_PAGE: break
s = float(sim[i][j])
if s < MIN_SIM: break
tgt = docs[j]
if tgt["url"] in d["outgoing"]: # the link is already there
continue
rows.append({"where": d["title_clean"][:44], "Where": tgt["title_clean"][:44],
"closeness": round(s,3), "anchor_hint": tgt["h1"][:48] or tgt["title_clean"][:48],
"url_from": d["url"], "url_where": tgt["url"]})
added += 1
plan = pd.DataFrame(rows)
print("\nnew links suggested:", len(plan))
print("already worth references in the texts:", sum(len(d["outgoing"]) for d in docs))
# orphans: to whom no one refers and to whom no one offers
incoming = collections.Counter()
for d in docs:
for o in d["outgoing"]: incoming[o] += 1
if partial:
print("articles without entries: skipped (incomplete crawl)")
orphans = []
else:
orphans = [d["title_clean"][:50] for d in docs if incoming[d["url"]] == 0]
print("articles without incoming links FROM TEXTS of other articles:", len(orphans))
for o in orphans[:5]: print(" -", o)
print("\ntop 8 offers:")
if len(plan):
for _, r in plan.sort_values("closeness", ascending=False).head(8).iterrows():
print(f' {r["closeness"]} {r["where"][:34]:34} -> {r["Where"][:34]}')
Press the start button. The first run takes a couple of minutes: the model is downloaded. Then the script bypasses the sitemap, downloads pages and counts.
Step 2. Read what he gave
The output is four things:
- How many links are already in the texts.Only links within paragraphs are counted – the menu, footer and author block do not count.
- Plan of new links – table: from, to, proximity, anchor hint. Sorted by proximity.
- Articles without incoming links are those same invisible ones. The metric is only fair if you crawl the site completely.
- The interlink_plan.csv file is the entire plan so you can work calmly and cross out what you’ve done.
Start at the top of the table: the most obvious pairs are there. Proximity 0.8 – almost certainly related. Below 0.4 is already a stretch, so the script doesn’t offer this.
Step 3. Place links correctly
A plan is a hint, not an order. Bet with your hands, according to three rules:
- Text only The link lives in the paragraph where you naturally mention the topic. Not in the footer, not in the “read also” block.
- Anchor – in the meaning of the target article. Not “here”, not “read also”, but a phrase describing where you are leading.
- Not in the headings. The link inside the h2 breaks and layout and markup. This is a separate rake that we have already sat on.
Ceiling – 4 links to the article. Twenty links are not twenty times better: the weight is eroded, the reader leaves. We have an average of 2.3 links per article on our blog, and that’s enough.
The rake I stepped on while writing this script
This is not a theory – I ran it on my blog, and the first versions lied. All four errors have already been fixed in the code above, but it’s useful to know about them if you write your own.
1. The menu was considered interlinking
The first version counted 32 internal links to the article with a real 2.5. Reason: links were collected from the entire page, including the menu and footer. Because of this, the script thought that the link was already there and did not offer what it needed. It can be cured by removing nav, header, footer, aside before collecting links.
2. Link to author
After the first edit, there were 4.15 links to the article instead of 2.3. The link to the author from the signature block turned out to be superfluous. Therefore, only links within
paragraphs are counted. After that, the score matched the base: 2.28 versus 2.3.
3. The headings filled the entire top
In a full crawl, the first places were taken by the pages “CRM”, “Design”, “Payments” with a proximity of 0.93. These are not articles, but columns: they have a short text, and they are similar to everything in their category at once. Screened by /category/, /tag/, /author/.
4. Empty header
The script cuts off the brand from the title (the tail of the form “— Site Name”), because this tail is on every page and spoils the similarity of all pairs. But if the title consists of a single brand – for example, on the main page – the cut nulled the title. Now there is a fallback option.
Our numbers
To have something to compare with. On a blog where linking is done automatically according to the meaning:
- 927 internal links to 372 articles.
- 98% of articles have links in the text – 366 out of 372.
- 2.3 links per article on average, the ceiling is 4.
- 47 articles with still no incoming links from the texts. Even with automation, there is a tail that needs to be retrieved by hand.
The last number is the most honest. Automation does not do the work for you one hundred percent, it removes 90% of the routine and shows where it remains.
How we automated this on our own site
The script above is a solid starting point: it tells you which articles are related, and you place the links yourself. With 372 articles that stopped being enough for us, so we automated it. Since we are showing you the method, here is the production version too.
- Embeddings come from mistral-embed, not from a local MiniLM. Vectors live in Pinecone: computed once, then only new articles get added.
- We look for the exact sentence, not just a related article. We embed every sentence of the article and every candidate title, then find the sentence that is genuinely about the same thing. Threshold: cosine 0.72.
- The anchor is taken from that sentence — real text, correct grammar. No “read also”.
- Hard thresholds. Candidates below 0.35 are dropped. If no sentence fits, we add no link at all. Zero links beat a link in the wrong place.
- A cap of 3 links per run. Over time an article reaches up to 4, averaging 2.3 — the numbers above.
Honest about the cost: this needs a paid embeddings API and a vector database. That is why the guide uses the version that costs nothing — it finds exactly the same pairs, you just place the links yourself. Under a hundred articles you will not feel the difference; at several hundred you will not keep up by hand.
Our entire method: how to repeat it yourself
Next – the whole kitchen without skipping. Five layers, real thresholds and why they are like that. You can repeat it on any stack, it’s not about PHP.
Layer 1. Indexing articles into a vector database
We take the title plus the first 6500 characters of text without HTML. We count the embedding and put it in Pinecone – in the namespace of your language. This is important: if you lump all languages together, translations of one article will become the closest relatives to each other, and all the linking will go into translations.
# 1) эмбеддинг статьи (1024 измерения)
curl https://api.mistral.ai/v1/embeddings \
-H "Authorization: Bearer $MISTRAL_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"mistral-embed","input":["Заголовок. Первые 6500 символов текста..."]}'
# 2) кладём вектор в Pinecone, неймспейс = язык статьи
curl https://ВАШ-ИНДЕКС.svc.ВАШ-РЕГИОН.pinecone.io/vectors/upsert \
-H "Api-Key: $PINECONE_KEY" \
-H "X-Pinecone-API-Version: 2025-01" \
-H "Content-Type: application/json" \
-d '{
"namespace": "ru",
"vectors": [{
"id": "1849",
"values": [0.013, -0.042, ...],
"metadata": {"post_id":1849, "lang":"ru", "title":"...", "url":"https://...", "cat":"SEO"}
}]
}' In the metadata we put post_id, language, title, URL and category – then they are needed to build a link and anchor without going to the database again. The model is limited to 7000 characters per input, so we cut the text in advance.
Layer 2. Looking for candidates
For a new article, we consider the same embedding and ask the nearest Pinecone – in the namespace of her language, we throw ourselves out of the search results.
curl https://ВАШ-ИНДЕКС.svc.ВАШ-РЕГИОН.pinecone.io/query \
-H "Api-Key: $PINECONE_KEY" \
-H "X-Pinecone-API-Version: 2025-01" \
-d '{"namespace":"ru", "vector":[...], "topK":15, "includeMetadata":true}'
# берём topK=14 кандидатов, порог score >= 0.35 — ниже уже не родняThe order in the pipeline is as follows: generated an article → linked → translated → and only then indexed. A new article is looking for relatives among already indexed cannot be its own candidate.
Layer 3. We find a sentence, and not just anywhere
This is the main difference from the script above. It’s not enough to know that articles are related—you have to understand in what sentence a link would be appropriate.
- We cut the article into sentences: we throw out headings, lists and ancillary comments, take only paragraphs, divide them by points, and leave sentences longer than 30 characters. Let’s take the first 60.
- We consider embeddings of all offers and all candidate headings to be one batch request.
- For each candidate, we look for a proposal with the maximum cosine. Threshold 0.72 — if not a single proposal is enough, we don’t put the link at all.
The last rule is more valuable than anything else: zero links are better than an inappropriate link. That is why we have 2.3 links to the article, and not 4 out of 4.
Layer 4. Live text anchor
We don’t invent an anchor, but cut it out of a found sentence. We slide the window into 6, 5, 4 and 3 words and count how many words of the window coincide with the words of the target title – we compare by the first five letters so that cases and endings do not interfere. We take the best window; We require at least one match and an anchor length of 8 characters.
Honestly about the weak point: the “one match” threshold is sometimes not enough. On this very article, the phrase “Internal links are the same” coincided with the title “Internal duplicate pages” with one root – and the stub of the phrase went into the anchor. On regular articles there are more matches and the anchor comes out normal, but if you repeat it, raise the threshold to two matches.
Layer 5. Insert safely
We look for the first occurrence of the anchor in HTML and run six checks before inserting it. Any one fails – we look for the next occurrence, if we don’t find any – the candidate is skipped.
- Not inside a tag. We compare the positions of the last “<” and “>” before the insertion point: if the opening bracket is later, we are inside the attribute.
- Not inside an existing link. We count open and closed “a” up to this point.
- Not in the title. If the last open h1-h6 comes after the last closed one, we are inside the header.
- Not inside the code. Same logic for
and
: the link in the example breaks what the reader is copying. We ran into this on this very article - the link got inside the prom. - Not on myself. We compare the target address with the address of the article itself. It sounds obvious, but we found one of these for 372 articles.
- Not twice for one article. If the text already contains a link to this address, we do not add a second one. All links lead to different articles.
Important detail: the last two checks live in the insertion function itself, not in the calling code. All three branches pass through it, so neither the algorithm nor the model can bypass the rule. The ceiling is 3 links per run; after processing, the article is marked with a service comment.
What's in total
Embeddings decide who is related to whom. Offer embeddings decide where exactly put a link. Morphology provides the anchor. Three checks prevent the layout from being broken. The 0.35 and 0.72 thresholds are what this is all about: they allow the system to remain silent when there is no suitable place.
Where can this be improved next?
Some of what we advised the reader, we have already done at our own place. We write honestly what already works and what doesn’t yet.
- Done: the model adds space for the link. Previously, it was like this: if there is no suitable offer, there is no link. Now the last step is a thread that asks the model to weave the mention into one paragraph and return it rewritten. Edits are accepted only if seven checks are passed: the anchor is 2-4 words, not from the ban list, is in the paragraph verbatim, not at the beginning, without HTML, the length of the paragraph is within ±30%, and most importantly - the meaning of the rewritten paragraph coincides with the original one, at least 0.90 in terms of embeddings. Even if one doesn’t pass, we don’t touch the paragraph.
- Done: anchor threshold raised to two matches, and the anchor itself is reduced to 2-4 words. On a sample of six articles, this replaced stubs like “The server automatically routes requests: if the model” with the normal “OpenAI-compatible API” and “private mesh networks.” There are fewer links - and rightly so: the system refuses to post when there is no meaningful anchor.
- Done: the insert bypasses the code. Checks for
and
have been added alongside the others. - All that remains is to check the results with Search Console. The similarity of meanings is a hypothesis. Fact: the two pages have the same queries. The connection “vector found → GSC confirmed” turns a guess into a solution. This is something we haven't done yet.
None of these points prevents you from starting: the basic version from the guide already gives results, and improvements make sense when there are a lot of articles and the cost of an error grows.
If you don't want to mess with the code
The same result can be obtained through Claude Code - insert the promt, it will do everything itself. All the rakes above are sewn inside the promt so that the model does not step on them again.
Create an internal linking plan for my website.
1. Collect the URL from the sitemap: https://MY-SITE/sitemap.xml, index - go around the nested ones.
Multilingual? Leave ONE language (for example /ru/).
2. Throw out the pages of categories, tags, author and the root of the site - these are not articles.
They have a short text, they are “similar” to everything in the section and will fill the report.
3. From the pages, take the title, h1 and the first 1500 characters of text.
First remove nav, header, footer, aside, script - otherwise the menu will be counted
as content and as “the link is already there.”
4. Collect existing internal links, but ONLY those inside <p> paragraphs.
The author block, signatures and “similar posts” widgets are not considered linking.
5. <a href="https://blog.krasovskiy.team/ru/seo-y-kontent-marketynh-praktycheskoe-rukovodstvo-dlia-brendov/">Cut off the brand tail from the title (after |</a>, ➣, —, -), but do not reset the title,
if it consists of one brand.
6. Count the embeddings: sentence-transformers,
paraphrase-multilingual-MiniLM-L12-v2, normalize_embeddings=True.
7. For each article, find the closest ones in meaning, skip those
which are already linked, and offer a maximum of 4 new ones.
Do not take proximity below 0.35 - this is no longer related.
8. Show articles to which no links FROM the TEXT of other articles lead.
If the crawl is incomplete, honestly say that this metric is unreliable.
9. Give the plan: where, where, proximity, anchor hint. Save it in interlink_plan.csv.What's next
Linking and cannibalization are two sides of the same coin, and are calculated on the same vectors. There you are looking for articles that compete with each other and reduce the positions of both, here, on the contrary, you connect relatives. If you have done one thing, do the second, the work has already been done.
Do the final check in Search Console: if two pages rank for the same queries, that is not kinship but cannibalization, and links will not fix it.
If something doesn’t add up or the script complains, write to us and we’ll sort it out.
Andrey Krasovskiy is a programmer and data scientist experienced in building complex automated systems with Python, Google Colab and n8n. His expertise spans SEO ecosystems, API integrations (Ahrefs, Google Ads, Search Console) and content pipelines. Andrey combines technical precision with an entrepreneurial mindset to build solutions that deliver real results.