How to Find and Fix Common Technical SEO Issues on Your Website

<p>Let's say you've built a website. You've invested in the design, bought a domain, and so on. But a website isn't only what you see on the screen. A lot happens behind the scenes. You have to understand how the internet works, as well as how Search Engine Optimization (SEO) works.</p> <p>Speed is a key part of it, too. A slow site can have very weak SEO, because most people won't spend much time waiting for a page to load, especially these days.</p> <p>Site speed is a well-known topic with many problems of its own to consider, like Largest Contentful Paint (LCP), Cumulative Layout Shift (CLS), and server speed. But I'll leave those for another article.</p> <p>This one is about the problems that stop search engines from reaching and reading your pages.</p> <p>In my opinion, technical SEO is much harder to apply than regular SEO, which focuses on content and keywords. The technical side usually requires a developer's knowledge, and might need some changes that can feel a bit uncomfortable if you don't have a technical background.</p> <p>A technical problem can stay unnoticed for months, because you usually don't see it. It's not a design problem. You only find it when you run specific checks, for example with <a href="https://pagespeed.web.dev/">Google PageSpeed Insights</a>.</p> <p>In this tutorial, you'll find and fix the most common ones on a small practice site. I didn't want to take a real website and shame its owners, so I built a practice site with the problems already in it.</p> <h2 id="heading-table-of-contents">Table of Contents</h2> <ul> <li><a href="#heading-prerequisites">Prerequisites</a></li> <li><a href="#heading-what-is-technical-seo">What Is Technical SEO?</a></li> <li><a href="#heading-how-common-are-these-problems-on-real-websites">How Common Are These Problems on Real Websites?</a></li> <li><a href="#heading-how-to-set-up-the-practice-site">How to Set Up the Practice Site</a></li> <li><a href="#heading-how-to-check-a-pages-http-status-code">How to Check a Page's HTTP Status Code</a><ul> <li><a href="#heading-what-is-a-soft-404">What Is a Soft 404?</a></li> </ul> </li> <li><a href="#heading-how-to-find-and-fix-redirect-chains">How to Find and Fix Redirect Chains</a></li> <li><a href="#heading-how-to-make-sure-robotstxt-isnt-blocking-important-pages">How to Make Sure robots.txt Isn't Blocking Important Pages</a></li> <li><a href="#heading-how-to-find-an-accidental-noindex">How to Find an Accidental noindex</a><ul> <li><a href="#heading-how-to-check-the-robots-meta-tag">How to Check the Robots Meta Tag</a></li> <li><a href="#heading-how-to-check-the-x-robots-tag-header">How to Check the X-Robots-Tag Header</a></li> </ul> </li> <li><a href="#heading-how-to-use-canonical-tags-correctly">How to Use Canonical Tags Correctly</a></li> <li><a href="#heading-how-to-fix-missing-and-duplicate-titles-and-meta-descriptions">How to Fix Missing and Duplicate Titles and Meta Descriptions</a></li> <li><a href="#heading-how-to-make-your-links-crawlable">How to Make Your Links Crawlable</a></li> <li><a href="#heading-how-to-check-whether-your-content-needs-javascript-to-appear">How to Check Whether Your Content Needs JavaScript to Appear</a></li> <li><a href="#heading-can-you-check-a-page-in-five-minutes">Can You Check a Page in Five Minutes?</a></li> <li><a href="#heading-conclusion">Conclusion</a></li> </ul> <h2 id="heading-prerequisites">Prerequisites</h2> <p>You don't need to be an experienced programmer to follow along. You'll need:</p> <ul> <li>A web browser, such as Chrome, Firefox, Edge or Safari. In Safari, first turn on <strong>Show features for web developers</strong> under Settings &gt; Advanced.</li> <li>A terminal: Terminal on macOS, PowerShell on Windows, or any terminal on Linux.</li> <li><code>curl</code>, a small program that downloads web pages. It comes with macOS, most Linux distributions, and Windows 10 (version 1803 and later) and Windows 11.</li> <li><a href="https://nodejs.org/">Node.js</a>, to run the practice site. The code was tested with Node.js 26.8 and curl 8.7.</li> </ul> <p>On Windows PowerShell, type <code>curl.exe</code> instead of <code>curl</code>, because there <code>curl</code> is a shortcut to a <a href="https://learn.microsoft.com/en-us/windows/curl/">different command</a>. Replace <code>| grep -i "text"</code> with <code>| Select-String "text"</code>, and <code>/dev/null</code> with <code>NUL</code>.</p> <h2 id="heading-what-is-technical-seo">What Is Technical SEO?</h2> <p>Google has programs called crawlers. They search the internet all the time for new websites and new pages. When Google discovers a new address, it may crawl the page and process what's on it (the text, images, and videos).</p> <p>If Google decides the page is worth indexing, it may add it to a very big database of web pages, called the index. Google can then consider the pages in its index for its search results, when they're relevant to what someone searches for. Google's guide to <a href="https://developers.google.com/search/docs/fundamentals/how-search-works">how Search works</a> calls these three steps crawling, indexing, and serving.</p> <p>So crawling doesn't guarantee indexing, and indexing doesn't guarantee a place in the results. The same guide says Google doesn't guarantee any of the three stages, even for a page that follows all of its rules.</p> <p>Fixing technical problems won't necessarily make your site rank first on Google. But it does strengthen your chances of ranking higher, because every problem you fix makes your site better.</p> <p>Technical SEO has to come together with regular SEO, like keywords, backlinks, engagement on social media, and so on.</p> <h2 id="heading-how-common-are-these-problems-on-real-websites">How Common Are These Problems on Real Websites?</h2> <p>The numbers below come from audits run with the site-auditing tool I build. Each website counts once, using its first audit, and this data comes from October 25th.</p> <p>Crawls check many pages on one site (a median of 10 pages here). Single-page audits check one page, usually the home page.</p> <table> <thead> <tr> <th>Problem</th> <th>How often</th> <th>Sample</th> </tr> </thead> <tbody><tr> <td>At least one page with no canonical tag</td> <td>64.2% of sites</td> <td>53 crawled sites, Aug 2025 to Oct 2026</td> </tr> <tr> <td>Two or more pages with exactly the same title</td> <td>61.8% of sites</td> <td>34 crawled sites with two or more pages that loaded</td> </tr> <tr> <td>At least one page with no meta description</td> <td>49.1% of sites</td> <td>53 crawled sites</td> </tr> <tr> <td>At least one link leading to a page that returns 404</td> <td>18.9% of sites</td> <td>53 crawled sites</td> </tr> <tr> <td>Page built mostly with JavaScript</td> <td>17.7% of pages</td> <td>277 single-page audits, May to Oct 2026</td> </tr> <tr> <td>At least one page with no title</td> <td>13.2% of sites</td> <td>53 crawled sites</td> </tr> <tr> <td>A link that search engines may not be able to follow</td> <td>8.3% of pages</td> <td>278 single-page audits</td> </tr> <tr> <td>A robots.txt file with errors, or one that failed to load</td> <td>7.9% of sites</td> <td>252 single-page audits, Jul to Oct 2026</td> </tr> <tr> <td>Page blocked by noindex or by robots.txt</td> <td>4.8% of pages</td> <td>251 single-page audits, Jul to Oct 2026</td> </tr> </tbody></table> <p>These are all technical problems, but they don't all weigh the same. Missing canonical tags are at the top, but Google doesn't require a canonical tag, so a page without one isn't broken by itself. It matters when the same or very similar content is reachable at more than one URL, as you'll see in the canonical section.</p> <p>Duplicate titles are something I had myself: when you write code, it's easy to miss things like that. That's why there are tools that check for these problems.</p> <p>Let's go over each issue in the table to make sure they're all clear:</p> <ul> <li><strong>Missing canonical tag:</strong> a canonical tag is basically one of the many tags you put in the <code>&lt;head&gt;</code> of your HTML. When two or three pages have pretty much the same content, it tells search engines like Google which URL is the original, main version, the one you prefer to get indexed. It's not required, though: without it, Google picks the main version itself, so it matters mostly for pages that have duplicates.</li> <li><strong>Duplicate titles:</strong> it's really easy to end up with the same title on two pages. You don't want a duplicated title because it's not helpful and can confuse Google. Each page should say what it's about.</li> <li><strong>Missing meta description:</strong> this one is really important, because it helps search engines as well as AI tools like ChatGPT. The meta description is there to explain very briefly what the page is about and what it's for. If you go to Google and search for a website, you'll see it under the actual link to the site.</li> <li><strong>A link to a 404 page:</strong> this basically means you have a broken link (maybe a link you probably don't use anymore) to a page that isn't there. So there's no reason for it to lead to something that doesn't exist. Use it to lead to something similar, or to something that does exist, so your links remain relevant.</li> <li><strong>A page built mostly with JavaScript:</strong> Google can run JavaScript, so this alone doesn't keep a page out of Google. But the content appears only after the JavaScript loads and runs, which is slower for visitors and one more step that can fail, and not every crawler runs JS. That's why I prefer <a href="https://www.freecodecamp.org/news/server-side-rendering-in-next-js-for-improved-seo/">server-side rendering</a> for the content that matters most.</li> <li><strong>A link search engines can't follow:</strong> this isn't a huge deal, and it really depends where the link goes. But say I wrote a really nice article and I want it to help me rank on Google. If search engines can't follow the link to it, then you invested all this time and energy in this article that relatively few people will likely see.</li> <li><strong>Errors in robots.txt:</strong> robots.txt basically helps you talk to search engines. If there are errors in this file, you might stop search engines from crawling pages you do want in search results.</li> <li><strong>A page blocked by noindex or robots.txt:</strong> these are two different things. A <code>noindex</code> rule asks search engines to keep a page out of their results, while a robots.txt rule only stops them from crawling it. You'll learn more about both, and why the difference matters, later in this tutorial.</li> </ul> <p>Soft 404s, redirect chains, and wrong canonical tags aren't in the table, because the audits don't record them.</p> <h2 id="heading-how-to-set-up-the-practice-site">How to Set Up the Practice Site</h2> <p>The practice site is a tiny website for a made-up shoe store called Acme Shoes. Every page on it has one problem built in. You don't need to understand every line of the code.</p> <p>Create a new folder, add a file called <code>server.js</code>, and paste in this code:</p> <pre><code class="language-javascript">// server.js: a tiny practice site with common technical SEO problems built in const http = require('node:http'); const PORT = 8080; const BASE = 'http://localhost:' + PORT; // Builds a simple HTML page. Leave out a field to leave that tag out. function page({ title, description, canonical, robots, body, script }) { return [ '&lt;!doctype html&gt;', '&lt;html lang="en"&gt;', '&lt;head&gt;', '&lt;meta charset="utf-8"&gt;', title &amp;&amp; '&lt;title&gt;' + title + '&lt;/title&gt;', description &amp;&amp; '&lt;meta name="description" content="' + description + '"&gt;', canonical &amp;&amp; '&lt;link rel="canonical" href="' + BASE + canonical + '"&gt;', robots &amp;&amp; '&lt;meta name="robots" content="' + robots + '"&gt;', '&lt;/head&gt;', '&lt;body&gt;', body, script &amp;&amp; '&lt;script&gt;' + script + '&lt;/script&gt;', '&lt;/body&gt;', '&lt;/html&gt;', ] .filter(Boolean) .join('\n'); } const routes = { '/': { html: page({ title: 'Acme Shoes: Running Shoes for Every Runner', description: 'Lightweight running shoes for road and trail, with free returns.', canonical: '/', body: [ '&lt;h1&gt;Acme Shoes&lt;/h1&gt;', '&lt;a href="/products/blue-runner"&gt;Blue Runner&lt;/a&gt;', '&lt;span onclick="location.href=\'/sale\'"&gt;Summer sale&lt;/span&gt;', ].join('\n'), }), }, // Problem: a vague title and no meta description '/products/blue-runner': { html: page({ title: 'Acme Shoes', canonical: '/products/blue-runner', body: '&lt;h1&gt;Blue Runner&lt;/h1&gt;', }), }, // Problem: a missing product that still answers "200 OK" (a soft 404) '/products/red-runner': { html: page({ title: 'Acme Shoes', body: '&lt;h1&gt;Sorry, that product could not be found.&lt;/h1&gt;', }), }, // Problem: a redirect chain, /old-sale -&gt; /sale-2025 -&gt; /sale '/old-sale': { redirect: '/sale-2025' }, '/sale-2025': { redirect: '/sale' }, '/sale': { html: page({ title: 'Summer Sale: Up to 40% Off Running Shoes | Acme Shoes', description: "Save up to 40% on last season's running shoes until August 31.", canonical: '/sale', body: '&lt;h1&gt;Summer sale&lt;/h1&gt;', }), }, // Problem: a noindex header nobody can see in the page itself '/about': { headers: { 'X-Robots-Tag': 'noindex' }, html: page({ title: 'About Acme Shoes', description: 'How a two-person workshop became a running shoe brand.', canonical: '/about', body: '&lt;h1&gt;About us&lt;/h1&gt;', }), }, // Problem: a canonical tag that points to the home page '/blog/choosing-trail-shoes': { html: page({ title: 'How to Choose Trail Running Shoes | Acme Shoes', description: 'Grip, drop and fit: what to look for in a trail running shoe.', canonical: '/', body: '&lt;h1&gt;How to choose trail running shoes&lt;/h1&gt;', }), }, // Problem: the content only exists after JavaScript runs '/catalog': { html: page({ title: 'All Running Shoes | Acme Shoes', description: 'Browse every Acme running shoe.', canonical: '/catalog', body: '&lt;div id="app"&gt;Loading...&lt;/div&gt;', script: "fetch('/api/products').then((r) =&gt; r.json()).then((products) =&gt; {" + "document.getElementById('app').innerHTML = products.map((p) =&gt; '&lt;h2&gt;' + p + '&lt;/h2&gt;').join('');" + '});', }), }, '/api/products': { type: 'application/json', html: JSON.stringify(['Blue Runner', 'Trail Grip 2', 'City Glide']), }, // Problem: "Disallow: /blog" blocks every path that starts with /blog '/robots.txt': { type: 'text/plain', html: 'User-agent: *\nDisallow: /blog\n', }, }; http .createServer((req, res) =&gt; { const path = new URL(req.url, BASE).pathname; const route = routes[path]; if (!route) { res.writeHead(404, { 'Content-Type': 'text/html; charset=utf-8' }); return res.end(page({ title: 'Page not found', body: '&lt;h1&gt;Page not found&lt;/h1&gt;' })); } if (route.redirect) { res.writeHead(301, { Location: route.redirect }); return res.end(); } res.writeHead(200, { 'Content-Type': (route.type || 'text/html') + '; charset=utf-8', ...route.headers, }); res.end(route.html); }) .listen(PORT, () =&gt; console.log('Practice site running at ' + BASE)); </code></pre> <p>Each key in <code>routes</code> is a page address. The comments that start with <code>Problem:</code> say what's wrong with each page.</p> <p>Start the server from the same folder:</p> <pre><code class="language-bash">node server.js </code></pre> <p>You should see:</p> <pre><code class="language-text">Practice site running at http://localhost:8080 </code></pre> <p>Leave the server running and open a second terminal for the commands below. When you change <code>server.js</code>, stop the server with Ctrl+C and start it again.</p> <h2 id="heading-how-to-check-a-pages-http-status-code">How to Check a Page's HTTP Status Code</h2> <p>I remember when I started building websites, I built all kinds of small things. I'm a full stack developer, and I kept trying experiments.</p> <p>Once I had a problem on the server side and saw in the Network tab that it returned a 500 error. Another time I removed a page on purpose, because I wanted to see if I'd get a 404 or other problems. If you just play around with the site you're building, you'll learn a lot from the results.</p> <p>HTTP status codes are the first thing a search engine reads. They're the server's way of talking to the client, such as your browser. The codes are split into five ranges: 1xx, 2xx, 3xx, 4xx, and 5xx. A 5xx code is a server problem, for example, and a 4xx code is a client-side problem.</p> <p>404 is the most common error. It means a page on the site wasn't found. You see 404s all the time, and the fix is typically just to maintain your site and your links.</p> <p>The codes you'll see in this tutorial are:</p> <ul> <li><code>200 OK</code>: here's the page.</li> <li><code>301 Moved Permanently</code>: the page lives at another address.</li> <li><code>404 Not Found</code>: there's no page here.</li> </ul> <p>Google doesn't index URLs that return a <code>4xx</code> code, according to its documentation on <a href="https://developers.google.com/search/docs/crawling-indexing/http-network-errors">HTTP status codes</a>.</p> <p>To see the status code of the practice site's home page, use <code>-I</code>, which shows only the headers, and <code>-s</code>, which hides curl's progress meter:</p> <pre><code class="language-bash">curl -sI http://localhost:8080/ </code></pre> <pre><code class="language-text">HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Date: Sun, 25 Oct 2026 17:42:53 GMT Connection: keep-alive Keep-Alive: timeout=5 </code></pre> <p>Now ask for an address that isn't in <code>routes</code>, to see what a correct 404 looks like:</p> <pre><code class="language-bash">curl -sI http://localhost:8080/products/does-not-exist </code></pre> <pre><code class="language-text">HTTP/1.1 404 Not Found Content-Type: text/html; charset=utf-8 Date: Sun, 25 Oct 2026 17:42:53 GMT Connection: keep-alive Keep-Alive: timeout=5 </code></pre> <p>In the browser, open the developer tools (F12, or Cmd+Option+I on a Mac), go to the <strong>Network</strong> tab, and reload. The <strong>Status</strong> column shows the code.</p> <h3 id="heading-what-is-a-soft-404">What Is a Soft 404?</h3> <p>Check a product the store no longer sells:</p> <pre><code class="language-bash">curl -sI http://localhost:8080/products/red-runner </code></pre> <pre><code class="language-text">HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Date: Sun, 25 Oct 2026 17:42:53 GMT Connection: keep-alive Keep-Alive: timeout=5 </code></pre> <p>In the browser, the same page says "Sorry, that product could not be found."</p> <p>A soft 404 is basically a "page not found", but the server still answers <code>200</code>. It confuses Google, because the address is still there, but what's on it isn't current anymore.</p> <p>To fix it, make missing pages return a real <code>404</code>, which is what <a href="https://support.google.com/webmasters/answer/7440203">Google recommends</a>. In the practice site, delete the <code>'/products/red-runner'</code> entry from <code>routes</code>. The server then answers <code>404</code>.</p> <p>Broken links lead to the same dead end. In the data, 18.9% of the 53 crawled sites had at least one link to a page that returned <code>404</code>.</p> <p>So don't leave links that lead to something that doesn't exist. Update them to lead to something similar, or to something that does exist.</p> <p>If the page moved, add a <code>301</code> redirect from the old address to the new one. A <code>404</code> is the right answer only for a page that's really gone.</p> <h2 id="heading-how-to-find-and-fix-redirect-chains">How to Find and Fix Redirect Chains</h2> <p>A redirect is basically when you want to send one URL (perhaps one that no longer exists) to another URL (that does exist). You might set up a redirect if, for example, you change your domain from <code>example-x.com</code> to <code>example-y.com</code>. You'll want everyone who types <code>example-x.com</code> to be redirected to <code>example-y.com</code> automatically.</p> <p>Redirects are fine, and they can be quite helpful when used properly. A <code>301</code> means the page moved for good. A <code>302</code> means the move is temporary. Google's documentation on <a href="https://developers.google.com/search/docs/crawling-indexing/301-redirects">redirects</a> explains how it treats each one, and its crawlers <a href="https://developers.google.com/search/docs/crawling-indexing/http-network-errors">follow up to 10 redirect hops</a>.</p> <p>But a redirect chain happens when an old address, such as a domain you used in the past, redirects to a new one, and then that one redirects again, and again...you get the picture. A few redirects aren't a big deal, but five, six, or seven can be a real problem.</p> <p>The first command below shows every step of the chain. The second gives you a one-line summary.</p> <p>Add <code>-L</code> so curl follows the redirects:</p> <pre><code class="language-bash">curl -sIL http://localhost:8080/old-sale </code></pre> <pre><code class="language-text">HTTP/1.1 301 Moved Permanently Location: /sale-2025 Date: Sun, 25 Oct 2026 17:42:53 GMT Connection: keep-alive Keep-Alive: timeout=5 HTTP/1.1 301 Moved Permanently Location: /sale Date: Sun, 25 Oct 2026 17:42:53 GMT Connection: keep-alive Keep-Alive: timeout=5 HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Date: Sun, 25 Oct 2026 17:42:53 GMT Connection: keep-alive Keep-Alive: timeout=5 </code></pre> <p><code>/old-sale</code> goes to <code>/sale-2025</code>, which goes to <code>/sale</code>. For a one-line summary:</p> <pre><code class="language-bash">curl -sL -o /dev/null -w "%{num_redirects} redirects, ended at %{url_effective}\n" http://localhost:8080/old-sale </code></pre> <pre><code class="language-text">2 redirects, ended at http://localhost:8080/sale </code></pre> <p><code>-o /dev/null</code> tells curl to write the page's HTML nowhere, so it doesn't fill your terminal and only the summary line is printed. <code>-w</code> sets what that summary line says.</p> <p>To fix it, point the old/original address straight at the final page:</p> <pre><code class="language-javascript">'/old-sale': { redirect: '/sale' }, </code></pre> <p>Restart the server and run the summary again. It now says <code>1 redirects, ended at http://localhost:8080/sale</code>.</p> <h2 id="heading-how-to-make-sure-robotstxt-isnt-blocking-important-pages">How to Make Sure robots.txt Isn't Blocking Important Pages</h2> <p>robots.txt is the site's way of telling search engines like Google (and their crawlers) what they're allowed to explore and where they have access.</p> <p>Look at the practice site's file:</p> <pre><code class="language-bash">curl -s http://localhost:8080/robots.txt </code></pre> <pre><code class="language-text">User-agent: * Disallow: /blog </code></pre> <p><code>Disallow: /blog</code> is dangerous because it stops crawlers from reading every path that starts with <code>/blog</code>. And usually, your articles are what promote your site the most.</p> <p>Google's <a href="https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt">robots.txt documentation</a> gives the same kind of example: <code>Disallow: /fish</code> also blocks <code>/fish.html</code> and <code>/fishheads</code>.</p> <p>To fix it, make the rule match only what you meant to block, for example a folder of drafts:</p> <pre><code class="language-text">User-agent: * Disallow: /blog-drafts/ </code></pre> <p>In the practice site, change the <code>html</code> line of the <code>'/robots.txt'</code> entry:</p> <pre><code class="language-javascript">html: 'User-agent: *\nDisallow: /blog-drafts/\n', </code></pre> <p>robots.txt controls crawling, not indexing. Google says it's <a href="https://developers.google.com/search/docs/crawling-indexing/robots/intro">not a mechanism for keeping a page out of Google</a>. For that, you need <code>noindex</code>.</p> <p>A URL blocked in robots.txt can even appear in search results, usually without a description, if other pages link to it. Google just never reads what's on the page.</p> <h2 id="heading-how-to-find-an-accidental-noindex">How to Find an Accidental noindex</h2> <p><code>noindex</code> tells Google not to index a particular page. It's harmful when it ends up on pages you do want indexed and ranked, like a product page or your home page.</p> <p>But it can be useful on pages you don't need to rank for, or that don't add much to the site, such as log in, sign in, and sign up pages.</p> <p><code>noindex</code> works only when the crawler can reach the page and read the rule. If robots.txt blocks the same URL, Google never sees the <code>noindex</code>, and the page can still show up in search results. So for a page you want out of Google, use <code>noindex</code> and leave the page crawlable.</p> <p>In the data, 11 of 457 audited pages (2.4%) had a <code>noindex</code> robots meta tag. There are <a href="https://developers.google.com/search/docs/crawling-indexing/block-indexing">two ways to set it</a>, and you need to check both.</p> <h3 id="heading-how-to-check-the-robots-meta-tag">How to Check the Robots Meta Tag</h3> <p>The first way is a tag in the page's <code>&lt;head&gt;</code>:</p> <pre><code class="language-html">&lt;meta name="robots" content="noindex"&gt; </code></pre> <p>Search the page for the word:</p> <pre><code class="language-bash">curl -s http://localhost:8080/about | grep -i noindex </code></pre> <p>On the About page, this prints nothing. You can also right-click the page, choose <strong>View Page Source,</strong> and search for <code>noindex</code>.</p> <p>Fixing a noindex is really easy. You basically remove it, and then search engines can index the page.</p> <p>Look for it in the page's code or template, and in your CMS or SEO plugin settings. In WordPress, the setting <strong>Discourage search engines from indexing this site</strong> adds it to every page.</p> <h3 id="heading-how-to-check-the-x-robots-tag-header">How to Check the X-Robots-Tag Header</h3> <p>The second way is an HTTP header. Check the About page's headers:</p> <pre><code class="language-bash">curl -sI http://localhost:8080/about </code></pre> <pre><code class="language-text">HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 X-Robots-Tag: noindex Date: Sun, 25 Oct 2026 17:42:59 GMT Connection: keep-alive Keep-Alive: timeout=5 </code></pre> <p>The header doesn't appear in View Page Source, only in the response headers. In the data, 5 of the 12 blocked pages had no <code>noindex</code> in their HTML: 2 had this header, and 3 were blocked from crawling by robots.txt.</p> <p>To fix it in the practice site, delete the <code>headers</code> line from the <code>'/about'</code> entry. On a real site, check the server configuration, the hosting provider's header settings, and any SEO plugin.</p> <h2 id="heading-how-to-use-canonical-tags-correctly">How to Use Canonical Tags Correctly</h2> <p>If you have several pages that talk about more or less the same thing, but without enough meat to be different, you want to designate one main page. The canonical tag tells search engines which page that is.</p> <p>The same page can also live at several addresses:</p> <pre><code class="language-text">https://example.com/blog/choosing-trail-shoes https://example.com/blog/choosing-trail-shoes?utm_source=newsletter https://www.example.com/blog/choosing-trail-shoes </code></pre> <p>These three addresses show the same article. The first one is the main one: the address you want Google to show in search results. So all three carry the same canonical tag, pointing to that main page:</p> <pre><code class="language-html">&lt;link rel="canonical" href="https://example.com/blog/choosing-trail-shoes"&gt; </code></pre> <p>When Google sees this tag on the <code>?utm_source=newsletter</code> version, it knows that page is a copy of the first address. The main address carries the same tag, pointing to itself. That's called a self-referencing canonical, and it says: I'm the main version.</p> <p>Google treats it as a <a href="https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls">strong signal, not a command</a>. If you use canonical tags, a self-referencing one on each main page is a good practice, and Google recommends it.</p> <p>But you don't have to use canonical tags at all. Google says specifying a canonical isn't required: without one, it picks the version it thinks is best to show users. A canonical tag is a good choice when the same or very similar content is reachable at more than one URL, like the three addresses above.</p> <p>That's why a missing canonical tag isn't a defect by itself. In the data, 27.1% of 457 audited pages had no canonical tag. What matters is whether any of them have duplicates, like a version with tracking parameters, that need one.</p> <p>Check the practice site's trail shoe article:</p> <pre><code class="language-bash">curl -s http://localhost:8080/blog/choosing-trail-shoes | grep -i canonical </code></pre> <pre><code class="language-text">&lt;link rel="canonical" href="http://localhost:8080/"&gt; </code></pre> <p>The article says its main version is the home page, so Google may show the home page instead of it, or ignore the tag.</p> <p>The article isn't a copy of anything, so it's its own main version. To fix it, point the article's canonical tag to the article itself:</p> <pre><code class="language-javascript">canonical: '/blog/choosing-trail-shoes', </code></pre> <h2 id="heading-how-to-fix-missing-and-duplicate-titles-and-meta-descriptions">How to Fix Missing and Duplicate Titles and Meta Descriptions</h2> <p>A good title or meta description is the right length, isn't overloaded, and says briefly what the page is about. It's sort of the essence of the page, like its introduction.</p> <p>Look at the Blue Runner product page:</p> <pre><code class="language-bash">curl -s http://localhost:8080/products/blue-runner | grep -i -E "&lt;title&gt;|name=\"description\"" </code></pre> <pre><code class="language-text">&lt;title&gt;Acme Shoes&lt;/title&gt; </code></pre> <p>The title says nothing about the page, and there's no description line at all. In PowerShell, use <code>| Select-String '&lt;title&gt;|name="description"'</code> for this one.</p> <p>Google's guides to <a href="https://developers.google.com/search/docs/appearance/title-link">titles</a> and <a href="https://developers.google.com/search/docs/appearance/snippet">snippets</a> ask for a unique, descriptive title and description on every page.</p> <p>To fix it in the practice site, replace the <code>title</code> line of the <code>'/products/blue-runner'</code> entry with:</p> <pre><code class="language-javascript">title: 'Blue Runner Road Running Shoe | Acme Shoes', description: 'A lightweight road running shoe with a cushioned sole and free returns.', </code></pre> <h2 id="heading-how-to-make-your-links-crawlable">How to Make Your Links Crawlable</h2> <p>Look at the practice site's home page:</p> <pre><code class="language-bash">curl -s http://localhost:8080/ </code></pre> <pre><code class="language-text">&lt;!doctype html&gt; &lt;html lang="en"&gt; &lt;head&gt; &lt;meta charset="utf-8"&gt; &lt;title&gt;Acme Shoes: Running Shoes for Every Runner&lt;/title&gt; &lt;meta name="description" content="Lightweight running shoes for road and trail, with free returns."&gt; &lt;link rel="canonical" href="http://localhost:8080/"&gt; &lt;/head&gt; &lt;body&gt; &lt;h1&gt;Acme Shoes&lt;/h1&gt; &lt;a href="/products/blue-runner"&gt;Blue Runner&lt;/a&gt; &lt;span onclick="location.href='/sale'"&gt;Summer sale&lt;/span&gt; &lt;/body&gt; &lt;/html&gt; </code></pre> <p>Everything else on this page is fine: the title, the description, the canonical tag and the Blue Runner link, which is a real <code>&lt;a&gt;</code> element with an <code>href</code>. The only problem is the "Summer sale" <code>&lt;span&gt;</code>.</p> <p>A <code>&lt;span&gt;</code> is basically HTML that isn't semantic, meaning it has no meaning. A real link does have meaning: search engines can understand it's a link, and screen readers can understand it's a link.</p> <p>Which links need to be crawlable? Basically, everything you want search engines to see. I mean the products you sell, your articles, your courses, and other things like that.</p> <p>Google's <a href="https://developers.google.com/search/docs/crawling-indexing/links-crawlable">link guidelines</a> say it can only reliably crawl an <code>&lt;a&gt;</code> element with an <code>href</code>.</p> <p>To fix it in the practice site, replace the <code>&lt;span&gt;</code> line with:</p> <pre><code class="language-javascript">'&lt;a href="/sale"&gt;Summer sale&lt;/a&gt;', </code></pre> <h2 id="heading-how-to-check-whether-your-content-needs-javascript-to-appear">How to Check Whether Your Content Needs JavaScript to Appear</h2> <p>With client-side rendering, the JavaScript has to load first, and only then can the other content load.</p> <p>JavaScript usually builds things like product lists, prices, reviews and menus. Without it, a client-side page often shows only a loading message or an empty box.</p> <p>Google does render JavaScript, but <a href="https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics">in a later phase</a>. The page waits in a rendering queue, which can take a few seconds or longer, and a script that fails or is blocked can leave the rendered page without its content.</p> <p>That's why server-side rendering is still a good idea, and it's what I prefer for the content that matters. Google's JavaScript guide says the same: server-side rendering makes the site faster for users and crawlers, and not all bots can run JavaScript.</p> <p>In the data, 17.7% of 277 audited pages were built mostly with JavaScript. That doesn't mean Google couldn't index them, only that their main content depended on JavaScript running correctly.</p> <p>With server-side rendering, the HTML content comes straight to the browser, and only after that does the JavaScript render.</p> <p>Google can index pages built either way. Its crawler runs JavaScript with an up-to-date version of Chrome, so server-side rendering isn't a requirement for getting into Google's index.</p> <p>The next test checks whether the important content is visible without JavaScript. It doesn't show what Google sees, because Google runs the JavaScript, but it shows what every crawler gets before any JavaScript runs.</p> <p>Open <code>http://localhost:8080/catalog</code> in your browser. You'll see three products. Now download it with curl, which doesn't run JavaScript:</p> <pre><code class="language-bash">curl -s http://localhost:8080/catalog | grep -i "trail grip" </code></pre> <p>This prints nothing, because the product names aren't in the HTML the server sends. Inside <code>&lt;body&gt;</code>, the server only sends:</p> <pre><code class="language-html">&lt;div id="app"&gt;Loading...&lt;/div&gt; &lt;script&gt;fetch('/api/products').then((r) =&gt; r.json()).then((products) =&gt; {document.getElementById('app').innerHTML = products.map((p) =&gt; '&lt;h2&gt;' + p + '&lt;/h2&gt;').join('');});&lt;/script&gt; </code></pre> <p>Without curl, use <strong>View Page Source</strong> in your browser. On this page, you'll see "Loading..." and the script, but none of the product names, because the browser adds them only after the JavaScript runs.</p> <p>On your own site, search the source for an important sentence from the page.</p> <p>To see the page the way Google renders it, use Google's own tools on a live page. On your own site, the URL Inspection tool in Search Console can run a live test (<strong>Test live URL</strong>, then <strong>View tested page</strong>) and show the HTML Google rendered and a screenshot.</p> <p>For any public page, the <a href="https://search.google.com/test/rich-results">Rich Results Test</a> shows the rendered HTML too. Google <a href="https://developers.google.com/search/docs/crawling-indexing/javascript/fix-search-javascript">recommends both</a> for finding JavaScript problems. Neither can reach <code>localhost</code>, so try them on your real site.</p> <p>If you want the product names in the HTML from the start, render them on the server. In the practice site, add the product list above <code>routes</code>:</p> <pre><code class="language-javascript">const products = ['Blue Runner', 'Trail Grip 2', 'City Glide']; </code></pre> <p>Replace the catalog's <code>body</code> and <code>script</code> lines with:</p> <pre><code class="language-javascript">body: '&lt;div id="app"&gt;' + products.map((p) =&gt; '&lt;h2&gt;' + p + '&lt;/h2&gt;').join('') + '&lt;/div&gt;', </code></pre> <p>And in the <code>'/api/products'</code> entry:</p> <pre><code class="language-javascript">html: JSON.stringify(products), </code></pre> <p>Restart the server and run the curl command again:</p> <pre><code class="language-text">&lt;div id="app"&gt;&lt;h2&gt;Blue Runner&lt;/h2&gt;&lt;h2&gt;Trail Grip 2&lt;/h2&gt;&lt;h2&gt;City Glide&lt;/h2&gt;&lt;/div&gt; </code></pre> <p>This works because the server now writes the product names into the HTML before it sends the page. Any crawler sees them, even one that can't run JavaScript, and you can still use JavaScript for the interactive parts.</p> <h2 id="heading-can-you-check-a-page-in-five-minutes">Can You Check a Page in Five Minutes?</h2> <p>Honestly, it's very hard. There can be a lot of problems, and five minutes isn't enough.</p> <p>To really know what the problems are, you need to run a full check, for example in Google PageSpeed Insights. If the site is yours, also use the <a href="https://support.google.com/webmasters/answer/9012289">URL Inspection tool</a> in Google Search Console.</p> <p>The commands from this tutorial are a good start:</p> <ol> <li><code>curl -sI</code> for the status code and an <code>X-Robots-Tag</code> header.</li> <li><code>curl -sIL</code> on an old address to count redirects.</li> <li><code>/robots.txt</code>, line by line.</li> <li>View Page Source for <code>noindex</code>, <code>canonical</code>, <code>&lt;title&gt;</code>, <code>description</code> and your important content.</li> </ol> <h2 id="heading-conclusion">Conclusion</h2> <p>In this tutorial, you fixed eight problems on a practice site: a soft 404, a redirect chain, a robots.txt rule, a hidden <code>noindex</code> header, a wrong canonical tag, a vague title and a missing description, a link crawlers couldn't follow, and content that needed JavaScript.</p> <p>The most important thing to remember is that you'll need to check your site all the time. Run scans, do maintenance, and make sure nothing breaks, especially when you make improvements or changes to the site.</p> <p>If you want to go further, my tutorial on <a href="https://www.freecodecamp.org/news/nextjs-json-ld-structured-data/">adding structured data (JSON-LD) to a Next.js site</a> is a good next step.</p> <p>If you'd like to check a live page for many of these issues in one go, my free tool <a href="https://www.greadme.com/page-audit">Greadme Page Audit</a> tests any URL with no sign-up needed.</p>