Technical SEO is the part of search engine optimisation that has nothing to do with what your pages say and everything to do with whether Google can find them, read them and trust them. If your content is good but you still don't show up, there is a fair chance a technical problem is holding you back.
This guide covers the technical SEO basics we check first on every audit, in the order we check them. For each one you'll see what it is, why it matters and how to check it yourself with free tools — mostly Google Search Console, which every site owner should set up before anything else.
1. Crawling: can Google reach your pages?
Before Google can rank a page it has to crawl it — fetch the URL and download the HTML. Googlebot discovers pages by following links and by reading your XML sitemap.
The things that stop crawling are usually simple:
- robots.txt blocks. A single line like
Disallow: /left over from a staging site blocks your whole website. Openyourdomain.com/robots.txtand read it. - Orphan pages. A page that no other page links to is hard for Google to find, even if it is in the sitemap.
- Server errors. Pages that return 5xx errors or time out get crawled less often. Your hosting matters.
- Links Google can't follow. Navigation built only with JavaScript click handlers instead of normal
<a href>links.
How to check: In Search Console, use URL Inspection on an important page and look at “Crawl allowed?” and “Page fetch”. The Settings → Crawl stats report shows how many requests Google makes each day and how many failed.
2. robots.txt: tell crawlers where not to go
robots.txt controls crawling, not indexing. That difference trips up a lot of people. If you block a URL in robots.txt, Google can't read it — so it also can't see a noindex tag on it, and the URL may stay in search results with no description.
Good uses of robots.txt:
- Internal search result pages (for example URLs with
?q=or?s=). - Endless filter and sort combinations on large shops.
- Admin, cart and checkout areas.
Never block the CSS and JavaScript files your pages need to render, and always add the full URL of your sitemap at the bottom: Sitemap: https://www.yourdomain.com/sitemap.xml.
3. Indexing: is the page actually in Google?
A crawled page is not automatically indexed. Google decides whether a page is worth keeping. In Search Console's Pages report (under Indexing) you'll see reasons such as:
- Crawled – currently not indexed: Google read it and decided not to index it, usually because the content is thin, duplicated or low value.
- Discovered – currently not indexed: Google knows the URL but hasn't crawled it yet, often a sign of weak internal links or too many low-value URLs.
- Excluded by ‘noindex’ tag: fine if you meant it, a disaster if a developer left it on your service pages.
- Duplicate, Google chose different canonical than user: Google thinks another URL is the main version.
A technical SEO fundamental that is often missed: not every page should be indexed. Thank-you pages, internal search results, tag archives and near-identical location pages usually do more harm than good in the index. A smaller set of genuinely useful pages beats thousands of thin ones.
4. XML sitemaps: a list of the pages you want indexed
An XML sitemap is a file that lists your important URLs. It helps Google discover new and updated pages faster. A clean sitemap:
- Contains only URLs that return HTTP 200, are indexable and are the canonical version.
- Leaves out redirects, 404s,
noindexpages and duplicates. - Uses the exact hostname and protocol you use everywhere else (for example always
https://www.). - Is split into several files with a sitemap index once you pass 50,000 URLs or 50 MB.
- Has a
<lastmod>date that changes only when the page really changes.
Submit it in Search Console under Sitemaps and check it again after big site changes.
5. Canonical tags: one URL per page
The same page can often be reached through several URLs: with and without www, with and without a trailing slash, with tracking parameters, or as /index.html and /. Google sees those as separate pages unless you tell it which one is the main version.
Add a self-referencing canonical tag to every indexable page:
<link rel="canonical" href="https://www.yourdomain.com/service-page.html">
Then make sure your sitemap, internal links and canonical tags all point to the same URL. Mixed signals are one of the most common reasons Google picks a different canonical than you wanted.
6. Redirects and HTTP status codes
Every URL returns a status code. The ones that matter for SEO:
| Code | Meaning | When to use it |
|---|---|---|
| 200 | OK | Every page you want indexed. |
| 301 | Moved permanently | A page has a new URL. Point it at the closest equivalent page. |
| 302 | Moved temporarily | Short-term moves only. Don't use it for permanent changes. |
| 404 | Not found | The page never existed or was removed with no replacement. |
| 410 | Gone | The page was removed on purpose and won't come back. |
| 5xx | Server error | Never on purpose. Fix the server or hosting. |
Two rules save a lot of trouble. First, avoid redirect chains (A → B → C); point A straight at C. Second, don't redirect every missing page to the homepage. Google treats that as a “soft 404”, and visitors who wanted a specific page get a confusing result.
7. HTTPS and one preferred hostname
Your whole site should load on HTTPS with a valid certificate, and every variant (http://, non-www, old subdomains) should 301-redirect in one hop to your preferred version. Check it by typing each variant into a browser and watching where it ends up.
8. Mobile-friendliness
Google uses the mobile version of your page for indexing and ranking. That means the mobile page must contain the same main content, headings, structured data and links as the desktop page. Check for:
- Text readable without zooming and buttons large enough to tap.
- No content hidden on mobile that exists on desktop.
- No pop-ups covering the main content on load.
9. Page speed and Core Web Vitals
Core Web Vitals are Google's user-experience metrics, measured on real visitors:
- LCP (Largest Contentful Paint): how fast the main content appears. Aim for 2.5 seconds or less.
- INP (Interaction to Next Paint): how quickly the page responds when someone taps or clicks. Aim for 200 milliseconds or less.
- CLS (Cumulative Layout Shift): how much the layout jumps while loading. Aim for 0.1 or less.
The fixes that help most small business sites: compress and resize images (WebP or AVIF), set width and height on images, load fonts efficiently, remove unused plugins and scripts, use caching, and choose hosting with fast response times in India if most of your visitors are here. Check your pages in PageSpeed Insights and the Core Web Vitals report in Search Console. See Core Web Vitals explained for more detail.
10. Structured data
Structured data (schema markup, usually written as JSON-LD) tells search engines what a page is about in a format they can read directly: your business name, address and phone number, a service, an article, a product with price and availability, or a list of FAQs. It can make pages eligible for rich results, though Google decides when to show them.
Only mark up what is visible on the page, keep it accurate, and test it with Google's Rich Results Test. Our guide to schema markup for SEO has examples.
11. Internal linking and site structure
Internal links tell Google which pages matter most and how topics connect. A simple structure works best: homepage → main service pages → supporting articles, with every important page reachable in three clicks or fewer. Use descriptive anchor text (“local SEO services”, not “click here”) and link from your articles back to the service page they support.
12. JavaScript rendering
Google can render JavaScript, but it takes extra time and resources, and some content never gets rendered properly. If your main content, links or metadata only appear after scripts run, check the rendered HTML with URL Inspection → View tested page. For important pages, serve the key content in the initial HTML.
Technical SEO checklist
- Search Console verified and sitemap submitted.
- robots.txt allows important pages, CSS and JS, and lists the sitemap.
- Important pages return 200, are indexable and have a self-referencing canonical.
- Sitemap lists only canonical, indexable 200 URLs.
- All http, non-www and old hostnames 301 to one preferred HTTPS version in one hop.
- No redirect chains, no mass redirects to the homepage, real 404s for missing pages.
- Thin, duplicate and utility pages set to
noindexor removed. - Mobile page has the same content as desktop.
- Core Web Vitals in the “good” range for your main templates.
- Valid structured data that matches visible content.
- Every important page linked from at least one other relevant page.
When to get help
Most of these checks take an afternoon with Search Console and PageSpeed Insights. If you find indexing problems you can't explain, a sudden drop in impressions, or a site with thousands of URLs, a full SEO audit is the next step. Our SEO services start with exactly this technical review before any content or link work.
