SEO feels like a foreign language to most developers out there. Trust me, I've been there (just recently). As a developer-turned-business-owner, I want to share my journey into the world of Search Engine Optimization. I’ve learned a lot and I want to share what I did, and what we are doing for our clients going forward.
Updated August 2026. The story below is from 2024, when I was working this out the hard way. Two years on, the basics have not moved. What did change is AI search, and it brought a failure mode I did not see coming. That part is at the end of the post.
The SEO Wake-Up Call
Picture this: I'm deep in code, probably dreaming in JavaScript, when my marketing manager drops a bombshell. "We need SiteMap.xml and Robots.txt files for our site." My first thought? "Uh... what now?" 😅
You see, I've always been the guy behind the scenes, crafting beautiful code and powerful functionality. SEO? That was for the marketing folks, right? Wrong. So wrong.
Who does SEO when you are the whole team?
Here's the thing about running a Micro-SaaS business. You wear all the hats. Developer, designer, marketer, you name it. And when it came to SEO for our own site, I couldn't just pass the buck. It was time to dive in headfirst.
Many of our Micro-SaaS tools are back-office apps with separate marketing sites. But for Hurrah.dev, I decided to take on the challenge myself. No more hiding behind pre-built solutions. It was time to understand this SEO beast.
What does robots.txt actually do?
First up: Robots.txt. Think of it as a friendly guide for search engine spiders. "Hey, Google Bot! Check this out, but maybe skip that admin section, okay?"
Creating this file was pretty straightforward. We set it up to exclude our admin sections and point towards our sitemap. Simple, right? But it got me thinking. How many developers overlook this crucial step?
Pro Tip: Always include a Robots.txt file. It's like leaving a map for search engines to find the good stuff on your site.
What belongs in sitemap.xml?
Now, SiteMap.xml. That's where things got interesting. With a limit of 50,000 URLs and 50MB, we had to be smart about what to include. It's not just about listing every single page; it's about highlighting what's important.
For our Single Page Application (SPA), this presented a unique challenge. How do you map out a site that technically lives on one page? The solution: focus on the content that matters most.
Pro Tip: For SPAs, consider server-side rendering or pre-rendering key pages. It helps search engines understand your content better.
How often should you update lastmod?
Diving deeper into SiteMap.xml, I encountered two critical elements: loc and lastmod. Sounds simple, right? Not so fast.
Creating the perfect URL for loc was like solving a puzzle. We ended up using article titles, replacing spaces with dashes. But then came the multiple domain issue. Turns out, canonical URLs are your best friend here.
As for lastmod, I learned that Google's pretty picky. Update it too often without significant changes, and you might as well be crying wolf. We tied it to our publish dates for blog posts and the latest content updates for our homepage.
Pro Tip: Be strategic with your lastmod dates. Only update them when you've made substantial changes to your content.
How do you build SEO into every client site?
Here's the kicker. We're not just doing this for ourselves. Every Micro-SaaS app we build needs to be SEO-friendly. So, we developed configurable settings to make our SEO features work for all our clients.
It's not about building a one-size-fits-all solution. It's about creating a framework that's flexible enough to adapt to each unique business need.
The Road Ahead: Always Learning, Always Improving
This journey into SEO has been eye-opening. We're not stopping at SiteMaps and Robots.txt. We're developing more tools to help our clients shine in search results. Think auto-configured meta tags, smart content suggestions, the works.
As developers, we have the power to make a huge impact on a site's visibility. It's not just about writing clean code anymore (though that's still super important). It's about writing code that search engines love too.
Updated for 2026: what AI search changed
Two years and a lot more audits later, including one on our own site, here is what I would add.
Do AI crawlers run your JavaScript?
No. And that one fact turns the 2024 "consider server-side rendering" tip into the whole ballgame.
We proved it on ourselves. In August 2026 we audited Hurrah.dev and found one of our own tools, the spreadsheet cost calculator, serving 11.7 KB of HTML with zero heading tags. Not one h1 through h6. Everything a reader saw was drawn by JavaScript after the page loaded. To a person with a browser the page looked finished. To a crawler it was an empty room.
Three signals agreed, and we had been reading each one separately for months: no headings, no FAQ schema, and an average Google position of 53. Its sibling page, the equity calculator, renders its explanatory copy on the server. That one carried 17.7 KB and eight headings.
The fix was not clever. We moved the explanatory copy out of the interactive component and into plain server-rendered markup outside it. The page now ships 18.2 KB with eight headings, all of it visible with JavaScript switched off, which puts it level with its sibling.
Pro Tip: Curl your own page and read what comes back. curl -s https://yoursite.com/some-page | grep -o '<h[1-6]' takes five seconds and shows you what a crawler actually gets. If it comes back empty, nothing else you do to that page matters.
For the record, Hurrah.dev is not the SPA I described in 2024. It runs on Blazor Server and prerenders its HTML, the same move we made for a client in our PHP to Blazor migration.
Should you block GPTBot and ClaudeBot in robots.txt?
This is the robots.txt question that did not exist in 2024. GPTBot, ClaudeBot, PerplexityBot and Google-Extended all read robots.txt, and all of them honour it.
We block none of them. Our robots.txt is still a single wildcard group that disallows /admin and /login and allows everything else. If you want an AI assistant to cite you, it has to be able to read you. Blocking the crawler is choosing to be invisible in the place people now ask their questions.
That is a decision, not a default. Publishers who license their archives make the opposite call for good reasons. Make it on purpose.
Do you need an llms.txt file?
Probably not, and I would not spend a sprint on it.
llms.txt returns a 404 on Hurrah.dev today, and we left it that way deliberately. Google Search does not consume it. It is a proposed convention with genuine momentum in some circles, but treating it as an AI visibility fix mistakes a wish for a mechanism. Fix your rendered HTML first.
Does any of this get you into AI Overviews?
Only indirectly, and this is the part most posts on the subject skip.
AI Overviews draw overwhelmingly from pages already ranking in the top few results. If your site averages position 15, the answer engine is not skipping you because your schema is wrong. It never got far enough down the results to see you at all. Structured data, headings and server rendering make you eligible. They do not make you rank.
So it is a rankings problem before it is an AI problem. Do the boring work first, in that order.
The lastmod advice held up, and it applies to this post
The 2024 version of me said only move lastmod when the content really changed. That still holds, and it is now the rule we hold ourselves to. A modified date that moves while the words stay the same does not just fail to help that page. It teaches a crawler to discount every other date on the site.
This post's modified date moved because everything under this 2026 heading is new. That is the deal.
Wrapping Up: The Developer's SEO Toolkit
So here's what two years of this has taught me:
- SEO isn't just for marketing. It's a crucial part of web development.
- Start with the basics: Robots.txt and SiteMap.xml are your foundation.
- Render your content on the server. No crawler, search or AI, runs your JavaScript.
- Check what a crawler receives rather than what your browser shows you.
- Be strategic with your URLs and update timestamps, including your own.
- Make your SEO features flexible and configurable.
- Eligibility is not ranking. Fix the rankings problem first.
- Never stop learning. SEO is always evolving, and now it has a second audience.
To all the fellow developers, just remember we're not just building websites anymore. We're building gateways for businesses to be discovered, by people and increasingly by the machines those people ask. So it's time to embrace SEO and make our apps not just functional, but discoverable too!
Building a new web app? Start with the right foundation with SEO included from day one. Or see how we catch problems before clients report them.
