Automating Sitemap Submissions to Google Search Console
When I added a Programmable Search Engine to my website, I expected it to improve how users found relevant content - I touched on how our expectations of search functionality is really high, and Squarespace integrated search just doesn’t quite meet those expectations. Adding Programmable Search Engine from Google definitely modernizes search to perform closer to expectations when it comes to interpreting intent, but the search results weren’t up-to-date! Results led to pages I’ve deleted, and newer pages that I published weren’t showing up.
Google hadn't crawled my site in a while, which left search functionality relying on stale indexing.
A great search engine is useless if it delivers outdated content. From a business perspective, poor search results is really frustrating, lowers engagement, and minimizes the opportunities your users have to get what they’re looking for. It would arguably be better to remove and rely on hierarchical navigation at that point. So, if I’m going to use this search engine I really need to make sure that my site is regularly crawled!
Solutioning
There are great solutions out there to do this - Yoast is probably the most popular. And, they’re really not that expensive considering how much they do for you. Good value! But, I don’t need something as robust as they offer just to make Search work better and I can do a proper feature analysis later on if I start having SEO issues that I care about. For now, I’m just a curious cheapskate and would rather build something myself that sorts out my specific use case.
If I just wanted to be able to quickly submit sitemaps, that’s a really easy ask. Squarespace auto-generates an XML of your sitemap (plenty of free ways to do this if you’re not using Squarespace), and Google Search Console has an API with an OAuth flow for submitting programmatically. There are a lot of examples “on the line” to do this if you want to run it locally. But, I don’t want to do it locally - I want it to run without me or my computer being involved. This was partly out of curiosity, but it was mostly because I’ll need very similar services set up for future projects. Might as well get it set up now and the effort for future projects can go down significantly.
Did it work?
Yep! And for those interested, I’ve detailed the steps in a playbook posted here.
It took awhile for me to figure this out, but it will pay dividends for future work. My solution uses a Cloud Run Service and is scheduled to submit my sitemap weekly through Cloud Scheduler. And, the cost is really minimal - while API usage and Cloud resources do have costs, a solution like this doesn’t run enough to invoke any costs. I did get a bill for $0.86 while testing, but I’ll charge that to my ignorance on handling authentication through service accounts.
All in, this will ensure that Google regularly receives updated sitemaps, prompting it to crawl my site more frequently. This simple yet effective automation removed any manual effort and ensured that my search engine reflects the latest site updates. I’ll need to think about other search engines in the future, especially considering the rise of ChatGPT Search and it’s reliance on Bing, but that’s a separate problem.
What’s Next?
With my highest priority item checked off my list when it comes to taxonomy implementation, I’m going to take a break from website optimization to work on a project. I would like to build out a data dashboard that can live on my website, so I’ll start with some business analysis on how that could work. This will diversify the content on my website, thus increasing the priority of some of my other taxonomy features like content tagging. For now, it all falls into the same buckets primarily so I’m having a hard time getting excited about adding something like that from a search and analytics perspective.
