Automating Sitemap Submissions to Google Search Console
This playbook provides a step-by-step guide to automating sitemap submissions to Google Search Console using Google Cloud technologies. It covers setting up the project, creating service accounts, developing and containerizing your application, deploying to Google Cloud Run, and scheduling the submission of your sitemap through Cloud Scheduler. This guide is one of many ways to automate your sitemap submission, and while it may not be the best way, it works and is easily reproducible.
While it would be much faster to navigate most of these steps through the CLI, it’s also easier to get lost along the way for those less familiar with Google Cloud.
Prerequisites
Before you begin, ensure you have the following:
A Google Cloud account with billing enabled. If you are not an administrator, make sure you have contact with someone who can grant the necessary permissions.
A GitHub account.
Step-by-Step Guide
Set Up Your Project
Go to Google Cloud Console to set up a new project or use a preferred existing project.
Select the Organization or Project in the top left and click ‘New Project’ or navigate to your preferred existing project. Name your project and select "Create" if you’ll be using a new project.
Enable Required APIs
Go to APIs & Services and select "ENABLE APIS AND SERVICES".
This brings you to the API Library. Search for and enable the following APIs:
Create and Authorize Service Account
To create the Service Account, navigate to IAM & Admin > Service Accounts and select "CREATE SERVICE ACCOUNT".
2. Follow Provide a name and description for the service account.
3. Grant the necessary permissions. The service account will require permissions that most align with the ‘Basic/Owner’ role, so I recommend starting with this role while setting up and then narrowing access later based on policy use.
4. Grant yourself permission to the service account and select "Done".
5. Select the newly created Service Account and navigate to the "Keys" tab. Generate a service account key to store in the Secret Manager. Create as a JSON file, and it will download. You’ll upload this file into the Secret Manager shortly.
6. Navigate to Secret Manager and select "Create Secret".
7. Enter a name and upload the key JSON file. Default values for other fields are acceptable, but customize as needed. Select "CREATE SECRET".
8. Finally, grant domain-wide permission to your new service account by navigating back to the newly created Service Account and opening "Advanced Settings". In the Domain-wide Delegation section, copy the Client ID.
9. Navigate to the Google Workspace Admin Console by clicking the provided link. On the left-hand side, go to Security > Access and data control > API controls.
10. Select "MANAGE DOMAIN WIDE DELEGATION".
11. Select "Add New".
12. Paste the copied client ID and the required OAuth scope. For this use case, the required scope is ‘https://www.googleapis.com/auth/webmasters’.
13. Select "Authorize".
Enable OAuth
Now that we have APIs enabled and a Service Account created, we should enable OAuth. Go back to APIs & Services and navigate to "OAuth consent screen".
2. Follow necessary steps for Project Configuration. Keep in mind we are only setting up for internal use so this will only be exposed during testing. Here is a table to guide configuration:
| Tab | Step | Field | Description |
|---|---|---|---|
| Overview | App Information | App Name | Name of app |
| Overview | App Information | User support email | Contact email for users |
| Overview | Audience | Audience | Select Internal |
| Overview | Contact Information | Email addresses | Contact email for Google |
| Clients | Clients | Application type | Select Web application |
| Clients | Clients | Name | Name oAuth client |
| Data Access | Add or Remove Scopes | Your non-sensitive scopes | Add Google Search Console scopes |
Additional information can be provided, but the above should meet the requirements for setting up your client for service account use.
Creating Cloud Function
The script can be cloned from here. The script will run as a containerized Dockerfile, and the only modification needed will be to create a new `.env` file based on your own variables. Reference the `.env.example` file for guidance and the below table provides descriptions.
| Environment Variable Name | Description |
|---|---|
PROJECT_ID |
Your Google Cloud Platform project ID. |
SECRET_ID |
The secret ID for accessing secure credentials stored in Secret Manager. |
SITE_URL |
The main URL of your website. |
SITEMAP_URL |
The URL of the sitemap to be submitted. |
SERVICEDELEGATE |
The email address of the service delegate for accessing Google APIs. |
2. Navigate to Cloud Run and select "CONNECT REPO".
3. Ensure GitHub is selected and click "SET UP WITH CLOUD BUILD".
4. Set up the connection to GitHub if not already set up, and then select the Source Repository you cloned the script into. Remember to update the `.env` file with your variables if you have not already.
5. Ensure the main branch is matched and the Build Type is Dockerfile. Save.
6. Configure name, region, and billing settings as needed, but make sure to adjust Authentication to "Require authentication".
7. In the Container, Volumes, Networking, Security section, navigate to the Settings tab and ensure the Port matches the Port set in the script (Dockerfile and main.py files). The script is set to Port 8080 by default.
8. Go to the SECURITY tab and adjust the Service Account to the created Service Account.
9. Select "Create" and wait for the build to complete. Take note of the service URL, as we’ll need this later.
10. Once complete, you can test through Google CLI. Select “Show Information on Service’s URLs”, “INVOKE IN CLOUD SHELL”, authorize Cloud Shell, and press Enter to run.
Or, simply open Google Cloud Shell and run:
curl -H \ "Authorization: Bearer $(gcloud auth print-identity-token)" \ <YOUR-SERVICE-URL>
11. If set up correctly, you’ll receive “Sitemap submitted successfully” as the response.
Congrats! At this point, we have successfully deployed the service. While it is deployed, we don’t have anything that will trigger it automatically yet, so unless manually triggered, the service will not run. We’ll set that up next.
Set Up Cloud Scheduler
Navigate to Cloud Scheduler and select "CREATE JOB".
Provide Name, Region, Description, Frequency, and Timezone as needed. You can learn about cron formatting for frequency here, but the frequency I’ve applied (0 0 * * 0) will run once per week on Sundays at midnight.
3. Configure the Execution with the URL of your service (noted in Cloud Run), add OIDC as the Authorization (assign the created Service Account), and limit the Audience to the service URL as well. Cloud Scheduler may add additional HTTP Headers by default.
4. Leave Optional Configurations as default or adjust as needed, and select "Create". Force Run the job to confirm it runs successfully. Your sitemap submission is now successfully scheduled!
Job Alerts
One of the best parts of automation is no longer needing to think about it. It’s just done. But, there’s one final piece that we need to no longer “think” about sitemap submissions - alerting. While the job is currently scheduled and working as expected, it would be helpful to know if it ever stops working. We’ll keep this basic, but there’s plenty of customization in alerting that could be implemented if preferred.
Go to Alerting and select Error Reporting.
Select Configure Notifications and then Manage Notification Channels
Add an Email Channel
Add the newly created channel to receive error messages. Keep in mind that this will alert you for all errors, so customization would be needed if you only want alerts for the job created.
All Done.
That’s a wrap! I hope you were able to follow along and now have one less recurring task to think about. If you got lost somewhere along the way, leave a comment below or reach out and I’ll be happy to assist.
