How Grok Bot handles this
A crawl takes longer than a chat, and a Bot is still at work after you leave. You publish a batch of product pages late in the day, attach @xspeedhub and ask the Bot to warm shop and tell you in the morning how it went. It calls start_preloader, and the plugin on the site queues the sitemap and fetches the pages in the background. Before it reports, the Bot reads get_preloader_status for the processed count, the last URL and any errors.
The plugin was added once from chat as a custom remote HTTPS MCP server and is account-wide, so any of your Bots can warm a site with it. If the crawl stalls with pages answering 403 or 406, the status message names the warmer's user-agent, xSpeed-Warmer/1.0, which a server firewall rule needs to allow, and the Bot can pass that on to whoever runs the server. If the module is off, the Bot can propose turning it on with update_settings, and stop_preloader is there if the crawl slows the site; pages already warmed stay cached.
Grok Bot offers Allow once, Always allow or Deny when an action needs approval. Always allow saves a rule, so later starts stop asking. Auto-review rules under Settings, General, Auto-review can force Ask first or allow matching actions, and Ask first wins when both match. xSpeed Hub has no confirmation step of its own, and it calls the preloader low-risk, asking the agent to confirm only when the site is unclear or the call covers every site. Add Ask first rules for start_preloader, stop_preloader and update_settings, so each one waits for your Allow once.
Set up Grok Bot once
Already connected? Skip to the prompts. Alternatives and troubleshooting are on the Grok Bot guide.
1Put your sites in xSpeed Hub
Sign in at app.xspeedcache.com with Google or email; the Hub is free and has no site cap. Then connect each WordPress site from its own dashboard: click Connect Hub in the xSpeed Cache top bar, then Connect via xSpeed Hub. Each site needs the free xSpeed Cache plugin.
2Ask your Bot to add xSpeed Hub
In a chat with your Bot, ask it to add a custom remote HTTPS MCP server with this URL. xAI says a plugin you add from your own chat is personal and account-wide, so every Bot on your account can use it. When the browser opens, approve access on the xSpeed Hub page.
Add a custom remote HTTPS MCP server called xspeedhub with the URL https://app.xspeedcache.com/xspeed/mcp and sign me in.
3Attach it and ask a first question
Type @ in a chat to attach the xspeedhub plugin to the task, then ask something that names xSpeed Hub. A list of your sites means the connection works.
Use @xspeedhub (xSpeed Hub) to list my sites and tell me which ones have page caching turned off.
Full Grok Bot setup, sign-in options and FAQ
Before you send a prompt that changes something
Reads change nothing on your sites, though contact_support emails xSpeed support. Writes run once your connection allows them, so your client's approval prompt and a read-only connection (the token's Read-only everywhere switch, or a Viewer sign-in) are the gates that matter. When an action needs approval, Grok Bot offers Allow once, Always allow or Deny. Always allow saves a rule, so the same kind of call no longer stops for you. Auto-review rules under Settings, General, Auto-review can force Ask first or allow matching actions automatically, and Ask first wins when both match. xSpeed Hub has no confirmation step of its own, so a Hub write tool you always allow runs as soon as your connection allows writes. Add Ask first rules for purge_cache, update_settings, toggle_cache, optimize_site and the other write tools. For read-only access, sign in as a Viewer member of the workspace.
What do I ask?
Three prompts written for Grok Bot. More for this job are below.
Use @xspeedhub (xSpeed Hub) to start the preloader on shop and report the processed count in the morning.
With xSpeed Hub, read the preloader status on every site and list any crawl that shows 403 or 406 errors.
Use xSpeed Hub to stop the preloader on blog. It is slowing the server during the import.
What happens, step by step
The first visitor to an uncached page pays for building it. The preloader fixes that by visiting your pages first. Through xSpeed Hub an agent can start the crawl on one site or on every site in a workspace, watch how far it has got, and stop it. The crawl runs inside WordPress in the background; the agent only starts it, reads its progress and stops it.
01
Check the cache and the preloader
get_cache_status tells the agent whether page caching is on, because there is nothing to fill when it is off. get_preloader_status shows whether a crawl is already running, so the agent does not start a second one.
02
Make sure the preloader is switched on
The preloader module is off on a fresh install, and the site refuses start_preloader until it is on. The agent can read the module with get_settings and, if you agree, switch it on with update_settings on the preloader module. The Hub tells the agent to briefly confirm intent before an update_settings call, but that is guidance to the agent, not a lock.
03
Start the crawl
start_preloader makes the site read its sitemap, follow any nested sitemap indexes, drop URLs that your cache exclusion rules skip, and queue the rest. The call returns once the queue is built; the pages are then fetched in the background, a few per run. For every site at once the agent passes site: "all", and the Hub starts the sites one after another with a one-second pause.
04
Watch the progress
get_preloader_status reports whether a crawl is running, how many URLs it has processed out of the total, the last URL it visited and the most recent errors. It works on one site at a time, so for several sites the agent calls it once per site.
05
Read what failed
A page that answers 403 or 406 usually means a server firewall is blocking the warmer by its user-agent, not that the page is broken. The status message names the user-agent to allow. A crawl that queues no URLs is reported as an error with the reason, not as a success.
06
Stop it if the server struggles
stop_preloader ends a running crawl. It is safe mid-run: pages already warmed stay cached. The agent can stop one site or every site in the workspace.
Reference
| Start and stop | start_preloader and stop_preloader (write) |
|---|---|
| Progress | get_preloader_status (read): running or not, processed out of total, last URL, recent errors |
| What gets crawled | The sitemap (the WordPress one at /wp-sitemap.xml unless you set another), nested indexes followed, excluded URLs skipped |
| Needs | The preloader module switched on (it is off by default) and page caching on |
| Pages per run | 5 by default, 1 to 50, set in the preloader module |
| Schedule setting | manual, hourly, daily or weekly, set in the preloader module |
| Every site at once | start_preloader and stop_preloader accept site: "all"; get_preloader_status does not |
| Fan-out pacing | Sites start one after another with a 1 second pause, one result per site |
| Warmer user-agent | xSpeed-Warmer/1.0, the string a firewall rule would need to allow |
| Stopping | Safe mid-run; warmed pages stay cached |
| Read-only access | A Viewer member's sign-in, or the connection token with Read-only everywhere on: start_preloader and stop_preloader are refused; get_preloader_status works. An OAuth sign-in by anyone else gets the scopes the client asks for. |
| Confirmation | None on the Hub; your client's prompt is the gate |
Rules worth keeping
- Warm after a purge, not instead of one. The Hub tells the agent that starting the preloader after a purge saves the next visitor from paying for the rebuild.
- A crawl requests every page in the sitemap, so it adds load to the origin. Start with one site, and on a small host keep Pages per run low or stop the crawl if the site slows.
- start_preloader and stop_preloader are write tools. The Hub calls the preloader low-risk and only asks the agent to confirm when the target site is unclear or the call covers every site, so your client's approval prompt and read-only access (a Viewer sign-in, or the connection token with Read-only everywhere on) are the gates that matter.
- A fan-out only starts the crawls. Check each site with get_preloader_status afterwards, and treat some sites failing to start as a partial result, not a failure of the whole call.
Good to know with Grok Bot
A routine is a good fit for reading a crawl and a poor fit for starting one. Routines run on the Bot's cloud computer on a schedule with nobody to answer an approval, and xAI notes that a Test run performs real work, including calls to connected tools, so testing a routine that starts the preloader starts a real crawl. The preloader module has its own Schedule setting in WordPress, manual, hourly, daily or weekly, which handles routine warming without an agent, so keep the Bot's routine to a morning report from get_preloader_status on each site. The Hub's Read-only everywhere switch belongs to the connection token, which this OAuth plugin does not send; a sign-in as a Viewer member is the read-only option. On a team plan, a Cursor connector policy can block the plugin, which shows as Disabled by team admin.
More prompts for this job
They work in any client connected to xSpeed Hub.
Use xSpeed Hub to start the preloader on shop and tell me how many URLs it queued.
Using xSpeed Hub, how far through its crawl is blog? Show processed against total and any errors.
With xSpeed Hub, purge the cache on docs, then warm it again.
Use xSpeed Hub to start the preloader on every site in my workspace and tell me which ones refused.
Use xSpeed Hub to stop the preloader on shop. It is slowing the server down.
The preloader on shop says it is disabled. Use xSpeed Hub to turn the module on, then start the crawl.
Using xSpeed Hub, which of my sites has no running crawl and a low hit ratio?
Frequently asked questions
Keep going
Warm the cache with other agents
More with Grok Bot
Documentation
- How to warm your cache ahead of visitors
- How to crawl your sitemap
- How to enable Page Cache
- Managing your fleet with xSpeed Hub
- How to write prompts for xSpeed Hub
- Grok Bot + xSpeed
- Every AI agent that works with xSpeed
From the blog