Teach Your Agent How Websites Work: llms.txt Discovery in OpenClaw 2026.2.17

OpenClaw 2026.2.17 quietly shipped a feature that could change how your AI agent interacts with the web: automatic llms.txt discovery. If you're not familiar with llms.txt, you're about to see why it matters.

What Is llms.txt?

The llms.txt standard is like robots.txt, but for AI. Instead of telling crawlers what not to index, it tells LLMs how to use a website:

• What APIs are available
• How to authenticate
• What actions are permitted
• Natural language descriptions of functionality
• Rate limits and usage guidelines

When a website publishes an llms.txt file at their root (e.g., https://example.com/llms.txt), any AI agent can read it to understand how to interact with that site properly.

How OpenClaw Uses It

With llms.txt discovery enabled (now the default in 2026.2.17), your OpenClaw agent will automatically:

1. Check for llms.txt when browsing or fetching from a new domain
2. Parse the instructions and incorporate them into its context
3. Follow the guidelines for that site's APIs and actions

This means less prompt engineering on your end. Instead of manually teaching your agent how to use each website, the sites themselves provide the instructions.

Real-World Example

Imagine asking your agent to interact with a SaaS product:

Before llms.txt:

After llms.txt:

Enabling/Disabling

The feature is enabled by default in 2026.2.17. To disable it:

agents:
  defaults:
    behavior:
      llmsDiscovery: false

Why This Matters

The llms.txt standard is gaining adoption. Major API providers, SaaS tools, and developer platforms are adding these files. By enabling discovery by default, OpenClaw agents become:

• More capable out of the box
• Less error-prone when interacting with new services
• Better citizens of the web (respecting rate limits, using proper endpoints)

Looking Ahead

As more sites adopt llms.txt, your agent will automatically benefit—no updates needed on your end. It's a prime example of how standards can multiply capability.

Check out the llms.txt spec at llmstxt.org if you're curious about the format, and consider adding one to your own projects.

Reference: OpenClaw PR #18158 — Thanks @yolo-maxi for the contribution!