I don't know which bots visit my site or what they take
We review server logs and your hosting or CDN settings (the network that serves your site from nearby servers). You get a list of AI bots that actually visited and the pages they opened.
Decide which AI bots read your website, and what they see
We check which AI crawlers visit your site and set clear rules for what they can read. We add an llms.txt file (a short guide to your site for language models) and tidy up robots.txt. You get a better chance of being found by AI assistants without exposing private pages.
Quote and schedule after your brief
Who it's for
- Site owners who want AI assistants to mention their business
- Companies with private areas: partner pricing, client portals, drafts
- Publishers, blogs and experts whose content may be reused without permission
- Online stores where bots put load on the server
Problems we solve
I want AI to recommend me, but I don't want to give away all my content
We separate bots that collect data for model training from bots that fetch answers for a user. You decide who gets in, and we write the rules.
My site is slow and bots add load
We add rate limits and close off utility pages. Your main content stays reachable.
I've heard of llms.txt but don't know if it matters
Honestly: it's not a standard every AI system reads. We add it as a cheap, safe hint and don't promise miracles.
What's included
- Audit of current robots.txt, sitemap and meta tags
- Log analysis: which AI bots visit and what they open
- Access policy: who is allowed, who is blocked
- Robots.txt rules for known AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others)
- Llms.txt with site structure and links to key pages
- Optional llms-full.txt with the text of important sections
- CDN or server-level rules for bots that ignore robots.txt
- Check that key pages are readable without heavy JavaScript
- Short handover document: what is configured and how to change it
Choose the scope
Use a package to describe what you need. We agree on the quote and schedule after reviewing your brief and requirements.
01
Start
Small site or landing page
- Robots.txt and sitemap audit
- AI bot access policy
- Llms.txt file
- Post-launch check
02
Control
Corporate site or online store
- Everything in Start
- Log analysis for the available period
- CDN or server-level rules
- Llms-full.txt for key sections
- Handover guide for your team
03
Monitored
High bot traffic or sensitive content
- Everything in Control
- Monthly report on AI bot visits
- Rule updates for new bots
- Review of page structure and content readability for crawlers
How we work
Brief and requirements
You describe the task and current workflow. We clarify requirements, choose a first stage and prepare a quote.
Audit and logs
We check what is already set up and which bots visit. We document the problems.
Access policy
We agree with you what to open and what to close, and explain the trade-offs.
Setup
We write robots.txt and llms.txt and add server rules. We test on a staging copy first if you have one.
Check and pilot
We agree on the schedule after clarifying scope and dependencies.
Scope and limits
- We don't promise AI will cite your site: those decisions belong to the AI systems
- We can't guarantee every bot obeys robots.txt: some ignore it
- We can't remove data already collected by models trained on your content earlier
FAQ
What is llms.txt?
It's a text file in your site's root with a short description of your business and links to main pages. It helps language models understand your structure faster. It's a community proposal, not a required standard.
Should I block AI bots?
It depends on your goal. If you want to be found through AI search, let the search bots in. Training bots can be blocked if your content is your main asset. We help you weigh it.
Will this affect my Google rankings?
We don't touch regular Googlebot. We configure Google-Extended separately, which covers Google's AI products. We show you every change before it goes live.
Do you need server access?
For the basics, access to site files or your CMS is enough. Log analysis and CDN rules also need hosting or CDN access, which can be limited and time-boxed.
How do you measure results?
We agree on a KPI before starting, for example visits from selected AI bots to key pages, or fewer unwanted requests. The data comes from logs or analytics, and you can see it yourself.
How much does it cost?
We prepare a quote after reviewing your brief and requirements. A first stage can cover: robots.txt and sitemap audit; AI bot access policy. Scope, payment and schedule are agreed before work starts.
Start with your task
Describe your business, the outcome you need and your current workflow. We will clarify the requirements and prepare a quote after your brief.