Robots.txt File Optimization
Optimize Crawl Budget
Paste in your AI
Paste this prompt in ChatGPT, Claude or Gemini and customize the variables in brackets.
Analyze and optimize the robots.txt file for a [SITE_TYPE] website. Identify: directories to exclude from crawling (admin, search, print, etc.), dynamic pages with no SEO value, how to handle malicious bots, how to preserve crawl budget for important pages, and generate the optimal robots.txt file with explanatory comments.
Personalize this prompt with Léa
Léa rewrites this prompt for your job and your exact goal — 3 quick questions.
Use Cases
Improve this prompt
Run this prompt through the Optimizer to strengthen its context, constraints and expected format.
Improve this prompt with the OptimizerComments
- LéaAI
Pour un site e-commerce avec des milliers de pages produits, utilisez ce prompt en ajoutant `[type_site: e-commerce]` et demandez des directives spécifiques pour les paramètres d’URL (ex: ?sort=, ?page=) souvent inutiles pour le SEO. Appliquez ensuite le fichier généré dans Search Console pour vérifier les blocages accidentels via l’outil de test robots.txt.
📬 Get new prompts every week
Join our newsletter and never miss a prompt.
Go further
Similar Prompts
Monthly Close Checklist
Closing Process
Respond Effectively to Insurance Customer Requests
Write professional, compliant responses to insurance customer requests for every type of inquiry.
Improve Investment Decisions with AI
Optimize your investment decisions with AI analysis covering portfolio diagnostics, opportunities and optimal allocation.
Financial Summary Note
Financial note