If you want AI models like Claude, ChatGPT, Gemini, and Perplexity to properly understand and reference your website, you need an llms.txt file. In this guide, I'll show you exactly how to generate one using the LLMS.txt Generator app.
What is llms.txt?
LLMS.txt is a standardized text file format that converts all your website content into clean markdown that AI models can efficiently process. Unlike regular HTML pages with navigation, popups, sidebars, and styling, llms.txt provides pure content that AI can index and cite accurately.
Why Your Website Needs llms.txt
- Better AI visibility: AI models can understand your content without parsing complex HTML
- Accurate citations: Clean content structure improves how AI references your pages
- Reduced processing: Text-only format means faster AI indexing
- Future-proof: As AI search grows, llms.txt becomes standard practice
Step-by-Step Tutorial
Step 1: Access the LLMS.txt Generator
Go to the LLMS.txt Generator on Apify and click "Try for free"
Step 2: Configure Your Input Settings
- Domain name: Enter your website URL (e.g., onescales.com)
- Website/Business name: Your company or site name
- Short description: A brief description of what your site offers
- Website category: Select from SaaS, e-commerce, service, media, blog, or other
These details will appear at the top of your llms.txt file to give AI models context about your website.
Step 3: Start the Crawl
Click "Save and start"
The app will now:
- Find your sitemap automatically
- Visit every page on your website
- Extract only the content (no headers, footers, navigation)
- Convert each page to markdown format
- Generate your llms.txt files
Processing time:
- Small sites (100 pages): 5-10 minutes
- Medium sites (1,000 pages): 30-60 minutes
- Large sites (10,000+ pages): Several hours
Step 4: Review Your Results
Once the run succeeds, scroll to the bottom to see your results table. You'll find:
- Individual .md files: One markdown file for each page on your site
- llms.txt: Summary of all pages with links to detailed content
- llms-full.txt: All page content combined in one file
- ZIP file: Download all files at once
Step 5: Download Your Files
Click the ZIP file link in the results table to download everything to your computer. Unzip the file to access:
- llms.txt (main file)
- llms-full.txt (combined content)
- Individual .md files for each page
Step 6: Upload to Your Website
Option A: Direct Upload to Root Directory
If your CMS allows root directory access:
- Upload llms.txt to yourdomain.com/llms.txt
- Upload llms-full.txt to yourdomain.com/llms-full.txt
- Upload all .md files to your root or a /content/ folder
Option B: Upload with Redirect (Shopify, Wix, etc.)
If you can't access the root directory:
- Upload files to your file storage (e.g., CDN, media library)
- Create a 301 redirect from yourdomain.com/llms.txt to the uploaded file location
- Repeat for llms-full.txt
Example for Shopify:
- Upload to: cdn.shopify.com/files/llms.txt
- Redirect: onescales.com/llms.txt → cdn.shopify.com/files/llms.txt
Step 7: Verify Your Installation
Visit yourdomain.com/llms.txt in your browser to confirm it's accessible. You should see clean text with your site summary and page listings.
Advanced Settings
Custom Sitemap URL
If your sitemap isn't at the default location (/sitemap.xml), specify the full URL:
https://yourdomain.com/custom-sitemap.xml
Specify Exact URLs
Instead of crawling the entire sitemap, you can list specific pages to process:
https://yourdomain.com/page1
https://yourdomain.com/page2
https://yourdomain.com/page3
Limit Maximum Pages
For large sites, limit the number of pages to process:
- Set "Maximum number of pages to process" to your desired limit (e.g., 10,000)
Pause Between Pages
If your server struggles with rapid requests, add a delay:
- Set "Pause between pages (ms)" to 1000-3000ms
Markdown File Path Configuration
If you're uploading .md files to a specific folder:
- Set "Path for markdown files" to match your upload location
- Example:
/cdn/shopify/files/content/
Skip Markdown Files
If you only want llms.txt and llms-full.txt without individual page files:
- Change "URL format in llms.txt" to "Original URLs"
- Leave "Path for markdown files" empty
Troubleshooting Failed URLs
If some pages fail to process:
- Go to the Export tab and download the CSV
- Check column D for failed URLs
- Copy the Run ID from the top of the page
- Start a new run with:
- The Run ID in the "Retry failed from run ID" field
- Failed URLs pasted in the "Start URLs" field
- Click Start
The app will reprocess only the failed pages and merge results with your successful crawl.
Understanding the File Formats
llms.txt Format
# One Scales
One Scales creates and monetizes apps on the Apify platform
## About Us
Learn about our mission and team
> /content/about-us.md
## Blog: AI Tools Guide
Complete guide to AI automation tools
> /content/blog-ai-tools.md
llms-full.txt Format
Contains all page content in one continuous file - useful for AI models that prefer processing everything at once rather than following links.
Individual .md Files
Each page gets its own markdown file with clean, formatted content extracted from the HTML.
Best Practices
- Update regularly: Regenerate your llms.txt file when you publish new content
- Monitor file size: llms-full.txt can get large - some AI models have file size limits
- Prioritize important pages: Use the URL list feature to process key pages first
- Test the output: Review generated markdown to ensure quality extraction
- Check redirects: Verify your redirects work correctly if not uploading to root
Conclusion
The LLMS.txt Generator makes it simple to optimize your website for AI indexing and citations. In just a few clicks, you can convert your entire site into AI-friendly format and improve your visibility across Claude, ChatGPT, Gemini, Perplexity, and other AI platforms.
Try it free: LLMS.txt Generator
Watch the video tutorial: YouTube Walkthrough
Tags: AI optimization, llms.txt, website SEO, AI-friendly content, Apify, web crawling, markdown conversion
