Robots.txt Blocking Test Page

This page demonstrates robots.txt blocking. Some subpages are blocked by robots.txt rules:

Blocked Product Pages: Products 2 and 4 (/robots-blocking/product/2, /robots-blocking/product/4)
Blocked Category: /robots-blocking/category/blocked
Allowed Pages: All other product pages and categories are allowed

Check /robots.txt to see the blocking rules.

About This Test Page

This page is designed to test the robots.txt compliance checking feature. The robots.txt file blocks specific paths:

When robots.txt compliance checking is enabled, workflows attempting to scrape these blocked URLs should fail with a ROBOTS_TXT_BLOCKED error.

$10

This is a test product

View Details

$20

This is another test product

⚠️ BLOCKED by robots.txt
View Details

$30

This is a third test product

View Details

$40

This is a fourth test product

⚠️ BLOCKED by robots.txt
View Details

$50

This is a fifth test product

View Details