# Brobfelds Hub - Robots.txt # Last updated: 2025-01-15 User-agent: * Allow: / Allow: /HomePage Allow: /InvestorLanding Allow: /StartupLanding Allow: /VideoLibrary Allow: /Company Disallow: /VideoPlayer Disallow: /AdminDashboard Disallow: /MyProfile Disallow: /Upload Disallow: /SEOSettings # Sitemap location Sitemap: https://hub.bronfelds.com/sitemap # Crawl delay (seconds) Crawl-delay: 1 # Common bots User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Slurp Allow: / User-agent: GPTBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / User-agent: * Allow: /