Website Pages
Website Pages provides an inventory of your website’s content, helping Asky understand your site structure and track which pages AI systems can access.
Overview
Website Pages tracks:
- All indexed pages on your website
- Page metadata (title, description, URL)
- Content embeddings for AI matching
- Page-level accessibility status
Accessing Website Pages
Go to Knowledge Base → Website Pages in the sidebar.
Page Inventory
Page List
The page list shows all indexed pages:
| Column | Description |
|---|---|
| URL | Full page URL |
| Title | Page title |
| Last Updated | When page was last indexed |
| Status | Indexing status |
How Pages Are Indexed
Initial Indexing
When you add a brand, Asky crawls your website:
- Starts from your homepage
- Follows internal links
- Indexes page content
- Creates embeddings for matching
Ongoing Updates
Pages are re-indexed when:
- You manually trigger a refresh
- Regular scheduled crawls occur
- Sitemap changes are detected
Sitemap Integration
If you have a sitemap.xml:
- Asky uses it to discover pages
- Ensures complete coverage
- Respects priority and lastmod
Indexing Errors
Common indexing errors:
| Error | Cause | Fix |
|---|---|---|
| 404 Not Found | Page doesn’t exist | Remove page or fix URL |
| 403 Forbidden | Access denied | Check server permissions |
| Timeout | Page too slow | Improve performance |
| Blocked | robots.txt restriction | Update robots.txt |
Managing Pages
Adding Pages
Pages are typically added automatically through crawling. To manually add:
- Click Add Page
- Enter the full URL
- Asky will index the page
Manual addition is useful for pages that aren’t linked from your main navigation or included in your sitemap, but you still want Asky to access it.
Page Embeddings
What Are Embeddings?
Embeddings are numerical representations of your content that enable:
- Semantic matching
- Content similarity analysis
- AI response correlation
How They’re Used
Embeddings help Asky:
- Match AI citations to your pages
- Understand content relationships
- Identify coverage gaps
- Track content performance
Embedding Updates
Embeddings update when:
- Page content changes significantly
- Manual refresh is triggered
- Scheduled re-indexing occurs
Integration with Features
Crawler Logs
Cross-reference with Crawler Logs:
- See which pages AI bots access
- Identify pages not being crawled
- Understand crawler preferences
Technical Opportunities
Page data feeds Technical Opportunities:
- Sitemap coverage analysis
- Broken URL detection
- Content quality scores
Content Opportunities
Page content contributes to Content Opportunities:
- Overall site content quality
- Topic coverage analysis
- Optimization opportunities
Troubleshooting
Pages Not Appearing
- Check robots.txt isn’t blocking
- Verify page is linked from your site
- Try manual addition
- Check for redirect issues
Missing Pages
If important pages are missing:
- Add to sitemap.xml
- Add internal links from indexed pages
- Manually add the pages
- Check for noindex meta tags
Next Steps
- Complete your Brand Profile
- Set up Crawler Logs to monitor bot activity
- Review Technical Opportunities for page health