Skip to main content

Whether you’re looking for quick answers or in-depth guidance, our resources are here to assist you effectively.

Websites Connector

SiftHub's Websites Connector enables you to integrate content from public websites such as help centers, documentation sites, and company websites into SiftHub, making them searchable and available for AI-generated answers.

Jump to setup

 

What Gets Connected

When connecting a website URL in SiftHub, apart from main page content, you have two configuration options:

1. Include linked subpages of this website (enabled by default)

  • All directly and indirectly linked pages under the same domain
  • For example, if you connect “acme.com/features”:
    • acme.com/features/page1 (included)
    • acme.com/features/page2 (included)
    • Other pages within acme.com/features/* (included)

2. Include all external linked pages (disabled by default)

  • Pages linked from external domains
  • For example, if enabled, links to external sites like twitter.com/acme would also be included

These settings can be configured for each website URL you connect to SiftHub.

Learn how to connect a specific website to SiftHub

 

Supported Content

SiftHub indexes the following from connected websites:

  • All text content on the page (images are not indexed)
  • Page metadata
    • Creation date
    • Last modified date
    • Author information (when available)
    • Page hierarchy and structure

Note: For dynamic websites, SiftHub captures the static version of the page.

 

How It Works

Once connected, SiftHub:

  • Indexes content from specified URLs based on your configuration
  • Syncs content updates every 7 days to capture changes
  • Makes content searchable and available for AI-generated answers
  • Maintains original page structure and hierarchy

 

Using Website Content in SiftHub

Your connected website content appears in:

  • Search results with relevant snippets and metadata
  • AI-generated Answers and document Autofill as verified sources
  • Conversations and Narratives with SiftMate

 

Content Updates

  • New or modified content syncs every 7 days
  • Updates include:
    •   New pages
    •   Content modifications
    •   Metadata changes
    •   Deletions
  • Deleted content is automatically removed from SiftHub results
  • Connection changes trigger immediate sync

 

Access Control

All connected website content is accessible to all SiftHub users in your account, as this connector is designed for public websites only.

Note: This connector is specifically for public websites. For websites requiring authentication, please contact support@sifthub.io for alternative solutions.

 

 

Website Connector Setup: Adding and Removing Websites

Required Permissions: Ensure you have the following access and permissions before you begin the setup, or contact your administrator for the same.

  • SiftHub Admin or Account Owner role: To activate the Connector from the SiftHub Web App > Connectors > Apps page.

 

Activate Connector

  1. Log in to the SiftHub web app and go to the ‘Connectors’ → ‘Apps’ list by clicking here.
  2. Click Connect on the Websites Connector

 

Connect Websites:

1. Enter the website URL in the "Add site URL" field (e.g., https://www.acme.com)

 

2. Configure connection options:

"Include linked subpages of this website" (enabled by default)

  • Automatically includes all pages under the same domain
  • For example: If connecting acme.com/features, it will include acme.com/features/page1, acme.com/features/page2)

"Include all external linked pages" (optional)

  • Indexes content from all external websites that are linked from your connected pages
  • For example: Links to twitter.com found on your pages

3. Click "Add" to connect the website

 

Manage Connected Websites:

All added websites appear under "Sites added" and are accessible via the “Manage” button in the connector. You can see the status of each connected site:

  • Connected: Successfully indexed and ready for use
  • Queued: Waiting to be processed
  • In Progress: Currently being indexed

Note: Websites will only be available for search and AI-generated answers once their status changes to "Connected"

You can modify the connection options and click "Update" to save any changes to your website connections

 

Remove Websites:

  • Navigate to the website connector and click on “Manage”
  • Click the trash icon (🗑️) next to the website URL
  • The removed website's content will no longer be available for search or AI-generated answers

Note: After adding, updating, or removing websites, SiftHub will automatically begin indexing new content or removing deleted content from your search results.

 

If you are experiencing issues with the connector or its setup, contact your SiftHub Customer Success Manager or reach out to support@sifthub.io.