WebCrawlerLimits
The rate limits for the URLs that you want to crawl. You should be authorized to crawl the URLs.
Contents
- maxPages
-
The max number of web pages crawled from your source URLs, up to 25,000 pages. If the web pages exceed this limit, the data source sync will fail and no web pages will be ingested.
Type: Integer
Valid Range: Minimum value of 1.
Required: No
- rateLimit
-
The max rate at which pages are crawled, up to 300 per minute per host.
Type: Integer
Valid Range: Minimum value of 1. Maximum value of 300.
Required: No
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
