Statements (22)
| Predicate | Object |
|---|---|
| gptkbp:instanceOf |
Web crawler activity
|
| gptkbp:affects |
Search engine ranking
Website server load |
| gptkbp:canBe |
Reverse DNS lookup
|
| gptkbp:controlledBy |
robots.txt
meta robots tag |
| gptkbp:documentation |
https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
|
| gptkbp:frequency |
Varies based on site popularity and update frequency
|
| https://www.w3.org/2000/01/rdf-schema#label |
Googlebot crawl requests
|
| gptkbp:identifiableBy |
User-Agent header
|
| gptkbp:IPRange |
Google-owned IP addresses
|
| gptkbp:mayBeBlockedBy |
robots.txt disallow rule
|
| gptkbp:mayBeSlowedBy |
Crawl-delay directive (not supported by Google)
|
| gptkbp:performedBy |
gptkb:Googlebot
|
| gptkbp:purpose |
Indexing web pages for Google Search
|
| gptkbp:response |
HTTP status codes
|
| gptkbp:supportsProtocol |
gptkb:HTTP
HTTPS |
| gptkbp:triggeredBy |
Crawl budget considerations
|
| gptkbp:userAgent |
gptkb:Googlebot
|
| gptkbp:bfsParent |
gptkb:Google_Search_Console_Crawl_Stats_Report
|
| gptkbp:bfsLayer |
7
|