Statements (103)
| Predicate | Object |
|---|---|
| gptkbp:instanceOf |
gptkb:software
|
| gptkbp:alsoKnownAs |
web robot
web bot web spider |
| gptkbp:block |
robots.txt
|
| gptkbp:canBe |
cybersecurity
news aggregation academic research fraud detection data mining lead generation market research competitive analysis sentiment analysis web archiving social media monitoring health data collection travel fare aggregation content aggregation SEO analysis website monitoring ad verification brand monitoring weather data collection event aggregation government data collection price monitoring coupon aggregation app data collection IPO data collection TV show data collection acquisition data collection auction data collection blockchain data collection book data collection celebrity data collection classified ads aggregation company data collection conference data collection contact information extraction content scraping course data collection court case data collection cryptocurrency data collection discussion board monitoring e-commerce monitoring email harvesting fashion data collection financial data collection flight data collection forum data collection funding data collection game data collection grant data collection hotel data collection image scraping ingredient data collection investment data collection job aggregation legal data collection legislation data collection link checking menu data collection merger data collection movie data collection music data collection nutrition data collection patent data collection podcast data collection political data collection product review aggregation public records aggregation real estate aggregation recipe aggregation regulation data collection restaurant data collection scholarly article aggregation scholarship data collection sports data collection startup data collection stock data collection tender data collection transportation data collection university data collection video scraping |
| gptkbp:cause |
server load
|
| gptkbp:example |
gptkb:DuckDuckBot
gptkb:Googlebot gptkb:Bingbot Baidu Spider Slurp Bot Sogou Spider Yandex Bot |
| gptkbp:function |
automatically browse the web
collect data from websites download web pages |
| https://www.w3.org/2000/01/rdf-schema#label |
WebCrawlers
|
| gptkbp:observedBy |
robots.txt
|
| gptkbp:purpose |
index web content
|
| gptkbp:usedBy |
search engines
|
| gptkbp:bfsParent |
gptkb:Alison_Segel
gptkb:Maria_Blasucci |
| gptkbp:bfsLayer |
7
|