|
gptkbp:instanceOf
|
Software tool
|
|
gptkbp:block
|
CAPTCHA
Rate limiting
robots.txt
|
|
gptkbp:can_be_written_as
|
gptkb:Java
gptkb:JavaScript
gptkb:Python
gptkb:Ruby
PHP
|
|
gptkbp:canAutomate
|
Yes
|
|
gptkbp:canBe
|
gptkb:Headless_browsers
APIs
Academic research
Market research
HTTP requests
Lead generation
Content aggregation
Cron jobs
Price monitoring
Sentiment analysis
Task schedulers
|
|
gptkbp:canBeBypassedBy
|
Proxy servers
Delay between requests
User-agent rotation
|
|
gptkbp:canBeDesktopBased
|
Yes
|
|
gptkbp:canBeIllegalIf
|
Violates copyright law
Violates terms of service
|
|
gptkbp:canBeLegalIf
|
Complies with website policies
Used for public data
|
|
gptkbp:canBeParsedBy
|
gptkb:HTML
gptkb:JSON
XML
|
|
gptkbp:canStore
|
CSV
Databases
Excel files
JSON files
|
|
gptkbp:cloudBased
|
Yes
|
|
gptkbp:commercialUse
|
Yes
|
|
gptkbp:detects
|
Bot detection systems
|
|
https://www.w3.org/2000/01/rdf-schema#label
|
Scraper
|
|
gptkbp:monitors
|
Webmasters
|
|
gptkbp:openSource
|
Yes
|
|
gptkbp:popularLibraries
|
gptkb:Cheerio
gptkb:Selenium
gptkb:BeautifulSoup
gptkb:Scrapy
playwright
|
|
gptkbp:usedFor
|
Data extraction
Web scraping
|
|
gptkbp:bfsParent
|
gptkb:Disc_plow
|
|
gptkbp:bfsLayer
|
5
|