robots-parser

samclarke
js
robots-parser JS library on GitHub robots-parser JS library on npm Download robots-parser JS library

A specification compliant robots.txt parser with wildcard (*) matching support.

Version 3.0.1 License MIT
robots-parser has no homepage
robots-parser JS library on GitHub
robots-parser JS library on npm
Download robots-parser JS library
Keywords
robots.txtparseruser-agentscraperspiderbotrobots-exclusion-standard
INSTALL
Version:
Learn more
Readme Files Statistics Browse CDN