robots-txt-parser
ChrisAkroyd js
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Version 2.0.3 License MIT
Keywords
robotstxtrobots.txtparsercrawlerspiderbotrobotstxtscraper
INSTALL
Statistics 
- month
- quarter
- year
- May 2025
- April 2025
- Q1 2025
- March 2025
- February 2025
- January 2025
- 2024
- Q4 2024
- December 2024
- November 2024
- October 2024
- Q3 2024
- September 2024
- August 2024
- July 2024
- Q2 2024
- June 2024
- May 2024
- April 2024
- Q1 2024
- March 2024
- February 2024
- January 2024
- 2023
- Q4 2023
- December 2023
- November 2023
- October 2023
- Q3 2023
- September 2023
- August 2023
- July 2023
- Q2 2023
- June 2023
- May 2023
- April 2023
- Q1 2023
- March 2023
- February 2023
- January 2023
- 2022
- Q4 2022
- December 2022
- November 2022
- October 2022
- Q3 2022
- September 2022
- August 2022
- July 2022
- Q2 2022
- June 2022
- May 2022
- April 2022
- Q1 2022
- March 2022
- February 2022
- January 2022
- 2021
- Q4 2021
- December 2021
- November 2021
- October 2021
- Q3 2021
- September 2021
- August 2021
- July 2021
- Q2 2021
- June 2021
- May 2021
- April 2021
- Q1 2021
- March 2021
- February 2021
- January 2021
- 2020
- Q4 2020
- December 2020
- November 2020
- October 2020
- Q3 2020
- September 2020
- August 2020
- July 2020
- Q2 2020
- June 2020
- May 2020
- April 2020
- Q1 2020
- March 2020
- February 2020
- January 2020
Requests 0
Bandwidth 0
Top version - 0
Full robots-txt-parser Download Stats 
ShareKeywords
robots
txt
robots.txt
parser
crawler
spider
bot
robotstxt
scraper
Get a badge for your package
