v 0.5.0 Updated: 2 months, 1 week ago
Protego is a pure-Python `robots.txt` parser with support for modern conventions.
https://pypi.python.org/pypi/Protego