Contributing
When websites change, spiders tend to break. I can’t make a promise to keep this repository up-to-date all by myself, so pull requests are more than welcome!
Adding a new spider
Create a branch for the spider, e.g.
mtvarticle
Add the spider name in pytest.ini and SPIDERS -file
Add the spider in tests/test_spiders.py similar to others
Add the spider API in finscraper/spiders.py
Write the Scrapy spider under finscraper/scrapy_spiders by naming the spider exactly with the spider name, e.g.
mtvarticle.py
- use of flake8 linting and Google style docstrings is recommendedMake sure the spider passes all non-benchmark tests within test_spiders.py
Push your branch into Github and make a pull request against master
Repository admin will merge to master and create a release after a review
Updating an existing spider
Steps 5. - 8. above.