Contributing

When websites change, spiders tend to break. I can’t make a promise to keep this repository up-to-date all by myself, so pull requests are more than welcome!

Adding a new spider

  1. Create a branch for the spider, e.g. mtvarticle

  2. Add the spider name in pytest.ini and .travis.yml

  3. Add the spider in tests/test_spiders.py similar to others

  4. Add the spider API in finscraper/spiders.py

  5. Write the Scrapy spider under finscraper/scrapy_spiders by naming the spider exactly with the spider name, e.g. mtvarticle.py - use of flake8 linting and Google style docstrings is recommended

  6. Make sure the spider passes all non-benchmark tests within test_spiders.py

  7. Push your branch into Github and make a pull request against master

  8. (OPTIONAL): Bump up the version in VERSION and re-build the documentation

Updating an existing spider

Steps 5. - 8. above.