When building webscrapers the following workflow usually works well for me:
- Use google chrome to inspect the html
- If it's plain html it should be very straigh forward
- If it has AJAX then you'll need to understand the network requests and make those requests accordingly. See resources for a link showing how to do this.
- If it is dynamic consider using Selenium
- Python libraries I commonly use
- Requests
- Beautiful Soup
- Re (for regular expressions)
- IDE
- Jupyter notebook/lab. I find is best, as there is a lot of explorative work to be done, and bit of hacking to get there.
- Always grab a previous scraped as a starting template
Resources