u/michaelnovati replied ·
Fun project. I learned a lot building scrappers for fun for a few learning projects. It's a very engineering mindset to try to do it from scratch. You have to identify reliable patterns, and build robust parsing code. The super proper way to do it is to use a more syntactical parsing framework, but if you want to really learn, try curling the page and parsing the HTML to extract the information reliably.
Just make sure you are not breaking any rules!