Implementing Python Scraper Day Two

Alright, I'm gonna see if I can get this done in one quick sprint. The goal for today is to finish the code and at least get it running with only a few errors (nothing fatally obviously).

Write the following:

  1. Start consumers
  2. Get a page
  3. Get the rows of films
  4. For each row
    1. Get all td's (columns)
    2. Get needed fields
      • If fields aren't there, skip
    3. Create Film and append to the first Queue
  5. When all Queues are empty:
    1. Add all Films to the dictionary
    2. Add all Actors to the dictionary
  6. For each film, get aggregate fields
  7. Save to mongodb
Live Brainstorm:
  • I had to pass self to consume which gets passed to a thread. Not sure how that'll work...
  • Really hoping that I don't run into proxy errors...
  • Time to start running the code @ 6:37PM on 12/1. I'm excited to see how much I did incorrectly!
  • First thing that I did wrong is that my SetQueue needs to be passed an ID var to put an item in the seen set.
  • Looks like I will probably need a bunch of IP's and probably a proxy...
    • Nevermind - I was just parsing the data wrong. Silly me.
  • For some reason, the NUM page doesn't error when increasing the page number like the other pages
  • It seems that requests automatically encodes a url - fancy!
  • Darnit..... I don't currently account for an actor that is also a director
    • Okay, I'm also gonna scrape whatever films an Actor has also directed
    • Film when aggregating will have to specify what to use
    • It shouldn't be a terrible thing to refactor
      • Okay, it was terrible. New plan:
      • Add director id to TODO queue with director-(id)
        • This will get processed and saved separate from the normal actor
        • This director-(id) will be striped when being called
  • Looks like I royally messed up the multithreading - should've expected that
    • Gonna switch it to row in order to see if I can debug it

Comments

Post a Comment

Popular posts from this blog

"Hello, World!" Blog Post

"My Big Data 'Wow' Moment" I'd been accepted to CCIS at Northeastern, but I wasn't super into computer science during the majority of my senior year in high school. It wasn't until I read this article ->  Forbes Article Link  in AP Psychology that my interest was peaked. (Ignoring the fact that it may have been fake) the fact that Target could predict a pregnancy using data boggled my mind. It was so gloriously creepy, and I wanted to be able to do that.  Because my computer science interest was sparked so late, I went into my co-op search with little to no coding experience. To avoid the stress of technical interviews, I took my first job in a role that did not require any advanced technical abilities. Strangely, I ended my 6 months having completed countless coding projects. From reverse engineering a hurricane prediction model to creating a web application to estimate storm surge at a location given a lat/long. Looking back, if I had more kno...
Read more

Python Scraper Brain Dump

Here's what I gotta do. I need to create a web scraper that crawls IMDb to get films to be used in the model. I'm probably going to start with this URL (IMDb Most Popular)  followed by the lists sorted by IMDb rating and Number of Votes. The scraper will store a set of seen movie ID's to avoid repeats. Once the three initial lists are parsed, the ID's will be put into a Queue so the program can be multi-threaded. The scraper will then take an item from the queue and start pulling out the needed data. I plan on creating a Python class called Film that knows how to gather information from IMDb. Pulling out fields like budget, runtime etc. will be pretty straightforward. It's actors that will be the doozy. If an actor has already been processed (check a set of actor ID's), then that actor is skipped. If an actor hasn't been seen, I'll create an Actor class to handle scraping that info. Any film an actor has been in will be added to the Queue of films to...
Read more

We Know Everything

Last Fall, I interviewed at, I'm going to call it Company W, for a second co-op. The position was for a Data Scientist co-op which, before I went in for the interview, I felt confident about. I had taken statistics in high school and was currently taking another course in college. I had also been asked to bring in a project where I had worked with 'big data' to present to my interviewers. I had just completed a project for a hackathon which I thought would be perfect. Company W was gonna be blown away. Stat Hacks  uses annual college crime statistics to create graphs to compare schools and provide statistics about the data set. The information in total came to about 4 years of about 19 statistics for 1,000 schools. Not very 'big' at all. But at the time, it felt like a massive amount as each year and school had to be scraped from an Excel document, scanned from a PDF, or manually entered (ouch...). When I finished my presentation, one interviewer asked what lan...
Read more