Implementing Python Scraper Day Three

This is (hopefully) the home stretch. What I'm going to accomplish today:

  1. Switch to running one at a time
  2. Save Films and Actors before running aggregate commands
  3. Make the R data model

Here we go... (12/4/2017 2:18pm):
  • I'm an idiot. My consumers didn't break (I don't think at least). I never started a while loop, so only one item was consumed and the function exited.
  • Some errors that appeared:
    • Dates on BOM didn't match IMDb which I'm using to map ID's
      • Fixed! Added a year tolerance that checks for multiple possible years
    • Some actors only have a year for birthday
      • Fixed! Set birthdate to 1-1-year
    • Budgets for films won't be in number format
      • Not the actual issue. Looks like the BOM -> IMDb is messing up
        • It was messing up because I checked for all years around the BOM year but not for the BOM year itself...smh
      • Once I fixed ^ the budget thing was an issue again. I was looking in the wrong spot for the budget. Should be fixed now.
  • Realizing my default date of year 3000 may be in issue if I don't check for it...
  • I almost didn't scrape any actresses because I only looked for the word actor
  • Seriously underestimated the RAM this needs. I'm going to save the films as they reach the output Queue so I need another consumer.
  • Next steps:
    • Refactor to save into MongoDB
    • Get running on an Intel NUC
    • Write code to get all from Database and then run aggregation
    • Start working on R code
  • If for some reason the process fails, I can restart and add already saved objects to be excluded from the SetQueue.
  • I should've purged the imdb_page and box office mojo page after the values were scraped...
  • Well the NUC crashed. I'm going to write code to purge the html pages and to populate the SetQueues
  • Smh such a rookie mistake. When passing sets to the SetQueues, I used a reference instead of a copy so every SetQueue was mutating the same sets. I'm a dingus.
  • Forgot to add Actors that were pending scraping back into the Queue in the event of a failure (which there have been several)

Comments

  1. Sumeet ChopraJune 6, 2025 at 1:23 AM

    I stumbled upon this post and I'm so glad I did! The way you've presented [mention a general idea or concept, e.g., 'the balance between ambition and well-being'

    Delhi Agra Jaipur Tour Packages
    Golden Triangle india Tour Package

    ReplyDelete

Post a Comment

Popular posts from this blog

R Brain Dump

My initial thoughts on the R portion of this assignment are that it should be pretty straightforward. I'll import the Film collection from Mongo into R. Turn it into a data.frame (all cleaning of na values would've been handled by the Python portion). I'll then try backward fitting a model. To test it, I will do a couple random splits to create test and training data then comparing. I will also try to take a real film that is upcoming and predict the revenue then wait and see. Once I get a model working, I will train it on all of the data and then create a function to predict. At this point, I will have to find a way to have Python call this function and return a result, but I will cross this bridge when I make the Python interface.
Read more

Video Demo General Script

Video Demo Rough Transcript Introduction to the Problem Gather Film and Actor data from IMDb and Box Office Mojo Following Features: FILM SPECIFIC weekday (1-7) day month budget length mpaa_num (converted from strings to ints) ACTOR SPECIFIC  avg_actor_age max_actor_film_revenue avg_actor_film_revenue max_actor_film_votes avg_actor_film_votes max_actor_film_stars avg_actor_film_stars max_actor_film_appearances avg_actor_film_appearances max_actor_film_metascore avg_actor_film_metascore DIRECTOR SPECIFIC director_age director_number_of_films max_director_film_revenue avg_director_film_revenue max_director_film_votes avg_director_film_votes max_director_film_stars avg_director_film_stars max_director_film_metascore avg_director_film_metascore Use Python to scrape the data Broken into two steps Scraping Aggregation Use MongoDb to save the objects (films and actors) Use R to build the multiple regression model Scraping...
Read more

Python Scraper Brain Dump

Here's what I gotta do. I need to create a web scraper that crawls IMDb to get films to be used in the model. I'm probably going to start with this URL (IMDb Most Popular)  followed by the lists sorted by IMDb rating and Number of Votes. The scraper will store a set of seen movie ID's to avoid repeats. Once the three initial lists are parsed, the ID's will be put into a Queue so the program can be multi-threaded. The scraper will then take an item from the queue and start pulling out the needed data. I plan on creating a Python class called Film that knows how to gather information from IMDb. Pulling out fields like budget, runtime etc. will be pretty straightforward. It's actors that will be the doozy. If an actor has already been processed (check a set of actor ID's), then that actor is skipped. If an actor hasn't been seen, I'll create an Actor class to handle scraping that info. Any film an actor has been in will be added to the Queue of films to...
Read more