Video Demo General Script

Video Demo Rough Transcript

  • Introduction to the Problem
    • Gather Film and Actor data from IMDb and Box Office Mojo
      • Following Features:
        • FILM SPECIFIC
          • weekday (1-7)
          • day
          • month
          • budget
          • length
          • mpaa_num (converted from strings to ints)
        • ACTOR SPECIFIC 
          • avg_actor_age
          • max_actor_film_revenue
          • avg_actor_film_revenue
          • max_actor_film_votes
          • avg_actor_film_votes
          • max_actor_film_stars
          • avg_actor_film_stars
          • max_actor_film_appearances
          • avg_actor_film_appearances
          • max_actor_film_metascore
          • avg_actor_film_metascore
        • DIRECTOR SPECIFIC
          • director_age
          • director_number_of_films
          • max_director_film_revenue
          • avg_director_film_revenue
          • max_director_film_votes
          • avg_director_film_votes
          • max_director_film_stars
          • avg_director_film_stars
          • max_director_film_metascore
          • avg_director_film_metascore
    • Use Python to scrape the data
      • Broken into two steps
        • Scraping
        • Aggregation
    • Use MongoDb to save the objects (films and actors)
    • Use R to build the multiple regression model
  • Scraping Process
    • Get list of movies from Box Office Mojo alphabetical
    • Use name and ~year~ to match to IMDb
    • Gather Film information
    • Gather Actor information
    • Tons of problems and had to make my own data structures
      • Restart if it fails and code needed to be changed
      • Had to move to a remote machine
      • 10GB of RAM and crashed the program
        • Now runs at about 200mb
  • Python Code
    • SetQueues
    • QueueConsumers
      • Find IMDb ID
      • Scrape IMDb
      • Scrape Actors
      • Save
    • ^ allowed me to restart from any point
    • Film
    • Actor
    • Improvements
      • Error handling
      • Multiprocessing
  • R Code
    • Linking to the database
    • Converting R values
    • Check for outliers (don't currently do anything about it)
    • Create the model
      • Used backwards fitting
      • Insert features removed here
    • Test using a 50/50 split and 30/70
    • Average error for both
    • Need to make two models depending on the film budget
  • Python Interface
    • Run the script from terminal
    • Get results
    • Good/Bad?
  • Documentation
    • Github
    • Blog (details on planning, working through problems, improvements etc.)
    • R Notebook
    • Comments

Comments

  1. AnonymousMarch 12, 2022 at 8:00 AM

    You are sharing a piece of nice information here. The information you have provided is genuinely instructive and significant for us. Thanks for sharing an article like this.Software Programming Course in Delhi

    ReplyDelete
  2. Sumeet ChopraApril 25, 2025 at 2:13 AM

    Thanks for sharing such valuable information. Your writing is always engaging, and I look forward to your future posts

    Mysore Ooty Coorg Tour Package
    Mysore Ooty Kodaikanal Tour Package

    ReplyDelete

Post a Comment

Popular posts from this blog

R Brain Dump

My initial thoughts on the R portion of this assignment are that it should be pretty straightforward. I'll import the Film collection from Mongo into R. Turn it into a data.frame (all cleaning of na values would've been handled by the Python portion). I'll then try backward fitting a model. To test it, I will do a couple random splits to create test and training data then comparing. I will also try to take a real film that is upcoming and predict the revenue then wait and see. Once I get a model working, I will train it on all of the data and then create a function to predict. At this point, I will have to find a way to have Python call this function and return a result, but I will cross this bridge when I make the Python interface.
Read more

Python Scraper Brain Dump

Here's what I gotta do. I need to create a web scraper that crawls IMDb to get films to be used in the model. I'm probably going to start with this URL (IMDb Most Popular)  followed by the lists sorted by IMDb rating and Number of Votes. The scraper will store a set of seen movie ID's to avoid repeats. Once the three initial lists are parsed, the ID's will be put into a Queue so the program can be multi-threaded. The scraper will then take an item from the queue and start pulling out the needed data. I plan on creating a Python class called Film that knows how to gather information from IMDb. Pulling out fields like budget, runtime etc. will be pretty straightforward. It's actors that will be the doozy. If an actor has already been processed (check a set of actor ID's), then that actor is skipped. If an actor hasn't been seen, I'll create an Actor class to handle scraping that info. Any film an actor has been in will be added to the Queue of films to...
Read more