"Hello, World!" Blog Post

"My Big Data 'Wow' Moment"

I'd been accepted to CCIS at Northeastern, but I wasn't super into computer science during the majority of my senior year in high school. It wasn't until I read this article -> Forbes Article Link in AP Psychology that my interest was peaked. (Ignoring the fact that it may have been fake) the fact that Target could predict a pregnancy using data boggled my mind. It was so gloriously creepy, and I wanted to be able to do that. 

Because my computer science interest was sparked so late, I went into my co-op search with little to no coding experience. To avoid the stress of technical interviews, I took my first job in a role that did not require any advanced technical abilities. Strangely, I ended my 6 months having completed countless coding projects. From reverse engineering a hurricane prediction model to creating a web application to estimate storm surge at a location given a lat/long. Looking back, if I had more knowledge on what to do with large sets of data, I could have worked on so many other extremely interesting projects.

My second co-op went about the same way. At Recorded Future, I had access to a very large and diverse data set. Over my 6 months there, I learned how to dip my toes into it, but I never got the chance, or had the know-how, to dive in. In my books, it's another large set of data that I missed a chance to do some really cool stuff with.

So far, I haven't been able to live up to 12th grade me's dreams of doing some creepy data magic. I'm looking forward to getting towards the end of this semester and getting to see what I can do for the final project. I hope that by the end of this course that not only will I have completed a successful data project, but I will know enough about the proper practices of a large data project so that I don't miss out on any other chances to ask/answer some tough 'big data' questions.

Comments

  1. Sumeet ChopraJuly 10, 2025 at 12:50 AM

    Such an insightful read! It's always great to come across content that makes you think. Keep up the fantastic work

    Goa Tour Package from Delhi
    Goa Packages with Flight from Delhi

    ReplyDelete

Post a Comment

Popular posts from this blog

Video Demo General Script

Video Demo Rough Transcript Introduction to the Problem Gather Film and Actor data from IMDb and Box Office Mojo Following Features: FILM SPECIFIC weekday (1-7) day month budget length mpaa_num (converted from strings to ints) ACTOR SPECIFIC  avg_actor_age max_actor_film_revenue avg_actor_film_revenue max_actor_film_votes avg_actor_film_votes max_actor_film_stars avg_actor_film_stars max_actor_film_appearances avg_actor_film_appearances max_actor_film_metascore avg_actor_film_metascore DIRECTOR SPECIFIC director_age director_number_of_films max_director_film_revenue avg_director_film_revenue max_director_film_votes avg_director_film_votes max_director_film_stars avg_director_film_stars max_director_film_metascore avg_director_film_metascore Use Python to scrape the data Broken into two steps Scraping Aggregation Use MongoDb to save the objects (films and actors) Use R to build the multiple regression model Scraping...
Read more

R Brain Dump

My initial thoughts on the R portion of this assignment are that it should be pretty straightforward. I'll import the Film collection from Mongo into R. Turn it into a data.frame (all cleaning of na values would've been handled by the Python portion). I'll then try backward fitting a model. To test it, I will do a couple random splits to create test and training data then comparing. I will also try to take a real film that is upcoming and predict the revenue then wait and see. Once I get a model working, I will train it on all of the data and then create a function to predict. At this point, I will have to find a way to have Python call this function and return a result, but I will cross this bridge when I make the Python interface.
Read more