F/X Scorn: Do Larger Budget Films Have Worse Ratings?

David Foster Wallace’s reactionary Gen X film criticism doesn’t stand up to quantitative analysis.

Sometimes being someone of my demographic who reads (and now, it appears, writes about) David Foster Wallace, reminds me a little of advice I overheard once about how to be a good religious person: the best strategy is to simply declare your identity upfront and then proceed to not act like a jerk.

He is not among my favourite writers, and I agree with much of the criticism of his professional work, personal behaviour, and degree of influence on the literary world. However, I do turn to him sometimes, chiefly for his shorter-form non-fiction, where I recognise and enjoy his perspective: he often finds something serious and sublime in subjects where other writers wouldn’t stoop to bestow such treatment. Two of his works I have turned to repeatedly are his essays on cruise ships and Roger Federer, which are as good places as any to start for those unfamiliar¹.

One essay of his that gently riled me recently had the surprisingly-pre-clickbait-era title of “F/X Porn”: the porn of the title is strictly metaphorical, used to describe a tendency he identified in then (1998) contemporary cinema towards shallow acts of visual technical spectacle (“F/X”), over either artistic style or substance. His main thesis is an underwhelmingly textbook GenX manifesto claim of favouring lower-budget films over higher-budget Hollywood extravaganzas. Two of the former films he praises are “The Terminator” and “Aliens”, and of the latter he scornfully nominates “Terminator 2: Judgement Day” and “Jurassic Park”.

He goes on to make the statement:

“T2” is thus also the first and best instance of a paradoxical law that appears to hold true for the entire F/X Porn genre. It is called the Inverse Cost and Quality Law (ICQL), and it states very simply that the larger a movie’s budget is, the shittier that movie is going to be.

Now, that is the kind of claim that makes a statistician’s ears prick up: a specific hypothesis about the functional relationship between two observable variables. Does the budget of a film really decrease the quality of the final project? This sounds somewhat dubious: read Virginia Woolf’s A Room of One’s Own to hear the exact opposite point made more convincingly concerning the need for the material resources of money, time, and space for (especially women) creatives to thrive.

I will do DFW the favour of not accompanying him on the cherry-picking expedition he implies by proposing a poorly-defined sub-genre of “F/X Porn” for which the claim holds true: the latent circular reasoning is disappointingly clear. Instead I will look to see whether this proposed inverse relationship between film budget and quality manifests in the films of the era in which he was writing.

The data used was retrieved from The Movie Data Base (TMDB), one of the more accessible large online movie data sets containing the key variables of budget, year of release, and average score from users. Anyone GenX-coded enough to question whether the artistic value of a film can really be captured by averaging over the opinions of the general public would do well to read “Let’s Talk About Love: A Journey to the End of Taste”, by Carl Wilson, a fantastic work of music criticism discussing “poptimisim” and the music of Céline Dion. Any readers who just rolled their eyes at the mention of Céline’s name without having either: a) listened to her music with an open mind, or b) reflected on how her music impacts the interior lives of her (mostly queer, older, female) audience, can fight me in the street.

The population of films was determined by the following filters:

• Original language in English, since this feels like an Anglosphere-specific conversation considering the films mentioned in the essay.
• Released in the peak GenX years between 1980 and 1999 inclusive.
• A budget larger than $1000: this is the minimum amount you might need to conceivably make something feature length, and removes spurious films with poorly-documented budgets.
• Three or more votes to avoid vanity projects.
• Runtimes between 40 minutes and 240 minutes to avoid short films and miniseries, respectively.
• No films marked as documentaries or concert films.

There was a final “study population” of 3,141 films matching all of these criteria: this is a fair amount of data, so we should be able to find something interesting if the effect size is not small. The key hypothesis concerned whether the budget of the film influenced the outcome of the average movie rating.

Let’s take a look at the budget plotted against the average TMDB ratings.

We have a cloud of points, with the four exemplar films mentioned previously being highlighted (T1 and Aliens in red, T2 and Jurassic Park in blue). On eyeballing this, I think I perceive a positive correlation between the two variables, but a regression analysis would be perfect to give more precise insight here. Let’s go for a linear regression with average rating as the dependent variable and independent variables of budget (after a logarithm), year of release and their interaction (the interaction is motivated by the essay’s suggestion that this is an emerging trend in films of this era).

\[% TMDB: budget x year (weighted, sensitivity check) \begin{array}{lrr} \textbf{Term} & \textbf{Estimate} & \textbf{95\% CI} \\ \hline \text{Intercept} & 6.183 & [6.151, 6.216] \\ \text{Budget (log, z-scored)} & 0.369 & [0.333, 0.406] \\ \text{Year (z-scored)} & -0.045 & [-0.077, -0.012] \\ \text{Budget × Year} & -0.040 & [-0.076, -0.004] \end{array}\]

If you don’t know how to read this table, don’t worry about it: this footnote is a very quick guide². It looks like the earlier eyeballing was leading us true: we have a very clear positive relationship between budget and rating (increasing budget increases the average rating), a smaller negative relationship with year of release and rating (films received lower ratings over time), and a very borderline negative relationship with their interaction (increasing budget is maybe less effective with time at increasing quality). This is a negative result concerning the central ICQL hypothesis of DFW’s article

Later in the article, he introduces two scientific-sounding “corollaries” of the proposed ICQL. the first is not testable with available data, the second may well be:

(ICQL (b)) There is no quicker or more efficient way to kill what is interesting and original about an interesting, original young director than to give that director a huge budget and lavish F/X resources.

He suggests that this can be observed as a director-specific effect, i.e. that a director will make worse films as they are given greater resources.

We fortunately have two items of statistical machinery to investigate this claim. Firstly, we introduce random effects per director into the previously specified model, which allows for isolation of effects by director. Secondly, we divide each film’s budget into a “between-director budget” variable, representing the average budget for the director and a “within-director budget” variable, comprising the deviation of each film from said average. This will allow us to interpret the influence of variations in average budget between directors and variation within the career of a single director. We should also be aware that this is a claim about interventions, whereas we only have observational data, so the following analysis results are of a prospective rather than causal nature:

\[\begin{array}{lrl} \textbf{Term} & \textbf{Estimate} & \textbf{95\% CI} \\ \hline \text{Intercept} & 6.045 & [6.004, 6.087] \\ \text{Budget (within-director)} & -0.007 & [-0.075, 0.060] \\ \text{Budget (between-director)} & 0.438 & [0.393, 0.483] \\ \text{Year (z-scored)} & -0.037 & [-0.070, -0.005] \\ \text{Budget (within-director) × Year} & 0.033 & [-0.036, 0.101] \\ \text{Budget (between-director) × Year} & -0.086 & [-0.125, -0.048] \end{array}\]

We see quite similar associations to the previous model, but crucially entirely restricted to the between-director budget variable. This suggests that, while increasing the budget will increase the film’s quality in general, individual directors tend to not vary in quality given the differences in budget from one film to the next. This again represents a negative result concerning the corollary proposed in the article.

Returning to Figure 2 from earlier of the specific films highlighted, it seems like the films mentioned in the article are also poor choices as being representative of the proposed inverse relationship: while the two films in blue are both more expensive than those in red, they are also both more highly rated on average, implying a positive relationship. “Terminator 2: Judgement Day” in particular looks quite close to being at the leading upper-right boundary of expensive films that make unprecedentedly good use of their budgets, i.e. Pareto-optimality.

What is Pareto optimality? Imagine all of these data points in the figure as being pins in a cork-board on a table in front of you. You have a ruler that is held flat against the board, which you can slide towards the collection of pins from any direction. Any pin that you can touch with the ruler would be “Pareto-optimal”, which is one method of identifying an extreme combination of budget and average rating. This can notably work from any direction, e.g. low or high budget films that make good or poor use of their resources would all qualify.

So which films are Pareto-optimal in our data set?

Of particular interest is the upper-right corner, featuring “The Shawshank Redemption” and “The Green Mile”, both of which feature no higher-rated film of a larger budget. These are excellent, expensive films, according to the standards of the TMND userbase. A little further clockwise we find “Titanic”, with no more expensive film of either higher or lower quality made in the timeframe considered.

The lower-right corner offers much juicier cherries to pick to support DFW’s hypothesis that higher budgets beget worse films, with the famous large-scale bombs of “Waterworld”, “Speed 2: Cruise Control” and “Inchon”. After a substantial step down in budget we find “Leonard Part 6”, a Bill Cosby vehicle which seems to already have been considered a poor film even before his fall from grace. Finally, we encounter “A State of Emergency”, a misguided science-fiction-meets-Christianity Michael Sheen vehicle.

In the lower left, we encounter a trio of films of both extremely low budget and quality: their presence here depends on a degree of randomness from noisier ratings, but may be of interest to the Z-movie-detritus aficionados out there.

Eventually, in the top left, we encounter the “indie darling” category: films of low budget and high quality. We meet the micro-budget comedy thriller “Blonde Death” and then finally the Limerick comedy duo D’Unbelievables’ TV Special “D’Telly”: I imagine my Irish readers will be equally heartened and surprised to learn that this is rated as the single best feature-length film of the era.

One does wonder how robust these results are to the population of people providing reviews: it is quite possible that TMDB users have one particular taste, and I am being harsh by using their ratings to evaluate DFW’s claims. The option remains to use a different set of ratings from a different community: for that purpose I turned to letterboxd. Letterboxd is a little like goodreads for films, and is famous for attracting a collection of zealously intense film devotees. As a sensitivity analysis of sorts, I ran the same analysis with ratings scraped from letterboxd instead of TMDB on exactly the same population of films. Here are the results from the regression analysis investigating the main budget-rating claim:

\[\begin{array}{lrl} \textbf{Term} & \textbf{Estimate} & \textbf{95\% CI} \\ \hline \text{Intercept} & 6.412 & [6.375, 6.450] \\ \text{Budget (log, z-scored)} & 0.124 & [0.081, 0.167] \\ \text{Year (z-scored)} & -0.047 & [-0.084, -0.009] \\ \text{Budget × Year} & -0.098 & [-0.140, -0.055] \end{array}\]

We again see a clear positive relationship with budget and rating, a negative relationship with time and rating, and now a significant negative interaction of budget and time with rating: while at no point does decreasing the budget of a film increase its quality, it does appear that increasing the budget became less effective with time in the period considered, which might be the effect that DFW was misidentifying as an overall dependence on budget. Let’s have a look at the analysis of the corollary claim concerning within-vs-between director variation on letterboxd ratings:

\[\begin{array}{lrl} \textbf{Term} & \textbf{Estimate} & \textbf{95\% CI} \\ \hline \text{Intercept} & 6.267 & [6.219, 6.315] \\ \text{Budget (within-director)} & -0.144 & [-0.217, -0.070] \\ \text{Budget (between-director)} & 0.137 & [0.083, 0.190] \\ \text{Year (z-scored)} & -0.061 & [-0.098, -0.025] \\ \text{Budget (within-director) × Year} & -0.019 & [-0.094, 0.055] \\ \text{Budget (between-director) × Year} & -0.155 & [-0.199, -0.111] \end{array}\]

We see a positive influence of between-director budget on rating, confirming the overall point that the more money a director has on average over their career, the better their films will be. We see negative associations as before time, the interaction of between-director budget and time, and intriguingly a negative association between within-director budget and rating. This would suggest that a normally high-budget director might make a better film than normal when given fewer resources, or, conversely, a normally low-budget director might make a worse film with more resources: this might be cautiously considered as evidence in support of the second corollary from the article.

And now, at last, here are the Pareto-optimal films derived from the letterboxd ratings:

We see a new highest-rating “film” of “Twin Peaks”: this is really a straight-to-video feature-length adaptation of the first episode of the arthouse TV series, which the community appears to be rating as representing the widely-beloved show in its entirety. We also see a new high-budget, bottom-right stinker in the form of “Mortal Kombat: Annihilation”, a failed 90s video-game cash-in.

And there? A new film in the top right, with no better film with a higher budget? “Terminator 2: Judgement Day”!

According to the most notorious online community of judgemental cinephile cognoscenti, “Terminator 2: Judgement Day” is among the handful of films with no other work better-justifying its budget. There exist many better examples of poor-quality, large-budget films, but I suppose it would not be as satisfying for DFW to snipe at unanimously-deserving targets.

So what is really fuelling his ire? Did he just enjoy criticising more famous people out of envy and forced subversion? Is that exactly what I have done myself in this essay, in the guise of a statistical “well-actually” debunking of his off-the-cuff claims? Well, here are my positive conclusions set out as vulnerably as possible:

• If you are going to resort to cherry-picking, be sure to choose examples that are not maximally opposed to your hypothesis.
• DFW would probably have felt more at home on letterboxd than TMND, on average, except for the “Terminator 2: Judgement Day” page in particular.
• Quantitative claims demand quantitative justifications: without the latter the former lose all meaning.

Readers: do you know of any other unexamined quantitative claims in surprising contexts that you would like to see examined by someone like me? Please let me know: it’s my idea of a good time to try them out in practice.