Step 3: Substantive Significance/Effect Size

Statistics can provide relatively concrete confidence intervals about whether the coefficient is unlikely to be zero and the direction of that coefficient. We can also use the numeric value of the coefficient to make imperfect guesses about the magnitude of the effect. The method for making these guesses with linear regressions is described below.

	% shift to Trump, 2012-2016
median income ($1,000s)	-0.158*
	(0.006)
Constant	8.337*
	(0.351)
Observations	3,111
Adjusted R2	0.203
Note:	* p<0.05

The regression function, including the estimated coefficients, is

\[\begin{align} y &= \beta x + \beta_0 + \epsilon \\ \text{shift to Trump} & = -0.158 * \text{median income (\$1,000s)} + 8.337 + \epsilon. \end{align}\]

Some simple calculations provide an imperfect guess about the substantive effect size for the coefficient on median income.

• Our best guess is that a $1,000 increase in county median income is associated with a -0.158% shift to Trump between 2012 and 2016.
• Looking at the scatterplot, the range of the values for county median income is from roughly $25,000 to $125,000. These points have been marked with red xes. This change in county median income is 100 units. So a county at the top end of the spectrum shifts to Trump by around \(100*0.158\% = 15.8\%\) less than a county at the highest income levels. If the best guess is close to being correct, this is a substantial change.
• Note: Statistical significance and substantive significance are different concepts. A statistically significant change in the outcome does not necessarily imply a substantively significant change.