Writing in the Disciplines: Political Science - Statistical Significance: Accounting for Errors in Measurement

Statistical Significance: Accounting for Errors in Measurement Writer's Web

By Daniel Palazzolo, Ph.D.
(printable version here)

We need to determine the statistical significance of a relationship when we use sample data because sample data contain error. We must therefore develop a null hypothesis, H₀, which states that there is no relationship between the variables, i.e. the differences between the means or proportions between the two variables in the sample are non-existent, they equal zero (Perhaps the differences we see in the sample means or proportions are just a result of sampling error). In order to substantiate the hypothesis, we want to reject the null hypothesis, so we ask, what is the probability that the differences in the sample are not equal to zero?

In order to test the null hypothesis, we construct a 95% confidence interval. This will allow us to determine if the null hypothesis for a relationship, measured by the differences in the sample data, is true or false. If the test statistic is greater than 2SE, or zero is not contained within the confidence interval, then we can say (with 95% confidence) that the differences in the sample are real, i.e. they are greater than zero, for the population as a whole. We reject the null hypothesis. Or, using the probability value (p-value), if p is less than .05, we can also reject the null hypothesis. If, on the other hand, the test statistic is less than 2SE, and zero falls within the confidence interval, and if the probability of getting the test statistic by chance is greater than .05, then we do not reject the null hypothesis.

The types of statistics we use, naturally, depend on the level of measurement for the variables. We can use the test statistic or the p-value to test the null hypothesis for two sample means and chi-square or the p-value to test the null hypothesis for relationships involving nominal/ordinal variables.

Data	Mean or Proportion	Distribution	Measure of Dispersion	Test statistic	Level of significance
Population	Population Mean	Normal Curve (fixed)	Standard Deviation	Z scores	95% ( + 2Z)
Sample	One Mean/One Proportion	T-Distribution (depends on n D of F)	Standard Error	t-ratio (Z when n is large)	95% ( + 2SE)
Sample	Two Means	T-Distribution (depends on n D of F)	Standard Error of Difference	t-ratio (Z when n is large) or p-value	95% ( + 2SE)
Sample	Two Proportions	T-Distribution (depends on n) D of F)	Standard Error of Difference	t-ratio (Z when n is large) or p-value	95% ( + 2SE)
Sample	Two ordinal and/or nominal variables	Chi-Square: depends on (r-1) (c-1)		Chi-square or p-value	95% (Critical Value)