Means and Variances of Linear Combinations of Random Variables

Means and Variances of Linear Combinations of Random Variables

In probability theory and statistics, linear combinations of random variables are common in various contexts, such as in regression analysis, portfolio theory, and signal processing. Understanding how the mean and variance behave for these linear combinations is crucial for analyzing the outcomes of such combinations.

1. Linear Combination of Random Variables

A linear combination of random variables is an expression that involves the random variables and constants (coefficients). For two random variables $X_1$ and $X_2$, a linear combination can be written as:

$$Z = aX_1 + bX_2 + c$$

where:

• $a$ and $b$ are constants (scalars),
• $X_1$ and $X_2$ are random variables, and
• $c$ is a constant (can be thought of as a shift).

More generally, for $n$ random variables $X_1, X_2, \dots, X_n$, a linear combination is:

$$Z = a_1 X_1 + a_2 X_2 + \dots + a_n X_n + c$$

where $a_1, a_2, \dots, a_n$ are constants (coefficients) and $c$ is also a constant.

2. Mean of a Linear Combination

The mean (or expected value) of a linear combination of random variables is computed using the linearity of expectation. The expected value of a linear combination of random variables is the linear combination of their expected values. Specifically:

$$E[Z] = E[a_1 X_1 + a_2 X_2 + \dots + a_n X_n + c]$$

Using the linearity of expectation:

$$E[Z] = a_1 E[X_1] + a_2 E[X_2] + \dots + a_n E[X_n] + c$$

This property holds regardless of whether the random variables $X_1, X_2, \dots, X_n$ are independent or not.

Example:

Suppose you have two random variables $X_1$ and $X_2$ with the following expected values:

• $E[X_1] = 3$
• $E[X_2] = 5$

Now, consider the linear combination $Z = 2X_1 - 3X_2 + 4$. The expected value of $Z$ is:

$$E[Z] = 2E[X_1] - 3E[X_2] + 4$$ $$E[Z] = 2(3) - 3(5) + 4 = 6 - 15 + 4 = -5$$

So, the expected value of $Z$ is -5.

3. Variance of a Linear Combination

The variance of a linear combination of random variables depends on both the variances of the individual random variables and the covariances between them. For two random variables $X_1$ and $X_2$, the variance of $Z = a_1 X_1 + a_2 X_2$ is given by:

$$\text{Var}(Z) = \text{Var}(a_1 X_1 + a_2 X_2)$$

Using the properties of variance, we can expand this as:

$$\text{Var}(Z) = a_1^2 \text{Var}(X_1) + a_2^2 \text{Var}(X_2) + 2a_1 a_2 \text{Cov}(X_1, X_2)$$

If there are more than two random variables (say, $X_1, X_2, \dots, X_n$), the variance of the linear combination $Z = a_1 X_1 + a_2 X_2 + \dots + a_n X_n$ becomes:

$$\text{Var}(Z) = a_1^2 \text{Var}(X_1) + a_2^2 \text{Var}(X_2) + \dots + a_n^2 \text{Var}(X_n) + 2 \sum_{i < j} a_i a_j \text{Cov}(X_i, X_j)$$

This formula accounts for the variance of each random variable as well as the covariance between each pair of random variables.

Key Points:

• The variance of a linear combination depends on the coefficients of the random variables, their individual variances, and the covariances between the variables.
• The covariance term captures how the random variables co-vary, and it will influence the total variance if the variables are not independent.

Example:

Suppose we have two random variables $X_1$ and $X_2$ with the following properties:

• $\text{Var}(X_1) = 4$
• $\text{Var}(X_2) = 9$
• $\text{Cov}(X_1, X_2) = 3$

Now, consider the linear combination $Z = 2X_1 - 3X_2$. The variance of $Z$ is:

$$\text{Var}(Z) = 2^2 \text{Var}(X_1) + (-3)^2 \text{Var}(X_2) + 2(2)(-3) \text{Cov}(X_1, X_2)$$ $$\text{Var}(Z) = 4 \times 4 + 9 \times 9 + 2 \times 2 \times (-3) \times 3$$ $$\text{Var}(Z) = 16 + 81 - 36 = 61$$

So, the variance of $Z$ is 61.

4. Special Cases: Independent Random Variables

If the random variables $X_1, X_2, \dots, X_n$ are independent, then the covariance between any pair of variables is zero, i.e., $\text{Cov}(X_i, X_j) = 0$ for $i \neq j$. In this case, the variance of the linear combination simplifies to:

$$\text{Var}(Z) = a_1^2 \text{Var}(X_1) + a_2^2 \text{Var}(X_2) + \dots + a_n^2 \text{Var}(X_n)$$

In the case of independent random variables, we only need to consider the individual variances, not the covariances.

Example (Independent Case):

If $X_1$ and $X_2$ are independent, with $\text{Var}(X_1) = 4$ and $\text{Var}(X_2) = 9$, and the linear combination is $Z = 2X_1 - 3X_2$, the variance is:

$$\text{Var}(Z) = 2^2 \text{Var}(X_1) + (-3)^2 \text{Var}(X_2)$$ $$\text{Var}(Z) = 4 \times 4 + 9 \times 9 = 16 + 81 = 97$$

Thus, the variance of $Z$ in the case of independent random variables is 97.

---

5. Summary of Formulas

1. Mean of a Linear Combination:

For random variables $X_1, X_2, \dots, X_n$ and constants $a_1, a_2, \dots, a_n$, the expected value of the linear combination $Z = a_1 X_1 + a_2 X_2 + \dots + a_n X_n + c$ is:

$$E[Z] = a_1 E[X_1] + a_2 E[X_2] + \dots + a_n E[X_n] + c$$

2. Variance of a Linear Combination:

For random variables $X_1, X_2, \dots, X_n$ with coefficients $a_1, a_2, \dots, a_n$, the variance of the linear combination $Z = a_1 X_1 + a_2 X_2 + \dots + a_n X_n$ is:

$$\text{Var}(Z) = a_1^2 \text{Var}(X_1) + a_2^2 \text{Var}(X_2) + \dots + a_n^2 \text{Var}(X_n) + 2 \sum_{i < j} a_i a_j \text{Cov}(X_i, X_j)$$

• If $X_1, X_2, \dots, X_n$ are independent, the covariance terms drop out, and the variance simplifies to:

$$\text{Var}(Z) = a_1^2 \text{Var}(X_1) + a_2^2 \text{Var}(X_2) + \dots + a_n^2 \text{Var}(X_n)$$

---

Conclusion

The mean and variance of linear combinations of random variables are important tools for understanding the behavior of sums, differences, and weighted combinations of random variables. The expected value of a linear combination is simply the linear combination of the expected values, while the variance involves not only the individual variances but also the covariances between the random variables. When the variables are independent, the covariance terms disappear, simplifying the calculation of variance.