Pseudo R-square : Understanding Goodness-of-Fit in Logistic Regression

$\textbf{1. Import Libraries}$

$\textbf{2. Simulate Data}$

$\textbf{3. Logistic Modelling}$

Here,

 Pseudo R-squ. : 0.2696

$\text{But, Why does logistic regression model report Pseudo - $R^2$ and not the R-squ. in its summary ?}$

Let's first understand, what is $R^2$ $?$

In Linear Regression, $R^2$ measures the proportion of variance in the dependent variable $(y)$ that is explained by the independent variables $(X)$ in a regression model, which indicate goodness of fit. It is defined as:

$R^2 = \frac{\text{SSR}}{\text{SST}}$

$\quad \text{where:}$

• $\text{SSR} = \sum_{i=1}^n \left( \hat{y}_i - \bar{y} \right)^2$

• $\text{SST} = \sum_{i=1}^n \left( y_i - \bar{y} \right)^2$

Now

• In linear regression, the dependent variable is continuous, which makes it possible to calculate measures like Sum of Squares Regression (SSR) and Total Sum of Squares (SST). These form the basis of $R^2$, a measure of how well the model explains the data.

• $R^2$ assumes a linear relationship and works by dividing the total variation in the dependent variable into explained and unexplained parts. This only works when the outcomes are continuous.

• In logistic regression, the dependent variable is binary (e.g., $0$ or $1$), so traditional variance-based calculations like $SSR$ and $SST$ don’t apply. Binary outcomes don’t have variance in the same way continuous data does.

• Logistic regression uses likelihood-based methods (like $log$-$likelihood$ or $deviance$) to measure how well the model fits the data. These methods replace $R^2$ for evaluating the model’s performance.

Remark

You might think that the variance $p(1-p)$ from the Bernoulli distribution, which logistic regression uses to model the probability of success $p$, could be split into "explained" and "unexplained" variance, like in linear regression.

However, this variance, $p(1-p)$, only reflects the uncertainty in the predicted probabilities. It doesn’t represent the variance of the actual target variable, which is binary ($0$ or $1$). Because of this, we can’t apply the traditional idea of explained and unexplained variance in logistic regression.

$\text{Reference: Agresti, A. (2002). Categorical Data Analysis. Wiley.}$

$\textbf{The Role of Pseudo-$R^2$}$

In logistic regression, traditional $R^2$ doesn't work because it's designed for continuous outcomes, not binary ones. To fill this gap, we use Pseudo-$R^2$ measures to assess how well the model fits the data by comparing it to a baseline (null model).

Here are some common types of Pseudo-$R^2$:

• McFadden's $R^2$ – This is reported in statsmodels library.
• Cox and Snell's $R^2$
• Nagelkerke's $R^2$

While Pseudo-$R^2$ do not directly measure explained variance, they serve as relative indicators of how well the model fits the data compared to a $null$ $model$. Pseudo-$R^2$ provides a way to interpret model effectiveness in a conceptually similar, but differently calculated, manner to traditional $R^2$.

$\text{What is null model ?}$

A null model is the simplest logistic regression model that includes no predictors, only an intercept. It predicts the same probability for all observations, equal to the proportion of $1$'s in the dataset.

• For example, If 30% of customers in a dataset made a purchase, the null model predicts: $p = P(purchase) = 0.3$ for every customer in the dataset.

1. McFadden's $R^2$

McFadden's $R^2$ evaluates the goodness-of-fit for logistic regression, analogous to $R^2$ in linear regression, but made for categorical response models in classification tasks.

It provides insight into how much the model improves upon the baseline scenario where no predictors are included, effectively answering the question: “Is the model doing significantly better than random guessing?”

Formula:

$R^2 = 1 - \frac{\text{log-likelihood of the fitted model}}{\text{log-likelihood of the null model}}$

$R^2_{\text{McFadden}} = 1 - \frac{\ln(L_{\text{model}})}{\ln(L_{\text{null}})}$

• $L_{\text{model}}$: The likelihood of the fitted model.
• $L_{\text{null}}$: The likelihood of the null model (a model with only an intercept).

Interpretation:

• Range: From $0$ to $1$, where higher values indicating better model fit.
• Values between 0.2–0.4 are generally considered a good fit.

Advantages: It benchmarks the improvement of the fitted model over the null model.

Limitations: It yields inherently smaller values than traditional $R^2$, so a low value does not always imply a poor model.

2. Adjusted McFadden's $R^2$

Similar to McFadden’s $R^2$, but penalizes for the number of predictors to account for model complexity.

Formula:

$R^2_{\text{Adjusted McFadden}} = 1 - \frac{\ln(L_{\text{model}}) - k}{\ln(L_{\text{null}})}$

Where $k$ is the number of predictors.

3. Cox and Snell's $R^2$

Formula:

$R^2 = 1 - \left( \frac{\text{likelihood of the fitted model}}{\text{likelihood of the null model}} \right)^{\frac{2}{N}}$

$= 1 - \left(\frac{L_{\text{null}}}{L_{\text{model}}}\right)^{\frac{2}{n}}$

• $L_{\text{model}}$: The likelihood of the fitted model.
• $L_{\text{null}}$: The likelihood of the null model (a model with only an intercept).
• $n$: The total number of observations.

Interpretation:

• Range: From $0$ to a theoretical maximum less than $1$, where higher values indicate better model fit.
• The maximum value depends on the sample size and the model’s characteristics.

Advantages: It provides an interpretable measure of fit, consistent with the $R^2$ concept from linear regression.

Limitations: Its range is from $0$ and maximum value is less than $1$, which can make comparisons between models or datasets less straightforward. Its maximum depends on the dataset and model characteristics.

4. Nagelkerke's $R^2$

It adjusts Cox and Snell's $R^2$ to ensure its maximum value is $1$, standardizing it for better comparability.

Formula:

$R^2 = \frac{\text{Cox-Snell R}^2}{1 - \text{(likelihood of the null model)}^{\frac{2}{N}}}$

$= \frac{R^2_{\text{Cox-Snell}}}{1 - \left(L_{\text{null}}\right)^{\frac{2}{n}}}$

• $R^2_{\text{Cox-Snell}}$: The Cox and Snell pseudo $R^2$.
• $L_{\text{null}}$: The likelihood of the null model (a model with only an intercept).
• $n$: The total number of observations.

Interpretation:

• Ranges from $0$ to $1$, with higher values indicating better model fit.

Advantages:

• Adjusts Cox and Snell's $R^2$ so it can go up to 1, making it easier to interpret.
• it Helps compare how well different models or datasets fit.
• Feels more similar to the $R^2$ used in linear regression, which makes it more familiar and intuitive.

Disadvantages:

• It doesn’t show how much variation in the data is explained, as it’s still an approximation.
• Depends on likelihood values, so it can be affected by the assumptions of the model.