Logistic Regression: Unlocking the Power of Classification

Logistic Regression: Unlocking the Power of Classification

Have you ever wondered how computers can predict whether an email is spam or not, or how a medical diagnosis can be made based on symptoms? The answer lies in a powerful technique called Logistic Regression. It's a statistical method used for classification problems, where the goal is to predict the probability of a certain outcome.

Understanding the Basics: From Scores to Probabilities

At its core, Logistic Regression takes a set of features (characteristics or attributes) of an instance and combines them to calculate a probability. Think of it as a weighted sum of these features, where each feature's weight indicates its importance in predicting the outcome. For example, if you want to predict if a customer will buy a product, features like "age," "income," and "past purchase history" would be considered.

To turn this weighted sum into a probability (a number between 0 and 1), we use a special function called the Sigmoid Function. The sigmoid function has a S shape and has a few benefits. It takes a real number and gives a value between 0 and 1, perfect for probabilities. It also squashes the outlier values toward 0 and 1. The formula is this:

P(y = 1) = 1 / (1 + exp(-(w · x + b)))

• P(y = 1): Probability of the outcome being "1" (e.g., customer will buy).
• w: Weights assigned to each feature.
• x: Feature values.
• b: Bias (a constant term).
• exp: The exponential function.

And for the probability of the outcome being "0":

P(y = 0) = 1 - P(y = 1)

The Logit: The value "w · x + b" (before applying the sigmoid) is often referred to as the "logit." It represents the log of the odds ratio (p / (1-p)). In other words, it is interpreted as a log odds.

Making Decisions: Classification with a Threshold

Now that we have a probability, how do we actually make a decision? We set a decision boundary, which is a threshold probability. If the predicted probability is above the threshold, we classify the instance as one class; otherwise, we classify it as the other. A common decision boundary is 0.5.

Example: Sentiment Analysis

Let's say we want to classify movie reviews as either positive or negative. We can define some features like the number of positive words in the review and the number of negative words. A logistic regression model can then assign weights to these features and predict the probability of the review being positive. If the probability is greater than 0.5, we classify it as positive; otherwise, we classify it as negative.

Consider a mini-review: "It's hokey. There are virtually no surprises, and the writing is second-rate. So why was it so enjoyable? For one thing, the cast is great. Another nice touch is the music. I was overcome with the urge to get off the couch and start dancing. It sucked me in, and it'll do the same to you."

Let's extract these features:

Feature	Description	Value
x1	Count of positive lexicon words	3
x2	Count of negative lexicon words	2
x3	Contains the word "no"	1
x4	Count of first and second person pronouns	3
x5	Contains an exclaimation mark "!"	0
x6	Natural log of word count	4.19

Suppose our model has learned the following weights: [2.5, -5.0, -1.2, 0.5, 2.0, 0.7] and a bias of 0.1. Then the probability of the review being positive is:

P(+) = sigmoid([2.5, -5.0, -1.2, 0.5, 2.0, 0.7] · [3, 2, 1, 3, 0, 4.19] + 0.1) = sigmoid(0.833) ≈ 0.70

Since 0.70 is greater than 0.5, we would classify the review as positive.

Real-World Applications and Feature Engineering

Logistic Regression is used in all sorts of real-world NLP tasks. Consider these features:

• Period Disambiguation: Determining if a period ends a sentence or is part of an abbreviation.
  • Feature Example 1: Is the word following the period lowercase? (e.g., "The cat sat. down")
  • Feature Example 2: Is the word containing the period in a list of acronyms? (e.g., "Prof.")

Designing vs. Learning Features: Features can be manually crafted, or they can be learned automatically by a machine-learning algorithm. Manually crafting features requires extensive human labor and expertise in language. Feature interactions can also be useful such as periods after upper case letters. Recent advances in NLP focus on representation learning, learning the features instead of designing them.

Scaling Input Features

Feature scaling can be very important! It standardizes the input features, which is very useful for comparing features. There are two main methods to do so:

• Standardization: Transforms values to have zero mean and unit variance.
• Normalization: Scales values between 0 and 1.

Data scaling is very important for large neural networks, because it speeds up gradient descent.

Processing Many Examples Efficiently

In practice, we'll want to process many examples at once. This can be done efficiently using matrix arithmetic. Instead of computing each output individually, we can pack all the input feature vectors into a single input matrix and use matrix multiplication to compute all the outputs in one operation. This greatly speeds up the computation, especially when dealing with large datasets.

Logistic Regression vs. Naive Bayes

Logistic Regression offers several advantages over Naive Bayes, another classification algorithm. Naive Bayes assumes that all features are independent of each other, which is often not true in real-world data. Logistic Regression is more robust to correlated features and generally works better on larger datasets. However, Naive Bayes can be surprisingly effective on very small datasets or short documents due to its simplicity and speed.

Multinomial Logistic Regression: Handling Multiple Classes

What if we have more than two classes? For example, we might want to classify sentiment as positive, negative, or neutral. In this case, we use Multinomial Logistic Regression, also called Softmax Regression. In this approach, there are K possible classes and K weight vectors.

The Softmax Function

Multinomial Logistic Regression uses the Softmax Function, a generalization of the sigmoid function. The softmax takes a vector of scores and transforms them into a probability distribution over the classes. Here is the formula:

softmax(z_i) = exp(z_i) / ∑_j=1^K exp(z_j)

• z_i: The score for class i.
• K: The total number of classes.

The softmax function has the benefit of squashing the values, ensuring they are all between 0 and 1. Like the sigmoid, we also refer to the input of the softmax as the logit.

Applying Softmax in Logistic Regression

To apply the softmax function to logistic regression, we'll need to have separate weight vectors and bias for each of the possible classes. So the probability of each of our output classes can thus be computed as:

P(y_k = 1 | x) = exp(w_k · x + b_k) / ∑_j=1^K exp(w_j · x + b_j)

The weights can also be represented by a matrix W, where each row k represents the vector of weights for class w_k, and b is a vector with each possible output class. These weights are optimized to achieve a high classification accuracy. Using this form, the following is an elegant equation:

y^ = softmax(Wx+b)

Features in Multinomial Logistic Regression

Like binary logistic regression, features in Multinomial Logistic Regression act to predict how a certain classification can be made.

For example, the exclamation feature in the sentiment example had one weight for positive and one for negative in binary classification. In the multinomial example, it has three, one each for neutral, negative, and positive.

Feature	Description	w5,+	w5,-	w5,0
Feature 5(x)	Contains exclamation point	3.5	3.1	-5.3

Learning in Logistic Regression

Logistic regression is a type of supervised learning, because we have the correct classification for each example in the training set. As such, we want to learn the model parameters, called θ, that allow it to be as close as possible to the true y values. This requires two things: the loss function and the optimization algorithm.

The Cross-Entropy Loss Function

This function says how close the predicted output y^, is to the true output, y. The goal is to maximize the probability of the correct label. This is done via conditional maximum likelihood estimation. The resulting loss function is called the Cross-Entropy Loss.

The Cross-Entropy Loss L_CE, can be given by:

L_CE(y^,y) = - [y * log y^ + (1-y)*log(1-y^)]

Why does minimizing the log likelihood do what we want? A perfect classifier has a probability of 1 and an incorrect classifier has a probability of 0. As such, we want the negative log of the probability, which gives us a loss metric. The cross-entropy loss also ensures that when we maximize the probability of the correct answer, we minimize that of the incorrect one.

Gradient Descent

To find the optimal weights, we use an optimization algorithm called Gradient Descent. This algorithm finds the minimum of a function by figuring out which direction in parameter space the function is rising the most steeply and moving in the opposite direction. It will search for the weights that minimize the cost function.

For logistic regression, the loss function is convex, so starting gradient descent at any point means it is guaranteed to find the minimum. It uses gradients that shows the directional components of the sharpest slope along each of the N parameters. This is done by asking: how much would a change in the gradient influence the loss function?.

Formally, the gradient of the function f is a vector in which each component expresses the partial derivative of f with respect to one of the variables. So the equation for updating θ based on the gradient is:

θ^t+1 = θ^t - η * ∇L(f(x;θ),y)

• θ^t+1 is the new parameters
• θ^t is the old parameters
• η is the learning rate
• ∇L(f(x;θ),y) is the gradient.

The Gradient for Logistic Regression

In order to update θ, we need to define the gradient. For each observation, it is:

∂L_CE(y^,y)/∂w_j = [σ(w*x+b) - y] * x_j = (y^ - y) * x_j

It is very intuitive to see that the gradient with respect to w_j represents a very intuitive value: the difference between the true y and our estimated y^ for that observation, multiplied by the corresponding input value x_j.

The Stochastic Gradient Descent Algorithm

This algorithm computes the gradient after each training example, nudging θ in the right direction. This makes it an online algorithm, since it processes input example-by-example. Stochastic Gradient Descent is called "stochastic" because it chooses a single random example at a time.

The learning rate, η, is a hyperparameter that must be adjusted. If it is too high, the learner will take steps that overshoot the minimum. If it is too low, the learner will take steps that are too small. Usually we start with a high learning rate then decrease it over time as a function of iteration, k.

Mini-Batch Training

Stochastic gradient descent can result in choppy movements, because it chooses a single random example at a time. As such, it can be useful to compute the gradient over batches of training instances. In batch training, the gradient is computed over the entire dataset. This offers a better estimate of which direction to move the weights at the cost of processing every single example in the training set. Mini-Batch training is a compromise: training on groups of examples (e.g. 512 or 1024), less than the entire dataset.

Regularization: Preventing Overfitting

A common problem in machine learning is overfitting. Overfitting occurs when the model learns the training data too well, including the noise and irrelevant details. As such, the model may not be able to generalize well to new and unseen data. To avoid overfitting, we add a new regularization term, R(θ), to the loss function. The new term penalizes large weights.

ˆθ = argmax_θ ∑^m_i=1 log P(y⁽ⁱ⁾|x⁽ⁱ⁾) - αR(θ)

Here, alpha is the parameter. If a feature is perfectly predictive of the outcome because it only occurs in one class, it will be assigned a very high weight, which we do not want. The regularization term is used to penalize large weights, so perfect matching with training data is not penalized less than settings that do not match the data as well.

There are two common ways to compute the regularization term R(θ).

1. L2 Regularization: A quadratic function of the weight values; it is easier to optimize. R(θ) = ||θ||₂² = ∑ⁿ_j=1 θ_j²
2. L1 Regularization: A linear function of the weight values. L1 prefers sparse solutions with weights set to zero. It results in weight vectors with fewer features. R(θ) = ||θ||₁ = ∑ⁿ_i=1 |θ_i|

Both types of regularization come from statistics. L1 is called lasso regression and L2 is called ridge regression. L1 is used more often in situations that require a lot of weights to be set to zero.

Learning in Multinomial Logistic Regression

The loss function for multinomial logistic regression extends the binary loss function from 2 to K classes. Also, the cross-entropy loss for a single example x is then:

L_CE(y^,y) = - ∑^K_k=1 y_k * log(y^_k) = - log(y^_c)

It turns out that the gradient for a single example is very similar to the binary gradient, (y^-y)*x.

Interpreting Models

One aspect of model building is to know why the classifier made its decision. That is, we want our decision to be interpretable. Because the features to logistic regression are often human-designed, one way to understand is to know the role each feature plays. This is enormously important for building transparent models. This can also be done by investigating the feature magnitude, as well as applying statistical tests.

Furthermore, logistic regression is used as a tool to test hypotheses, for example, whether logically negative words are associated with negative sentiment. However, it is necessary to control for potential confounds like the movie genre, the review length, and other factors that may influence sentiment.

In Summary

• Logistic Regression is a model that extracts features from input, multiplies them by a weight, and sums them via a sigmoid function to generate a probability.
• It can be used with two classes, or multiple classes with multinomial logistic regression.
• The softmax function can be used to compute probabilities.
• The weights are learned from a loss function like cross-entropy loss, which is minimized by iterative algorithms like gradient descent.
• Regularization is used to avoid overfitting.
• Lastly, logistic regression is a tool that can be used to study the importance of individual features.