Feature Selection Flashcards | Learn Forward Selection, Backward Elimination, and LASSO
Arrow keys or swipe to navigate cards
Master feature selection methods with our flashcards. Learn Forward Selection, Backward Elimination, and LASSO easily. Explore now on Stellar Study Cards!
What is forward selection in the context of feature selection? Forward selection is a stepwise selection technique where models are incrementally built by adding features one at a time, starting from an empty model and adding the feature that improves the model the most at each step.
How does forward selection determine which feature to add next? At each step, forward selection evaluates the addition of each remaining feature based on a criterion such as AIC, BIC, or adjusted R-squared, and selects the feature that provides the greatest improvement in the specified criterion.
What are the advantages of using forward selection over full model fitting? Forward selection can be computationally faster and less prone to overfitting compared to fitting models with all possible feature subsets, especially in scenarios with a large number of features.
Can forward selection be used with non-linear models like decision trees? Yes, forward selection can be applied to any model that can be evaluated using a given criterion, including non-linear models such as decision trees. However, its effectiveness might vary depending on the model's complexity and data characteristics.
What is a potential drawback of using forward selection? A potential drawback of forward selection is that it is a greedy algorithm, meaning it makes locally optimal choices at each step, which may not result in a globally optimal model.
What is Backward Elimination in the context of feature selection? Backward Elimination is a feature selection technique where you start with all features and iteratively remove the least significant feature, based on a statistical test like p-value, to improve the model's performance.
What is the stopping criterion in Backward Elimination? The stopping criterion in Backward Elimination is when all remaining features have p-values less than a pre-defined significance level (e.g., 0.05), indicating they are statistically significant.
How do you determine which feature to remove during Backward Elimination? During Backward Elimination, you determine the feature to remove by selecting the feature with the highest p-value from a statistical test like an F-test or t-test, which exceeds the significance threshold.
What type of model is typically used in Backward Elimination? Backward Elimination is typically used with linear regression models, but it can also be applied to other types of models that allow statistical tests for significance of features.
What are the advantages of using Backward Elimination for feature selection? The advantages of Backward Elimination include simplicity in implementation, the ability to improve model interpretability by reducing irrelevant features, and enhancing model performance by minimizing overfitting.
What does LASSO stand for and what is its primary purpose in machine learning models? LASSO stands for Least Absolute Shrinkage and Selection Operator. Its primary purpose is to perform both variable selection and regularization to enhance the prediction accuracy and interpretability of statistical models.
What is the key mathematical expression that represents the LASSO penalty term? The LASSO penalty term is represented as \( \lambda \sum_{j=1}^{p} |\beta_j| \), where \( \lambda \) is a tuning parameter and \( \beta_j \) are the coefficients of the model.
How does LASSO differ from Ridge Regression in terms of variable selection? LASSO can shrink some coefficients to exactly zero, effectively selecting a subset of features. In contrast, Ridge Regression shrinks coefficients but does not set them exactly to zero, so it includes all features in the model.
What impact does the tuning parameter \( \lambda \) have in LASSO? The tuning parameter \( \lambda \) controls the strength of the penalty. A larger \( \lambda \) results in more coefficients being set to zero (more feature selection), while a smaller \( \lambda \) keeps more features with non-zero coefficients.
In which scenarios is LASSO particularly useful? LASSO is particularly useful in scenarios where automatic feature selection is desired and where the model needs to be interpretable by reducing the number of variables used, especially when there is multicollinearity among features.