Feature Engineering
Feature engineering is the process of using domain knowledge to select, transform, or create input variables (features) from raw data to improve the performance of machine learning models. It bridges raw data and predictive algorithms by producing representations that algorithms can learn from more effectively. Techniques include normalization, one-hot encoding, polynomial feature creation, and dimensionality reduction.
Common Feature Engineering Techniques and Their Use Cases
| Technique | Input Type | Output | Purpose | Example |
|---|---|---|---|---|
| Normalization | Numerical | Scaled [0,1] | Remove magnitude bias | Age: 25 → 0.25 |
| One-Hot Encoding | Categorical | Binary vector | Encode categories | Color: Red → [1,0,0] |
| Log Transform | Skewed numeric | Log-scaled | Reduce skewness | Income: log(50000) |
| Binning | Continuous | Ordinal groups | Discretize values | Age → Young/Mid/Old |
| Polynomial Features | Numeric | Higher-degree | Capture nonlinearity | x → x, x², x³ |
| TF-IDF | Text | Numeric vector | Text representation | Word frequency weights |
Interactive Tools
Wikimedia Commons, CC BY-SA
Related Terms
Decision Tree (ML)
A decision tree is a supervised machine learning model that splits data into branches based on feature values, forming a tree structure where each internal node represents a feature test, each branch represents an outcome, and each leaf node holds a prediction. Trees are trained by choosing splits that maximise information gain or minimise Gini impurity at each step. They are highly interpretable and serve as the building block for ensemble methods like random forests and gradient boosting.
Random Forest
A random forest is an ensemble machine learning algorithm that constructs a large number of decision trees during training and outputs the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees. Each tree is trained on a bootstrap sample of the data and uses a random subset of features at each split, introducing diversity that reduces variance and overfitting. Introduced by Leo Breiman in 2001, random forests are among the most widely used and robust general-purpose algorithms.
Regularization (ML)
Regularization in machine learning refers to techniques that add a penalty term to the loss function to discourage model complexity, thereby reducing overfitting and improving generalisation to unseen data. The two most common forms are L1 (Lasso) regularization, which promotes sparsity by penalising the absolute values of weights, and L2 (Ridge) regularization, which penalises the squared values, shrinking all weights toward zero. Regularization is a fundamental concept in statistical learning theory, closely tied to the bias–variance trade-off.
From Latin "feature" (factura, a making or formation) and "engineering" (from Latin ingenium, cleverness). The compound term emerged in the machine learning community in the 1990s–2000s, popularised by practitioners building early ML pipelines.