5- Импутация с использованием многомерного вменения с помощью цепного уравнения (MICE)
Этот тип вменения работает, заполняя недостающие данные несколько раз. Многочисленные вменения (МИ) намного лучше, чем одно вменение, так как они лучше измеряют неопределенность отсутствующих значений Подход цепных уравнений также очень гибок и может обрабатывать различные переменные разных типов данных (т. Е. Непрерывные или двоичные), а также такие сложности, как границы или схемы пропуска при опросе. Для получения дополнительной информации о механике алгоритма, вы можете обратиться к R электронная бумага
6- Вменение с использованием глубокого обучения (Datawig):
Этот метод очень хорошо работает с категориальными и нечисловыми функциями. Это библиотека, которая изучает модели машинного обучения с использованием глубоких нейронных сетей для расчета отсутствующих значений в кадре данных. Он также поддерживает как CPU, так и GPU для обучения.
Вменение с использованием Datawig
Pros:
- Довольно точный по сравнению с другими методами.
- Он имеет некоторые функции, которые могут обрабатывать категориальные данные (Feature Encoder).
- Он поддерживает процессоры и графические процессоры.
Минусы:
- Одноколонное вменение.
- Может быть довольно медленным с большими наборами данных.
- Вы должны указать столбцы, которые содержат информацию о целевом столбце, который будет вменяться.
Другие методы вменения:
Стохастическая регрессия вменения:
Это очень похоже на вменение регрессии, которое пытается предсказать пропущенные значения путем регрессии из других связанных переменных в том же наборе данных плюс некоторое случайное остаточное значение.
Экстраполяция и интерполяция:
Он пытается оценить значения из других наблюдений в диапазоне дискретного набора известных точек данных.
Горячее вложение:
Работает путем случайного выбора отсутствующего значения из набора связанных и похожих переменных.
В заключение, нет идеального способа компенсировать отсутствующие значения в наборе данных. Каждая стратегия может работать лучше для определенных наборов данных и отсутствующих типов данных, но может работать намного хуже для других типов наборов данных. Существуют некоторые правила, которые определяют, какую стратегию использовать для определенных типов пропущенных значений, но помимо этого вам следует поэкспериментировать и проверить, какая модель лучше всего подходит для вашего набора данных.
Ссылки:
- [1] Buuren, S.V. & Groothuis-Oudshoorn, K. (2011). Мыши: многовариантное вложение с помощью цепных уравнений в R. Journal of Statistical Software
- https://impyute.readthedocs.io/en/master/index.html
Handling Missing Data with KNN Imputer
Handling missing data is a crucial step in the data preprocessing phase before building machine learning models. Missing data can cause issues in analysis and modeling, as many algorithms do not handle missing values directly. One commonly used method for handling missing data is the K-Nearest Neighbors (KNN) Imputer, which offers several benefits over other techniques:
3 min read
The Why?
1. Retains Data: KNN Imputer retains the most data compared to other techniques such as removing rows or columns with missing values. It replaces missing values with imputed values, ensuring that the entire dataset is used for analysis and modeling.
2. Preserves Relationships: KNN Imputer imputes missing values based on the nearest neighbors, which means it preserves the underlying relationships in the data. It takes into account the feature similarities between data points to estimate the missing values, making it more contextually relevant.
3. Non-parametric: KNN Imputer is a non-parametric method, which means it does not make assumptions about the data’s distribution. It is suitable for both numeric and categorical data, making it versatile in handling various types of missing values.
4. Flexibility: The KNN Imputer allows you to specify the number of nearest neighbors to use for imputation (controlled by the n_neighbors parameter). This flexibility lets you adapt the imputation to the specific characteristics of your dataset.
5. Less Bias: By using multiple neighboring data points to estimate missing values, KNN Imputer reduces the bias that may be introduced when using simple imputation techniques like mean or median imputation.
6. Improved Accuracy: In scenarios where missing values are not entirely at random, KNN Imputer can perform better than traditional imputation methods, leading to more accurate models and analyses.
7. Easy Implementation: Scikit-Learn provides a user-friendly implementation of the KNN Imputer, making it easy to incorporate into your data preprocessing pipeline.
The code begins by importing the necessary libraries and loading the dataset Building_permits.csv using Pandas. It also displays the first 5 rows of the dataset to give an overview.
Identifying Missing Values
The code calculates the total missing values in each dataset column using the isnull().sum() function .
Preparing Data for KNN Imputer
The code then prepares the data for KNN imputation by selecting only the numerical float columns from the dataset.
Applying KNN Imputer
The KNN Imputer is applied to fill in the missing values in the selected data. The n_neighbors parameter is set to 5, meaning the imputer will use the mean value of the 5 nearest neighbors to impute the missing values.
Displaying the Results
the code displays the first 5 rows of the imputed Data Frame to show the filled-in values after KNN imputation.
The When?
KNN Imputer is a powerful and versatile method for handling missing data, offering advantages such as data retention, relationship preservation, and adaptability to different data types. It is particularly useful when dealing with non-random missingness and can lead to more accurate and reliable machine-learning models.
sklearn.impute .KNNImputer¶
class sklearn.impute. KNNImputer ( * , missing_values = nan , n_neighbors = 5 , weights = ‘uniform’ , metric = ‘nan_euclidean’ , copy = True , add_indicator = False , keep_empty_features = False ) [source] ¶
Imputation for completing missing values using k-Nearest Neighbors.
Each sample’s missing values are imputed using the mean value from n_neighbors nearest neighbors found in the training set. Two samples are close if the features that neither is missing are close.
Read more in the User Guide .
New in version 0.22.
Parameters : missing_values int, float, str, np.nan or None, default=np.nan
The placeholder for the missing values. All occurrences of missing_values will be imputed. For pandas’ dataframes with nullable integer dtypes with missing values, missing_values should be set to np.nan, since pd.NA will be converted to np.nan.
n_neighbors int, default=5
Number of neighboring samples to use for imputation.
weights or callable, default=’uniform’
Weight function used in prediction. Possible values:
- ‘uniform’ : uniform weights. All points in each neighborhood are weighted equally.
- ‘distance’ : weight points by the inverse of their distance. in this case, closer neighbors of a query point will have a greater influence than neighbors which are further away.
- callable : a user-defined function which accepts an array of distances, and returns an array of the same shape containing the weights.
Distance metric for searching neighbors. Possible values:
- ‘nan_euclidean’
- callable : a user-defined function which conforms to the definition of _pairwise_callable(X, Y, metric, **kwds) . The function accepts two arrays, X and Y, and a missing_values keyword in kwds and returns a scalar distance value.
If True, a copy of X will be created. If False, imputation will be done in-place whenever possible.
add_indicator bool, default=False
If True, a MissingIndicator transform will stack onto the output of the imputer’s transform. This allows a predictive estimator to account for missingness despite imputation. If a feature has no missing values at fit/train time, the feature won’t appear on the missing indicator even if there are missing values at transform/test time.
keep_empty_features bool, default=False
If True, features that consist exclusively of missing values when fit is called are returned in results when transform is called. The imputed value is always 0 .
New in version 1.2.
Attributes : indicator_ MissingIndicator
Indicator used to add binary indicators for missing values. None if add_indicator is False.
n_features_in_ int
Number of features seen during fit .
New in version 0.24.
feature_names_in_ ndarray of shape ( n_features_in_ ,)
Names of features seen during fit . Defined only when X has feature names that are all strings.
New in version 1.0.
Univariate imputer for completing missing values with simple strategies.
Multivariate imputer that estimates values to impute for each feature with missing values from all the others.
- Olga Troyanskaya, Michael Cantor, Gavin Sherlock, Pat Brown, Trevor Hastie, Robert Tibshirani, David Botstein and Russ B. Altman, Missing value estimation methods for DNA microarrays, BIOINFORMATICS Vol. 17 no. 6, 2001 Pages 520-525.
>>> import numpy as np >>> from sklearn.impute import KNNImputer >>> X = [[1, 2, np.nan], [3, 4, 3], [np.nan, 6, 5], [8, 8, 7]] >>> imputer = KNNImputer(n_neighbors=2) >>> imputer.fit_transform(X) array([[1. , 2. , 4. ], [3. , 4. , 3. ], [5.5, 6. , 5. ], [8. , 8. , 7. ]])
Fit the imputer on X.
Fit to data, then transform it.
Get output feature names for transformation.
Get metadata routing of this object.
Get parameters for this estimator.
Set output container.
Set the parameters of this estimator.
Impute all missing values in X.
Fit the imputer on X.
Parameters : X array-like shape of (n_samples, n_features)
Input data, where n_samples is the number of samples and n_features is the number of features.
y Ignored
Not used, present here for API consistency by convention.
Returns : self object
The fitted KNNImputer class instance.
Fit to data, then transform it.
Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X .
Parameters : X array-like of shape (n_samples, n_features)
y array-like of shape (n_samples,) or (n_samples, n_outputs), default=None
Target values (None for unsupervised transformations).
**fit_params dict
Additional fit parameters.
Returns : X_new ndarray array of shape (n_samples, n_features_new)
get_feature_names_out ( input_features = None ) [source] ¶
Get output feature names for transformation.
Parameters : input_features array-like of str or None, default=None
- If input_features is None , then feature_names_in_ is used as feature names in. If feature_names_in_ is not defined, then the following input feature names are generated: [«x0», «x1», . «x(n_features_in_ — 1)»] .
- If input_features is an array-like, then input_features must match feature_names_in_ if feature_names_in_ is defined.
Transformed feature names.
Get metadata routing of this object.
Please check User Guide on how the routing mechanism works.
Returns : routing MetadataRequest
A MetadataRequest encapsulating routing information.
Get parameters for this estimator.
Parameters : deep bool, default=True
If True, will return the parameters for this estimator and contained subobjects that are estimators.
Returns : params dict
Parameter names mapped to their values.
Set output container.
See Introducing the set_output API for an example on how to use the API.
Parameters : transform , default=None
Configure output of transform and fit_transform .
- «default» : Default output format of a transformer
- «pandas» : DataFrame output
- None : Transform configuration is unchanged
Set the parameters of this estimator.
The method works on simple estimators as well as on nested objects (such as Pipeline ). The latter have parameters of the form __ so that it’s possible to update each component of a nested object.
Parameters : **params dict
Returns : self estimator instance
Impute all missing values in X.
Parameters : X array-like of shape (n_samples, n_features)
The input data to complete.
Returns : X array-like of shape (n_samples, n_output_features)
The imputed dataset. n_output_features is the number of features that is not always missing during fit .
A Guide To KNN Imputation
Missing values exist in almost all datasets and it is essential to handle them properly in order to construct reliable machine learning models with optimal statistical power. In this article, we will talk about what missing values are, how to identify them, and how to replace them by using the K-Nearest Neighbors imputation method. To demonstrate this method, we will use the famous Titanic dataset in this guide.
What are Missing Values?
A missing value can be defined as the data value that is not captured nor stored for a variable in the observation of interest. There are 3 types of missing values —
Missing Completely at Random (MCAR)
MCAR occurs when the missing on the variable is completely unsystematic. When our dataset is missing values completely at random, the probability of missing data is unrelated to any other variable and unrelated to the variable with missing values itself. For example, MCAR would occur when data is missing because the responses to a research survey about depression are lost in the mail.
Missing at Random (MAR)
MAR occurs when the probability of the missing data on a variable is related to some other measured variable but unrelated to the variable with missing values itself. For example, the data values are missing because males are less likely to respond to a depression survey. In this case, the missing data is related to the gender of the respondents. However, the missing data is not related to the level of depression itself.
Missing Not at Random (MNAR)
MNAR occurs when the missing values on a variable are related to the variable with the missing values itself. In this case, the data values are missing because the respondents failed to fill in the survey due to their level of depression.
Effects of Missing Values
Having missing values in our datasets can have various detrimental effects. Here are a few examples —
- Missing data can limit our ability to perform important data science tasks such as converting data types or visualizing data
- Missing data can reduce the statistical power of our models which in turn increases the probability of Type II error. Type II error is the failure to reject a false null hypothesis.
- Missing data can reduce the representativeness of the samples in the dataset.
- Missing data can distort the validity of the scientific trials and can lead to invalid conclusions.
Identifying Missing Values
Finding missing values with Python is straightforward. First, we will import Pandas and create a data frame for the Titanic dataset.
import pandas as pddf = pd.read_csv(‘titanic.csv’)
Next, we will remove some of the independent variable columns that have little use for KNN Imputer or the machine learning algorithm if we are building one. These columns include passenger names, passenger IDs, cabin and ticket numbers.
df = df.drop(['Unnamed: 0', 'PassengerId', 'Name',
'Ticket', 'Cabin'], axis=1)
We will then use Pandas’ data frame attributes, ‘.isna()’ and ‘.isany()’, to detect missing values. These attributes will return Boolean values where ‘True’ indicates that there are missing values in the particular column.
df.isna().isany()
As we can see, the columns ‘Age’ and ‘Embarked’ have missing values. Instead of ‘.isany()’, we can also use ‘.sum()’ to find out the number of missing values in the columns.
df.isna().sum()
There you go. Now, we know that ‘Age’ has 177 and ‘Embarked’ has 2 missing values.
KNN Imputer
KNN Imputer was first supported by Scikit-Learn in December 2019 when it released its version 0.22. This imputer utilizes the k-Nearest Neighbors method to replace the missing values in the datasets with the mean value from the parameter ‘n_neighbors’ nearest neighbors found in the training set. By default, it uses a Euclidean distance metric to impute the missing values.
To see this imputer in action, we will import it from Scikit-Learn’s impute package —
from sklearn.impute import KNNImputer
One thing to note here is that the KNN Imputer does not recognize text data values. It will generate errors if we do not change these values to numerical values. For example, in our Titanic dataset, the categorical columns ‘Sex’ and ‘Embarked’ have text data.
A good way to modify the text data is to perform one-hot encoding or create “dummy variables”. The idea is to convert each category into a binary data column by assigning a 1 or 0. Other options would be to use LabelEncoder or OrdinalEncoder from Scikit-Learn’s preprocessing package.
In this tutorial, we will stick to one-hot encoding. First, we will make a list of categorical variables with text data and generate dummy variables by using ‘.get_dummies’ attribute of Pandas data frame package. An important caveat here is we are setting “drop_first” parameters as True in order to prevent the Dummy Variable Trap.
Note: You can also use Scikit-Learn’s LabelBinarizer method here.
cat_variables = df[[‘Sex’, ‘Embarked’]]
cat_dummies = pd.get_dummies(cat_variables, drop_first=True)
cat_dummies.head()
Now we have 3 dummy variable columns. In the “Sex_male” column, 1 indicates that the passenger is male and 0 is female. The “Sex_female” column is dropped since the “drop_first” parameter is set as True. Similarly, there are only 2 columns for “Embarked” because the third one has been dropped.
Next, we will drop the original “Sex” and “Embarked” columns from the data frame and add the dummy variables.
df = df.drop(['Sex', 'Embarked'], axis=1)
df = pd.concat([df, cat_dummies], axis=1)
df.head()
Another critical point here is that the KNN Imptuer is a distance-based imputation method and it requires us to normalize our data. Otherwise, the different scales of our data will lead the KNN Imputer to generate biased replacements for the missing values. For simplicity, we will use Scikit-Learn’s MinMaxScaler which will scale our variables to have values between 0 and 1.
from sklearn.preprocessing import MinMaxScalerscaler = MinMaxScaler()
df = pd.DataFrame(scaler.fit_transform(df), columns = df.columns)
df.head()
Now that our dataset has dummy variables and normalized, we can move on to the KNN Imputation. Let’s import it from Scikit-Learn’s Impute package and apply it to our data. In this example, we are setting the parameter ‘n_neighbors’ as 5. So, the missing values will be replaced by the mean value of 5 nearest neighbors measured by Euclidean distance.
from sklearn.impute import KNNImputerimputer = KNNImputer(n_neighbors=5)
df = pd.DataFrame(imputer.fit_transform(df),columns = df.columns)
Ok, the verdict is in! Let’s see the results.
df.isna().any()
df.isna().sum()
As demonstrated above, our data frame no longer has missing values. They have been imputed as the means of k-Nearest Neighbor values.
Conclusion
There are different ways to handle missing data. Some methods such as removing the entire observation if it has a missing value or replacing the missing values with mean, median or mode values. However, these methods can waste valuable data or reduce the variability of your dataset. In contrast, KNN Imputer maintains the value and variability of your datasets and yet it is more precise and efficient than using the average values.
References
- https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3668100/
- http://www.stat.columbia.edu/~gelman/arm/missing.pdf
- https://machinelearningmastery.com/knn-imputation-for-missing-values-in-machine-learning/
- https://scikit-learn.org/stable/modules/generated/sklearn.impute.KNNImputer.html
- https://www.iriseekhout.com/missing-data/