Package 'VIM'

Description

Calculate or plot the amount of missing/imputed values in each variable and the amount of missing/imputed values in certain combinations of variables.

Print method for objects of class "aggr".

Summary method for objects of class "aggr".

Print method for objects of class "summary.aggr".

Usage

aggr(x, delimiter = NULL, plot = TRUE, ...)

## S3 method for class 'aggr'
plot(
  x,
  col = c("skyblue", "red", "orange"),
  bars = TRUE,
  numbers = FALSE,
  prop = TRUE,
  combined = FALSE,
  varheight = FALSE,
  only.miss = FALSE,
  border = par("fg"),
  sortVars = FALSE,
  sortCombs = TRUE,
  ylabs = NULL,
  axes = TRUE,
  labels = axes,
  cex.lab = 1.2,
  cex.axis = par("cex"),
  cex.numbers = par("cex"),
  gap = 4,
  ...
)

## S3 method for class 'aggr'
print(x, ..., digits = NULL)

## S3 method for class 'aggr'
summary(object, ...)

## S3 method for class 'summary.aggr'
print(x, ...)

Arguments

Details

Often it is of interest how many missing/imputed values are contained in each variable. Even more interesting, there may be certain combinations of variables with a high number of missing/imputed values.

If combined is FALSE, two separate plots are drawn for the missing/imputed values in each variable and the combinations of missing/imputed and non-missing values. The barplot on the left hand side shows the amount of missing/imputed values in each variable. In the aggregation plot on the right hand side, all existing combinations of missing/imputed and non-missing values in the observations are visualized. Available, missing and imputed data are color coded as given by col. Additionally, there are two possibilities to represent the frequencies of occurrence of the different combinations. The first option is to visualize the proportions or frequencies by a small bar plot and/or numbers. The second option is to let the cell heights be given by the frequencies of the corresponding combinations. Furthermore, variables may be sorted by the number of missing/imputed values and combinations by the frequency of occurrence to give more power to finding the structure of missing/imputed values.

If combined is TRUE, a small version of the barplot showing the amount of missing/imputed values in each variable is drawn on top of the aggregation plot.

The graphical parameter oma will be set unless supplied as an argument.

Value

for aggr, a list of class "aggr" containing the following components:

• x the data used.

• combinations a character vector representing the combinations of variables.

• count the frequencies of these combinations.

• percent the percentage of these combinations.

• missings a data.frame containing the amount of missing/imputed values in each variable.

• tabcomb the indicator matrix for the combinations of variables.

a list of class "summary.aggr" containing the following components:

• missings a data.frame containing the amount of missing or imputed values in each variable.

• combinations a data.frame containing a character vector representing the combinations of variables along with their frequencies and percentages.

Note

Some of the argument names and positions have changed with version 1.3 due to extended functionality and for more consistency with other plot functions in VIM. For back compatibility, the arguments labs and names.arg can still be supplied to ...{} and are handled correctly. Nevertheless, they are deprecated and no longer documented. Use ylabs and labels instead.

Author(s)

Andreas Alfons, Matthias Templ, modifications for displaying imputed values by Bernd Prantner

Matthias Templ, modifications by Andreas Alfons and Bernd Prantner

Matthias Templ, modifications by Andreas Alfons

Andreas Alfons, modifications by Bernd Prantner

References

M. Templ, A. Alfons, P. Filzmoser (2012) Exploring incomplete data using visualization tools. Advances in Data Analysis and Classification, Online first. DOI: 10.1007/s11634-011-0102-y.

See Also

print.aggr(), summary.aggr()

aggr()

print.summary.aggr(), aggr()

summary.aggr(), aggr()

Other plotting functions: barMiss(), histMiss(), marginmatrix(), marginplot(), matrixplot(), mosaicMiss(), pairsVIM(), parcoordMiss(), pbox(), scattJitt(), scattMiss(), scattmatrixMiss(), spineMiss()

Examples

data(sleep, package="VIM")
## for missing values
a <- aggr(sleep)
a
summary(a)

## for imputed values
sleep_IMPUTED <- kNN(sleep)
a <- aggr(sleep_IMPUTED, delimiter="_imp")
a
summary(a)


data(sleep, package = "VIM")
a <- aggr(sleep, plot=FALSE)
a


data(sleep, package = "VIM")
summary(aggr(sleep, plot=FALSE))


data(sleep, package = "VIM")
s <- summary(aggr(sleep, plot=FALSE))
s

Description

Convert colors to semitransparent colors.

Usage

alphablend(col, alpha = NULL, bg = NULL)

Arguments

Value

a vector containing the semitransparent colors.

Author(s)

Andreas Alfons

Examples

alphablend("red", 0.6)

Description

Average log brain and log body weights for 28 Species

Format

A data frame with 28 observations on the following 2 variables.

lbody

log body weight

lbrain

log brain weight

Details

The original data can be found in package MASS. 10 values on brain weight are set to be missing.

Source

P. J. Rousseeuw and A. M. Leroy (1987) Robust Regression and Outlier Detection. Wiley, p. 57.

References

Venables, W. N. and Ripley, B. D. (1999) Modern Applied Statistics with S-PLUS. Third Edition. Springer.

Templ, M. (2022) Visualization and Imputation of Missing Values. Springer Publishing. Upcoming book.

Examples

data(Animals_na)
aggr(Animals_na)

Description

Barplot with highlighting of missing/imputed values in other variables by splitting each bar into two parts. Additionally, information about missing/imputed values in the variable of interest is shown on the right hand side.

Usage

barMiss(
  x,
  delimiter = NULL,
  pos = 1,
  selection = c("any", "all"),
  col = c("skyblue", "red", "skyblue4", "red4", "orange", "orange4"),
  border = NULL,
  main = NULL,
  sub = NULL,
  xlab = NULL,
  ylab = NULL,
  axes = TRUE,
  labels = axes,
  only.miss = TRUE,
  miss.labels = axes,
  interactive = TRUE,
  ...
)

Arguments

Details

If more than one variable is supplied, the bars for the variable of interest are split according to missingness/number of imputed missings in the additional variables.

If only.miss=TRUE, the missing/imputed values in the variable of interest are visualized by one bar on the right hand side. If additional variables are supplied, this bar is again split into two parts according to missingness/number of imputed missings in the additional variables.

Otherwise, a small barplot consisting of two bars is drawn on the right hand side. The first bar corresponds to observed values in the variable of interest and the second bar to missing/imputed values. Since these two bars are not on the same scale as the main barplot, a second y-axis is plotted on the right (if axes=TRUE). Each of the two bars are again split into two parts according to missingness/number of imputed missings in the additional variables. Note that this display does not make sense if only one variable is supplied, therefore only.miss is ignored in that case.

If interactive=TRUE, clicking in the left margin of the plot results in switching to the previous variable and clicking in the right margin results in switching to the next variable. Clicking anywhere else on the graphics device quits the interactive session. When switching to a continuous variable, a histogram is plotted rather than a barplot.

Value

a numeric vector giving the coordinates of the midpoints of the bars.

Note

Some of the argument names and positions have changed with version 1.3 due to extended functionality and for more consistency with other plot functions in VIM. For back compatibility, the arguments axisnames, names.arg and names.miss can still be supplied to ...{} and are handled correctly. Nevertheless, they are deprecated and no longer documented. Use labels and miss.labels instead.

Author(s)

Andreas Alfons, modifications to show imputed values by Bernd Prantner

References

M. Templ, A. Alfons, P. Filzmoser (2012) Exploring incomplete data using visualization tools. Advances in Data Analysis and Classification, Online first. DOI: 10.1007/s11634-011-0102-y.

See Also

spineMiss(), histMiss()

Other plotting functions: aggr(), histMiss(), marginmatrix(), marginplot(), matrixplot(), mosaicMiss(), pairsVIM(), parcoordMiss(), pbox(), scattJitt(), scattMiss(), scattmatrixMiss(), spineMiss()

Examples

data(sleep, package = "VIM")
## for missing values
x <- sleep[, c("Exp", "Sleep")]
barMiss(x)
barMiss(x, only.miss = FALSE)

## for imputed values
x_IMPUTED  <- kNN(sleep[, c("Exp", "Sleep")])
barMiss(x_IMPUTED, delimiter = "_imp")
barMiss(x_IMPUTED, delimiter = "_imp", only.miss = FALSE)

Description

Dataset containing the original Wisconsin breast cancer data.

Format

A data frame with 699 observations on the following 11 variables.

ID

Sample ID

clump_thickness

as integer from 1 - 10

uniformity_cellsize

as integer from 1 - 10

uniformity_cellshape

as integer from 1 - 10

adhesion

as integer from 1 - 10

epithelial_cellsize

as integer from 1 - 10

bare_nuclei

as integer from 1 - 10, includes 16 missings

chromatin

as integer from 1 - 10

normal_nucleoli

as integer from 1 - 10

mitoses

as integer from 1 - 10

class

benign or malignant

References

The data downloaded and conditioned for R from the UCI machine learning repository, see https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Original) This breast cancer databases was obtained from the University of Wisconsin Hospitals, Madison from Dr. William H. Wolberg. If you publish results when using this database, then please include this information in your acknowledgements. Also, please cite one or more of: O. L. Mangasarian and W. H. Wolberg: "Cancer diagnosis via linear programming", SIAM News, Volume 23, Number 5, September 1990, pp 1 & 18. William H. Wolberg and O.L. Mangasarian: "Multisurface method of pattern separation for medical diagnosis applied to breast cytology", Proceedings of the National Academy of Sciences, U.S.A., Volume 87, December 1990, pp 9193-9196. O. L. Mangasarian, R. Setiono, and W.H. Wolberg: "Pattern recognition via linear programming: Theory and application to medical diagnosis", in: "Large-scale numerical optimization", Thomas F. Coleman and Yuying Li, editors, SIAM Publications, Philadelphia 1990, pp 22-30. K. P. Bennett & O. L. Mangasarian: "Robust linear programming discrimination of two linearly inseparable sets", Optimization Methods and Software 1, 1992, 23-34 (Gordon & Breach Science Publishers).

Examples

data(bcancer)
aggr(bcancer)

Description

Plot a background map.

Usage

bgmap(map, add = FALSE, ...)

Arguments

Author(s)

Andreas Alfons

References

M. Templ, A. Alfons, P. Filzmoser (2012) Exploring incomplete data using visualization tools. Advances in Data Analysis and Classification, Vol. 6, pp. 29-47. DOI: 10.1007/s11634-011-0102-y.

See Also

growdotMiss(), mapMiss()

Examples

data(kola.background, package = "VIM")
bgmap(kola.background)

Description

A plastic product is produced in three parallel reactors (TK104, TK105, or TK107). For each row in the dataset, we have the same batch of raw material that was split, and fed to the 3 reactors. These values are the brittleness index for the product produced in the reactor. A simulated data set.

Format

A data frame with 23 observations on the following 3 variables.

TK104

Brittleness for batches of raw material in reactor 104

TK105

Brittleness for batches of raw material in reactor 105

TK107

Brittleness for batches of raw material in reactor 107

Source

https://openmv.net/info/brittleness-index

Examples

data(brittleness)
aggr(brittleness)

Description

For each continuous cell (i,j), computes the conditional expectation $E(x_{ij} | x_{i,-j})$ under a robust Gaussian model (estimated via MCD), standardizes the residual, and applies a weight function. This captures multivariate outlier structure that univariate standardization misses.

Usage

cellWeightsMCD(X, method = "tukey", alpha = NULL)

Arguments

Value

an n x p numeric matrix of weights in $[0, 1]$

Author(s)

Matthias Templ

Description

This data set is the same as in package mvoutlier, except that values below the detection limit are coded as NA.

Format

A data frame with 606 observations on the following 110 variables.

*ID

a numeric vector

XCOO

a numeric vector

YCOO

a numeric vector

Ag

a numeric vector

Ag_INAA

a numeric vector

Al

a numeric vector

Al2O3

a numeric vector

As

a numeric vector

As_INAA

a numeric vector

Au_INAA

a numeric vector

B

a numeric vector

Ba

a numeric vector

Ba_INAA

a numeric vector

Be

a numeric vector

Bi

a numeric vector

Br_IC

a numeric vector

Br_INAA

a numeric vector

Ca

a numeric vector

Ca_INAA

a numeric vector

CaO

a numeric vector

Cd

a numeric vector

Ce_INAA

a numeric vector

Cl_IC

a numeric vector

Co

a numeric vector

Co_INAA

a numeric vector

EC

a numeric vector

Cr

a numeric vector

Cr_INAA

a numeric vector

Cs_INAA

a numeric vector

Cu

a numeric vector

Eu_INAA

a numeric vector

F_IC

a numeric vector

Fe

a numeric vector

Fe_INAA

a numeric vector

Fe2O3

a numeric vector

Hf_INAA

a numeric vector

Hg

a numeric vector

Hg_INAA

a numeric vector

Ir_INAA

a numeric vector

K

a numeric vector

K2O

a numeric vector

La

a numeric vector

La_INAA

a numeric vector

Li

a numeric vector

LOI

a numeric vector

Lu_INAA

a numeric vector

wt_INAA

a numeric vector

Mg

a numeric vector

MgO

a numeric vector

Mn

a numeric vector

MnO

a numeric vector

Mo

a numeric vector

Mo_INAA

a numeric vector

Na

a numeric vector

Na_INAA

a numeric vector

Na2O

a numeric vector

Nd_INAA

a numeric vector

Ni

a numeric vector

Ni_INAA

a numeric vector

NO3_IC

a numeric vector

P

a numeric vector

P2O5

a numeric vector

Pb

a numeric vector

pH

a numeric vector

PO4_IC

a numeric vector

Rb

a numeric vector

S

a numeric vector

Sb

a numeric vector

Sb_INAA

a numeric vector

Sc

a numeric vector

Sc_INAA

a numeric vector

Se

a numeric vector

Se_INAA

a numeric vector

Si

a numeric vector

SiO2

a numeric vector

Sm_INAA

a numeric vector

Sn_INAA

a numeric vector

SO4_IC

a numeric vector

Sr

a numeric vector

Sr_INAA

a numeric vector

SUM_XRF

a numeric vector

Ta_INAA

a numeric vector

Tb_INAA

a numeric vector

Te

a numeric vector

Th

a numeric vector

Th_INAA

a numeric vector

Ti

a numeric vector

TiO2

a numeric vector

U_INAA

a numeric vector

V

a numeric vector

W_INAA

a numeric vector

Y

a numeric vector

Yb_INAA

a numeric vector

Zn

a numeric vector

Zn_INAA

a numeric vector

ELEV

a numeric vector

*COUN

a numeric vector

*ASP

a numeric vector

TOPC

a numeric vector

LITO

a numeric vector

Al_XRF

a numeric vector

Ca_XRF

a numeric vector

Fe_XRF

a numeric vector

K_XRF

a numeric vector

Mg_XRF

a numeric vector

Mn_XRF

a numeric vector

Na_XRF

a numeric vector

P_XRF

a numeric vector

Si_XRF

a numeric vector

Ti_XRF

a numeric vector

Note

For a more detailed description of this data set, see the help file chorizon in package mvoutlier.

Source

Kola Project (1993-1998)

References

Reimann, C., Filzmoser, P., Garrett, R.G. and Dutter, R. (2008) Statistical Data Analysis Explained: Applied Environmental Statistics with R. Wiley.

Examples

data(chorizonDL, package = "VIM")
summary(chorizonDL)

Description

This is a modified version of the original training data set taken from the UCI repository, see reference. The modifications are only related to having appropriate levels for factor variables. This data set is about horse diseases where the task is to determine, if the lesion of the horse was surgical or not.

Format

A training data frame with 300 observations on the following 31 variables.

surgery

yes or no

age

1 equals an adult horse, 2 is a horse younger than 6 months

hospitalID

ID

temp_rectal

rectal temperature

pulse

heart rate in beats per minute

respiratory_rate

a normal rate is between 8 and 10

temp_extreme

temperature of extremities

pulse_peripheral

factor with four categories

capillayr_refill_time

a clinical judgement. The longer the refill, the poorer the circulation. Possible values are 1 = < 3 seconds and 2 = >= 3 seconds

pain

a subjective judgement of the horse's pain level

peristalsis

an indication of the activity in the horse's gut. As the gut becomes more distended or the horse becomes more toxic, the activity decreases

abdominal_distension

An animal with abdominal distension is likely to be painful and have reduced gut motility. A horse with severe abdominal distension is likely to require surgery just tio relieve the pressure

nasogastric_tube

This refers to any gas coming out of the tube. A large gas cap in the stomach is likely to give the horse discomfort

nasogastric_reflux

posible values are 1 = none, 2 = > 1 liter, 3 = < 1 liter. The greater amount of reflux, the more likelihood that there is some serious obstruction to the fluid passage from the rest of the intestine

nasogastric_reflux_PH

scale is from 0 to 14 with 7 being neutral. Normal values are in the 3 to 4 range

rectal_examination

Rectal examination. Absent feces probably indicates an obstruction

abdomen

abdomen. possible values 1 = normal, 2 = other, 3 = firm feces in the large intestine, 4 = distended small intestine, 5 = distended large intestine

cell_volume

packed cell volume. normal range is 30 to 50. The level rises as the circulation becomes compromised or as the animal becomes dehydrated.

protein

total protein. Normal values lie in the 6-7.5 (gms/dL) range. The higher the value the greater the dehydration

abdominocentesis_appearance

Abdominocentesis appearance. A needle is put in the horse's abdomen and fluid is obtained from the abdominal cavity

abdomcentesis_protein

abdomcentesis total protein. The higher the level of protein the more likely it is to have a compromised gut. Values are in gms/dL

outcome

What eventually happened to the horse?

surgical_lesion

retrospectively, was the problem (lesion) surgical?

lesion_type1

type of lesion

lesion_type2

type of lesion

lesion_type3

type of lesion

cp_data

temp_extreme_ordered

temperature of extremities (ordered)

mucous_membranes_col

mucous membranes. A subjective measurement of colour

mucous_membranes_group

different recodings of mucous membrances

Source

https://archive.ics.uci.edu/ml/datasets/Horse+Colic Creators: Mary McLeish & Matt Cecile, Department of Computer Science, University of Guelph, Guelph, Ontario, Canada N1G 2W1 Donor: Will Taylor

Examples

data(colic)
aggr(colic)

Description

Subset of the collision data from December 20. to December 31. 2018 from NYCD.

Details

Each record represents a collision in NYC by city, borough, precinct and cross street.

Source

https://data.cityofnewyork.us/Public-Safety/NYPD-Motor-Vehicle-Collisions/h9gi-nx95

Examples

data(collisions)
aggr(collisions)

Description

Colored map in which the proportion or amount of missing/imputed values in each region is coded according to a continuous or discrete color scheme. The sequential color palette may thereby be computed in the HCL or the RGB color space.

Usage

colormapMiss(
  x,
  region,
  map,
  imp_index = NULL,
  prop = TRUE,
  polysRegion = 1:length(x),
  range = NULL,
  n = NULL,
  col = c("red", "orange"),
  gamma = 2.2,
  fixup = TRUE,
  coords = NULL,
  numbers = TRUE,
  digits = 2,
  cex.numbers = 0.8,
  col.numbers = par("fg"),
  legend = TRUE,
  interactive = TRUE,
  ...
)

colormapMissLegend(
  xleft,
  ybottom,
  xright,
  ytop,
  cmap,
  n = 1000,
  horizontal = TRUE,
  digits = 2,
  cex.numbers = 0.8,
  col.numbers = par("fg"),
  ...
)

Arguments

Details

The proportion or amount of missing/imputed values in x of each region is coded according to a continuous or discrete color scheme in the color range defined by col. In addition, the proportions or numbers can be shown as labels in the regions.

If interactive is TRUE, clicking in a region displays more detailed information about missing/imputed values on the console. Clicking outside the borders quits the interactive session.

Value

colormapMiss returns a list with the following components:

• nmiss a numeric vector containing the number of missing/imputed values in each region.

• nobs a numeric vector containing the number of observations in each region.

• pmiss a numeric vector containing the proportion of missing values in each region.

• prop a logical indicating whether the proportion of missing/imputed values have been used rather than the total amount.

• range the range of the proportion or amount of missing/imputed values corresponding to the color range.

• n either a positive integer giving the number of equally spaced cut-off points for a discretized color scheme, or NULL for a continuous color scheme.

• start the start color of the color scheme.

• end the end color of the color scheme.

• space a character string giving the color space (either "rgb" for RGB colors or "hcl" for HCL colors).

• gamma numeric; the display gamma value (see colorspace::hex()).

• fixup a logical indicating whether the colors have been corrected to valid RGB values (see colorspace::hex()).

Note

Some of the argument names and positions have changed with versions 1.3 and 1.4 due to extended functionality and for more consistency with other plot functions in VIM. For back compatibility, the arguments cex.text and col.text can still be supplied to ...{} and are handled correctly. Nevertheless, they are deprecated and no longer documented. Use cex.numbers and col.numbers instead.

Author(s)

Andreas Alfons, modifications to show imputed values by Bernd Prantner

References

M. Templ, A. Alfons, P. Filzmoser (2012) Exploring incomplete data using visualization tools. Advances in Data Analysis and Classification, Online first. DOI: 10.1007/s11634-011-0102-y.

See Also

colSequence(), growdotMiss(), mapMiss()

Description

Compute color sequences by linear interpolation based on a continuous color scheme between certain start and end colors. Color sequences may thereby be computed in the HCL or RGB color space.

Usage

colSequence(p, start, end, space = c("hcl", "rgb"), ...)

colSequenceRGB(p, start, end, fixup = TRUE, ...)

colSequenceHCL(p, start, end, fixup = TRUE, ...)

Arguments

Value

A character vector containing hexadecimal strings of the form "#RRGGBB".

Author(s)

Andreas Alfons

References

Zeileis, A., Hornik, K., Murrell, P. (2009) Escaping RGBland: Selecting colors for statistical graphics. Computational Statistics & Data Analysis, 53 (9), 1259–1270.

See Also

colorspace::hex(), colorspace::sequential_hcl()

Examples

p <- c(0, 0.3, 0.55, 0.8, 1)

## HCL colors
colSequence(p, c(0, 0, 100), c(0, 100, 50))
colSequence(p, polarLUV(L=90, C=30, H=90), c(0, 100, 50))

## RGB colors
colSequence(p, c(1, 1, 1), c(1, 0, 0), space="rgb")
colSequence(p, RGB(1, 1, 0), "red")

Description

Count the number of infinite or missing values in a vector.

Usage

countInf(x)

Arguments

Value

countInf returns the number of infinite values in x. countNA returns the number of missing values in x.

Author(s)

Andreas Alfons

Examples

data(sleep, package="VIM")
countInf(log(sleep$Dream))
countNA(sleep$Dream)

Description

A synthetic replacement for the Pima Indians Diabetes data: 768 observations on 9 variables, a binary target (Outcome) and medical predictors such as the number of pregnancies, plasma glucose, BMI, insulin level and age. Every value is synthetic; no row corresponds to a real person.

Format

A data frame with 768 observations on the following 9 variables.

Pregnancies

Number of times pregnant (0 is a valid value)

Glucose

Plasma glucose concentration at 2 hours in an oral glucose tolerance test

BloodPressure

Diastolic blood pressure (mm Hg)

SkinThickness

Triceps skin fold thickness (mm)

Insulin

2-Hour serum insulin (mu U/ml)

BMI

Body mass index (weight in kg/(height in m)^2)

DiabetesPedigreeFunction

Diabetes pedigree function

Age

Age in years

Outcome

Diabetes (yes or no)

Details

The original data were collected by the National Institute of Diabetes and Digestive and Kidney Diseases from female patients of Pima Indian heritage aged at least 21. The team that collected them has stated that the data should not have been distributed and has asked for redistribution to stop, while a large body of teaching material and package examples depends on them. This synthetic version exists so that such material keeps working without real patient records being circulated.

The schema follows PimaIndiansDiabetes2 from package mlbench: physically impossible values (a zero glucose level, blood pressure, skin fold thickness, insulin level or BMI) are coded as NA, whereas 0 pregnancies is a valid value. Column names, types and the levels of Outcome are those VIM has used since version 6, so existing code runs unchanged; only the values differ from the real data shipped before VIM 7.3.0.

Missing values in the original are not scattered at random: BloodPressure, SkinThickness and Insulin are frequently missing together, and the pattern is age-related. The pattern itself was synthesised as a variable and the missing values imposed from it, so that complete-case and imputation examples behave as they did before (placing missing values independently would produce the joint pattern about five times instead of the observed 19 to 24).

Results computed on these data resemble published analyses of the original without matching them: a logistic regression reaches an AUC of 0.868 against 0.862 on the original, and a model fitted only on the synthetic data reaches 0.854 on the real records. Individual correlations are less reliable – most come out slightly weaker than in the original, but the association between number of pregnancies and age is considerably stronger (0.74 against 0.54). Use the data for teaching, examples and method demonstrations; any result computed on them describes the synthesis, not the Pima population, and must not be read as evidence about diabetes.

Author(s)

Matthias Templ [email protected] (synthetic version)

Source

Synthetic. Generated by Matthias Templ from PimaIndiansDiabetes2 (package mlbench) with the synvey package – sequential robust synthesis with a random-forest backend and the missingness pattern treated as a structural variable (synvey 0.1.0, R 4.5.2, seed 20260804):

SynthDiabetes <- synvey::robsynth(PimaIndiansDiabetes2,
                                  method      = "robust_rf",
                                  missingness = "structural",
                                  coarsen     = "age",
                                  target      = "sample",
                                  seed        = 20260804)$synth

This configuration was selected from a comparison of five synthesis packages under two designs for the missingness mechanism. Candidates reproducing any original record verbatim were excluded; the data set contains no such record. synvey is not on CRAN; it is available from https://github.com/matthias-da/synvey. For VIM the columns were renamed to the historical names (pregnant, glucose, pressure, triceps, insulin, mass, pedigree, age, diabetes became Pregnancies, Glucose, BloodPressure, SkinThickness, Insulin, BMI, DiabetesPedigreeFunction, Age, Outcome), counts are stored as integers and the Outcome levels neg/pos as no/yes. The original data are neither distributed with VIM nor needed to use this data set.

References

Smith, J.W., Everhart, J.E., Dickson, W.C., Knowler, W.C., & Johannes, R.S. (1988). Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the Symposium on Computer Applications and Medical Care (pp. 261–265). IEEE Computer Society Press. https://pmc.ncbi.nlm.nih.gov/articles/PMC2245318/ (The study behind the original data; the original data are not redistributed here.)

Examples

data(diabetes)
summary(diabetes)

# missingness is structured rather than scattered
aggr(diabetes)

Description

Various error measures evaluating the quality of imputations

Usage

evaluation(x, y, m, vartypes = "guess", where = NULL)

nrmse(x, y, m)

pfc(x, y, m)

msecov(x, y)

msecor(x, y)

Arguments

Details

This function has been mainly written for procudures that evaluate imputation or replacement of rounded zeros. The ni parameter can thus, e.g. be used for expressing the number of rounded zeros.

Value

the error measures value

Author(s)

Matthias Templ

References

M. Templ, A. Kowarik, P. Filzmoser (2011) Iterative stepwise regression imputation using standard and robust methods. Computational Statistics & Data Analysis, Vol. 55, pp. 2793-2806.

Examples

data(iris)
iris_orig <- iris_imp <- iris
iris_imp$Sepal.Length[sample(1:nrow(iris), 10)] <- NA
iris_imp$Sepal.Width[sample(1:nrow(iris), 10)] <- NA
iris_imp$Species[sample(1:nrow(iris), 10)] <- NA
m <- is.na(iris_imp)
iris_imp <- kNN(iris_imp, imp_var = FALSE)
evaluation(iris_orig, iris_imp, m = m, vartypes = c(rep("numeric", 4), "factor"))
msecov(iris_orig[, 1:4], iris_imp[, 1:4])

Description

The relative consumption of certain food items in European and Scandinavian countries.

Format

A data frame with 16 observations on the following 21 variables.

Details

The numbers represent the percentage of the population consuming that food type.

Source

https://openmv.net/info/food-consumption

Examples

data(food)
str(food)
aggr(food)

Description

Computes the average missing value gap of a vector.

Usage

gapMiss(x, what = mean)

Arguments

Details

The length of each sequence of missing values (gap) in a vector is calculated and the mean gap is reported

Value

The gap statistics

Author(s)

Matthias Templ based on a suggestion and draft from Huang Tian Yuan.

Examples

v <- rnorm(20)
v[3] <- NA
v[6:9] <- NA
v[13:17] <- NA
v
gapMiss(v)
gapMiss(v, what = median)
gapMiss(v, what = function(x) mean(x, trim = 0.1))
gapMiss(v, what = var)

Description

The function gowerD is used by kNN to compute the distances for numerical, factor ordered and semi-continous variables.

Usage

gowerD(
  data.x,
  data.y = data.x,
  weights = rep(1, ncol(data.x)),
  numerical = colnames(data.x),
  factors = vector(),
  orders = vector(),
  mixed = vector(),
  levOrders = vector(),
  mixed.constant = rep(0, length(mixed)),
  returnIndex = FALSE,
  nMin = 1L,
  returnMin = FALSE,
  methodStand = "range"
)

Arguments

Details

Numerical and semi-continuous (mixed) variables are standardised by their range (or IQR, see methodStand) before the distance is computed; for mixed variables the point mass (mixed.constant) is excluded from the range estimate and scaled along with the data.

Missing values in a distance variable are replaced by a sentinel (the pooled column maximum plus one) after standardisation. Two records both missing the same variable therefore have distance 0 on it, and a missing-vs-observed pair contributes up to about 2 (rather than a variable's nominal maximum of 1). This sentinel convention departs from Gower's (1971) omit-and-renormalise rule and biases nearest-neighbour selection towards records that share the same missingness pattern.

The distance computation is parallelised with OpenMP where R was built with OpenMP support. The number of threads is getOption("VIM.ncores"); if the option is unset, at most 2 threads are used while the package is checked (⁠R CMD check⁠ sets ⁠_R_CHECK_LIMIT_CORES_⁠; CRAN policy allows at most two cores) and OpenMP's default – typically all cores, or OMP_NUM_THREADS – otherwise. Results do not depend on the number of threads.

returnIndex=FALSE: a numerical matrix n x m with the computed distances returnIndex=TRUE: a named list with "ind" containing the requested indices and "mins" the computed distances

Examples

data(sleep)
# all variables used as numerical
gowerD(sleep)

# split in numerical an
gowerD(sleep, numerical = c("BodyWgt", "BrainWgt", "NonD", "Dream", "Sleep", "Span", "Gest"),
  orders = c("Pred","Exp","Danger"), levOrders = c(5,5,5))

# as before but only returning the index of the closest observation
gowerD(sleep, numerical = c("BodyWgt", "BrainWgt", "NonD", "Dream", "Sleep", "Span", "Gest"),
  orders = c("Pred","Exp","Danger"), levOrders = c(5,5,5), returnIndex = TRUE)

Description

Map with dots whose sizes correspond to the values in a certain variable. Observations with missing/imputed values in additional variables are highlighted.

Usage

growdotMiss(
  x,
  coords,
  map,
  pos = 1,
  delimiter = NULL,
  selection = c("any", "all"),
  log = FALSE,
  col = c("skyblue", "red", "skyblue4", "red4", "orange", "orange4"),
  border = par("bg"),
  alpha = NULL,
  scale = NULL,
  size = NULL,
  exp = c(0, 0.95, 0.05),
  col.map = grey(0.5),
  legend = TRUE,
  legtitle = "Legend",
  cex.legtitle = par("cex"),
  cex.legtext = par("cex"),
  ncircles = 6,
  ndigits = 1,
  interactive = TRUE,
  ...
)

Arguments

Details

The smallest dots correspond to the 10\ the 99\ defining the shape of the exponential function. Missings/imputed missings in the variable of interest will be drawn as rectangles.

If interactive=TRUE, detailed information for an observation can be printed on the console by clicking on the corresponding point. Clicking in a region that does not contain any points quits the interactive session.

Note

The function was renamed to growdotMiss in version 1.3. bubbleMiss is a (deprecated) wrapper for growdotMiss for back compatibility with older versions. However, due to extended functionality, some of the argument positions have changed.

The code is based on (removed from CRAN) bubbleFIN from package StatDA.

Author(s)

Andreas Alfons, Matthias Templ, Peter Filzmoser, Bernd Prantner

References

M. Templ, A. Alfons, P. Filzmoser (2012) Exploring incomplete data using visualization tools. Advances in Data Analysis and Classification, Online first. DOI: 10.1007/s11634-011-0102-y.

See Also

bgmap(), mapMiss(), colormapMiss()

Examples

data(chorizonDL, package = "VIM")
data(kola.background, package = "VIM")
coo <- chorizonDL[, c("XCOO", "YCOO")]
## for missing values
x <- chorizonDL[, c("Ca","As", "Bi")]
growdotMiss(x, coo, kola.background, border = "white")

## for imputed values
x_imp <- kNN(chorizonDL[,c("Ca","As","Bi" )])
growdotMiss(x_imp, coo, kola.background, delimiter = "_imp", border = "white")

Description

Histogram with highlighting of missing/imputed values in other variables by splitting each bin into two parts. Additionally, information about missing/imputed values in the variable of interest is shown on the right hand side.

Usage

histMiss(
  x,
  delimiter = NULL,
  pos = 1,
  selection = c("any", "all"),
  breaks = "Sturges",
  right = TRUE,
  col = c("skyblue", "red", "skyblue4", "red4", "orange", "orange4"),
  border = NULL,
  main = NULL,
  sub = NULL,
  xlab = NULL,
  ylab = NULL,
  axes = TRUE,
  only.miss = TRUE,
  miss.labels = axes,
  interactive = TRUE,
  ...
)

Arguments

Details

If more than one variable is supplied, the bins for the variable of interest will be split according to missingness/number of imputed missings in the additional variables.

If only.miss=TRUE, the missing/imputed values in the variable of interest are visualized by one bar on the right hand side. If additional variables are supplied, this bar is again split into two parts according to missingness/number of imputed missings in the additional variables.

Otherwise, a small barplot consisting of two bars is drawn on the right hand side. The first bar corresponds to observed values in the variable of interest and the second bar to missing/imputed values. Since these two bars are not on the same scale as the main barplot, a second y-axis is plotted on the right (if axes=TRUE). Each of the two bars are again split into two parts according to missingness/number of imputed missings in the additional variables. Note that this display does not make sense if only one variable is supplied, therefore only.miss is ignored in that case.

If interactive=TRUE, clicking in the left margin of the plot results in switching to the previous variable and clicking in the right margin results in switching to the next variable. Clicking anywhere else on the graphics device quits the interactive session. When switching to a categorical variable, a barplot is produced rather than a histogram.

Value

a list with the following components:

• breaks the breakpoints.

• counts the number of observations in each cell.

• missings the number of highlighted observations in each cell.

• mids the cell midpoints.

Note

Some of the argument names and positions have changed with version 1.3 due to extended functionality and for more consistency with other plot functions in VIM. For back compatibility, the arguments axisnames and names.miss can still be supplied to ...{} and are handled correctly. Nevertheless, they are deprecated and no longer documented. Use miss.labels instead.

Author(s)

Andreas Alfons, Bernd Prantner

References

M. Templ, A. Alfons, P. Filzmoser (2012) Exploring incomplete data using visualization tools. Advances in Data Analysis and Classification, Online first. DOI: 10.1007/s11634-011-0102-y.

See Also

spineMiss(), barMiss()

Other plotting functions: aggr(), barMiss(), marginmatrix(), marginplot(), matrixplot(), mosaicMiss(), pairsVIM(), parcoordMiss(), pbox(), scattJitt(), scattMiss(), scattmatrixMiss(), spineMiss()

Examples

data(tao, package = "VIM")
## for missing values
x <- tao[, c("Air.Temp", "Humidity")]
histMiss(x)
histMiss(x, only.miss = FALSE)

## for imputed values
x_IMPUTED <- kNN(tao[, c("Air.Temp", "Humidity")])
histMiss(x_IMPUTED, delimiter = "_imp")
histMiss(x_IMPUTED, delimiter = "_imp", only.miss = FALSE)

Description

Implementation of the popular Sequential, Random (within a domain) hot-deck algorithm for imputation.

Usage

hotdeck(
  data,
  variable = NULL,
  ord_var = NULL,
  domain_var = NULL,
  makeNA = NULL,
  NAcond = NULL,
  impNA = TRUE,
  donorcond = NULL,
  imp_var = TRUE,
  imp_suffix = "imp"
)

Arguments

Value

the imputed data set.

Note

If the sequential hotdeck does not lead to a suitable, a random donor in the group will be used.

Author(s)

Alexander Kowarik

References

A. Kowarik, M. Templ (2016) Imputation with R package VIM. Journal of Statistical Software, 74(7), 1-16.

See Also

Other imputation methods: impPCA(), imputeCellEM(), imputeCellIRMI(), imputeCellM(), imputeCellMCD(), imputeCellwise(), imputeRobust(), imputeRobustChain(), irmi(), kNN(), matchImpute(), medianSamp(), rangerImpute(), regressionImp(), sampleCat(), vimmi, vimpute(), xgboostImpute()

Examples

data(sleep)
sleepI <- hotdeck(sleep)
sleepI2 <- hotdeck(sleep,ord_var="BodyWgt",domain_var="Pred")

# Usage of donorcond in a simple example
sleepI3 <- hotdeck(
  sleep,
  variable = c("NonD", "Dream", "Sleep", "Span", "Gest"),
  ord_var = "BodyWgt", domain_var = "Pred",
  donorcond = list(">4", "<17", ">1.5", "%between%c(8,13)", ">5")
)

set.seed(132)
nRows <- 1e3
# Generate a data set with nRows rows and several variables
x <- data.frame(
  x = rnorm(nRows), y = rnorm(nRows),
  z = sample(LETTERS, nRows, replace = TRUE),
  d1 = sample(LETTERS[1:3], nRows, replace = TRUE),
  d2 = sample(LETTERS[1:2], nRows, replace = TRUE),
  o1 = rnorm(nRows), o2 = rnorm(nRows), o3 = rnorm(100)
)
origX <- x
x[sample(1:nRows,nRows/10), 1] <- NA
x[sample(1:nRows,nRows/10), 2] <- NA
x[sample(1:nRows,nRows/10), 3] <- NA
x[sample(1:nRows,nRows/10), 4] <- NA
xImp <- hotdeck(x,ord_var = c("o1", "o2", "o3"), domain_var = "d2")

Description

Greedy algorithm for EM-PCA including robust methods

Usage

impPCA(
  x,
  method = "classical",
  m = 1,
  eps = 0.5,
  k = ncol(x) - 1,
  maxit = 100,
  boot = FALSE,
  verbose = TRUE
)

Arguments

Value

the imputed data set. If boot = FALSE this is a data.frame. If boot = TRUE this is a list where each list element contains a data.frame.

Author(s)

Matthias Templ

References

Serneels, Sven and Verdonck, Tim (2008). Principal component analysis for data containing outliers and missing elements. Computational Statistics and Data Analysis, Elsevier, vol. 52(3), pages 1712-1727

See Also

Other imputation methods: hotdeck(), imputeCellEM(), imputeCellIRMI(), imputeCellM(), imputeCellMCD(), imputeCellwise(), imputeRobust(), imputeRobustChain(), irmi(), kNN(), matchImpute(), medianSamp(), rangerImpute(), regressionImp(), sampleCat(), vimmi, vimpute(), xgboostImpute()

Examples

data(Animals, package = "MASS")
Animals$brain[19] <- Animals$brain[19] + 0.01
Animals <- log(Animals)
colnames(Animals) <- c("log(body)", "log(brain)")
Animals_na <- Animals
probs <- abs(Animals$`log(body)`^2)
probs <- rep(0.5, nrow(Animals))
probs[c(6,16,26)] <- 0
set.seed(1234)
Animals_na[sample(1:nrow(Animals), 10, prob = probs), "log(brain)"] <- NA
w <- is.na(Animals_na$`log(brain)`)
impPCA(Animals_na)
impPCA(Animals_na, method = "mcd")
impPCA(Animals_na, boot = TRUE, m = 10)
impPCA(Animals_na, method = "mcd", boot = TRUE)[[1]]
plot(`log(brain)` ~ `log(body)`, data = Animals, type = "n", ylab = "", xlab="")
mtext(text = "impPCA robust", side = 3)
points(Animals$`log(body)`[!w], Animals$`log(brain)`[!w])
points(Animals$`log(body)`[w], Animals$`log(brain)`[w], col = "grey", pch = 17)
imputed <- impPCA(Animals_na, method = "mcd", boot = TRUE)[[1]]
colnames(imputed) <- c("log(body)", "log(brain)")
points(imputed$`log(body)`[w], imputed$`log(brain)`[w], col = "red", pch = 20, cex = 1.4)
segments(x0 = Animals$`log(body)`[w], x1 = imputed$`log(body)`[w], y0 = Animals$`log(brain)`[w],
y1 = imputed$`log(brain)`[w], lty = 2, col = "grey")
legend("topleft", legend = c("non-missings", "set to missing", "imputed values"),
pch = c(1,17,20), col = c("black","grey","red"), cex = 0.7)
mape <- round(100* 1/sum(is.na(Animals_na$`log(brain)`)) * sum(abs((Animals$`log(brain)` -
imputed$`log(brain)`) / Animals$`log(brain)`)), 2)
s2 <- var(Animals$`log(brain)`)
nrmse <- round(sqrt(1/sum(is.na(Animals_na$`log(brain)`)) * sum(abs((Animals$`log(brain)` -
imputed$`log(brain)`) / s2))), 2)
text(x = 8, y = 1.5, labels = paste("MAPE =", mape))
text(x = 8, y = 0.5, labels = paste("NRMSE =", nrmse))

Description

EM algorithm with latent contamination indicators that jointly estimates clean distribution parameters and identifies cellwise outliers. Each continuous cell has a posterior probability of being clean vs contaminated. The clean distribution is modeled as multivariate normal for continuous variables, with categorical variables handled via conditional multinomial logistic regression.

Usage

imputeCellEM(
  data,
  maxit_em = 100,
  eps_em = 0.005,
  gamma_init = 3,
  eps_init = 0.1,
  uncert = "conditional",
  conditioning = "weighted",
  trust_min = 0.5,
  trace = FALSE
)

Arguments

Details

The algorithm proceeds as follows:

1. Initialization. Missing values are filled by initialise (medians for continuous, modes for categorical). Initial location and scale are estimated robustly (median and MAD). Cell weights are initialized to 1.

2. E-step (for each continuous variable $j$, each observation $i$):

   • Compute the conditional mean and variance of $x_{ij}$ given the other continuous variables, using the current $\mu$ and $\Sigma$.

   • For observed cells: compute the posterior probability that the cell is clean vs contaminated, yielding cell weight $w_{ij}$.

   • For missing cells: impute from the conditional distribution (with optional PMM).

3. M-step:

   • Update $\mu$: cell-weighted mean.

   • Update $\Sigma$: pairwise cell-weighted covariance using $w_{ij} \cdot w_{ik}$ (not row-level min).

   • Update contamination rates: $\varepsilon_j = 1 - \mathrm{mean}(w_{ij})$ over observed cells.

   • Update contamination scale $\gamma_j$ from weighted variance of contaminated cells.

   • For categorical variables: fit weighted multinomial logistic with row weights derived from continuous cell weights.

4. Convergence. Check relative change in estimated parameters ($\mu$, $\Sigma$); stop when below eps_em or after maxit_em iterations. The observed-data log-likelihood (sum of per-variable conditional mixture log-likelihoods) is tracked for diagnostics.

This method differs from cellGMM (Zaccaria et al., 2025) in using a single clean component rather than a mixture of clean clusters, and in supporting mixed continuous + categorical data.

The pseudo_loglik component is a composite (pseudo) log-likelihood: the sum of per-variable conditional mixture log-likelihoods, not the proper observed-data joint log-likelihood. It is useful for monitoring convergence but should not be compared across models or used for model selection criteria such as AIC/BIC.

This implementation is an ECM (Expectation Conditional Maximization) variant rather than a pure EM algorithm, because the conditional variance in the E-step is computed using the updated Sigma from the current M-step rather than the Sigma from the previous iteration. As a result, strict log-likelihood monotonicity is not guaranteed, but is observed empirically in practice.

Value

A list with components:

Note

Model uncertainty via bootstrap (Rubin's combining rules for multiple imputation) is not yet implemented. The current version provides single imputation with stochastic uncertainty (PMM or residual draw). For valid multiple imputation, call the function repeatedly with different seeds and combine using Rubin's rules.

Author(s)

Matthias Templ

References

A.P. Dempster, N.M. Laird, D.B. Rubin (1977) Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society: Series B, 39(1), 1–38.

C.F.J. Wu (1983) On the Convergence Properties of the EM Algorithm. The Annals of Statistics, 11(1), 95–103.

J. Raymaekers, P.J. Rousseeuw (2024) The cellwise minimum covariance determinant estimator. Journal of the American Statistical Association, 119(548), 2610–2621.

G. Zaccaria, L.A. Garcia-Escudero, F. Greselin, A. Mayo-Iscar (2025) Cellwise outlier detection in heterogeneous populations. Technometrics, 67(4), 643–654.

M. Templ, A. Kowarik, P. Filzmoser (2011) Iterative stepwise regression imputation using standard and robust methods. Computational Statistics & Data Analysis, Vol. 55, pp. 2793–2806.

See Also

imputeCellIRMI, imputeCellM, initialise, irmi

Other imputation methods: hotdeck(), impPCA(), imputeCellIRMI(), imputeCellM(), imputeCellMCD(), imputeCellwise(), imputeRobust(), imputeRobustChain(), irmi(), kNN(), matchImpute(), medianSamp(), rangerImpute(), regressionImp(), sampleCat(), vimmi, vimpute(), xgboostImpute()

Examples

data(sleep, package = "VIM")
result <- imputeCellEM(sleep)
head(result$data_imputed)

# Inspect estimated contamination rates
result$epsilon

# Cell weight matrix (1 = clean, low = likely contaminated)
image(result$cellweights, main = "Cell weights")

# Log-likelihood trace
plot(result$pseudo_loglik, type = "b", xlab = "Iteration",
     ylab = "Pseudo log-likelihood")

# With predictive mean matching for imputation
result2 <- imputeCellEM(sleep, uncert = "pmm", trace = TRUE)

# Mixed data example
data(testdata)
result3 <- imputeCellEM(testdata$wna)

Description

Estimates the mean structure $B$, scatter $\Sigma$ and cell weights $W$ of the model $x_i | u_i \sim N(B' u_i, \Sigma)$, where $u_i$ is a design row built from the categorical variables. Detection uses the conditional residual of a cell given the other continuous cells in its row and the row's categorical pattern.

Usage

imputeCellGLoc(
  data,
  design = ~.,
  weights = c("soft", "binary"),
  maxit = 200,
  eps = 0.005,
  alpha = 0.75,
  psi_c = 4.685,
  peer_w_min = 0.5,
  peer_band = .gloc_peer_band,
  damp = NULL,
  cw_crit = 1e-08,
  trace = FALSE,
  start = c("robust", "classical"),
  categorical = c("em", "level")
)

Arguments

Details

With design = ~ 1 the estimator reduces to the published continuous-only estimators: to the cellwise MCD of Raymaekers and Rousseeuw (2024) with weights = "binary", and to the cellwise weighted maximum likelihood estimator of Rousseeuw (2026) with weights = "soft".

The soft corner corrects the weighted scatter for its downweighting. Without a correction the bisquare downweighting deflates the scatter by about 21% at the default tuning, which would inflate the standardised residuals and make the estimator over-flag. The correction is per column and depends on $\Sigma$, not a single constant: the weights act on conditional residuals, so they shrink only the unpredictable part of each cell, and they leave its covariances with the other cells unbiased to first order. Since 7.5.1 the correction therefore adds the missing part of each conditional variance to the diagonal and keeps the off-diagonals; see .gloc_correct_scatter. It vanishes at psi_c = Inf. It is a first-order correction: on clean data with five variables at correlation 0.7 (n = 10000) it returns conditional variances 4.5% too large and a block correlation of 0.682, where the symmetric rescaling used until 7.5.0 returned conditional variances 13% too small and a block correlation of 0.742.

The estimator is the triple $(B, \Sigma, W)$ and iteration stops only when all three have settled: $\max|U(B - B_{old})| / \sqrt{\max \mathrm{diag}(\Sigma)} < eps$, $\max|\Sigma - \Sigma_{old}| / \max \mathrm{diag}(\Sigma) < eps$ and $\max|W - W_{old}| / damp < eps$. The first term is a change in the fitted values measured in units of the scatter, not a change in a coefficient measured against a location, so the criterion is invariant to shifting the data; the second is relative, so it is invariant to rescaling it; the third is the weight map's fixed-point residual, relaxation divided out. converged = TRUE therefore means that nothing this function returns is still moving, which is the only reading of convergence a detection method can defend: $W$ is not a nuisance quantity here, it is the flag set the caller uses.

The scatter condition is a formal guard rather than an active one. Across every fit measured so far it has never been the binding constraint: at the iteration where the other two conditions are first met it stands at most a seventh of its own tolerance. It is there because $\Sigma$ is returned and was the one returned quantity the rule did not test, and because it costs nothing; it is not there because it was observed to catch anything.

The soft corner's weight update is relaxed (see .gloc_damp) to keep the fixed-point iteration inside its contraction radius, and the factor is adapted rather than fixed; see damp. Relaxation alone does not make the criterion attainable: conditioning on a peer only when its weight exceeds a threshold makes the weight map discontinuous, and a discontinuous self-map of $[0,1]^{n \times p}$ need not have a fixed point at all, so before 7.4.0 the iteration could be asked to reach a state that did not exist – and in the design = ~ . arm it usually was. The peer band (see .gloc_peer_band) removes that, and the two work together rather than one of them doing the work: over 260 fits of the categorical-mean-structure arm, converged is 131 under the 7.3.1 schedule and hard cut, 181 with the relaxation floor at 0.5 and the scatter condition but the cut still hard, and 245 with the band as well. About 40% of the gain is the floor, the rest the band.

Convergence is not universal and should not be assumed. On that 260-fit grid it is 245, or 94%, with the failures concentrated at correlations of 0.6 and above combined with 20% of cells contaminated.

All convergence figures in this section were measured before 7.4.1, with the classical cold start (start = "classical"); the robust start has not been measured on that grid.

When the design does not use a factor's own contrasts attribute – the factor has an unused level, or, under categorical = "level", its missing values become a level of their own – the fit says so once for that factor (since 7.5.1; see .gloc_contrast_lost), and base R's own "contrasts dropped" warning is muffled at every design site here; the fitted means, $\Sigma$, $W$ and the imputations are the same under any full-rank coding, so only B's rows change.

Continuous columns that are integer in data stay integer in $imputed; their conditional expectations are rounded.

A missing continuous cell is imputed by its conditional expectation given the unflagged cells in its row, selected by the same peer rule detection uses (peer_w_min with the peer band), in both weight corners. Until 7.4.0 the imputation conditioned on every observed cell, flagged ones included, so a grossly contaminated cell was carried into the imputation of its row-mates; B, Sigma and W were not affected. See .gloc_impute.

Per-level detection (since 7.5.1). Under categorical = "em" in the soft corner, a row with a missing categorical cell is judged once per candidate level, or per level combination when several cells are missing. Each candidate carries its own cell weights: its cells are compared with its own fitted mean and conditioned on the cells that are clean under it. The E-step scores a level by the cellwise-penalised likelihood of the binary corner, the Gaussian log-density of the cells retained under the level minus $\lambda_j / 2$ for each cell flagged under it, with $\lambda_j = \chi^2_{1;0.99} + \log 2\pi + \log c_j$, $c_j = 1 / (\Sigma^{-1})_{jj}$ from the current scatter, and the peer band in between (see .gloc_cat_score). B and Sigma are updated with each candidate's weights times its posterior probability, and W reports the posterior mixture of the candidates' weights, which cat_weights returns. Why: until 7.5.0 detection in such a row ran at its expected design row, where a cell that decides the level looks outlying whenever the true level is far from the prior mean. That cell was flagged, the E-step dropped it, and the posterior could not move, a self-locking fixed point. On the datasets of the categorical arms of the 7.5.0 simulation at eps = 0 (n = 200, six continuous and six categorical variables, 10 replicates per pattern), the decisive cell was flagged in 17.6-19.8% of the rows missing the factor, and the imputed level was right in 0.809-0.861 of those rows, against 0.965-0.986 for a classifier that uses the fit's own $B$ and $\Sigma$, the true prior probabilities and every observed continuous cell. A contaminated cell is flagged under every level and pays the same penalty under each, so it still does not steer the level. The binary corner keeps one weight row per observation, since cellWise::cellMCD returns one: its detection stays at the expected design row, and the lock can occur there. The soft corner is the default. The missing continuous cells of such a row are imputed by the posterior mixture of its candidates' conditional expectations, each taken at that candidate's design row and conditioned on the cells that are clean under it (.gloc_impute_mix); see imputed under Value. (Corrected: until 7.5.0 they were imputed once, at the expected design row with the row's weights, which was the exact mixture expectation only while every level shared one weight row.)

Value

a list with B, Sigma, W, U, imputed, converged, iterations, criterion, cat_posterior, cat_prob_observed, cat_multi_missing, cat_priors, em_starts and cat_weights.

W is the $n x p$ matrix of cell weights of the continuous variables, 0 on missing cells. Under per-level detection ("em", soft corner, a missing categorical cell; see Details) a row with a missing categorical cell reports the posterior mixture $W_i = \sum_k r_k W_k$ of its candidates' weights under the final E-step; a row without one, and a row above the combination cap, reports its own.

imputed is data with its missing cells filled: a continuous cell by its conditional expectation given the unflagged cells of its row, a categorical cell (under "em") by its posterior mode. Under per-level detection a missing continuous cell of a row that also misses a categorical cell is imputed instead by the posterior mixture $\sum_k r_k E[x_{mis} | \textrm{cells clean under } k, u_k]$ over that row's candidates: each candidate contributes the conditional expectation at its own design row, conditioned on the cells that are clean under it, and the candidates are mixed by their posterior probabilities $r_k$. That is the exact posterior mean of the cell. Every other row, a row above the combination cap included, is imputed at its expected design row with the returned W, as in 7.5.0.

cat_weights is NULL unless per-level detection ran. Then it is a list with one entry per candidate of the rows below the combination cap that miss a categorical cell, in the order of the candidate table: row, the data row (integer); levels, a data frame of the candidate's values of every categorical variable, observed values included – its columns are the candidate table's factors, on the level sets the fit observed, so a character or logical column of data comes back as a factor; prob, the candidate's posterior probability, the product of the marginal posteriors when several cells are missing; and W, the candidates' cell weights, a matrix with one row per candidate and one column per continuous variable. W[i, ] of the fit equals colSums(prob * W) over row i's entries.

Under categorical = "em", U holds each row's expected design row, so U %*% B are the fitted means. cat_posterior is a list with one matrix per categorical variable that has a missing cell: its rows are those cells' rows (row names are row indices), its columns the levels, its entries posterior probabilities. imputed holds the posterior mode (ties: the first level). For multiple imputation, draw the level from these probabilities first and the continuous cells given it. cat_prob_observed is an $n x k$ matrix, NA where the cell is missing: for every observed categorical cell, the posterior probability of its own level computed as if the cell were missing, without a refit. A small value points at a miscoded cell; the estimation does not use it. It is NA in every column of a row above the combination cap as well, since such a row has no pseudo-rows to compute it from. Under per-level detection it is not on the same scale as cat_posterior: the diagnostic keeps 7.5.0's un-penalised Gaussian density, evaluated at the returned (mixture) W, while the fit's own posteriors now come from the cellwise-penalised per-candidate score (.gloc_cat_score). Per-level scoring of observed cells is deliberately deferred, so compare a value of this matrix with other values of it and with the same fit's flagging rate, not with an entry of cat_posterior. cat_multi_missing is the share of rows with two or more missing categorical cells. cat_priors holds the final multinomial prior models, so that level posteriors for other rows can be computed under this fit (for example in bootstrap multiple imputation). All four are NULL under categorical = "level", and under "em" when the data have no categorical column.

em_starts is NULL unless the fit ran from two starts (see categorical). Then it is a list: chosen, "complete" for the first start or "na_level" for the second; objective, converged and iterations, each a vector named complete and na_level; and lambda, the penalty per continuous variable in the objective. When the second start failed, its three entries are NA and error holds its error message. iterations, converged and criterion of the fit are those of the chosen run.

criterion is the named vector (means, scatter, weights, scatter_spread, categorical): the three stopping residuals as of the last iteration, compared against eps, plus the elementwise spread of $\Sigma$ over the last .gloc_stall_iters iterations relative to its largest variance. scatter_spread is reported only when converged is FALSE, and is NA otherwise: on a converged fit the window still holds the last steps of the approach, so a value there would say nothing about stability. When the fit did not converge it says how much the returned scatter depends on where maxit happened to stop – a settled cycle reports its amplitude, a run still drifting reports the drift – and a caller can test it instead of parsing a warning string. The fifth entry, categorical, is the largest change of a categorical posterior probability in the last iteration; it is 0 without the EM, NA when no iteration ran, and part of the stopping rule with the EM. Under per-level detection weights is taken over the candidates: the largest change of a candidate's cell weight times the candidate's posterior probability (1 for a row that is a single candidate or lies above the combination cap), divided by the relaxation factor. The adaptive relaxation and the stall detector use the same number, so a candidate the posterior has all but ruled out cannot hold up convergence.

It does not tell those two apart, and it is not an error estimate. On 10 non-converged weights = "binary" fits it was positive for all 10, while raising maxit left 8 of them bit-identical and moved 2 (by 0.008 and 0.096 relative, the latter matching its reported spread exactly). That is the intended behaviour – all 10 return a scatter that depends on the stopping point – but the rank correlation with what raising maxit actually does is only 0.16, so read it as "this answer is not settled", never as "it would move by this much". Every entry is NA at maxit = 0.

References

Raymaekers, J. and Rousseeuw, P. J. (2024). The cellwise minimum covariance determinant estimator. JASA 119(548), 2610-2621. doi:10.1080/01621459.2023.2267777

Rousseeuw, P. J. (2026). Analyzing cellwise weighted data. Econometrics and Statistics 38, 31-41. doi:10.1016/j.ecosta.2023.01.007

Description

Extends IRMI (Templ, Kowarik, and Filzmoser, 2011) with cellwise contamination handling. Each conditional regression uses a cell-weighted IRWLS engine where per-cell weights in the design matrix downweight contaminated cells without discarding entire observations.

Usage

imputeCellIRMI(
  data,
  method = "tukey",
  alpha = NULL,
  maxit = 100,
  maxit_irwls = 50,
  eps = 0.005,
  eps_irwls = 1e-06,
  uncert = "pmm",
  weight_update = "multivariate",
  init_weights = "ddc",
  hard_threshold = 0.5,
  trace = FALSE
)

Arguments

Details

The algorithm works iteratively: in each outer iteration, every variable with missing values is used as response in a conditional regression on all remaining variables. For continuous responses, the custom cellIRWLS() engine fits a weighted regression where each cell in the design matrix receives its own weight reflecting potential cellwise contamination. For categorical responses, a weighted multinomial model is used. After each regression, cell weights for the response variable are updated from the residuals.

The algorithm proceeds as follows:

1. Missing values are initialised using initialise.

2. Initial cell weights are computed with cellWeights() on all continuous variables in the initialised data.

3. Outer loop (up to maxit iterations):

   • For each variable $j$ with missing values:

     • Form predictor matrix $X$ (all other variables) and response $y$ (variable $j$).

     • If $j$ is continuous: fit cellIRWLS(X, y, w_cell, w_response) and impute missing values in $j$ using the fitted model plus uncertainty.

     • If $j$ is categorical: fit nnet::multinom() with row weights derived from the cell weight matrix and impute by sampling from predicted probabilities.

     • Update cell weights for $j$ from residuals via cellWeightsFromResiduals().

   • Check convergence: relative change in imputed values falls below eps.

Value

A list with components:

Author(s)

Matthias Templ

References

Templ, M., Kowarik, A. and Filzmoser, P. (2011). Iterative stepwise regression imputation using standard and robust methods. Computational Statistics & Data Analysis, 55(10), 2793–2806.

See Also

imputeCellM, imputeCellEM, initialise, irmi

Other imputation methods: hotdeck(), impPCA(), imputeCellEM(), imputeCellM(), imputeCellMCD(), imputeCellwise(), imputeRobust(), imputeRobustChain(), irmi(), kNN(), matchImpute(), medianSamp(), rangerImpute(), regressionImp(), sampleCat(), vimmi, vimpute(), xgboostImpute()

Examples

data(sleep, package = "VIM")
result <- imputeCellIRMI(sleep)
head(result$data_imputed)
image(result$cellweights, main = "Cell weights")

# With Huber weights (less aggressive downweighting)
result2 <- imputeCellIRMI(sleep, method = "huber", trace = TRUE)

# Mixed data example
data(testdata)
result3 <- imputeCellIRMI(testdata$wna)

Description

Impute missing values using a cell-weighted M-estimation approach. Each cell in the predictor matrix receives its own weight reflecting potential cellwise contamination, so that contaminated predictor cells are downweighted without discarding entire observations.

Usage

imputeCellM(
  formula,
  data,
  method = "tukey",
  alpha = NULL,
  maxit_irwls = 50,
  eps_irwls = 1e-06,
  uncert = "pmm",
  value_back = "all",
  maxit = 10,
  eps = 0.005,
  trace = FALSE
)

Arguments

Details

The function has two interfaces: with a model formula, a single response variable is imputed from the specified predictors; with a data.frame (or matrix) as first argument, all variables with missing values are imputed by chained equations, i.e. each such variable is regressed on all remaining variables and the sweeps are iterated until the imputed values stabilise.

The formula interface is a lightweight single-response alternative to imputeCellIRMI. It fits one cell-weighted IRWLS regression using cellIRWLS() and imputes the missing values in the response variable. This is appropriate when only one variable needs imputation and a specific model formula is desired.

The data.frame interface runs the same per-variable machinery as a chained-equations algorithm: missing values are initialised (median/mode), then each variable with missing values in turn is used as response in a formula containing all remaining variables. Sweeps use deterministic predictions and are iterated until the relative change of the imputed values falls below eps (or maxit is reached); the requested uncert step is applied once after convergence.

For categorical response variables, a weighted multinomial model via multinom is fitted instead. Categorical predictors are not subject to the cellwise contamination model (their cell weights are always 1).

Value

If value_back = "ymiss", a named vector of imputed values (for rows that were originally missing) is returned. Otherwise, a list with components:

data_imputed

the imputed data.frame (same structure as input)

cellweights

n x p matrix of final cell weights (1 = clean, 0 = fully downweighted). Categorical columns always have weight 1.

converged

logical; always TRUE for single-formula imputation, convergence of the sweeps for the data.frame interface

iterations

integer; always 1L for single-formula imputation, the number of sweeps for the data.frame interface

Note

Model uncertainty via bootstrap (Rubin's combining rules for multiple imputation) is not yet implemented. The current version provides single imputation with stochastic uncertainty (PMM or residual draw). For valid multiple imputation, call the function repeatedly with different seeds and combine using Rubin's rules.

Author(s)

Matthias Templ

References

M. Templ, A. Kowarik, P. Filzmoser (2011) Iterative stepwise regression imputation using standard and robust methods. Computational Statistics & Data Analysis, Vol. 55, pp. 2793-2806.

See Also

imputeCellIRMI, imputeRobust

Other imputation methods: hotdeck(), impPCA(), imputeCellEM(), imputeCellIRMI(), imputeCellMCD(), imputeCellwise(), imputeRobust(), imputeRobustChain(), irmi(), kNN(), matchImpute(), medianSamp(), rangerImpute(), regressionImp(), sampleCat(), vimmi, vimpute(), xgboostImpute()

Examples

data(sleep, package = "VIM")
# Impute Dream using BodyWgt and BrainWgt as predictors
result <- imputeCellM(Dream ~ BodyWgt + BrainWgt, data = sleep)
head(result)

# Return only imputed values
impvals <- imputeCellM(Dream ~ BodyWgt + BrainWgt, data = sleep,
                       value_back = "ymiss")

# Huber weights (less aggressive downweighting)
result2 <- imputeCellM(Dream ~ BodyWgt + BrainWgt, data = sleep,
                       method = "huber")

# Chained-equations interface: impute all variables with missings
result3 <- imputeCellM(sleep)
head(result3$data_imputed)

Description

Extends the cellwise MCD approach (Raymaekers & Rousseeuw 2024) to mixed continuous + categorical data. Uses MCD for robust covariance estimation of the continuous block, computes cellwise weights from conditional residuals, then imputes missing values via conditional expectations (continuous) and weighted multinomial regression (categorical). Iterates until convergence.

Usage

imputeCellMCD(
  data,
  maxit = 50,
  eps = 0.005,
  method = "tukey",
  alpha = NULL,
  mcd_alpha = 0.75,
  hard_threshold = 0.5,
  mcd_observed = "all",
  init_method = "median",
  uncert = "conditional",
  m = 1L,
  boot = FALSE,
  trace = FALSE
)

Arguments

Value

A list with components:

Note

Multiple imputation with proper parameter-uncertainty propagation is not yet implemented for this method (see boot). m > 1 returns repeated imputations whose between-imputation variability comes only from the stochastic uncertainty step (uncert) and estimator randomness, so downstream Rubin pooling understates total variance. Treat the current m > 1 output as approximate.

Author(s)

Matthias Templ

References

Raymaekers, J. and Rousseeuw, P.J. (2024). The cellwise minimum covariance determinant estimator. Journal of the American Statistical Association, 119(548), 2610–2621.

See Also

imputeCellIRMI, imputeCellM, imputeCellEM

Other imputation methods: hotdeck(), impPCA(), imputeCellEM(), imputeCellIRMI(), imputeCellM(), imputeCellwise(), imputeRobust(), imputeRobustChain(), irmi(), kNN(), matchImpute(), medianSamp(), rangerImpute(), regressionImp(), sampleCat(), vimmi, vimpute(), xgboostImpute()

Examples

## Not run: 
data(sleep, package = "VIM")
result <- imputeCellMCD(sleep)
head(result$data_imputed)

# Inspect cell weights
image(result$cellweights, main = "Cell weights")

# With pairwise robust covariance
result2 <- imputeCellMCD(sleep, mcd_observed = "pairwise", trace = TRUE)

## End(Not run)

Description

Uses a cell-weighted MM-estimator for each variable regression: S-step on unweighted data (high breakdown), M-step with cell weights (fine-grained downweighting). This preserves the S-estimator's breakdown guarantee while incorporating cellwise information.

Usage

imputeCellMM(
  data,
  maxit = 50,
  eps = 0.005,
  alpha_weight = 4.685,
  init_weights = "mcd",
  uncert = "pmm",
  trace = FALSE
)

Arguments

Value

list with data_imputed, cellweights, converged, iterations

Author(s)

Matthias Templ

Description

IRMI-style imputation using cellwise-robust regression as the inner engine. Three engines are available: CRM (Filzmoser et al. 2020), Shooting S (Öllerer et al. 2016), and a cellwise-weighted MM hybrid.

Usage

imputeCellReg(
  data,
  engine = "crm",
  maxit = 50,
  eps = 0.005,
  uncert = "pmm",
  trace = FALSE
)

Arguments

Details

The function cycles through all variables with missing values (IRMI framework), fitting a cellwise-robust regression of each variable on all others. The engine argument selects the regression method:

"crm"

CRM (Cellwise Robust M-regression) from the crmReg package. Uses SPADIMO for cellwise outlier detection within each regression. Requires crmReg.

"cellwise-mm"

Hybrid: compute cell weights via MCD conditional residuals, then fit MM-estimation (lmrob) with row weights derived from cell weights. The MM-estimator provides high breakdown point.

"shooting-s"

Shooting S-estimator (Öllerer et al. 2016). Iterates between cellwise detection and S-estimation. Implemented from the published algorithm.

Categorical variables are imputed via weighted multinomial logistic regression, with row weights derived from the continuous cell weights.

Value

A list with components:

Author(s)

Matthias Templ

References

P. Filzmoser, S. Höppner, I. Ortner, S. Serneels, S. Van Aelst (2020) Cellwise robust M regression. Computational Statistics and Data Analysis, 147, 106944.

V. Öllerer, A. Alfons, C. Croux (2016) The shooting S-estimator for robust regression. Computational Statistics, 31(3), 829–844.

Description

Convenience wrapper that dispatches to one of the three cellwise-robust imputation methods: imputeCellIRMI, imputeCellM, or imputeCellEM.

Usage

imputeCellwise(data, method = c("cellIRMI", "cellM", "cellEM"), ...)

Arguments

Value

The return value of the dispatched function. See the documentation of the individual methods for details.

Note

Model uncertainty via bootstrap (Rubin's combining rules for multiple imputation) is not yet implemented. The current version provides single imputation with stochastic uncertainty (PMM or residual draw). For valid multiple imputation, call the function repeatedly with different seeds and combine using Rubin's rules.

Author(s)

Matthias Templ

See Also

imputeCellIRMI, imputeCellM, imputeCellEM

Other imputation methods: hotdeck(), impPCA(), imputeCellEM(), imputeCellIRMI(), imputeCellM(), imputeCellMCD(), imputeRobust(), imputeRobustChain(), irmi(), kNN(), matchImpute(), medianSamp(), rangerImpute(), regressionImp(), sampleCat(), vimmi, vimpute(), xgboostImpute()

Examples

data(sleep, package = "VIM")
result <- imputeCellwise(sleep, method = "cellIRMI")
head(result$data_imputed)

Description

Multiple imputation using classical and robust methods accounting for model and imputation uncertainty.

Usage

imputeRobust(
  form,
  data,
  boot = TRUE,
  robustboot = "stratified",
  method = "MM",
  takeAll = TRUE,
  alpha = 0.75,
  uncert = "pmm",
  family = "Gaussian",
  value_back = "all"
)

Arguments

Details

Complex formulas can be provided for each variable in your data set.

Value

Imputed data set.

See Also

initialise lmrob gam pdist

Other imputation methods: hotdeck(), impPCA(), imputeCellEM(), imputeCellIRMI(), imputeCellM(), imputeCellMCD(), imputeCellwise(), imputeRobustChain(), irmi(), kNN(), matchImpute(), medianSamp(), rangerImpute(), regressionImp(), sampleCat(), vimmi, vimpute(), xgboostImpute()

Examples

## Not run: 
if(interactive()){
 #EXAMPLE1
 }

## End(Not run)

Description

FUNCTION_DESCRIPTION

Usage

imputeRobustChain(
  formulas = vector("list", ncol(data)),
  data,
  boot = TRUE,
  robustboot = TRUE,
  method = "lts",
  multinom.method = "multinom",
  takeAll = TRUE,
  eps = 0.5,
  maxit = 4,
  alpha = 0.5,
  uncert = "pmm",
  familiy = "Gaussian",
  value_back = "matrix",
  trace = FALSE
)

Arguments

Details

DETAILS

Value

OUTPUT_DESCRIPTION

See Also

initialise outCoDa

Other imputation methods: hotdeck(), impPCA(), imputeCellEM(), imputeCellIRMI(), imputeCellM(), imputeCellMCD(), imputeCellwise(), imputeRobust(), irmi(), kNN(), matchImpute(), medianSamp(), rangerImpute(), regressionImp(), sampleCat(), vimmi, vimpute(), xgboostImpute()

Examples

## Not run: 
if(interactive()){
 #EXAMPLE1
 }

## End(Not run)

Description

Rough estimation of missing values in a vector according to its type.

Usage

initialise(x, mixed, method = "kNN", mixed.constant = NULL)

Arguments

Details

Missing values are imputed with the mean for vectors of class "numeric", with the median for vectors of class "integer", and with the mode for vectors of class "factor". Hence, x should be prepared in the following way: assign class "numeric" to numeric vectors, assign class "integer" to ordinal vectors, and assign class "factor" to nominal or binary vectors.

Value

the initialized vector.

Note

The function is used internally by some imputation algorithms.

Author(s)

Matthias Templ, modifications by Andreas Alfons

Description

In each step of the iteration, one variable is used as a response variable and the remaining variables serve as the regressors.

Usage

irmi(
  x,
  eps = 5,
  maxit = 100,
  mixed = NULL,
  mixed.constant = NULL,
  count = NULL,
  step = FALSE,
  robust = FALSE,
  takeAll = TRUE,
  noise = TRUE,
  noise.factor = 1,
  force = FALSE,
  robMethod = "lmrob",
  force.mixed = TRUE,
  mi = 1,
  addMixedFactors = FALSE,
  trace = FALSE,
  init.method = "kNN",
  modelFormulas = NULL,
  multinom.method = "multinom",
  imp_var = TRUE,
  imp_suffix = "imp"
)

Arguments

Details

The method works sequentially and iterative. The method can deal with a mixture of continuous, semi-continuous, ordinal and nominal variables including outliers.

A full description of the method can be found in the mentioned reference.

Value

the imputed data set.

Note

Deprecated: irmi() is considered legacy functionality. For new projects, please use vimpute() instead, which provides a more modern, flexible and robust imputation framework.

Author(s)

Matthias Templ, Alexander Kowarik

References

M. Templ, A. Kowarik, P. Filzmoser (2011) Iterative stepwise regression imputation using standard and robust methods. Computational Statistics & Data Analysis, Vol. 55, pp. 2793-2806.

A. Kowarik, M. Templ (2016) Imputation with R package VIM. Journal of Statistical Software, 74(7), 1-16.

M. Templ (2023) Visualization and Imputation of Missing Values. Springer Publishing. Series in Computational Statistics. Cham. Switzerland. 463 pages. DOI: 10.1007/978-3-031-30073-8

See Also

Other imputation methods: hotdeck(), impPCA(), imputeCellEM(), imputeCellIRMI(), imputeCellM(), imputeCellMCD(), imputeCellwise(), imputeRobust(), imputeRobustChain(), kNN(), matchImpute(), medianSamp(), rangerImpute(), regressionImp(), sampleCat(), vimmi, vimpute(), xgboostImpute()

Examples

data(sleep)
irmi(sleep)

data(testdata)
imp_testdata1 <- irmi(testdata$wna, mixed = testdata$mixed)

# mixed.constant != 0 (-10)
testdata$wna$m1[testdata$wna$m1 == 0] <- -10
testdata$wna$m2 <- log(testdata$wna$m2 + 0.001)
imp_testdata2 <- irmi(
  testdata$wna,
  mixed = testdata$mixed,
  mixed.constant = c(-10,log(0.001))
)
imp_testdata2$m2 <- exp(imp_testdata2$m2) - 0.001

#example with fixed formulas for the variables with missing
form = list(
  NonD  = c("BodyWgt", "BrainWgt"),
  Dream = c("BodyWgt", "BrainWgt"),
  Sleep = c("BrainWgt"           ),
  Span  = c("BodyWgt"            ),
  Gest  = c("BodyWgt", "BrainWgt")
)
irmi(sleep, modelFormulas = form, trace = TRUE)

# Example with ordered variable
td <- testdata$wna
td$c1 <- as.ordered(td$c1)
irmi(td)

Description

k-Nearest Neighbour Imputation based on a variation of the Gower Distance for numerical, categorical, ordered and semi-continous variables.

Usage

kNN(
  data,
  variable = colnames(data),
  k = 5,
  dist_var = colnames(data),
  weights = NULL,
  numFun = median,
  catFun = maxCat,
  makeNA = NULL,
  NAcond = NULL,
  impNA = TRUE,
  donorcond = NULL,
  mixed = vector(),
  mixed.constant = NULL,
  trace = FALSE,
  imp_var = TRUE,
  imp_suffix = "imp",
  addRF = FALSE,
  onlyRF = FALSE,
  addRandom = FALSE,
  useImputedDist = TRUE,
  weightDist = FALSE,
  methodStand = "range",
  ordFun = medianSamp
)

Arguments

Details

Numerical and semi-continuous (mixed) distance variables are standardised by their range (or IQR, see methodStand) so that each variable contributes on a comparable scale; for mixed variables the point mass (mixed.constant) is excluded from the range estimate.

Missing values in the distance variables (dist_var) are handled with a sentinel: after standardisation an NA is replaced by the maximum of the pooled column plus one. As a consequence two records that are both missing the same distance variable have distance 0 on that variable (they are treated as identical there), while a record missing the variable and one observing it contribute up to about twice a normal variable's share. In effect kNN is drawn towards donors that share the recipient's pattern of missingness in the distance variables; under a MAR mechanism this can select unrepresentative donors. This departs from Gower's (1971) convention of omitting non-comparable variables and renormalising the remaining weights. To limit the effect, prefer dist_var variables with few or no missing values.

Value

the imputed data set.

Author(s)

Alexander Kowarik, Statistik Austria

References

A. Kowarik, M. Templ (2016) Imputation with R package VIM. Journal of Statistical Software, 74(7), 1-16.

See Also

Other imputation methods: hotdeck(), impPCA(), imputeCellEM(), imputeCellIRMI(), imputeCellM(), imputeCellMCD(), imputeCellwise(), imputeRobust(), imputeRobustChain(), irmi(), matchImpute(), medianSamp(), rangerImpute(), regressionImp(), sampleCat(), vimmi, vimpute(), xgboostImpute()

Examples

data(sleep)
kNN(sleep)
library(laeken)
kNN(sleep, numFun = weightedMean, weightDist=TRUE)

Description

Coordinates of the Kola background map.

Source

Kola Project (1993-1998)

References

Reimann, C., Filzmoser, P., Garrett, R.G. and Dutter, R. (2008) Statistical Data Analysis Explained: Applied Environmental Statistics with R. Wiley, 2008.