dataquieR

The goal of dataquieR is to provide functions for assessing data quality issues in studies, that can be used alone or in a data quality pipeline. dataquieR also implements one generic pipeline producing htmltools based HTML5 reports.

See also

https://dataquality.qihs.uni-greifswald.de

Installation

You can install the released version of dataquieR from CRAN with:

install.packages("dataquieR")

The suggested packages can be directly installed by:

install.packages("dataquieR", dependencies = TRUE)

The developer version from GitLab.com can be installed using:

if (!requireNamespace("remotes")) {
  install.packages("remotes")
}
remotes::install_gitlab("libreumg/dataquier")

For examples and additional documentation, please refer to our website.

dataquieR usage questionnaire

To help us improve dataquieR, we invite you to provide your feedback by completing this short survey (English or German version).

Suggested packages

dataquieR reports can now use plotly if installed. That means that, in the final report, you can zoom in the figures and get information by hovering on the points, etc. To install plotly type:

install.packages("plotly")

To install all suggested packages, run:

prep_check_for_dataquieR_updates()

This command can also check for new beta releases of dataquieR from our own server, so not from CRAN:

prep_check_for_dataquieR_updates(beta = TRUE)

By default, dataquieR is kept small for CRAN installation. The convenience update helper installs dataquieR from source with byte-compiled R code:

prep_check_for_dataquieR_updates()

If you want to keep the installed package smaller by installing without byte compilation, use:

prep_check_for_dataquieR_updates(byte_compile = FALSE)

Hint If you are running dataquieR in an un-trusted setting, namely, inside a server application, please consider disabling the import of R-serialization files to prevent users from importing RData (or RDS or even R) files, that trigger code execution on your machine, see, e.g., Ivan Krylov’s blog for the reason:

# prevent rio from reading potentially code-containing files
options(rio.import.trust = FALSE)

If you do so, the example data won’t be loaded any more.

If you are using a version >= 2.0.0 of rio, this will be the default, so for running our examples, then, you’ll have to trust our files by using e.g. withr::with_options(list(rio.import.trust = FALSE), prep_get_data_frame("study_data")) for loading our example study data into the data-frame cache, initially and trusting our files loaded from

Cluster use

dq_report2() and friends can compute indicator calls in parallel. The cores argument accepts:

If you have already registered a cluster via parallel::setDefaultCluster(), pass cores = NULL to reuse it. dataquieR then leaves its lifecycle to you. An explicit cores value or cluster takes precedence over the registered default cluster.

Local multicore (default)

For local multicore work the default (PSOCK socket cluster) is usually fine:

dq_report2(study_data, meta_data, ..., cores = 4)

MPI clusters (Rmpi / snow)

Any cluster created through parallel::makeCluster(..., type = "MPI") (or via the snow / Rmpi chain) plugs in like any other parallel cluster. Either pass it as cores =:

library(Rmpi)
cl <- parallel::makeCluster(mpi.universe.size() - 1L, type = "MPI")
withr::defer(parallel::stopCluster(cl))

dq_report2(study_data, meta_data, ..., cores = cl)

or register it as the default cluster up front and pass cores = NULL to dq_report2() so that it uses the registered cluster:

cl <- parallel::makeCluster(8, type = "MPI")
parallel::setDefaultCluster(cl)
withr::defer({ parallel::setDefaultCluster(NULL); parallel::stopCluster(cl) })

dq_report2(study_data, meta_data, ..., cores = NULL)

The same pattern works for any other makeCluster()-style backend (PSOCK to remote hosts, FORK on Unix, etc.).

HPC schedulers (SLURM, SGE, Torque/PBS, LSF, …)

For batch schedulers, use mode = "futures" together with a future plan from future.batchtools:

install.packages(c("future", "future.batchtools"))
library(future)
library(future.batchtools)

plan(batchtools_slurm, template = "slurm.tmpl")   # or batchtools_sge / _torque / _lsf

dq_report2(study_data, meta_data, ..., mode = "futures")

For parallel HTML rendering, the main process and rendering workers need a shared, writable file system at the same path. This is usually already true for local clusters; on distributed nodes, arrange a shared directory (for example via NFS). If the file-system check fails, dataquieR writes the pages serially instead.

Earlier versions reached the same schedulers through options(parallelMap.mode = "BatchJobs") (or "batchtools"). That dispatch path is no longer wired up; the labels are still accepted by the new backend for source-level compatibility but degrade to sequential execution. mode = "futures" is the supported replacement.

References

Funding – see also here