jmatrixpp

Juan Domingo

Load package

library(parallelpam)

Please, read this

This is a copy of the vignette of package jmatrix (Domingo (2023a)). It is included here since jmatrix is underlying this package and you will need to know how to prepare your data to be processed by parallelpam. But you must NOT load package jmatrix; all the functions detailed below are already included into parallelpam (Domingo (2023b)) and are available just by loading it with library(parallelpam).

Purpose

The package jmatrix (Domingo (2023a)) was originally conceived as a tool for other packages, namely parallelpam (Domingo (2023b)) and scellpam (Domingo (2023c)) which needed to deal with very big matrices which might not fit in the memory of the computer, particularly if their elements are of type double (in most modern machines, 8 bytes per element) whereas they could fit if they were matrices of other data types, in particular of floats (4 bytes per element).

Unfortunately, R is not a strongly typed language. Double is the default type in R and it is not easy to work with other data types. Trials like the package float (Schmidt (2022)) have been done, but to use them you have to coerce a matrix already loaded in R memory to a float matrix, and then you can delete it. But, what happens if you computer has not memory enough to hold the matrix in the first place?. This is the problem this package tries to address.

Our idea is to use the disk as temporarily storage of the matrix in a file with a internal binary format (jmatrix format). This format has a header of 128 bytes with information like type of matrix (full, sparse or symmetric), data type of each element (char, short, int, long, float, double or long double), number of rows and columns and endianness; then comes the content as binary data (in sparse matrices zeros are not stored; in symmetric matrices only the lower-diagonal is stored) and finally the metadata (currently, names for rows/columns if needed and an optional comment).

Such files are created and loaded by functions written in C++ which are accessible from R with Rcpp (Eddelbuettel and François (2011)). The file, once loaded, uses strictly the needed memory for its data type and can be processed by other C++ functions (like the PAM algorithm or any other numeric library written in C++) also from inside R.

The matrix contained in a binary data file in jmatrix format cannot be loaded directly in R memory as a R matrix (that would be impossible, anyway, since precisely this package is done for the cases in which such matrix would NOT fit into the available RAM). Nevertheless, limited access through some functions is provided to read one or more rows or one or more columns as R vectors or matrices (obviously, coerced to double).

The package jmatrix must not be considered as a final, finished software. Currently is mostly an instrumental solution to address our needs and we make available as a separate package just in case it could be useful for anyone else.

Workflow

Debug messages

First of all, the package can show quite informative (but sometimes verbose) messages in the console. To turn on/off such messages you can use.

ParallelpamSetDebug(TRUE,TRUE)
#> Debugging for PAM algorithm set to ON.
#> Debugging for jmatrix inside parallelpam package set to ON.
# Initially, state of debug is FALSE.

which turns on messages related with the PAM algorithm (first TRUE) and also with the loading/saving of the binary matrices (second TRUE)

Data storage

As stated before, the binary matrix files should normally be created from C++ getting the data from an external source like a data file in a format used in bioinformatics or a .csv file. These files should be read by chunks. As an example, look at function CsvToJMat in package scellpam.

As a convenience and only for testing purposes (to be used in this vignette), we provide the function JWriteBin to write a R matrix as a jmatrix file.

# Create a 6x8 matrix of random values
Rf <- matrix(runif(48),nrow=6)
# Set row and column names for it
rownames(Rf) <- c("A","B","C","D","E","F")
colnames(Rf) <- c("a","b","c","d","e","f","g","h")
# Let's see the matrix
Rf
#>           a         b         c         d           e         f          g
#> A 0.6535695 0.7618674 0.7586606 0.5504418 0.340457673 0.7004754 0.21545662
#> B 0.8472642 0.2057142 0.6356137 0.5221313 0.636560047 0.6408356 0.07924971
#> C 0.5486340 0.8362046 0.3493792 0.3131979 0.008064953 0.4319581 0.52104012
#> D 0.2784492 0.1415840 0.2738906 0.4614996 0.294583396 0.1718650 0.21092166
#> E 0.2843412 0.9749653 0.0210961 0.5962367 0.877271971 0.9198650 0.76991743
#> F 0.3070112 0.8650555 0.1120602 0.7708062 0.316901997 0.6886493 0.96400709
#>           h
#> A 0.4604245
#> B 0.8550426
#> C 0.8112752
#> D 0.6149907
#> E 0.2364623
#> F 0.3633564
# and write it as the binary file Rfullfloat.bin
JWriteBin(Rf,"Rfullfloat.bin",dtype="float",dmtype="full",
          comment="Full matrix of floats")
#> The passed matrix has row names for the 6 rows and they will be used.
#> The passed matrix has column names for the 8 columns and they will be used.
#> Writing binary matrix Rfullfloat.bin of (6x8)
#> End of block of binary data at offset 320
#>    Writing row names (6 strings written, from A to F).
#>    Writing column names (8 strings written, from a to h).
#>    Writing comment: Full matrix of floats
# Also, you can write it with double data type:
JWriteBin(Rf,"Rfulldouble.bin",dtype="double",dmtype="full",
          comment="Full matrix of doubles")
#> The passed matrix has row names for the 6 rows and they will be used.
#> The passed matrix has column names for the 8 columns and they will be used.
#> Writing binary matrix Rfulldouble.bin of (6x8)
#> End of block of binary data at offset 512
#>    Writing row names (6 strings written, from A to F).
#>    Writing column names (8 strings written, from a to h).
#>    Writing comment: Full matrix of doubles

To get information about the stored file the function JMatInfo is provided. Of course, this funcion does not read the complete file in memory but just the header.

# Information about the float binary file
JMatInfo("Rfullfloat.bin")
#> File:               Rfullfloat.bin
#> Matrix type:        FullMatrix
#> Number of elements: 48
#> Data type:          float
#> Endianness:         little endian (same as this machine)
#> Number of rows:     6
#> Number of columns:  8
#> Metadata:           Stored names of rows and columns.
#> Metadata comment:  "Full matrix of floats"
# Same information about the double binary file
JMatInfo("Rfulldouble.bin")
#> File:               Rfulldouble.bin
#> Matrix type:        FullMatrix
#> Number of elements: 48
#> Data type:          double
#> Endianness:         little endian (same as this machine)
#> Number of rows:     6
#> Number of columns:  8
#> Metadata:           Stored names of rows and columns.
#> Metadata comment:  "Full matrix of doubles"

A jmatrix binary file can be exported to .csv/.tsv table. This is done with the function JMatToCsv

# Create a 6x8 matrix of random values
Rf <- matrix(runif(48),nrow=6)
# Set row and column names for it
rownames(Rf) <- c("A","B","C","D","E","F")
colnames(Rf) <- c("a","b","c","d","e","f","g","h")
# Store it as the binary file Rfullfloat.bin
JWriteBin(Rf,"Rfullfloat.bin",dtype="float",dmtype="full",
          comment="Full matrix of floats")
#> The passed matrix has row names for the 6 rows and they will be used.
#> The passed matrix has column names for the 8 columns and they will be used.
#> Writing binary matrix Rfullfloat.bin of (6x8)
#> End of block of binary data at offset 320
#>    Writing row names (6 strings written, from A to F).
#>    Writing column names (8 strings written, from a to h).
#>    Writing comment: Full matrix of floats
# Save the content of this .bin as a .csv file
JMatToCsv("Rfullfloat.bin","Rfullfloat.csv",csep=",",withquotes=FALSE)
#> Read full matrix with size (6,8)

The generated file will not have quotes neither around the column names (in its first line) nor around each row name (at the beginning of each line) since withquotes is FALSE but it can be set to TRUE for the opposite behavior. Also, a .tsv (tabulator separated values) would have been generated using csep=“\t”.

Also, a jmatrix binary file can also be generated from a .csv/.tsv file. Such file must have a first line with the names of the columns (possibly surrounded by double quotes, including a first empty double-quote, since the column of row names has no name itself). The rest of its lines must start with a string (possibly surrounded by double quotes) with the row name and the values. In all cases (first line and data lines) each column must be separated from the next by a separation character (usually, a comma). No separation character must be added at the end of each line. This format is compatible with the .csv generated by R with the function write.csv.

The function to read .csv files is CsvToJMat

# Create a 6x8 matrix of random values
Rf <- matrix(runif(48),nrow=6)
# Set row and column names for it
rownames(Rf) <- c("A","B","C","D","E","F")
colnames(Rf) <- c("a","b","c","d","e","f","g","h")
# Save it as a .csv file with the standard R function...
write.csv(Rf,"rf.csv")
# ...and read it to create a jmatrix binary file
CsvToJMat("rf.csv","rf.bin",mtype="full",csep=",",ctype="raw",valuetype="float",transpose=FALSE,comment="Test matrix generated reading a .csv file")
#> 8 columns of values (not including the column of names) in file rf.csv.
#> 6 lines (excluding header) in file rf.csv
#> Data will be read from each line and stored as float values.
#> Reading line... 0 
#> Read 6 data lines of file rf.csv, as expected.
#> Writing binary matrix rf.bin of (6x8)
#> End of block of binary data at offset 320
#>    Writing row names (6 strings written, from A to F).
#>    Writing column names (8 strings written, from a to h).
#>    Writing comment: Test matrix generated reading a .csv file
# Let's see the characteristics of the binary file
JMatInfo("rf.bin")
#> File:               rf.bin
#> Matrix type:        FullMatrix
#> Number of elements: 48
#> Data type:          float
#> Endianness:         little endian (same as this machine)
#> Number of rows:     6
#> Number of columns:  8
#> Metadata:           Stored names of rows and columns.
#> Metadata comment:  "Test matrix generated reading a .csv file"

Special note for symmetric matrices:

The parameter mtype=“symmetric” will consider the content of the .csv file as a symmetric matrix. This implies that it must be a square matrix (same number of rows and columns) but the upper-diagonal matrix that must be present (it does not matter with which values) will be read, and immediately ignored, i.e.: only the lower-diagonal matrix (including the main diagonal) will be stored.

Data load

As stated before, no function is provided to read the whole matrix in memory which would contradict the philosophy of this package, but you can get rows or columns from a file.

# Reads row 1 into vector vf. Float values inside the file are
# promoted to double.
(vf<-GetJRow("Rfullfloat.bin",1))
#>          a          b          c          d          e          f          g 
#> 0.19678958 0.23088335 0.98931634 0.49710301 0.16694224 0.02685002 0.09892655 
#>          h 
#> 0.55508590

Obviously, storage in float provokes a loosing of precision. We have observed this not to be relevant for PAM (partitioning around medoids) algorihm but it can be important in other cases. It is the price to pay for halving the needed space.

# Checks the precision lost
max(abs(Rf[1,]-vf))
#> [1] 0.7879247

Nevertheless, storing as double obviously keeps the data intact.

vd<-GetJRow("Rfulldouble.bin",1)
max(abs(Rf[1,]-vd))
#> [1] 0.5572689

Now, let us see examples of some functions to read rows or columns by number or by name, or to read several rows/columns as a R matrix. In all examples numbers for rows and columns are in R-convention (i.e.  starting at 1)

# Read column number 3
(vf<-GetJCol("Rfullfloat.bin",3))
#>         A         B         C         D         E         F 
#> 0.9893163 0.7672790 0.6814054 0.8444259 0.8263064 0.7493779
# Test precision
max(abs(Rf[,3]-vf))
#> [1] 0.7879247
# Read row with name C
(vf<-GetJRowByName("Rfullfloat.bin","C"))
#>         a         b         c         d         e         f         g         h 
#> 0.4845684 0.2979238 0.6814054 0.6412764 0.7182701 0.1317413 0.5847336 0.4967039
# Read column with name c
(vf<-GetJColByName("Rfullfloat.bin","c"))
#>         A         B         C         D         E         F 
#> 0.9893163 0.7672790 0.6814054 0.8444259 0.8263064 0.7493779
# Get the names of all rows or columns as vectors of R strings
(rn<-GetJRowNames("Rfullfloat.bin"))
#> [1] "A" "B" "C" "D" "E" "F"
(cn<-GetJColNames("Rfullfloat.bin"))
#> [1] "a" "b" "c" "d" "e" "f" "g" "h"
# Get the names of rows and columns simultaneosuly as a list of two elements
(l<-GetJNames("Rfullfloat.bin"))
#> $rownames
#> [1] "A" "B" "C" "D" "E" "F"
#> 
#> $colnames
#> [1] "a" "b" "c" "d" "e" "f" "g" "h"
# Get several rows at once. The returned matrix has the rows in the
# same order as the passed list,
# and this list can contain even repeated values
(vm<-GetJManyRows("Rfullfloat.bin",c(1,4)))
#>           a         b         c         d         e          f          g
#> A 0.1967896 0.2308833 0.9893163 0.4971030 0.1669422 0.02685002 0.09892655
#> D 0.4201853 0.7878940 0.8444259 0.3794409 0.4742073 0.59423566 0.65029418
#>           h
#> A 0.5550859
#> D 0.7248745

# Of course, columns can be extrated equally
(vc<-GetJManyCols("Rfulldouble.bin",c(1,4)))
#>           a         d
#> A 0.6535695 0.5504418
#> B 0.8472642 0.5221313
#> C 0.5486340 0.3131979
#> D 0.2784492 0.4614996
#> E 0.2843412 0.5962367
#> F 0.3070112 0.7708062
# and similar functions are provided for extracting by names:
(vm<-GetJManyRowsByNames("Rfulldouble.bin",c("A","D")))
#>           a         b         c         d         e         f         g
#> A 0.6535695 0.7618674 0.7586606 0.5504418 0.3404577 0.7004754 0.2154566
#> D 0.2784492 0.1415840 0.2738906 0.4614996 0.2945834 0.1718650 0.2109217
#>           h
#> A 0.4604245
#> D 0.6149907
(vc<-GetJManyColsByNames("Rfulldouble.bin",c("a","d")))
#>           a         d
#> A 0.6535695 0.5504418
#> B 0.8472642 0.5221313
#> C 0.5486340 0.3131979
#> D 0.2784492 0.4614996
#> E 0.2843412 0.5962367
#> F 0.3070112 0.7708062

The package can manage and store sparse and symmetric matrices, too.

# Generation of a 6x8 sparse matrix
Rsp <- matrix(rep(0,48),nrow=6)
sparsity <- 0.1
nnz <- round(48*sparsity)
where <- floor(47*runif(nnz))
val <- runif(nnz)
for (i in 1:nnz)
{
 Rsp[floor(where[i]/8)+1,(where[i]%%8)+1] <- val[i]
}
rownames(Rsp) <- c("A","B","C","D","E","F")
colnames(Rsp) <- c("a","b","c","d","e","f","g","h")
# Let's see the matrix
Rsp
#>   a         b         c d         e f         g h
#> A 0 0.6882563 0.9259237 0 0.0000000 0 0.0000000 0
#> B 0 0.0000000 0.0000000 0 0.0613993 0 0.0000000 0
#> C 0 0.0000000 0.0000000 0 0.0000000 0 0.7708025 0
#> D 0 0.0000000 0.0000000 0 0.0000000 0 0.0000000 0
#> E 0 0.0000000 0.0000000 0 0.0000000 0 0.0000000 0
#> F 0 0.0000000 0.0000000 0 0.0000000 0 0.0000000 0
# Write the matrix as sparse with type float
JWriteBin(Rsp,"Rspafloat.bin",dtype="float",dmtype="sparse",
          comment="Sparse matrix of floats")
#> The passed matrix has row names for the 6 rows and they will be used.
#> The passed matrix has column names for the 8 columns and they will be used.
#> Writing binary matrix Rspafloat.bin of (6x8)
#> End of block of binary data at offset 184
#>    Writing row names (6 strings written, from A to F).
#>    Writing column names (8 strings written, from a to h).
#>    Writing comment: Sparse matrix of floats

Notice that the condition of being a sparse matrix and the storage space used can be known with the matrix info.

JMatInfo("Rspafloat.bin")
#> File:               Rspafloat.bin
#> Matrix type:        SparseMatrix
#> Number of elements: 48
#> Data type:          float
#> Endianness:         little endian (same as this machine)
#> Number of rows:     6
#> Number of columns:  8
#> Metadata:           Stored names of rows and columns.
#> Metadata comment:  "Sparse matrix of floats"
#> Binary data size:   56 bytes, which is 29.1667% of the full matrix size (which would be 192 bytes).

Be careful: trying to store as sparse a matrix which is not (it has not a majority of 0-entries) works, but produces a matrix larger than the corresponding full matrix.

With respect to symmetric matrices, JWriteBin works the same way. Let us generate a \(7 \times 7\) symmetric matrix.

Rns <- matrix(runif(49),nrow=7)
Rsym <- 0.5*(Rns+t(Rns))
rownames(Rsym) <- c("A","B","C","D","E","F","G")
colnames(Rsym) <- c("a","b","c","d","e","f","g")
# Let's see the matrix
Rsym
#>           a         b         c         d         e         f          g
#> A 0.8271694 0.5111366 0.8057600 0.6237288 0.3487583 0.3395977 0.64504252
#> B 0.5111366 0.1050049 0.8322203 0.6738105 0.5140863 0.7249584 0.42778007
#> C 0.8057600 0.8322203 0.6140115 0.6303769 0.4853426 0.3791378 0.76673656
#> D 0.6237288 0.6738105 0.6303769 0.5494543 0.5290579 0.7935519 0.33613545
#> E 0.3487583 0.5140863 0.4853426 0.5290579 0.7484415 0.5722244 0.10826163
#> F 0.3395977 0.7249584 0.3791378 0.7935519 0.5722244 0.5289929 0.71760296
#> G 0.6450425 0.4277801 0.7667366 0.3361354 0.1082616 0.7176030 0.02336276
# Write the matrix as symmetric with type float
JWriteBin(Rsym,"Rsymfloat.bin",dtype="float",dmtype="symmetric",
          comment="Symmetric matrix of floats")
#> The passed matrix has row names for the 7 rows and they will be used.
#> Writing binary matrix Rsymfloat.bin
#> End of block of binary data at offset 240
#>    Writing row names (7 strings written, from A to G).
#>    Writing comment: Symmetric matrix of floats
# Get the information 
JMatInfo("Rsymfloat.bin")
#> File:               Rsymfloat.bin
#> Matrix type:        SymmetricMatrix
#> Number of elements: 49 (28 really stored)
#> Data type:          float
#> Endianness:         little endian (same as this machine)
#> Number of rows:     7
#> Number of columns:  7
#> Metadata:           Stored only names of rows.
#> Metadata comment:  "Symmetric matrix of floats"

Notice that if you store a R matrix which is NOT symmetric as a symmetric jmatrix, only the lower triangular part (including the main diagonal) will be saved. The upper-triangular part will be lost.

The functions to read rows/colums stated before works equally independently of the matrix character (full, sparse or symmetric) so you can play with them using the Rspafloat.bin and Rsymfloat.bin file to check they work.

Finally, if the jmatrix stored in a binary file has names associated to rows or columns, you can filter it using them and generate another jmatrix file with only the rows or columns you wish to keep. The function to do so is ‘FilterJMatByName’.

Rns <- matrix(runif(49),nrow=7)
rownames(Rns) <- c("A","B","C","D","E","F","G")
colnames(Rns) <- c("a","b","c","d","e","f","g")
# Let's see the matrix
Rns
#>            a         b         c           d          e         f         g
#> A 0.17052417 0.9204240 0.7505034 0.228405223 0.88242361 0.4865331 0.6458089
#> B 0.17444946 0.1660279 0.3839469 0.431906950 0.71387389 0.7469430 0.5750215
#> C 0.32451102 0.1125252 0.6518157 0.653104109 0.30782816 0.4877045 0.5044671
#> D 0.84140265 0.5060611 0.7527313 0.002565448 0.95500877 0.4845132 0.8222577
#> E 0.18026296 0.6883146 0.7382379 0.243447394 0.96109301 0.2725671 0.6478524
#> F 0.70892719 0.7586087 0.4803552 0.125862863 0.83738070 0.3579156 0.2791915
#> G 0.06937287 0.8540948 0.9997607 0.214751761 0.09709168 0.7267408 0.7775108
# Write the matrix as full with type float
JWriteBin(Rns,"Rfullfloat.bin",dtype="float",dmtype="full",
          comment="Full matrix of floats")
#> The passed matrix has row names for the 7 rows and they will be used.
#> The passed matrix has column names for the 7 columns and they will be used.
#> Writing binary matrix Rfullfloat.bin of (7x7)
#> End of block of binary data at offset 324
#>    Writing row names (7 strings written, from A to G).
#>    Writing column names (7 strings written, from a to g).
#>    Writing comment: Full matrix of floats
# Extract the first two and the last two columns
FilterJMatByName("Rfullfloat.bin",c("a","b","f","g"),"Rfullfloat_fourcolumns.bin",namesat="cols")
#> Read full matrix with size (7,7)
#> Writing binary matrix Rfullfloat_fourcolumns.bin of (7x4)
#> End of block of binary data at offset 240
#>    Writing row names (7 strings written, from A to G).
#>    Writing column names (4 strings written, from a to g).
#>    Writing comment: Full matrix of floats
# Let's load the matrix and let's see it
vm<-GetJManyRows("Rfullfloat_fourcolumns.bin",c(1,7))
vm
#>            a         b         f         g
#> A 0.17052418 0.9204240 0.4865331 0.6458089
#> G 0.06937287 0.8540948 0.7267408 0.7775108
Domingo, Juan. 2023a. Jmatrix: Read from/Write to Disk Matrices with Any Data Type in a Binary Format.
———. 2023b. Parallelpam: Applies the Partitioning-Around-Medoids (PAM) Clustering Algorithm to Big Sets of Data Using Parallel Implementation, If Several Cores Are Available.
———. 2023c. Scellpam: Applying Partitioning Around Medoids to Single Cell with High Number of Cells.
Eddelbuettel, Dirk, and Romain François. 2011. Rcpp: Seamless R and C++ Integration.” Journal of Statistical Software 40 (8): 1–18. https://doi.org/10.18637/jss.v040.i08.
Schmidt, Drew. 2022. float: 32-Bit Floats.” https://cran.r-project.org/package=float.