Jan Stanstrup

Heatmaps the right way

Jan Stanstrup — Fri, 27 Nov 2015 00:00:00 GMT

In this post I will go through some of the pitfalls of heatmap generation in R. I found that at least for me the heatmap functions do not always do what you think they do. When working with my own data I got convinced that something was wrong and investigated. I found that scaling was not done the way it would seem from the documentation of several functions. I eventually ended in on stackoverflow where Thomas W. Leja had explained what was going on.

Asking several colleagues no-one was aware of the issues I had found. I have therefore written up my findings here and in this post I will show in detail what to me doesn’t work as I expected and how do gain control of what precisely is done when you make a heatmap.
If you are surprised as well by the below please pass this on to everybody you know that might not be making heatmaps the way they think they are.

heatplot from made4

Lets start with heatplot from the made4 package just using the default settings.

library(made4)

x <- as.matrix(mtcars)
heatplot(x,margins=c(4,7))

Fig. 1

Figure 1. Using heatplot defaults.

Yikes! This is not good for this data. This is because heatplot by default does row scaling only. Fair enough. Lets use column scaling instead.

x <- as.matrix(mtcars)
heatplot(x,scale="column",margins=c(4,7))

Fig. 2

Figure 2. Heatplot with column scaling for the heat data only.

Much better! But wait! Did you see it? The colors in the plot changed. But the dendrogram did not! It turns out that the scale argument only refers to the scaling of the heat data, NOT what happens to the scaling before calculation of the dendrograms.
So how is the data for the dendrogram actually scaled now??? It turns out that if you look at the code for the function dualScale it does NOT do what it says. dualScale applies row scaling before dendrogram calculations… And that is it. So now we have row scaling for the dendrograms and have added additional column scaling for the heat data. Hmmmm… not what we want either.

So lets turn off the row scaling and we do the column scaling before we give the data to heatplot.

x <- scale(as.matrix(mtcars))
heatplot(x,scale="none",dualScale=FALSE,margins=c(4,7))

Fig. 3

Figure 3. Column scaled data and then heatplot. No row scaling in heatplot.

Good then. That seems better, all is well. Well… There is a thing more that probably did not work as you had thought but is not very visible in this case.
heatplot has the argument zlim which by default reassigns all values smaller than -3 and all values above 3 to -3 and 3 respectively. This makes a lot of sense when you use unit variance scaling. If you don’t do that most of your values will be in the middle of the distribution which gives you very pale colors since only the very extreme values will use the full range of the color gradient.
The problem here is that zlim is actually turned off when dualScale=FALSE which was probably not obvious to you either. In this case it doesn’t do much damage but if the destribution of you data is tailed this won’t look nice.

Pheww… Now we understand heatplot.

heatmap from stats and heatmap.2 from gplots

Now lets see if we can do the same plot with heatmap from stats. heatmap uses different defaults for distance calculation and clustering so lets change heatmap to use the same calculations and also make the color the same.

library(RColorBrewer)

x <- scale(as.matrix(mtcars))
heatmap(x,
        scale     = "none",
        col       = rev(colorRampPalette(brewer.pal(10, "RdBu"))(256)),
        distfun   = function(x) as.dist(1-cor(t(x))), 
        hclustfun = function(x) hclust(x, method="ave")
        )

Fig. 4

Figure 4. Column scaled data and then heatmap. No other scaling.

Oh no! Both dendrogram and the heat data is different. It turns out there are two reasons for this. For the dendrogram it turns out that heatplot uses the dendrogram directly whereas heatmap reorders the dendrogram based on row means. As far as I could find you cannot change this without calculating the dendrograms seperately.

For the colors it turns out that heatplot uses so-called symmetric breaks which means that 0 is put in the middle of the distribution of the data.
Btw.: And this is very important, heatmap and heatmap.2 (we will get to that one) has the same “feature” as heatplot: scale refers ONLY to the heat data, NOT the dendrogram calculation.

Lets jump right to heatmap.2 which is basically a version of heatmap with more options.

x <- scale(as.matrix(mtcars))
heatmap.2(x,
          scale     = "none",
          trace     = "none",
          col       = rev(colorRampPalette(brewer.pal(10, "RdBu"))(256)),
          distfun   = function(x) as.dist(1-cor(t(x))), 
          hclustfun = function(x) hclust(x, method="ave"),
          margins=c(4,7)
          )

Fig. 5

Figure 5. Column scaled data and then heatmap.2. No other scaling.

This looks similar to heatplot but heatmap.2 reorders as above while heatplot does not.

Gaining full controls

If you are not busy hyperventilating at this point I will offer here a solution that will give the control we seek over when exactly what happens to the data.

I have created (well actually I have stolen and expanded the code provided by Thomas W. Leja) a function, heat.clust, that will generate scaled data and dendrograms that can be passed to heatmap.2so that it can be controlled what happens at which point. You can find the function in my package massageR.
Lets jump into an example.

library(massageR)
library(gplots)

x <- as.matrix(mtcars)
z <- heat.clust(x,
                scaledim="column",
                zlim=c(-3,3),
                zlim_select = c("dend","outdata"),
                reorder=c("column","row"),
                distfun  = function(x) as.dist(1-cor(t(x))), 
                hclustfun= function(x) hclust(x, method="complete"),
                scalefun = scale)


heatmap.2(z$data,
          Rowv=z$Rowv, 
          Colv=z$Colv,
          trace="none",
          scale="none",
          symbreaks = TRUE,
          col=rev(colorRampPalette(brewer.pal(10, "RdBu"))(256)),
          margins=c(4,7)
          )

Fig. 6

Figure 6. Preparing data using heat.clust and then heatplot.2.

The parameters for heat.clust do the following:

scaledim: Selects to scale “column” or “row” or both (scaledim = c("row","column")).
zlim: Selects the limits of the data after scaling. Note that if you use anything else than unit variance scaling you should consider these limits. -3 to 3 probably doesn’t make sense in any other case. You can disable the limits using c(-Inf,Inf).
zlim_select: Selects when to apply zlim. For the dendrogram and/or for the outputted (scaled) data that is passed to heatmap.2.
reorder: Selects if to reorder the dendrogram by “column” and/or “row” means.
distfun: The function for distance calculation.
hclustfun: The function for clustering.
scalefun: the function to use for scaling.

Next we simply pass on the data and row and column dendrograms to heatmap.2. We can then do additional scaling and set symmetric breaks if we want.
I hope you enjoyed the read and now feel you know exactly what the heatmap funcions do.

Exemplifying why it matters when you do scaling

If you are not convinced that the above is very important I will explain here why scaling is really really important.
Lets take the same data as before and do no scaling at all.

x <- as.matrix(mtcars)

z <- heat.clust(x,
                scaledim="none",
                zlim=c(-3,3),
                zlim_select = "none",
                reorder=c("column","row"),
                distfun  = function(x) as.dist(1-cor(t(x))), 
                hclustfun= function(x) hclust(x, method="complete"),
                scalefun = scale)


heatmap.2(z$data,
          Rowv=z$Rowv, 
          Colv=z$Colv,
          trace="none",
          scale="none",
          symbreaks = TRUE,
          col=rev(colorRampPalette(brewer.pal(10, "RdBu"))(256)),
          margins=c(4,7)
          )

Fig. 7

Figure 7. Preparing data using heat.clust with no scaling and then heatplot.2.

We can clearly see that this is no good. Now if we do the same just with scaling for the heat data we get.

x <- as.matrix(mtcars)

z <- heat.clust(x,
                scaledim="none",
                zlim=c(-3,3),
                zlim_select = "none",
                reorder=c("column","row"),
                distfun  = function(x) as.dist(1-cor(t(x))), 
                hclustfun= function(x) hclust(x, method="complete"),
                scalefun = scale)


heatmap.2(z$data,
          Rowv=z$Rowv, 
          Colv=z$Colv,
          trace="none",
          scale="column",
          symbreaks = TRUE,
          col=rev(colorRampPalette(brewer.pal(10, "RdBu"))(256)),
          margins=c(4,7)
          )

Fig. 8

Figure 8. Preparing data using heat.clust with no scaling and then heatplot.2 with column scaling of heat data.

Doesn’t look so bad right? Wrong! Compared to the last plot we did in the previous section you might think the clustering doesn’t look that different. But lets look at what actually happened to the correlation matrix for the rows when the columns are not scaled.

library(corrplot)

x <- as.matrix(mtcars)
corrplot(cor(t(x)), method="color")

Fig. 9

Figure 9. Correlation plot of rows in the mtcars data.

Dang! Everything is super correlated! And if you plot two rows against each other we see why.

x <- as.matrix(mtcars)
plot(x[6,],x[7,])

Fig. 10

Figure 10. Variable 7 vs. variable 6 of the mtcars data. Data not scaled.

cor(x[6,],x[7,])

[1] 0.9814781

Because the variables are on so different scales fitting a line through seems pretty linear and the correlation is high…
But if we do the scaling we see the truth.

x <- scale(as.matrix(mtcars))
plot(x[6,],x[7,])

Fig. 11

Figure 11. Variable 7 vs. variable 6 of the mtcars data. Data column scaled first.

cor(x[6,],x[7,])

[1] -0.1641199

Not correlated at all it turns out.

Equivalent plots using heat.clust + heatmap.2

In this section I replicate all the plots above so you can see exactly what each command do when using full control of what happens.

Fig. 1 equivalent (click to show/hide)

Fig. 2 equivalent (click to show/hide)

Fig. 3 equivalent (click to show/hide)

Fig. 4 equivalent (click to show/hide)

Fig. 5 equivalent (click to show/hide)

Annotating LC-MS data with PredRet in R

Jan Stanstrup — Sun, 22 Nov 2015 00:00:00 GMT

In this tutorial I will show you how you can use the PredRet database to annotate your LC-MS metabolomics data directly in R. The annotation function I will be using is general-purpose, though, so you can use it to annotate any data where you have a list of compounds with known m/z’s and retention times (RTs).

What is PredRet?

In short PredRet is a user-driven database of compound retention times. The purpose of PredRet is to be able to predict the RT of a compound in one (your!) chromatographic system if it has been experimentally determined in another chromatographic system by someone, somewhere in the world. You can download the paper here and visit the project’s home page at PredRet.org.

Pulling data from PredRet

So lets get going.

First we will download the PredRet database with the PredRetR package. We will get both the experimental and the predicted values for the chromatographic system “LIFE_old”.

library(PredRetR)

predret_db <- PredRet_get_db(exp_pred = c("exp","pred"),
                             systems = "LIFE_old"
                            )

Then lets take a look at the structure of the data.frame we have retrieved. If the recorded_rt column has data we have an experimentally determined RT, if the predicted_rt column has data we have a RT predicted with the PredRet systems. The ci_lower and ci_upper columns show the prediction interval for the predictions.

library(knitr)
library(dplyr)

dim(predret_db)

[1] 3367   12

head(predret_db) %>% kable(format="html", table.attr = 'class="nicetable"')

system	name	pubchem	inchi	date added	username	predicted	suspect	recorded_rt	predicted_rt	ci_lower	ci_upper
LIFE_old	(DL)-p-hydroxyphenyllactic acid	NA	InChI=1S/C9H10O4/c10-7-3-1-6(2-4-7)5-8(11)9(12)13/h1-4,8,10-11H,5H2,(H,12,13)	2019-09-06 17:29:46	jan	FALSE	FALSE	1.544350	NA	NA	NA
LIFE_old	(R)-2-hydroxybutyric acid	NA	InChI=1S/C4H8O3/c1-2-3(5)4(6)7/h3,5H,2H2,1H3,(H,6,7)	2019-09-06 17:29:46	jan	FALSE	FALSE	1.399400	NA	NA	NA
LIFE_old	1-O-1'-(Z)-octadecenyl-2-hydroxy-sn-glycero-3-phosphocholine (LysoPC(P-18:0))	NA	InChI=1S/C26H54NO6P/c1-5-6-7-8-9-10-11-12-13-14-15-16-17-18-19-20-22-31-24-26(28)25-33-34(29,30)32-23-21-27(2,3)4/h20,22,26,28H,5-19,21,23-25H2,1-4H3	2019-09-06 17:29:46	jan	FALSE	FALSE	4.947717	NA	NA	NA
LIFE_old	2-Hydroxy-2-methylbutyric acid	NA	InChI=1S/C5H10O3/c1-3-5(2,8)4(6)7/h8H,3H2,1-2H3,(H,6,7)	2019-09-20 02:26:09	jan	FALSE	TRUE	1.625783	NA	NA	NA
LIFE_old	2-Hydroxy-3-methylbutyric acid	NA	InChI=1S/C5H10O3/c1-3(2)4(6)5(7)8/h3-4,6H,1-2H3,(H,7,8)	2019-09-20 02:26:09	jan	FALSE	TRUE	1.685467	NA	NA	NA
LIFE_old	2-methylacetoacetic acid	NA	InChI=1S/C5H8O3/c1-3(4(2)6)5(7)8/h3H,1-2H3,(H,7,8)	2019-09-06 17:29:46	jan	FALSE	FALSE	1.612850	NA	NA	NA

We have the RT directly in the database above but we do not have the m/z. Since we have the InChI the easiest way to get the m/z is to extract the molecular formula and then use the Rdisop package to get the mass.

library(Rdisop)
library(stringr)

formulas   <- str_split_fixed(predret_db[,"inchi"],"/",3)[,2]
masses     <- sapply(formulas, function(x) getMass(getMolecule(x)))

predret_db <- cbind.data.frame(predret_db, mz_pos = masses + 1.0073)

We can then split the database in two. One for the experimental RTs and one for the predicted. Here I have used some pipe/dplyr style code and even a forward assign. If you are not familiar with this type of R code I urge you to look into it. It really makes code much more readable.

predret_db %>% filter(!is.na(recorded_rt)) -> predret_db_exp
predret_db %>% filter(is.na(recorded_rt))  -> predret_db_pred

Annotating a dataset

We now have the database ready for annotation.

So we can load a dataset/peaklist. This peaklist was previously created with XCMS and fragments/adducts annotated with CAMERA. But again any peaklist will do.

library(readxl)

data <- read_excel("../../static/material/data/peaklist_pos.xlsx") %>%
        replace(is.na(.), "") %>%
        as.data.frame

Lets take a look at the interesting columns we have in the dataset.

info_cols <- c("mz","rt","isotopes","adduct","pcgroup")
dim(data)

[1] 4232  470

head(data[,info_cols])

         mz       rt   isotopes              adduct pcgroup
1  62.06013 273.8827            [M+H-C5H8O]+ 145.11       1
2  95.08600 271.5567                                      1
3  98.09642 268.1695                                      1
4 104.10687 271.2400   [12][M]+ [M+H-COCH2]+ 145.11       1
5 105.11030 271.2415 [12][M+1]+                           1
6 114.09119 268.9771            [M+H-CH3OH]+ 145.11       1

Now we can use db.comp.assign from my chemhelper package to annotate the dataset.

We would probably want one column in our peaklist for the RTs we have determined experimentally and one for the predicted RTs. So we do the annotation twice.

The first two arguments to the function are the m/z and RT of the dataset. The next three are the name, m/z and RT of the database of known compounds. Lastly we give the tolerance for the m/z and RT for a database match.

library(chemhelper)

anno_exp <- db.comp.assign(mz           = data[,"mz"],
                           rt           = data[,"rt"],
                           comp_name_db = predret_db_exp[,"name"],
                           mz_db        = predret_db_exp[,"mz_pos"],
                           rt_db        = predret_db_exp[,"recorded_rt"] *60, # *60 since XCMS works in seconds but PredRet is in minutes.
                           mzabs        = 0.01,
                           ppm          = 15,
                           ret_tol      = 10
                           )

                                        [,1]
Unique hits                               89
Non-unique hits                           10
Compounds not found                      201
Markers had multiple compounds assigned   53

anno_pred <- db.comp.assign(mz          = data[,"mz"],
                           rt           = data[,"rt"],
                           comp_name_db = predret_db_pred[,"name"],
                           mz_db        = predret_db_pred[,"mz_pos"],
                           rt_db        = predret_db_pred[,"predicted_rt"] *60,
                           mzabs        = 0.01,
                           ppm          = 15,
                           ret_tol      = 10
                           )

                                        [,1]
Unique hits                              209
Non-unique hits                           15
Compounds not found                     2843
Markers had multiple compounds assigned   72

Now lets put the annotations together with our dataset.

data_annotated <- cbind.data.frame(data,anno_exp,anno_pred)

The result

Then lets take a look at one of the feature groups (from CAMERA) where we got an annotation. In this example we have a feature that was annotated as tryptophan. The “OR” is because tryptophan was in the database several times. If multiple compounds would fit the m/z and RT they would be written with “OR” between them.

There is also a fragment annotated as adipic acid and 2-methylglutaric acid using a hit from the predicted RTs. In this case we have almost perfect CAMERA annotation suggesting the pseudo-molecular ion is the m/z = 205.0979 feature and the compound is very likely tryptophan.

data_annotated %>%
  filter(pcgroup==16) %>%
  select(one_of(info_cols,"anno_exp","anno_pred")) %>%
  kable(format="html", table.attr = 'class="nicetable"')

mz	rt	isotopes	adduct	pcgroup	anno_exp	anno_pred
118.0657	91.31296		[M+H-NH3-CO-COCH2]+ 204.09	16
130.0655	91.40512			16		QUINOLINE
132.0808	91.27887	[44][M]+	[M+H-NH3-CO-CO]+ 204.09	16
133.0845	91.19346	[44][M+1]+		16
142.0647	91.20588		[M+H-NH3-HCOOH]+ 204.09	16
144.0416	91.25665			16
144.0814	91.34230	[55][M]+	[M+H-NH3-CO2]+ 204.09	16		OSI1265 OR OSI1138 (non-unique hit)
145.0846	91.28341	[55][M+1]+		16
146.0599	91.33676	[58][M]+	[M+H-NH3-COCH2]+ 204.09	16		4-Hydroxyquinoline
147.0647	91.36894	[58][M+1]+		16	Adipic acid	ADIPATE OR 2-METHYLGLUTARATE OR METHYGLUTARATE
159.0922	91.28882	[71][M]+	[M+H-HCOOH]+ 204.09	16		OSI6
160.0851	91.35138	[71][M+1]+		16
170.0608	91.34747		[M+H-NH3-H2O]+ 204.09	16
188.0711	91.34805	[93][M]+	[M+H-NH3]+ 204.09	16
189.0743	91.28935	[93][M+1]+		16
205.0979	91.32364	[102][M]+	[M+H]+ 204.09	16	tryptophan OR Tryptophan OR TRYPTOPHAN	L-tryptopan (15N2, 98%)
206.1034	91.31068	[102][M+1]+		16
245.1301	91.91434		[M+H+(CH3)2CO-H2O]+ (acetone cond.) 204.09	16
276.1823	91.41425			16		[3-Carboxy-2-(3-hydroxyhexanoyloxy)propyl]-trimethylazanium
409.1873	91.28252	[265][M]+	[2M+H]+ 204.09	16
410.1906	91.26287	[265][M+1]+		16
447.1336	91.29100		[2M+K]+ 204.09 [4M+2K]2+ 204.09	16

Lets take a look at another feature. This time we have experimental RTs that say the feature is either 1,7-dimethylxanthine OR theobromine but a prediction from the PredRet database also suggest that the feature could be theophylline as well.

data_annotated %>%
  filter(pcgroup==400) %>%
  select(one_of(info_cols,"anno_exp","anno_pred")) %>%
  kable(format="html", table.attr = 'class="nicetable"')

mz	rt	isotopes	adduct	pcgroup	anno_exp	anno_pred
124.0508	89.67717			400
181.0725	89.56150	[86][M]+	[M+H-NH3-CO2-NH3-H2O]+ 276.119 [M+H-C4H6-COCH2]+ 276.119	400	1,7-dimethylxanthine OR theobromine	Theobromine OR Theophylline OR Paraxanthine OR OSI101
182.0761	89.59507	[86][M+1]+		400
315.0806	89.70856		[M+K]+ 276.119	400		HMDB0005033

That’s it!