Recap of pre-processing

The previous section walked-through the pre-processing and transformation of the count data. Here, for completeness, we list the minimal steps required to process the data prior to differential expression analysis.

# Read the sample information into a data frame
sampleinfo <- read_tsv("data/SampleInfo.txt")

# Read the data into R
seqdata <- read_tsv("data/GSE60450_Lactation.featureCounts", comment = "#")

# Transform the data to matrix of counts
countdata <- as.data.frame(seqdata) %>% 
    column_to_rownames("Geneid") %>% # turn the geneid column into rownames
    rename_all(str_remove, ".bam") %>% # remove the ".bam" from the column names
    select(sampleinfo$Sample) %>% # keep sample columns using sampleinfo
    as.matrix()

# filter the data to remove genes with few counts
keep <- rowSums(countdata) > 5
countdata <- countdata[keep,]

Load the data

Alternatively we can load the `objects with the RData file we created in the pre-processing tutorial.

# load the RData object we created in the previous session
load("Robjects/preprocessing.RData")
ls()
[1] "countdata"  "sampleinfo" "wibble"    
dim(countdata)
[1] 22013    12
sampleinfo

The model formula and design matrices

First load the packages we need.

library(tidyverse)
library(DESeq2)

Now that we are happy that that the quality of the data looks good, we can proceed to testing for differentially expressed genes. There are a number of packages to analyse RNA-Seq data. Most people use DESeq2 or edgeR. They are both equally applicable. There is an informative and honest blog post here by Mike Love, one of the authors of DESeq2, about deciding which to use.

We will use DESeq2 for the rest of this practical.

Create a DESeqDataSet object with the raw data

Creating the design model formula

First we need to create a design model formula for our analysis. DESeq2 will use this to generate the model matrix, as we have seen in the linear models lecture.

We have two variables: “status”" and “cell type”. We will fit two models under two assumptions: no interaction and interaction of these two factors.

Let’s start with the model with only main effects, that is no interaction. The main assumption here is that the effect of the status is the same in both type of cells.

# Use the standard R 'formula' syntax for an additive model
design <- as.formula(~ CellType + Status)

What does this look like as a model matrix?

modelMatrix <- model.matrix(design, data = sampleinfo)
modelMatrix
   (Intercept) CellTypeluminal Statuspregnant Statusvirgin
1            1               0              0            1
2            1               0              0            1
3            1               0              1            0
4            1               0              1            0
5            1               0              0            0
6            1               0              0            0
7            1               1              0            1
8            1               1              0            1
9            1               1              1            0
10           1               1              1            0
11           1               1              0            0
12           1               1              0            0
attr(,"assign")
[1] 0 1 2 2
attr(,"contrasts")
attr(,"contrasts")$CellType
[1] "contr.treatment"

attr(,"contrasts")$Status
[1] "contr.treatment"

It would be nice if virgin were the the base line/intercept. To get R to use virgin as the intercept we need to use a factor. Let’s set factor levels on Status to use virgin as the intercept.

sampleinfo$Status <- factor(sampleinfo$Status, 
                            levels = c("virgin", "pregnant", "lactate"))
modelMatrix <- model.matrix(design, data = sampleinfo)
modelMatrix
   (Intercept) CellTypeluminal Statuspregnant Statuslactate
1            1               0              0             0
2            1               0              0             0
3            1               0              1             0
4            1               0              1             0
5            1               0              0             1
6            1               0              0             1
7            1               1              0             0
8            1               1              0             0
9            1               1              1             0
10           1               1              1             0
11           1               1              0             1
12           1               1              0             1
attr(,"assign")
[1] 0 1 2 2
attr(,"contrasts")
attr(,"contrasts")$CellType
[1] "contr.treatment"

attr(,"contrasts")$Status
[1] "contr.treatment"

Build a DESeq2DataSet

We don’t actually need to pass DESeq2 the model matrix, instead we pass it the design formula and the sampleinfo it will build the matrix itself.

# create the DESeqDataSet object
ddsObj.raw <- DESeqDataSetFromMatrix(countData = countdata,
                                     colData = sampleinfo,
                                     design = design)
some variables in design formula are characters, converting to factors

Data exploration

Let’s plot a PCA from vst transformed data. Can you anticipate if the interaction term will be important?

vstcounts <- vst(ddsObj.raw, blind=TRUE)
plotPCA(vstcounts, intgroup=c("Status", "CellType"))

Differential expression analysis with DESeq2

The DESeq2 work flow

The main DESeq2 work flow is carried out in 3 steps:

First, Calculate the “median ratio” normalisation size factors…

ddsObj <- estimateSizeFactors(ddsObj.raw)

… then estimate dispersion …

ddsObj <- estimateDispersions(ddsObj)
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates

… finally, apply Negative Binomial GLM fitting and calculate Wald statistics

ddsObj <- nbinomWaldTest(ddsObj)

The DESeq command

In practice the 3 steps above can be performed in a single step using the DESeq wrapper function. Performing the three steps separately is useful if you wish to alter the default parameters of one or more steps, otherwise the DESeq function is fine.

# Run DESeq
ddsObj <- DESeq(ddsObj.raw)
estimating size factors
estimating dispersions
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing

Generate a results table

We can generate a table of differential expression results from the DDS object using the results function of DESeq2.

res <- results(ddsObj, alpha=0.05)
head(res)
log2 fold change (MLE): Status lactate vs virgin 
Wald test p-value: Status lactate vs virgin 
DataFrame with 6 rows and 6 columns
                            baseMean     log2FoldChange             lfcSE
                           <numeric>          <numeric>         <numeric>
ENSMUSG00000051951  193.628316332322  0.685666258944297 0.756712220192481
ENSMUSG00000102331 0.552133378452342   1.82614307383877  2.73990068528126
ENSMUSG00000025900  2.09442070179461  -3.02782655402916  1.54150999536423
ENSMUSG00000025902  52.4208304161579 -0.703252517806886 0.417356581488203
ENSMUSG00000098104 0.684213614702782  0.201738946690919  1.68661571665909
ENSMUSG00000103922  27.5871076674379   1.63592009157741 0.669531803485365
                                stat             pvalue               padj
                           <numeric>          <numeric>          <numeric>
ENSMUSG00000051951 0.906112311454264  0.364876409573908  0.520197122671947
ENSMUSG00000102331 0.666499732508119  0.505091734596282                 NA
ENSMUSG00000025900 -1.96419521322256 0.0495074577244765  0.117366437063074
ENSMUSG00000025902 -1.68501600070434 0.0919854844526348  0.189956255812803
ENSMUSG00000098104 0.119611684332298  0.904790763421973                 NA
ENSMUSG00000103922  2.44337921374511 0.0145504392990085 0.0446034399345576

Independent filtering

You will notice that some of the adjusted p-values (padj) are NA. Remember in Session 2 we said that there is no need to pre-filter the genes as DESeq2 will do this through a process it calls ‘independent filtering’. The genes with NA are the ones DESeq2 has filtered out.

From DESeq2 manual: “The results function of the DESeq2 package performs independent filtering by default using the mean of normalized counts as a filter statistic. A threshold on the filter statistic is found which optimizes the number of adjusted p values lower than a [specified] significance level”.

The default significance level for independent filtering is 0.1, however, you should set this to the FDR cut off you are planning to use. We will use 0.05 - this was the purpose of the alpha argument in the previous command.

The default contrast of results

The results function has returned the results for the contrast “lactate vs virgin”. Let’s have a look at the model matrix to understand why DESeq2 has given us this particular contrast.

modelMatrix
   (Intercept) CellTypeluminal Statuspregnant Statuslactate
1            1               0              0             0
2            1               0              0             0
3            1               0              1             0
4            1               0              1             0
5            1               0              0             1
6            1               0              0             1
7            1               1              0             0
8            1               1              0             0
9            1               1              1             0
10           1               1              1             0
11           1               1              0             1
12           1               1              0             1
attr(,"assign")
[1] 0 1 2 2
attr(,"contrasts")
attr(,"contrasts")$CellType
[1] "contr.treatment"

attr(,"contrasts")$Status
[1] "contr.treatment"

By default, results has returned the contrast encoded by the final column in the model matrix. DESeq2 has the command resultsNames that allows us to view the contrasts that are available directly from the model matrix.

resultsNames(ddsObj)
[1] "Intercept"                 "CellType_luminal_vs_basal"
[3] "Status_pregnant_vs_virgin" "Status_lactate_vs_virgin" 

Let’s just rename res so that we know which contrast results it contains.

resLvV <- res
rm(res)

If we want a different contrast we can just pass the results function the name of the design matrix column that encodes it. Let’s retrieve the results for pregant versus virgin

resPvV <- results(ddsObj, 
                  name="Status_pregnant_vs_virgin", 
                  alpha = 0.05)
resPvV
log2 fold change (MLE): Status pregnant vs virgin 
Wald test p-value: Status pregnant vs virgin 
DataFrame with 22013 rows and 6 columns
                            baseMean     log2FoldChange             lfcSE
                           <numeric>          <numeric>         <numeric>
ENSMUSG00000051951  193.628316332322 -0.954402462919142 0.758163669842639
ENSMUSG00000102331 0.552133378452342  0.793605030301116  2.76321615794719
ENSMUSG00000025900  2.09442070179461 -0.422312119520873  1.28960999953244
ENSMUSG00000025902  52.4208304161579 -0.372373337372365 0.399035417679523
ENSMUSG00000098104 0.684213614702782  -1.47471635546294  1.79948663916206
...                              ...                ...               ...
ENSMUSG00000064367  54055.8663105821  0.344588799853028 0.164346846848978
ENSMUSG00000064368  13454.3113895505  0.426945077829144 0.156109922741008
ENSMUSG00000064370  111885.216173771  0.415862003187525 0.225727285622729
ENSMUSG00000064372   396.00873428211  0.262372947574106 0.261696769419487
ENSMUSG00000095742  632.014515495881 -0.191806454656634 0.355891725136564
                                 stat             pvalue               padj
                            <numeric>          <numeric>          <numeric>
ENSMUSG00000051951  -1.25883433997468  0.208090174010857  0.411522854118992
ENSMUSG00000102331  0.287203383643606  0.773956595640506                 NA
ENSMUSG00000025900 -0.327472739567765  0.743310358008101                 NA
ENSMUSG00000025902 -0.933183674616645  0.350725142291883  0.564484945558628
ENSMUSG00000098104 -0.819520591800363  0.412489458421745                 NA
...                               ...                ...                ...
ENSMUSG00000064367   2.09671683065316  0.036018649711218  0.124182953553429
ENSMUSG00000064368   2.73490032108632  0.006239916868708 0.0341430901523808
ENSMUSG00000064370   1.84232048881578 0.0654282845131489  0.190916096242412
ENSMUSG00000064372     1.002583823087  0.316061704203191  0.530023878511752
ENSMUSG00000095742 -0.538946092615762  0.589924051630021  0.762012734103729

Let’s get the top 100 genes by adjusted p-value

topGenesPvV <- as.data.frame(resPvV) %>%
    rownames_to_column("GeneID") %>% 
    arrange(padj) %>% 
    head(100)
topGenesPvV

Challenge 1

Obtain results for luminal vs basal and find the top 200 genes. Call the new results object resBvL.

Contrasts

Suppose we want to find differentially expressed genes between pregnant and lactate. We don’t have a parameter that explicitly will allow us to test that hypothesis. We need to provide a contrast.

resultsNames(ddsObj)
[1] "Intercept"                 "CellType_luminal_vs_basal"
[3] "Status_pregnant_vs_virgin" "Status_lactate_vs_virgin" 
resPvL <- results(ddsObj,
                  contrast=c("Status", "pregnant", "lactate"), 
                  alpha = 0.05)
resPvL
log2 fold change (MLE): Status pregnant vs lactate 
Wald test p-value: Status pregnant vs lactate 
DataFrame with 22013 rows and 6 columns
                            baseMean      log2FoldChange             lfcSE
                           <numeric>           <numeric>         <numeric>
ENSMUSG00000051951  193.628316332322   -1.64006872186344  0.76011381700911
ENSMUSG00000102331 0.552133378452342   -1.03253804353765  2.70726895584589
ENSMUSG00000025900  2.09442070179461    2.60551443450829  1.58113233162031
ENSMUSG00000025902  52.4208304161579   0.330879180434521  0.42458801011868
ENSMUSG00000098104 0.684213614702782   -1.67645530215386  1.83315246550495
...                              ...                 ...               ...
ENSMUSG00000064367  54055.8663105821  -0.302824789042765 0.164348578199341
ENSMUSG00000064368  13454.3113895505    -0.2514951308436 0.156113680277036
ENSMUSG00000064370  111885.216173771  -0.196775657448966 0.225728145700564
ENSMUSG00000064372   396.00873428211 -0.0171568600400709 0.262491263103216
ENSMUSG00000095742  632.014515495881   0.280640913843416 0.357511558787644
                                  stat             pvalue              padj
                             <numeric>          <numeric>         <numeric>
ENSMUSG00000051951   -2.15766203055849 0.0309541180695751 0.106766273830347
ENSMUSG00000102331  -0.381394704544615  0.702910389782284                NA
ENSMUSG00000025900    1.64787879066277 0.0993775464909503                NA
ENSMUSG00000025902   0.779294686964981   0.43580614370291 0.636130155679824
ENSMUSG00000098104   -0.91452038698378  0.360443470413369                NA
...                                ...                ...               ...
ENSMUSG00000064367   -1.84257626296872 0.0653909022821323 0.183136793004815
ENSMUSG00000064368    -1.6109743258714  0.107185318448968 0.258163585175872
ENSMUSG00000064370  -0.871737358397456  0.383351674855471 0.588905143356533
ENSMUSG00000064372 -0.0653616422780692  0.947886063772488 0.973183515621002
ENSMUSG00000095742   0.784984168889803  0.432462850616724 0.633552881520095

Comparing two design models

Suppose we thought that maybe status were irrelevant and really the only differences might be between cell types. We could fit a simpler model, this would give us more degrees of freedom and therefore more power, but how would we know if it was a better model of not? We can compare the two models using the “log ratio test” (LRT).

designC <- as.formula(~ CellType )
# Compare the designs
ddsObjC <- DESeq(ddsObj, test="LRT", reduced=designC)
using pre-existing size factors
estimating dispersions
found already estimated dispersions, replacing these
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing
resCvCS <- results(ddsObjC)
resCvCS
log2 fold change (MLE): Status lactate vs virgin 
LRT p-value: '~ CellType + Status' vs '~ CellType' 
DataFrame with 22013 rows and 6 columns
                            baseMean     log2FoldChange             lfcSE
                           <numeric>          <numeric>         <numeric>
ENSMUSG00000051951  193.628316332322  0.685666258944297 0.756712220192481
ENSMUSG00000102331 0.552133378452342   1.82614307383877  2.73990068528126
ENSMUSG00000025900  2.09442070179461  -3.02782655402916  1.54150999536423
ENSMUSG00000025902  52.4208304161579 -0.703252517806886 0.417356581488203
ENSMUSG00000098104 0.684213614702782  0.201738946690919  1.68661571665909
...                              ...                ...               ...
ENSMUSG00000064367  54055.8663105821  0.647413588895793 0.164350605695024
ENSMUSG00000064368  13454.3113895505  0.678440208672744 0.156129934723267
ENSMUSG00000064370  111885.216173771  0.612637660636491 0.225729381046246
ENSMUSG00000064372   396.00873428211  0.279529807614177 0.262569324325279
ENSMUSG00000095742  632.014515495881  -0.47244736850005 0.357092689972414
                                stat               pvalue                 padj
                           <numeric>            <numeric>            <numeric>
ENSMUSG00000051951  4.40903494919066    0.110303736820745    0.203011375932541
ENSMUSG00000102331 0.397659679657469    0.819689359948631                   NA
ENSMUSG00000025900  4.68578354201197   0.0960494831334411    0.182314817580936
ENSMUSG00000025902  2.89099132419601    0.235629255115333    0.361374169977365
ENSMUSG00000098104   1.1310712566686    0.568055798474907                   NA
...                              ...                  ...                  ...
ENSMUSG00000064367  15.2088899461407 0.000498231876884956  0.00246460256311396
ENSMUSG00000064368  18.6633472727525 8.85738725215165e-05 0.000552135624089729
ENSMUSG00000064370  7.28491634375843   0.0261878904433618   0.0653923895110851
ENSMUSG00000064372  1.38080328700443    0.501374654748133    0.622036554157161
ENSMUSG00000095742  1.63512832055875    0.441505783095712    0.569776660113684

The null hypothesis is that there is no significant difference between the two models, i.e. the simpler model is sufficient to explain the variation in gene expression between the samples. If the the adjusted p-value for a gene passes a significance threshold (e.g. padj < 0.05) then we should consider using the more complex model for this gene. In practice we would usually choose one model or the other and apply it to all genes.

Challenge 2

When we looked at the PCA it did seem that an interaction model might be warranted. Let’s test that.
1.Fit a model with interaction. 2. Use the LRT to compare the two models.
3. Is the number of replicates good enough to include the interaction?
4. Is the interaction needed in the model?

Finally save the results in a new RData object

save(resLvV, ddsObj, sampleinfo, file="results/DE.RData")

References

LS0tCnRpdGxlOiAiUk5BLXNlcSBhbmFseXNpcyBpbiBSIgpzdWJ0aXRsZTogIkRpZmZlcmVudGlhbCBFeHByZXNzaW9uIG9mIFJOQS1zZXEgZGF0YSIKYXV0aG9yOiAiU3RlcGhhbmUgQmFsbGVyZWF1LCBNYXJrIER1bm5pbmcsIEFiYmkgRWR3YXJkcywgT3NjYXIgUnVlZGEsIEFzaGxleSBTYXdsZSIKZGF0ZTogJ2ByIGZvcm1hdChTeXMudGltZSgpLCAiTGFzdCBtb2RpZmllZDogJWQgJWIgJVkiKWAnCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgdG9jOiB5ZXMKICBodG1sX2RvY3VtZW50OgogICAgdG9jOiB5ZXMKbWludXRlczogMzAwCmxheW91dDogcGFnZQpiaWJsaW9ncmFwaHk6IHJlZi5iaWIKZWRpdG9yX29wdGlvbnM6IAogIGNodW5rX291dHB1dF90eXBlOiBpbmxpbmUKLS0tCgojIFJlY2FwIG9mIHByZS1wcm9jZXNzaW5nCgpUaGUgcHJldmlvdXMgc2VjdGlvbiB3YWxrZWQtdGhyb3VnaCB0aGUgcHJlLXByb2Nlc3NpbmcgYW5kIHRyYW5zZm9ybWF0aW9uIG9mIHRoZQpjb3VudCBkYXRhLiBIZXJlLCBmb3IgY29tcGxldGVuZXNzLCB3ZSBsaXN0IHRoZSBtaW5pbWFsIHN0ZXBzIHJlcXVpcmVkIHRvIApwcm9jZXNzIHRoZSBkYXRhIHByaW9yIHRvIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2lzLgoKYGBge3IgcmVjYXAsIGV2YWwgPSBGQUxTRX0KIyBSZWFkIHRoZSBzYW1wbGUgaW5mb3JtYXRpb24gaW50byBhIGRhdGEgZnJhbWUKc2FtcGxlaW5mbyA8LSByZWFkX3RzdigiZGF0YS9TYW1wbGVJbmZvLnR4dCIpCgojIFJlYWQgdGhlIGRhdGEgaW50byBSCnNlcWRhdGEgPC0gcmVhZF90c3YoImRhdGEvR1NFNjA0NTBfTGFjdGF0aW9uLmZlYXR1cmVDb3VudHMiLCBjb21tZW50ID0gIiMiKQoKIyBUcmFuc2Zvcm0gdGhlIGRhdGEgdG8gbWF0cml4IG9mIGNvdW50cwpjb3VudGRhdGEgPC0gYXMuZGF0YS5mcmFtZShzZXFkYXRhKSAlPiUgCiAgICBjb2x1bW5fdG9fcm93bmFtZXMoIkdlbmVpZCIpICU+JSAjIHR1cm4gdGhlIGdlbmVpZCBjb2x1bW4gaW50byByb3duYW1lcwogICAgcmVuYW1lX2FsbChzdHJfcmVtb3ZlLCAiLmJhbSIpICU+JSAjIHJlbW92ZSB0aGUgIi5iYW0iIGZyb20gdGhlIGNvbHVtbiBuYW1lcwogICAgc2VsZWN0KHNhbXBsZWluZm8kU2FtcGxlKSAlPiUgIyBrZWVwIHNhbXBsZSBjb2x1bW5zIHVzaW5nIHNhbXBsZWluZm8KICAgIGFzLm1hdHJpeCgpCgojIGZpbHRlciB0aGUgZGF0YSB0byByZW1vdmUgZ2VuZXMgd2l0aCBmZXcgY291bnRzCmtlZXAgPC0gcm93U3Vtcyhjb3VudGRhdGEpID4gNQpjb3VudGRhdGEgPC0gY291bnRkYXRhW2tlZXAsXQpgYGAKCiMjIExvYWQgdGhlIGRhdGEKCkFsdGVybmF0aXZlbHkgd2UgY2FuIGxvYWQgdGhlIGBvYmplY3RzIHdpdGggdGhlIFJEYXRhIGZpbGUgd2UgY3JlYXRlZCBpbiB0aGUgCnByZS1wcm9jZXNzaW5nIHR1dG9yaWFsLgoKYGBge3IgbG9hZERhdGF9CiMgbG9hZCB0aGUgUkRhdGEgb2JqZWN0IHdlIGNyZWF0ZWQgaW4gdGhlIHByZXZpb3VzIHNlc3Npb24KbG9hZCgiUm9iamVjdHMvcHJlcHJvY2Vzc2luZy5SRGF0YSIpCmxzKCkKZGltKGNvdW50ZGF0YSkKc2FtcGxlaW5mbwpgYGAKCiMgVGhlIG1vZGVsIGZvcm11bGEgYW5kIGRlc2lnbiBtYXRyaWNlcwoKRmlyc3QgbG9hZCB0aGUgcGFja2FnZXMgd2UgbmVlZC4KCmBgYHtyIHNldHVwLCBtZXNzYWdlID0gRkFMU0V9CmxpYnJhcnkodGlkeXZlcnNlKQpsaWJyYXJ5KERFU2VxMikKYGBgCgpOb3cgdGhhdCB3ZSBhcmUgaGFwcHkgdGhhdCB0aGF0IHRoZSBxdWFsaXR5IG9mIHRoZSBkYXRhIGxvb2tzIGdvb2QsIHdlIGNhbiAKcHJvY2VlZCB0byB0ZXN0aW5nIGZvciBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgZ2VuZXMuIFRoZXJlIGFyZSBhIG51bWJlciBvZiAKcGFja2FnZXMgdG8gYW5hbHlzZSBSTkEtU2VxIGRhdGEuIE1vc3QgcGVvcGxlIHVzZSAKW0RFU2VxMl0oaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sKSAKb3IgW2VkZ2VSXShodHRwOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvaHRtbC9lZGdlUi5odG1sKS4gVGhleSAKYXJlIGJvdGggZXF1YWxseSBhcHBsaWNhYmxlLiBUaGVyZSBpcyBhbiBpbmZvcm1hdGl2ZSBhbmQgaG9uZXN0IGJsb2cgcG9zdApbaGVyZV0oaHR0cHM6Ly9taWtlbG92ZS53b3JkcHJlc3MuY29tLzIwMTYvMDkvMjgvZGVzZXEyLW9yLWVkZ2VyLykgYnkgTWlrZSBMb3ZlLApvbmUgb2YgdGhlIGF1dGhvcnMgb2YgREVTZXEyLCBhYm91dCBkZWNpZGluZyB3aGljaCB0byB1c2UuCgpXZSB3aWxsIHVzZSAqKkRFU2VxMioqIGZvciB0aGUgcmVzdCBvZiB0aGlzIHByYWN0aWNhbC4KCiMjIENyZWF0ZSBhIERFU2VxRGF0YVNldCBvYmplY3Qgd2l0aCB0aGUgcmF3IGRhdGEKCiMjIyBDcmVhdGluZyB0aGUgZGVzaWduIG1vZGVsIGZvcm11bGEKCkZpcnN0IHdlIG5lZWQgdG8gY3JlYXRlIGEgZGVzaWduIG1vZGVsIGZvcm11bGEgZm9yIG91ciBhbmFseXNpcy4gYERFU2VxMmAgd2lsbCAKdXNlIHRoaXMgdG8gZ2VuZXJhdGUgdGhlIG1vZGVsIG1hdHJpeCwgYXMgd2UgaGF2ZSBzZWVuIGluIHRoZSBsaW5lYXIgbW9kZWxzIApsZWN0dXJlLiAKCldlIGhhdmUgdHdvIHZhcmlhYmxlczogInN0YXR1cyIiIGFuZCAiY2VsbCB0eXBlIi4gV2Ugd2lsbCBmaXQgdHdvIG1vZGVscyB1bmRlciAKdHdvIGFzc3VtcHRpb25zOiBubyBpbnRlcmFjdGlvbiBhbmQgaW50ZXJhY3Rpb24gb2YgdGhlc2UgdHdvIGZhY3RvcnMuIAoKTGV0J3Mgc3RhcnQgd2l0aCB0aGUgbW9kZWwgd2l0aCBvbmx5IG1haW4gZWZmZWN0cywgdGhhdCBpcyBubyBpbnRlcmFjdGlvbi4gClRoZSBtYWluIGFzc3VtcHRpb24gaGVyZSBpcyB0aGF0IHRoZSBlZmZlY3Qgb2YgdGhlIHN0YXR1cyBpcyB0aGUgc2FtZSBpbiBib3RoIAp0eXBlIG9mIGNlbGxzLgoKYGBge3IgbW9kZWxGb3J1bWxhfQojIFVzZSB0aGUgc3RhbmRhcmQgUiAnZm9ybXVsYScgc3ludGF4IGZvciBhbiBhZGRpdGl2ZSBtb2RlbApkZXNpZ24gPC0gYXMuZm9ybXVsYSh+IENlbGxUeXBlICsgU3RhdHVzKQpgYGAKCldoYXQgZG9lcyB0aGlzIGxvb2sgbGlrZSBhcyBhIG1vZGVsIG1hdHJpeD8KYGBge3IgbW9kZWxNYXRyaXh9Cm1vZGVsTWF0cml4IDwtIG1vZGVsLm1hdHJpeChkZXNpZ24sIGRhdGEgPSBzYW1wbGVpbmZvKQptb2RlbE1hdHJpeApgYGAKCkl0IHdvdWxkIGJlIG5pY2UgaWYgYHZpcmdpbmAgd2VyZSB0aGUgdGhlIGJhc2UgbGluZS9pbnRlcmNlcHQuIFRvIGdldCBSIHRvIAp1c2UgYHZpcmdpbmAgYXMgdGhlIGludGVyY2VwdCB3ZSBuZWVkIHRvIHVzZSBhIGBmYWN0b3JgLiBMZXQncyBzZXQgZmFjdG9yIGxldmVscwpvbiBTdGF0dXMgdG8gdXNlIHZpcmdpbiBhcyB0aGUgaW50ZXJjZXB0LgoKYGBge3Igc2V0RmFjdG9yc30Kc2FtcGxlaW5mbyRTdGF0dXMgPC0gZmFjdG9yKHNhbXBsZWluZm8kU3RhdHVzLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxldmVscyA9IGMoInZpcmdpbiIsICJwcmVnbmFudCIsICJsYWN0YXRlIikpCm1vZGVsTWF0cml4IDwtIG1vZGVsLm1hdHJpeChkZXNpZ24sIGRhdGEgPSBzYW1wbGVpbmZvKQptb2RlbE1hdHJpeApgYGAKCiMgQnVpbGQgYSBERVNlcTJEYXRhU2V0CgpXZSBkb24ndCBhY3R1YWxseSBuZWVkIHRvIHBhc3MgYERFU2VxMmAgdGhlIG1vZGVsIG1hdHJpeCwgaW5zdGVhZCB3ZSBwYXNzIGl0IHRoZSAKZGVzaWduIGZvcm11bGEgYW5kIHRoZSBgc2FtcGxlaW5mb2AgaXQgd2lsbCBidWlsZCB0aGUgbWF0cml4IGl0c2VsZi4KCmBgYHtyIG1ha2VERFNPYmp9CiMgY3JlYXRlIHRoZSBERVNlcURhdGFTZXQgb2JqZWN0CmRkc09iai5yYXcgPC0gREVTZXFEYXRhU2V0RnJvbU1hdHJpeChjb3VudERhdGEgPSBjb3VudGRhdGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb2xEYXRhID0gc2FtcGxlaW5mbywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRlc2lnbiA9IGRlc2lnbikKYGBgCgojIERhdGEgZXhwbG9yYXRpb24KCkxldCdzIHBsb3QgYSBQQ0EgZnJvbSBgdnN0YCB0cmFuc2Zvcm1lZCBkYXRhLiAKQ2FuIHlvdSBhbnRpY2lwYXRlIGlmIHRoZSBpbnRlcmFjdGlvbiB0ZXJtIHdpbGwgYmUgaW1wb3J0YW50PwoKYGBge3IgcGNhUGxvdCwgZmlnLndpZHRoPTUsIGZpZy5oZWlnaHQ9NX0KdnN0Y291bnRzIDwtIHZzdChkZHNPYmoucmF3LCBibGluZD1UUlVFKQpwbG90UENBKHZzdGNvdW50cywgaW50Z3JvdXA9YygiU3RhdHVzIiwgIkNlbGxUeXBlIikpCmBgYAoKIyBEaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcyB3aXRoIERFU2VxMgoKIyMgVGhlIGBERVNlcTJgIHdvcmsgZmxvdwoKVGhlIG1haW4gYERFU2VxMmAgd29yayBmbG93IGlzIGNhcnJpZWQgb3V0IGluIDMgc3RlcHM6CgpGaXJzdCwgQ2FsY3VsYXRlIHRoZSAibWVkaWFuIHJhdGlvIiBub3JtYWxpc2F0aW9uIHNpemUgZmFjdG9ycy4uLgoKYGBge3IgY29tbW9uU2l6ZUZhY3RvcnN9CmRkc09iaiA8LSBlc3RpbWF0ZVNpemVGYWN0b3JzKGRkc09iai5yYXcpCmBgYAoKLi4uIHRoZW4gZXN0aW1hdGUgZGlzcGVyc2lvbiAuLi4KCmBgYHtyIGdlbmV3aXNlRGlzcGVyc2lvbn0KZGRzT2JqIDwtIGVzdGltYXRlRGlzcGVyc2lvbnMoZGRzT2JqKQpgYGAKCi4uLiBmaW5hbGx5LCBhcHBseSBOZWdhdGl2ZSBCaW5vbWlhbCBHTE0gZml0dGluZyBhbmQgY2FsY3VsYXRlIFdhbGQgc3RhdGlzdGljcwpgYGB7ciBhcHBseUdMTX0KZGRzT2JqIDwtIG5iaW5vbVdhbGRUZXN0KGRkc09iaikKYGBgCgojIyBUaGUgYERFU2VxYCBjb21tYW5kCgpJbiBwcmFjdGljZSB0aGUgMyBzdGVwcyBhYm92ZSBjYW4gYmUgcGVyZm9ybWVkIGluIGEgc2luZ2xlIHN0ZXAgdXNpbmcgdGhlIApgREVTZXFgIHdyYXBwZXIgZnVuY3Rpb24uIFBlcmZvcm1pbmcgdGhlIHRocmVlIHN0ZXBzIHNlcGFyYXRlbHkgaXMgdXNlZnVsIGlmIHlvdQp3aXNoIHRvIGFsdGVyIHRoZSBkZWZhdWx0IHBhcmFtZXRlcnMgb2Ygb25lIG9yIG1vcmUgc3RlcHMsIG90aGVyd2lzZSB0aGUgYERFU2VxYApmdW5jdGlvbiBpcyBmaW5lLgoKYGBge3IgdGhlU2hvcnRWZXJzaW9ufQojIFJ1biBERVNlcQpkZHNPYmogPC0gREVTZXEoZGRzT2JqLnJhdykKYGBgCgojIyBHZW5lcmF0ZSBhIHJlc3VsdHMgdGFibGUKCldlIGNhbiBnZW5lcmF0ZSBhIHRhYmxlIG9mIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIHJlc3VsdHMgZnJvbSB0aGUgRERTIG9iamVjdAp1c2luZyB0aGUgYHJlc3VsdHNgIGZ1bmN0aW9uIG9mIERFU2VxMi4KCmBgYHtyIHJlc3VsdHNUYWJsZX0KcmVzIDwtIHJlc3VsdHMoZGRzT2JqLCBhbHBoYT0wLjA1KQpoZWFkKHJlcykKYGBgCgojIyMgSW5kZXBlbmRlbnQgZmlsdGVyaW5nCgpZb3Ugd2lsbCBub3RpY2UgdGhhdCBzb21lIG9mIHRoZSBhZGp1c3RlZCBwLXZhbHVlcyAoYHBhZGpgKSBhcmUgTkEuIFJlbWVtYmVyIAppbiBTZXNzaW9uIDIgd2Ugc2FpZCB0aGF0IHRoZXJlIGlzIG5vIG5lZWQgdG8gcHJlLWZpbHRlciB0aGUgZ2VuZXMgYXMgREVTZXEyCndpbGwgZG8gdGhpcyB0aHJvdWdoIGEgcHJvY2VzcyBpdCBjYWxscyAnaW5kZXBlbmRlbnQgZmlsdGVyaW5nJy4gVGhlIGdlbmVzIAp3aXRoIGBOQWAgYXJlIHRoZSBvbmVzIGBERVNlcTJgIGhhcyBmaWx0ZXJlZCBvdXQuCgpGcm9tIGBERVNlcTJgIG1hbnVhbDoKIlRoZSByZXN1bHRzIGZ1bmN0aW9uIG9mIHRoZSBgREVTZXEyYCBwYWNrYWdlIHBlcmZvcm1zIGluZGVwZW5kZW50IGZpbHRlcmluZyBieQpkZWZhdWx0IHVzaW5nIHRoZSBtZWFuIG9mIG5vcm1hbGl6ZWQgY291bnRzIGFzIGEgZmlsdGVyIHN0YXRpc3RpYy4gQSB0aHJlc2hvbGQgCm9uIHRoZSBmaWx0ZXIgc3RhdGlzdGljIGlzIGZvdW5kIHdoaWNoIG9wdGltaXplcyB0aGUgbnVtYmVyIG9mIGFkanVzdGVkIHAgdmFsdWVzCmxvd2VyIHRoYW4gYSBbc3BlY2lmaWVkXSBzaWduaWZpY2FuY2UgbGV2ZWwiLgoKVGhlIGRlZmF1bHQgc2lnbmlmaWNhbmNlIGxldmVsIGZvciBpbmRlcGVuZGVudCBmaWx0ZXJpbmcgaXMgYDAuMWAsIGhvd2V2ZXIsIHlvdQpzaG91bGQgc2V0IHRoaXMgdG8gdGhlIEZEUiBjdXQgb2ZmIHlvdSBhcmUgcGxhbm5pbmcgdG8gdXNlLiBXZSB3aWxsIHVzZSBgMC4wNWAgLQp0aGlzIHdhcyB0aGUgcHVycG9zZSBvZiB0aGUgYGFscGhhYCBhcmd1bWVudCBpbiB0aGUgcHJldmlvdXMgY29tbWFuZC4KCiMjIyBUaGUgZGVmYXVsdCBjb250cmFzdCBvZiBgcmVzdWx0c2AKClRoZSBgcmVzdWx0c2AgZnVuY3Rpb24gaGFzIHJldHVybmVkIHRoZSByZXN1bHRzIGZvciB0aGUgY29udHJhc3QgImxhY3RhdGUgdnMgCnZpcmdpbiIuIExldCdzIGhhdmUgYSBsb29rIGF0IHRoZSBtb2RlbCBtYXRyaXggdG8gdW5kZXJzdGFuZCB3aHkgYERFU2VxMmAgaGFzIApnaXZlbiB1cyB0aGlzIHBhcnRpY3VsYXIgY29udHJhc3QuCgpgYGB7ciB2aWV3TW9kZWxNYXRyaXh9Cm1vZGVsTWF0cml4CmBgYAoKQnkgZGVmYXVsdCwgYHJlc3VsdHNgIGhhcyByZXR1cm5lZCB0aGUgY29udHJhc3QgZW5jb2RlZCBieSB0aGUgZmluYWwgY29sdW1uIGluCnRoZSBtb2RlbCBtYXRyaXguIGBERVNlcTJgIGhhcyB0aGUgY29tbWFuZCBgcmVzdWx0c05hbWVzYCB0aGF0IGFsbG93cyB1cyB0byB2aWV3IAp0aGUgY29udHJhc3RzIHRoYXQgYXJlIGF2YWlsYWJsZSBkaXJlY3RseSBmcm9tIHRoZSBtb2RlbCBtYXRyaXguCgpgYGB7ciByZXN1bHRzTmFtZXN9CnJlc3VsdHNOYW1lcyhkZHNPYmopCmBgYAoKTGV0J3MganVzdCByZW5hbWUgYHJlc2Agc28gdGhhdCB3ZSBrbm93IHdoaWNoIGNvbnRyYXN0IHJlc3VsdHMgaXQgY29udGFpbnMuCgpgYGB7cn0KcmVzTHZWIDwtIHJlcwpybShyZXMpCmBgYAoKSWYgd2Ugd2FudCBhIGRpZmZlcmVudCBjb250cmFzdCB3ZSBjYW4ganVzdCBwYXNzIHRoZSBgcmVzdWx0c2AgZnVuY3Rpb24gdGhlIG5hbWUKb2YgdGhlIGRlc2lnbiBtYXRyaXggY29sdW1uIHRoYXQgZW5jb2RlcyBpdC4KTGV0J3MgcmV0cmlldmUgdGhlIHJlc3VsdHMgZm9yIHByZWdhbnQgdmVyc3VzIHZpcmdpbgoKYGBge3IgcmVzdWx0UHZWfQpyZXNQdlYgPC0gcmVzdWx0cyhkZHNPYmosIAogICAgICAgICAgICAgICAgICBuYW1lPSJTdGF0dXNfcHJlZ25hbnRfdnNfdmlyZ2luIiwgCiAgICAgICAgICAgICAgICAgIGFscGhhID0gMC4wNSkKcmVzUHZWCmBgYAoKTGV0J3MgZ2V0IHRoZSB0b3AgMTAwIGdlbmVzIGJ5IGFkanVzdGVkIHAtdmFsdWUKCmBgYHtyIHRvcEdlbmVzUHZWLCBtZXNzYWdlID0gRn0KdG9wR2VuZXNQdlYgPC0gYXMuZGF0YS5mcmFtZShyZXNQdlYpICU+JQogICAgcm93bmFtZXNfdG9fY29sdW1uKCJHZW5lSUQiKSAlPiUgCiAgICBhcnJhbmdlKHBhZGopICU+JSAKICAgIGhlYWQoMTAwKQp0b3BHZW5lc1B2VgpgYGAKCj4gIyMjIyBDaGFsbGVuZ2UgMSB7LmNoYWxsZW5nZX0KPiBPYnRhaW4gcmVzdWx0cyBmb3IgbHVtaW5hbCB2cyBiYXNhbCBhbmQgZmluZCB0aGUgdG9wIDIwMCBnZW5lcy4KPiBDYWxsIHRoZSBuZXcgcmVzdWx0cyBvYmplY3QgYHJlc0J2TGAuCgpgYGB7ciBzb2x1dGlvbkNoYWxsZW5nZTF9CmBgYAoKIyMgQ29udHJhc3RzCgpTdXBwb3NlIHdlIHdhbnQgdG8gZmluZCBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgZ2VuZXMgYmV0d2VlbiAqKnByZWduYW50KiogYW5kIAoqKmxhY3RhdGUqKi4gV2UgZG9uJ3QgaGF2ZSBhIHBhcmFtZXRlciB0aGF0IGV4cGxpY2l0bHkgd2lsbCBhbGxvdyB1cyB0byB0ZXN0IAp0aGF0IGh5cG90aGVzaXMuIFdlIG5lZWQgdG8gcHJvdmlkZSBhIGNvbnRyYXN0LgoKYGBge3IgbWFrZUNvbnRyYXN0fQpyZXN1bHRzTmFtZXMoZGRzT2JqKQoKcmVzUHZMIDwtIHJlc3VsdHMoZGRzT2JqLAogICAgICAgICAgICAgICAgICBjb250cmFzdD1jKCJTdGF0dXMiLCAicHJlZ25hbnQiLCAibGFjdGF0ZSIpLCAKICAgICAgICAgICAgICAgICAgYWxwaGEgPSAwLjA1KQpyZXNQdkwKYGBgCgojIENvbXBhcmluZyB0d28gZGVzaWduIG1vZGVscwoKU3VwcG9zZSB3ZSB0aG91Z2h0IHRoYXQgbWF5YmUgYHN0YXR1c2Agd2VyZSBpcnJlbGV2YW50IGFuZCByZWFsbHkgdGhlIG9ubHkgCmRpZmZlcmVuY2VzIG1pZ2h0IGJlIGJldHdlZW4gY2VsbCB0eXBlcy4gV2UgY291bGQgZml0IGEgc2ltcGxlciBtb2RlbCwgdGhpcyAKd291bGQgZ2l2ZSB1cyBtb3JlIGRlZ3JlZXMgb2YgZnJlZWRvbSBhbmQgdGhlcmVmb3JlIG1vcmUgcG93ZXIsIGJ1dCBob3cKd291bGQgd2Uga25vdyBpZiBpdCB3YXMgYSBiZXR0ZXIgbW9kZWwgb2Ygbm90PyBXZSBjYW4gY29tcGFyZSB0aGUgdHdvIG1vZGVscwp1c2luZyB0aGUgImxvZyByYXRpbyB0ZXN0IiAoTFJUKS4KCmBgYHtyIGNvbXBhcmVNb2RlbHN9CmRlc2lnbkMgPC0gYXMuZm9ybXVsYSh+IENlbGxUeXBlICkKCiMgQ29tcGFyZSB0aGUgZGVzaWducwpkZHNPYmpDIDwtIERFU2VxKGRkc09iaiwgdGVzdD0iTFJUIiwgcmVkdWNlZD1kZXNpZ25DKQpyZXNDdkNTIDwtIHJlc3VsdHMoZGRzT2JqQykKcmVzQ3ZDUwpgYGAKClRoZSBudWxsIGh5cG90aGVzaXMgaXMgdGhhdCB0aGVyZSBpcyBubyBzaWduaWZpY2FudCBkaWZmZXJlbmNlIGJldHdlZW4gdGhlIHR3bwptb2RlbHMsIGkuZS4gdGhlIHNpbXBsZXIgbW9kZWwgaXMgc3VmZmljaWVudCB0byBleHBsYWluIHRoZSB2YXJpYXRpb24gaW4gZ2VuZQpleHByZXNzaW9uIGJldHdlZW4gdGhlIHNhbXBsZXMuIElmIHRoZSB0aGUgYWRqdXN0ZWQgcC12YWx1ZSBmb3IgYSBnZW5lIHBhc3NlcwphIHNpZ25pZmljYW5jZSB0aHJlc2hvbGQgKGUuZy4gcGFkaiA8IDAuMDUpIHRoZW4gd2Ugc2hvdWxkIGNvbnNpZGVyIHVzaW5nIHRoZSAKbW9yZSBjb21wbGV4IG1vZGVsIGZvciB0aGlzIGdlbmUuIEluIHByYWN0aWNlIHdlIHdvdWxkIHVzdWFsbHkgY2hvb3NlIG9uZSBtb2RlbApvciB0aGUgb3RoZXIgYW5kIGFwcGx5IGl0IHRvIGFsbCBnZW5lcy4KCj4gIyMjIENoYWxsZW5nZSAyIHsuY2hhbGxlbmdlfQo+IFdoZW4gd2UgbG9va2VkIGF0IHRoZSBQQ0EgaXQgZGlkIHNlZW0gdGhhdCBhbiBpbnRlcmFjdGlvbiBtb2RlbCBtaWdodCBiZQo+IHdhcnJhbnRlZC4gTGV0J3MgdGVzdCB0aGF0LiAgCj4gMS5GaXQgYSBtb2RlbCB3aXRoIGludGVyYWN0aW9uLgo+IDIuIFVzZSB0aGUgTFJUIHRvIGNvbXBhcmUgdGhlIHR3byBtb2RlbHMuICAKPiAzLiBJcyB0aGUgbnVtYmVyIG9mIHJlcGxpY2F0ZXMgZ29vZCBlbm91Z2ggdG8gaW5jbHVkZSB0aGUgaW50ZXJhY3Rpb24/ICAgIAo+IDQuIElzIHRoZSBpbnRlcmFjdGlvbiBuZWVkZWQgaW4gdGhlIG1vZGVsPyAgCgpgYGB7ciBzb2x1dGlvbkNoYWxsZW5nZTJ9CmBgYAoKIyMgRmluYWxseSBzYXZlIHRoZSByZXN1bHRzIGluIGEgbmV3IFJEYXRhIG9iamVjdAoKYGBge3Igc2F2ZU9iamVjdHMsIGV2YWw9RkFMU0V9CnNhdmUocmVzTHZWLCBkZHNPYmosIHNhbXBsZWluZm8sIGZpbGU9InJlc3VsdHMvREUuUkRhdGEiKQpgYGAKCi0tLS0tLS0tLS0tLS0tLS0tLS0tCgojIFJlZmVyZW5jZXMK