The list of differentially expressed genes is sometimes so long that its interpretation becomes cumbersome and time consuming. A common downstream procedure is gene set testing. It aims to understand which pathways or gene networks the differentially expressed genes are implicated in.
Various ways exist to test for enrichment of biological pathways.
GSEA analysis
Gene Set Enrichment Analysis GSEA was tests whether a set of genes of interest, e.g. genes (Subramanian et al. 2005). The software is distributed by the Broad Institute and is freely available for use by academic and non-profit organisations.
In addition to the GSEA software the Broad also provide a number of very well curated gene set
s for testing against your data - the Molecular Signatures Database (MSigDB). Unfortunately, these are collections of human genes, however, these lists have been translated to mouse equivalents by the Walter+Eliza Hall Institutes Bioinformatics service and made avaialble for download.
The analysis is performed by:
- ranking all genes in the
data set
- identifying the rank positions of all members of the
gene set
in the ranked data set
- calculating an enrichment score (ES) that represents the difference between the observed rankings and that which would be expected assuming a random rank distribution.
commentary on GSEA. The article describing the original software is available here.
fgsea
The fgsea
package (Sergushichev 2016) implements the same algorithm in R vignette “fast preranked gene set enrichment analysis (GSEA)”.
library(fgsea)
Loading required package: Rcpp
load("Robjects/Annotated_Results_LvV.RData")
Create ranks
Rank all genes based on their fold change. We need to exclude genes for which we do not have Entrez IDs. Also, we should use the shrunk LFC values.
gseaDat <- filter(shrinkLvV, !is.na(Entrez))
ranks <- gseaDat$logFC
names(ranks) <- gseaDat$Entrez
head(ranks)
497097 19888 20671 27395 18777 21399
0.5851914 -1.0498457 -0.5893634 0.4313540 0.6478714 -0.1110589
Plot the ranked fold changes.
barplot(sort(ranks, decreasing = T))
Load pathways
load("Robjects/mouse_H_v5.RData")
pathwaysH <- Mm.H
Conduct analysis
fgseaRes <- fgsea(pathwaysH, ranks, minSize=15, maxSize = 500, nperm=1000)
There are ties in the preranked stats (0.05% of the list).
The order of those tied genes will be arbitrary, which may produce unexpected results.
The warning produced indicates that there are few genes that have the same fold change and so are ranked equally. fgsea
with arbitrarily order determine which comes first in the ranked list. As long as this number is small it shouldn’t significantly effect the results. If the number is large something is suspicious about the fold change results.
Lets look at the top 10 results.
head(fgseaRes[order(padj, -abs(NES)), ], n=10)
Enrichment score plot
plotEnrichment(pathwaysH[["HALLMARK_ESTROGEN_RESPONSE_EARLY"]], ranks)
Remember to check the GSEA article for the complete explanation.
GSEA table plot
The function plotGseaTable
allows us to plot a summary figue showing the results for multiple pathways.
topUp <- fgseaRes %>%
filter(ES > 0) %>%
top_n(10, wt=-padj)
topDown <- fgseaRes %>%
filter(ES < 0) %>%
top_n(10, wt=-padj)
topPathways <- bind_rows(topUp, topDown) %>%
arrange(-ES)
plotGseaTable(pathwaysH[topPathways$pathway],
ranks,
fgseaRes,
gseaParam = 0.5)
Challenge 1
Another common way to rank the genes is to order by pvalue, but also, sorting so that upregulated genes are at start and downregulated at the other - you can do this combining the sign of the fold change and the pvalue.
1. Rank the genes by statisical significance - you will need to create a new ranking value using -log10({p value}) * sign({Fold Change})
2. Load the “C2” pathways from the the data/mouse_c2_v5.RData
file
3. Run fgsea
using the new ranked genes and the C2 pathways
4. Run fgsea
using the new ranked genes and the H pathways. How do these results differ from the ones we got when ranking by the fold change alone?
GO enrichment analysis
goseq
GOseq is a method to conduct Gene Ontology (GO) analysis suitable for RNA-seq data as it accounts for the gene length bias in detection of over-representation (Young et al. 2010)
From the GOseq vignette:
- GOseq first needs to quantify the length bias present in the dataset under consideration.
- This is done by calculating a Probability Weighting Function or PWF which can be thought of as a function which gives the probability that a gene will be differentially expressed (DE), based on its length alone.
- The PWF is calculated by fitting a monotonic spline to the binary data series of differential expression (1=DE, 0=Not DE) as a function of gene length.
- The PWF is used to weight the chance of selecting each gene when forming a null distribution for GO category membership.
- The fact that the PWF is calculated directly from the dataset under consideration makes this approach robust, only correcting for the length bias present in the data.
“GO analysis of RNA-seq data requires the use of random sampling in order to generate a suitable null distribution for GO category membership and calculate each category’s significance for over representation amongst DE genes. … In most cases, the Wallenius distribution can be used to approximate the true null distribution, without any significant loss in accuracy. The goseq package implements this approximation as its default option.”
library(goseq)
supportedOrganisms() %>% filter(str_detect(Genome, "mm"))
NAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercionNAs introduced by coercion
Create a list of differentially expressed genes
isSigGene <- shrinkLvV$FDR < 0.01 & !is.na(shrinkLvV$FDR)
genes <- as.integer(isSigGene)
names(genes) <- shrinkLvV$GeneID
Fit the Probability Weighting Function (PWF)
pwf <- nullp(genes, "mm10", "ensGene", bias.data = shrinkLvV$medianTxLength)
initial point very close to some inequality constraints
Conduct GO enrichment analysis
goResults <- goseq(pwf, "mm10","ensGene", test.cats=c("GO:BP"))
Plot the top 10
goResults %>%
top_n(10, wt=-over_represented_pvalue) %>%
mutate(hitsPerc=numDEInCat*100/numInCat) %>%
ggplot(aes(x=hitsPerc,
y=term,
colour=over_represented_pvalue,
size=numDEInCat)) +
geom_point() +
expand_limits(x=0) +
labs(x="Hits (%)", y="GO term", colour="p value", size="Count")
KEGG pathway enrichment analysis
clusterProfiler
We can analyse for enrichment of KEGG pathways in much the same way as for GO terms. We could also use goseq
for this, but this time we’re going to use clusterProfiler
(Yu et al. 2012). clusterprofiler
supports direct online access of the current KEGG database, rather than relying on R annotation packages, it also provides some nice visualisation options.
library(clusterProfiler)
search_kegg_organism('mmu', by='kegg_code')
KEGG enrichment analysis
sigGenes <- shrinkLvV$Entrez[ shrinkLvV$FDR < 0.01 &
!is.na(shrinkLvV$FDR) &
abs(shrinkLvV$logFC) > 1 ]
sigGenes <- na.exclude(sigGenes)
kk <- enrichKEGG(gene = sigGenes, organism = 'mmu')
head(kk, n=10)
Visualise a pathway
In a browser
clusterProfile
has a function browseKegg
that allows you to view the Kegg pathway with the genes colours in in your browser.
browseKEGG(kk, 'mmu03320')
As a file
The package pathview
(Luo et al. 2013) can be used to generate figures of KEGG pathways. One advantage over clusterProfiles
browser methos is that the genes are coloured according to their expression levels in our data. The package plots the KEGG pathway to a png
file in the working directory.
library(pathview)
Loading required package: org.Hs.eg.db
##############################################################################
Pathview is an open source software package distributed under GNU General
Public License version 3 (GPLv3). Details of GPLv3 is available at
http://www.gnu.org/licenses/gpl-3.0.html. Particullary, users are required to
formally cite the original Pathview paper (not just mention it) in publications
or products. For details, do citation("pathview") within R.
The pathview downloads and uses KEGG data. Non-academic uses may require a KEGG
license agreement (details at http://www.kegg.jp/kegg/legal.html).
##############################################################################
logFC <- annotLvV$logFC
names(logFC) <- annotLvV$Entrez
pathview(gene.data = logFC,
pathway.id = "mmu03320",
species = "mmu",
limit = list(gene=5, cpd=1))
Info: Downloading xml files for mmu03320, 1/1 pathways..
Info: Downloading png files for mmu03320, 1/1 pathways..
'select()' returned 1:1 mapping between keys and columns
Info: Working in directory /Users/sawle01/Documents/RNAseq_course/CRUK_CI_RNAseq_course/Course_Materials
Info: Writing image file mmu03320.pathview.png
mmu03320.pathview.png:
Challenge 3
- Use
pathview
to export a figure for “mmu04060”, but this time only use genes that are statistically significant at FDR < 0.01
LS0tCnRpdGxlOiAiUk5BLXNlcSBhbmFseXNpcyBpbiBSIgphdXRob3I6ICJTdGVwaGFuZSBCYWxsZXJlYXUsIE1hcmsgRHVubmluZywgQWJiaSBFZHdhcmRzLCBPc2NhciBSdWVkYSwgQXNobGV5IFNhd2xlIgpkYXRlOiAnYHIgZm9ybWF0KFN5cy50aW1lKCksICJMYXN0IG1vZGlmaWVkOiAlZCAlYiAlWSIpYCcKb3V0cHV0OgogIGh0bWxfbm90ZWJvb2s6CiAgICB0b2M6IHllcwogIGh0bWxfZG9jdW1lbnQ6CiAgICB0b2M6IHllcwptaW51dGVzOiAzMDAKbGF5b3V0OiBwYWdlCnN1YnRpdGxlOiBHZW5lIFNldCBUZXN0aW5nIGZvciBSTkEtc2VxCmJpYmxpb2dyYXBoeTogcmVmLmJpYgplZGl0b3Jfb3B0aW9uczogCiAgY2h1bmtfb3V0cHV0X3R5cGU6IGlubGluZQotLS0KCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKYGBgCgpUaGUgbGlzdCBvZiBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgZ2VuZXMgaXMgc29tZXRpbWVzIHNvIGxvbmcgdGhhdCBpdHMgaW50ZXJwcmV0YXRpb24gYmVjb21lcyBjdW1iZXJzb21lIGFuZCB0aW1lIGNvbnN1bWluZy4gQSBjb21tb24gZG93bnN0cmVhbSBwcm9jZWR1cmUgaXMgZ2VuZSBzZXQgdGVzdGluZy4gSXQgYWltcyB0byB1bmRlcnN0YW5kIHdoaWNoIHBhdGh3YXlzIG9yIGdlbmUgbmV0d29ya3MgdGhlIGRpZmZlcmVudGlhbGx5IGV4cHJlc3NlZCBnZW5lcyBhcmUgaW1wbGljYXRlZCBpbi4KClZhcmlvdXMgd2F5cyBleGlzdCB0byB0ZXN0IGZvciBlbnJpY2htZW50IG9mIGJpb2xvZ2ljYWwgcGF0aHdheXMuCgojIEdTRUEgYW5hbHlzaXMKCkdlbmUgU2V0IEVucmljaG1lbnQgQW5hbHlzaXMgR1NFQSB3YXMgdGVzdHMgd2hldGhlciBhIHNldCBvZiBnZW5lcyBvZiBpbnRlcmVzdCwgCmUuZy4gZ2VuZXMKW0BTdWJyYW1hbmlhbjE1NTQ1XS4gVGhlIHNvZnR3YXJlIGlzIGRpc3RyaWJ1dGVkIGJ5IHRoZSBbQnJvYWQgCkluc3RpdHV0ZV0oaHR0cDovL3NvZnR3YXJlLmJyb2FkaW5zdGl0dXRlLm9yZy9nc2VhL2luZGV4LmpzcCkgYW5kIGlzIGZyZWVseSAKYXZhaWxhYmxlIGZvciB1c2UgYnkgYWNhZGVtaWMgYW5kIG5vbi1wcm9maXQgb3JnYW5pc2F0aW9ucy4gCgpJbiBhZGRpdGlvbiB0byB0aGUgR1NFQSBzb2Z0d2FyZSB0aGUgQnJvYWQgYWxzbyBwcm92aWRlIGEgbnVtYmVyIG9mIHZlcnkgd2VsbCAKY3VyYXRlZCBgZ2VuZSBzZXRgcyBmb3IgdGVzdGluZyBhZ2FpbnN0IHlvdXIgZGF0YSAtIHRoZSBbTW9sZWN1bGFyIFNpZ25hdHVyZXMgCkRhdGFiYXNlIChNU2lnREIpXShodHRwOi8vc29mdHdhcmUuYnJvYWRpbnN0aXR1dGUub3JnL2dzZWEvbXNpZ2RiL2luZGV4LmpzcCkuIApVbmZvcnR1bmF0ZWx5LCB0aGVzZSBhcmUgY29sbGVjdGlvbnMgb2YgaHVtYW4gZ2VuZXMsIGhvd2V2ZXIsIHRoZXNlIGxpc3RzCmhhdmUgYmVlbiB0cmFuc2xhdGVkIHRvIG1vdXNlIGVxdWl2YWxlbnRzIGJ5IHRoZSBXYWx0ZXIrRWxpemEgSGFsbCBJbnN0aXR1dGVzCkJpb2luZm9ybWF0aWNzIHNlcnZpY2UgYW5kIG1hZGUgYXZhaWFsYmxlIGZvciBbZG93bmxvYWRdKGh0dHA6Ly9iaW9pbmYud2VoaS5lZHUuYXUvc29mdHdhcmUvTVNpZ0RCLykuCgoKVGhlIGFuYWx5c2lzIGlzIHBlcmZvcm1lZCBieToKCi0gKGkpIHJhbmtpbmcgYWxsIGdlbmVzIGluIHRoZSBgZGF0YSBzZXRgCi0gKGlpKSBpZGVudGlmeWluZyB0aGUgcmFuayBwb3NpdGlvbnMgb2YgYWxsIG1lbWJlcnMgb2YgdGhlIGBnZW5lIHNldGAgaW4gdGhlIApyYW5rZWQgYGRhdGEgc2V0YAotIChpaWkpIGNhbGN1bGF0aW5nIGFuIGVucmljaG1lbnQgc2NvcmUgKEVTKSB0aGF0IHJlcHJlc2VudHMgdGhlIGRpZmZlcmVuY2UgCmJldHdlZW4gdGhlIG9ic2VydmVkIHJhbmtpbmdzIGFuZCB0aGF0IHdoaWNoIHdvdWxkIGJlIGV4cGVjdGVkIGFzc3VtaW5nIGEgcmFuZG9tIApyYW5rIGRpc3RyaWJ1dGlvbi4KCltjb21tZW50YXJ5IG9uIEdTRUFdKGh0dHBzOi8vd3d3Lm5jYmkubmxtLm5paC5nb3YvcG1jL2FydGljbGVzL1BNQzEyNjYxMzEvKS4gClRoZSBhcnRpY2xlIGRlc2NyaWJpbmcgdGhlIG9yaWdpbmFsIHNvZnR3YXJlIGlzIGF2YWlsYWJsZSAKW2hlcmVdKGh0dHA6Ly93d3cucG5hcy5vcmcvY29udGVudC8xMDIvNDMvMTU1NDUubG9uZykuCgojIyBgZmdzZWFgCgpUaGUgYGZnc2VhYCBwYWNrYWdlIFtAU2VyZ3VzaGljaGV2MjAxNl0gaW1wbGVtZW50cyB0aGUgc2FtZSBhbGdvcml0aG0gaW4gUiBbdmlnbmV0dGVdKGh0dHA6Ly93d3cuYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvdmlnbmV0dGVzL2Znc2VhL2luc3QvZG9jL2Znc2VhLXR1dG9yaWFsLmh0bWwpICJmYXN0IHByZXJhbmtlZCBnZW5lIHNldCBlbnJpY2htZW50IGFuYWx5c2lzIChHU0VBKSIuCgpgYGB7ciBmZ3NlYX0KbGlicmFyeShmZ3NlYSkKYGBgCgpgYGB7ciBsb2FkRGF0YX0KbG9hZCgiUm9iamVjdHMvQW5ub3RhdGVkX1Jlc3VsdHNfTHZWLlJEYXRhIikKYGBgCgojIyBDcmVhdGUgcmFua3MKClJhbmsgYWxsIGdlbmVzIGJhc2VkIG9uIHRoZWlyIGZvbGQgY2hhbmdlLiBXZSBuZWVkIHRvIGV4Y2x1ZGUgZ2VuZXMgZm9yCndoaWNoIHdlIGRvIG5vdCBoYXZlIEVudHJleiBJRHMuIEFsc28sIHdlIHNob3VsZCB1c2UgdGhlIHNocnVuayBMRkMgdmFsdWVzLgoKYGBge3IgcHJlcGFyZWRhdGF9CmdzZWFEYXQgPC0gZmlsdGVyKHNocmlua0x2ViwgIWlzLm5hKEVudHJleikpCgpyYW5rcyA8LSBnc2VhRGF0JGxvZ0ZDCm5hbWVzKHJhbmtzKSA8LSBnc2VhRGF0JEVudHJlegpoZWFkKHJhbmtzKQpgYGAKClBsb3QgdGhlIHJhbmtlZCBmb2xkIGNoYW5nZXMuCgpgYGB7cn0KYmFycGxvdChzb3J0KHJhbmtzLCBkZWNyZWFzaW5nID0gVCkpCmBgYAoKIyMgTG9hZCBwYXRod2F5cwoKYGBge3J9CmxvYWQoIlJvYmplY3RzL21vdXNlX0hfdjUuUkRhdGEiKQpwYXRod2F5c0ggPC0gTW0uSApgYGAKCiMjIENvbmR1Y3QgYW5hbHlzaXMKCmBgYHtyfQpmZ3NlYVJlcyA8LSBmZ3NlYShwYXRod2F5c0gsIHJhbmtzLCBtaW5TaXplPTE1LCBtYXhTaXplID0gNTAwLCBucGVybT0xMDAwKQpgYGAKClRoZSB3YXJuaW5nIHByb2R1Y2VkIGluZGljYXRlcyB0aGF0IHRoZXJlIGFyZSBmZXcgZ2VuZXMgdGhhdCBoYXZlIHRoZSBzYW1lIGZvbGQKY2hhbmdlIGFuZCBzbyBhcmUgcmFua2VkIGVxdWFsbHkuIGBmZ3NlYWAgd2l0aCBhcmJpdHJhcmlseSBvcmRlciBkZXRlcm1pbmUgd2hpY2gKY29tZXMgZmlyc3QgaW4gdGhlIHJhbmtlZCBsaXN0LiBBcyBsb25nIGFzIHRoaXMgbnVtYmVyIGlzIHNtYWxsIGl0IHNob3VsZG4ndApzaWduaWZpY2FudGx5IGVmZmVjdCB0aGUgcmVzdWx0cy4gSWYgdGhlIG51bWJlciBpcyBsYXJnZSBzb21ldGhpbmcgaXMgCnN1c3BpY2lvdXMgYWJvdXQgdGhlIGZvbGQgY2hhbmdlIHJlc3VsdHMuCgpMZXRzIGxvb2sgYXQgdGhlIHRvcCAxMCByZXN1bHRzLgoKYGBge3J9CmhlYWQoZmdzZWFSZXNbb3JkZXIocGFkaiwgLWFicyhORVMpKSwgXSwgbj0xMCkKYGBgCgojIyBFbnJpY2htZW50IHNjb3JlIHBsb3QKCmBgYHtyfQpwbG90RW5yaWNobWVudChwYXRod2F5c0hbWyJIQUxMTUFSS19FU1RST0dFTl9SRVNQT05TRV9FQVJMWSJdXSwgcmFua3MpCmBgYAoKUmVtZW1iZXIgdG8gY2hlY2sgdGhlIFtHU0VBIGFydGljbGVdKGh0dHA6Ly93d3cucG5hcy5vcmcvY29udGVudC8xMDIvNDMvMTU1NDUuZnVsbCkgZm9yIHRoZSBjb21wbGV0ZSBleHBsYW5hdGlvbi4KCiMjIEdTRUEgdGFibGUgcGxvdAoKVGhlIGZ1bmN0aW9uIGBwbG90R3NlYVRhYmxlYCBhbGxvd3MgdXMgdG8gcGxvdCBhIHN1bW1hcnkgZmlndWUgc2hvd2luZyB0aGUgCnJlc3VsdHMgZm9yIG11bHRpcGxlIHBhdGh3YXlzLgoKYGBge3J9CnRvcFVwIDwtIGZnc2VhUmVzICU+JSAKICAgIGZpbHRlcihFUyA+IDApICU+JSAKICAgIHRvcF9uKDEwLCB3dD0tcGFkaikKdG9wRG93biA8LSBmZ3NlYVJlcyAlPiUgCiAgICBmaWx0ZXIoRVMgPCAwKSAlPiUgCiAgICB0b3BfbigxMCwgd3Q9LXBhZGopCnRvcFBhdGh3YXlzIDwtIGJpbmRfcm93cyh0b3BVcCwgdG9wRG93bikgJT4lIAogICAgYXJyYW5nZSgtRVMpCgpwbG90R3NlYVRhYmxlKHBhdGh3YXlzSFt0b3BQYXRod2F5cyRwYXRod2F5XSwgCiAgICAgICAgICAgICAgcmFua3MsIAogICAgICAgICAgICAgIGZnc2VhUmVzLCAKICAgICAgICAgICAgICBnc2VhUGFyYW0gPSAwLjUpCmBgYAoKPiAjIyBDaGFsbGVuZ2UgMSB7LmNoYWxsZW5nZX0KPgo+IEFub3RoZXIgY29tbW9uIHdheSB0byByYW5rIHRoZSBnZW5lcyBpcyB0byBvcmRlciBieSBwdmFsdWUsIGJ1dCBhbHNvLCBzb3J0aW5nCj4gc28gdGhhdCB1cHJlZ3VsYXRlZCBnZW5lcyBhcmUgYXQgc3RhcnQgYW5kIGRvd25yZWd1bGF0ZWQgYXQgdGhlIG90aGVyIC0gCj4geW91IGNhbiBkbyB0aGlzIGNvbWJpbmluZyB0aGUgc2lnbiBvZiB0aGUgZm9sZCBjaGFuZ2UgYW5kIHRoZSBwdmFsdWUuICAKPiAxLiBSYW5rIHRoZSBnZW5lcyBieSBzdGF0aXNpY2FsIHNpZ25pZmljYW5jZSAtIHlvdSB3aWxsIG5lZWQgdG8gY3JlYXRlCj4gYSBuZXcgcmFua2luZyB2YWx1ZSB1c2luZyBgLWxvZzEwKHtwIHZhbHVlfSkgKiBzaWduKHtGb2xkIENoYW5nZX0pYCAgCj4gMi4gTG9hZCB0aGUgIkMyIiBwYXRod2F5cyBmcm9tIHRoZSB0aGUgYGRhdGEvbW91c2VfYzJfdjUuUkRhdGFgIGZpbGUgICAKPiAzLiBSdW4gYGZnc2VhYCB1c2luZyB0aGUgbmV3IHJhbmtlZCBnZW5lcyBhbmQgdGhlIEMyIHBhdGh3YXlzICAKPiA0LiBSdW4gYGZnc2VhYCB1c2luZyB0aGUgbmV3IHJhbmtlZCBnZW5lcyBhbmQgdGhlIEggcGF0aHdheXMuIEhvdyBkbyB0aGVzZSAKPiByZXN1bHRzIGRpZmZlciBmcm9tIHRoZSBvbmVzIHdlIGdvdCB3aGVuIHJhbmtpbmcgYnkgdGhlIGZvbGQgY2hhbmdlIGFsb25lPyAgCgpgYGB7ciBzb2x1dGlvbjF9CgpgYGAKCiMgR08gZW5yaWNobWVudCBhbmFseXNpcwoKIyMgYGdvc2VxYAoKR09zZXEgaXMgYSBtZXRob2QgdG8gY29uZHVjdCBHZW5lIE9udG9sb2d5IChHTykgYW5hbHlzaXMgc3VpdGFibGUgZm9yIFJOQS1zZXEgZGF0YSBhcyBpdCBhY2NvdW50cyBmb3IgdGhlIGdlbmUgbGVuZ3RoIGJpYXMgaW4gZGV0ZWN0aW9uIG9mIG92ZXItcmVwcmVzZW50YXRpb24gW1tAWW91bmcyMDEwXV0oaHR0cHM6Ly9nZW5vbWViaW9sb2d5LmJpb21lZGNlbnRyYWwuY29tL2FydGljbGVzLzEwLjExODYvZ2ItMjAxMC0xMS0yLXIxNCkKCkZyb20gdGhlIFtHT3NlcSB2aWduZXR0ZV0oaHR0cHM6Ly93d3cuYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvdmlnbmV0dGVzL2dvc2VxL2luc3QvZG9jL2dvc2VxLnBkZik6CgotIEdPc2VxIGZpcnN0IG5lZWRzIHRvIHF1YW50aWZ5IHRoZSBsZW5ndGggYmlhcyBwcmVzZW50IGluIHRoZSBkYXRhc2V0IHVuZGVyIGNvbnNpZGVyYXRpb24uCi0gVGhpcyBpcyBkb25lIGJ5IGNhbGN1bGF0aW5nIGEgUHJvYmFiaWxpdHkgV2VpZ2h0aW5nIEZ1bmN0aW9uIG9yIFBXRiB3aGljaCBjYW4gYmUgdGhvdWdodCBvZiBhcyBhIGZ1bmN0aW9uIHdoaWNoIGdpdmVzIHRoZSBwcm9iYWJpbGl0eSB0aGF0IGEgZ2VuZSB3aWxsIGJlIGRpZmZlcmVudGlhbGx5IGV4cHJlc3NlZCAoREUpLCBiYXNlZCBvbiBpdHMgbGVuZ3RoIGFsb25lLgotIFRoZSBQV0YgaXMgY2FsY3VsYXRlZCBieSBmaXR0aW5nIGEgbW9ub3RvbmljIHNwbGluZSB0byB0aGUgYmluYXJ5IGRhdGEgc2VyaWVzIG9mIGRpZmZlcmVudGlhbCBleHByZXNzaW9uICgxPURFLCAwPU5vdCBERSkgYXMgYSBmdW5jdGlvbiBvZiBnZW5lIGxlbmd0aC4KLSBUaGUgUFdGIGlzIHVzZWQgdG8gd2VpZ2h0IHRoZSBjaGFuY2Ugb2Ygc2VsZWN0aW5nIGVhY2ggZ2VuZSB3aGVuIGZvcm1pbmcgYSBudWxsIGRpc3RyaWJ1dGlvbiBmb3IgR08gY2F0ZWdvcnkgbWVtYmVyc2hpcC4KLSBUaGUgZmFjdCB0aGF0IHRoZSBQV0YgaXMgY2FsY3VsYXRlZCBkaXJlY3RseSBmcm9tIHRoZSBkYXRhc2V0IHVuZGVyIGNvbnNpZGVyYXRpb24gbWFrZXMgdGhpcyBhcHByb2FjaCByb2J1c3QsIG9ubHkgY29ycmVjdGluZyBmb3IgdGhlIGxlbmd0aCBiaWFzIHByZXNlbnQgaW4gdGhlIGRhdGEuCgoiR08gYW5hbHlzaXMgb2YgUk5BLXNlcSBkYXRhIHJlcXVpcmVzIHRoZSB1c2Ugb2YgcmFuZG9tIHNhbXBsaW5nIGluIG9yZGVyIHRvIGdlbmVyYXRlIGEgc3VpdGFibGUgbnVsbCBkaXN0cmlidXRpb24gZm9yIEdPIGNhdGVnb3J5IG1lbWJlcnNoaXAgYW5kIGNhbGN1bGF0ZSBlYWNoIGNhdGVnb3J5J3Mgc2lnbmlmaWNhbmNlIGZvciBvdmVyIHJlcHJlc2VudGF0aW9uIGFtb25nc3QgREUgZ2VuZXMuIC4uLiBJbiAgbW9zdCAgY2FzZXMsICB0aGUgIFdhbGxlbml1cyBkaXN0cmlidXRpb24gY2FuIGJlIHVzZWQgdG8gYXBwcm94aW1hdGUgdGhlIHRydWUgbnVsbCBkaXN0cmlidXRpb24sIHdpdGhvdXQgYW55IHNpZ25pZmljYW50IGxvc3MgaW4gYWNjdXJhY3kuIFRoZSBnb3NlcSBwYWNrYWdlIGltcGxlbWVudHMgdGhpcyBhcHByb3hpbWF0aW9uIGFzIGl0cyBkZWZhdWx0IG9wdGlvbi4iCgpgYGB7ciBnb1NlcVBhY2thZ2UsIG1lc3NhZ2U9RkFMU0V9CmxpYnJhcnkoZ29zZXEpCnN1cHBvcnRlZE9yZ2FuaXNtcygpICU+JSBmaWx0ZXIoc3RyX2RldGVjdChHZW5vbWUsICJtbSIpKQpgYGAKCiMjIENyZWF0ZSBhIGxpc3Qgb2YgZGlmZmVyZW50aWFsbHkgZXhwcmVzc2VkIGdlbmVzCgpgYGB7ciBnZXRERUdzfQppc1NpZ0dlbmUgPC0gc2hyaW5rTHZWJEZEUiA8IDAuMDEgJiAhaXMubmEoc2hyaW5rTHZWJEZEUikKZ2VuZXMgPC0gYXMuaW50ZWdlcihpc1NpZ0dlbmUpCm5hbWVzKGdlbmVzKSA8LSBzaHJpbmtMdlYkR2VuZUlECmBgYAoKIyMgRml0IHRoZSBQcm9iYWJpbGl0eSBXZWlnaHRpbmcgRnVuY3Rpb24gKFBXRikKCmBgYHtyIHB3RnVuY3Rpb259CnB3ZiA8LSBudWxscChnZW5lcywgIm1tMTAiLCAiZW5zR2VuZSIsIGJpYXMuZGF0YSA9IHNocmlua0x2ViRtZWRpYW5UeExlbmd0aCkKYGBgCgojIyBDb25kdWN0IEdPIGVucmljaG1lbnQgYW5hbHlzaXMKCmBgYHtyIHJ1bkdvc2VxLCBtZXNzYWdlPUZBTFNFfQpnb1Jlc3VsdHMgPC0gZ29zZXEocHdmLCAibW0xMCIsImVuc0dlbmUiLCB0ZXN0LmNhdHM9YygiR086QlAiKSkKYGBgCgojIyBQbG90IHRoZSB0b3AgMTAKCmBgYHtyIHBsb3RHT30KZ29SZXN1bHRzICU+JSAKICAgIHRvcF9uKDEwLCB3dD0tb3Zlcl9yZXByZXNlbnRlZF9wdmFsdWUpICU+JSAKICAgIG11dGF0ZShoaXRzUGVyYz1udW1ERUluQ2F0KjEwMC9udW1JbkNhdCkgJT4lIAogICAgZ2dwbG90KGFlcyh4PWhpdHNQZXJjLCAKICAgICAgICAgICAgICAgeT10ZXJtLCAKICAgICAgICAgICAgICAgY29sb3VyPW92ZXJfcmVwcmVzZW50ZWRfcHZhbHVlLCAKICAgICAgICAgICAgICAgc2l6ZT1udW1ERUluQ2F0KSkgKwogICAgICAgIGdlb21fcG9pbnQoKSArCiAgICAgICAgZXhwYW5kX2xpbWl0cyh4PTApICsKICAgICAgICBsYWJzKHg9IkhpdHMgKCUpIiwgeT0iR08gdGVybSIsIGNvbG91cj0icCB2YWx1ZSIsIHNpemU9IkNvdW50IikKYGBgCgojIyBHZXQgdGhlIEdPIGluZm9ybWF0aW9uIGZvciB0aGUgR08gYWNjZXNzaW9ucwoKYGBge3IgZ2V0R09pbmZvfQpsaWJyYXJ5KEdPLmRiKQpHT1RFUk1bW2dvUmVzdWx0cyRjYXRlZ29yeVsxXV1dCmBgYAoKPiAjIyBDaGFsbGVuZ2UgMiB7LmNoYWxsZW5nZX0KPgo+IDEuIENyZWF0ZSBhIGxpc3Qgb2YgZ2VuZXMgdGhhdCBhcmUgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudCBhdCBGRFIgPCAwLjAxIGFuZAo+IGFyZSB1cC1yZWd1bGF0ZWQgYnkgYXQgbGVhc3QgNHggKGxvZ0ZDPjIpCj4gaW4gbGFjdGF0aW5nIG1pY2UKPiAyLiBSdW4gYSBgZ29zZXFgIGFuYWx5c2lzIG9uIHRoaXMgZ2VuZSBsaXN0Cj4gMy4gUGxvdCB0aGUgcmVzdWx0cwo+IDQuIEhvdyBpcyB0aGlzIHJlc3VsdCBkaWZmZXJlbnQgdG8gdGhlIHByZXZpb3VzIEdPIGFuYWx5c2lzPwoKYGBge3Igc29sdXRpb24yLCBldmFsPUZ9CgpgYGAKCiMgS0VHRyBwYXRod2F5IGVucmljaG1lbnQgYW5hbHlzaXMKCiMjIGBjbHVzdGVyUHJvZmlsZXJgCgpXZSBjYW4gYW5hbHlzZSBmb3IgZW5yaWNobWVudCBvZiBLRUdHIHBhdGh3YXlzIGluIG11Y2ggdGhlIHNhbWUgd2F5IGFzIGZvciBHTyAKdGVybXMuIFdlIGNvdWxkIGFsc28gdXNlIGBnb3NlcWAgZm9yIHRoaXMsIGJ1dCB0aGlzIHRpbWUgd2UncmUgZ29pbmcgdG8gdXNlIApgY2x1c3RlclByb2ZpbGVyYCBbQFl1MjAxMl0uIGBjbHVzdGVycHJvZmlsZXJgIHN1cHBvcnRzIGRpcmVjdCBvbmxpbmUgYWNjZXNzIG9mCnRoZSBjdXJyZW50IEtFR0cgZGF0YWJhc2UsIHJhdGhlciB0aGFuIHJlbHlpbmcgb24gUiBhbm5vdGF0aW9uIHBhY2thZ2VzLCBpdAphbHNvIHByb3ZpZGVzIHNvbWUgbmljZSB2aXN1YWxpc2F0aW9uIG9wdGlvbnMuCgpgYGB7ciBsb2FkQ2x1c3RlclByb2ZpbGVyLCBtZXNzYWdlPUZBTFNFfQpsaWJyYXJ5KGNsdXN0ZXJQcm9maWxlcikKc2VhcmNoX2tlZ2dfb3JnYW5pc20oJ21tdScsIGJ5PSdrZWdnX2NvZGUnKQpgYGAKCiMjIEtFR0cgZW5yaWNobWVudCBhbmFseXNpcwoKYGBge3IgZW5yaWNoS0VHR30Kc2lnR2VuZXMgPC0gc2hyaW5rTHZWJEVudHJlelsgc2hyaW5rTHZWJEZEUiA8IDAuMDEgJiAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIWlzLm5hKHNocmlua0x2ViRGRFIpICYKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYWJzKHNocmlua0x2ViRsb2dGQykgPiAxIF0Kc2lnR2VuZXMgPC0gbmEuZXhjbHVkZShzaWdHZW5lcykKa2sgPC0gZW5yaWNoS0VHRyhnZW5lID0gc2lnR2VuZXMsIG9yZ2FuaXNtID0gJ21tdScpCmhlYWQoa2ssIG49MTApCmBgYAoKIyMgVmlzdWFsaXNlIGEgcGF0aHdheQoKIyMjIEluIGEgYnJvd3NlcgoKYGNsdXN0ZXJQcm9maWxlYCBoYXMgYSBmdW5jdGlvbiBgYnJvd3NlS2VnZ2AgdGhhdCBhbGxvd3MgeW91IHRvIHZpZXcgdGhlIEtlZ2cKcGF0aHdheSB3aXRoIHRoZSBnZW5lcyBjb2xvdXJzIGluIGluIHlvdXIgYnJvd3Nlci4KCmBgYHtyIGJyb3dzZUtlZ2d9CmJyb3dzZUtFR0coa2ssICdtbXUwMzMyMCcpCmBgYAoKIyMjIEFzIGEgZmlsZQoKVGhlIHBhY2thZ2UgYHBhdGh2aWV3YCBbQEx1bzIwMTNdIGNhbiBiZSB1c2VkIHRvIGdlbmVyYXRlIGZpZ3VyZXMgb2YgS0VHRyAKcGF0aHdheXMuIE9uZSBhZHZhbnRhZ2Ugb3ZlciBgY2x1c3RlclByb2ZpbGVzYCBicm93c2VyIG1ldGhvcyBpcyB0aGF0IHRoZSBnZW5lcwphcmUgY29sb3VyZWQgYWNjb3JkaW5nIHRvIHRoZWlyIGV4cHJlc3Npb24gbGV2ZWxzIGluIG91ciBkYXRhLiBUaGUgcGFja2FnZSBwbG90cwp0aGUgS0VHRyBwYXRod2F5IHRvIGEgYHBuZ2AgZmlsZSBpbiB0aGUgd29ya2luZyBkaXJlY3RvcnkuCgpgYGB7ciBwYXRodmlld30KbGlicmFyeShwYXRodmlldykKCmxvZ0ZDIDwtIGFubm90THZWJGxvZ0ZDCm5hbWVzKGxvZ0ZDKSA8LSBhbm5vdEx2ViRFbnRyZXoKcGF0aHZpZXcoZ2VuZS5kYXRhID0gbG9nRkMsIAogICAgICAgICBwYXRod2F5LmlkID0gIm1tdTAzMzIwIiwgCiAgICAgICAgIHNwZWNpZXMgPSAibW11IiwgCiAgICAgICAgIGxpbWl0ID0gbGlzdChnZW5lPTUsIGNwZD0xKSkKYGBgCgptbXUwMzMyMC5wYXRodmlldy5wbmc6CgohW21tdTAzMzIwIC0gUFBBUiBzaWduYWxpbmcgcGF0aHdheV0oLi4vaW1hZ2VzL21tdTAzMzIwLnBhdGh2aWV3LnBuZykKCj4gIyMgQ2hhbGxlbmdlIDMgey5jaGFsbGVuZ2V9Cj4KPiAxLiBVc2UgYHBhdGh2aWV3YCB0byBleHBvcnQgYSBmaWd1cmUgZm9yICJtbXUwNDA2MCIsIGJ1dCB0aGlzIHRpbWUgb25seQo+IHVzZSBnZW5lcyB0aGF0IGFyZSBzdGF0aXN0aWNhbGx5IHNpZ25pZmljYW50IGF0IEZEUiA8IDAuMDEKCmBgYHtyIHNvbHV0aW9uMywgZXZhbD1GfQoKYGBgCgotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KCiMgUmVmZXJlbmNlcw==