The list of differentially expressed genes is sometimes so long that its interpretation becomes cumbersome and time consuming. A common downstream procedure is gene set testing. It aims to understand which pathways or gene networks the differentially expressed genes are implicated in.

Various ways exist to test for enrichment of biological pathways.

GSEA analysis

Gene Set Enrichment Analysis GSEA was tests whether a set of genes of interest, e.g. genes (Subramanian et al. 2005). The software is distributed by the Broad Institute and is freely available for use by academic and non-profit organisations.

In addition to the GSEA software the Broad also provide a number of very well curated gene sets for testing against your data - the Molecular Signatures Database (MSigDB). Unfortunately, these are collections of human genes, however, these lists have been translated to mouse equivalents by the Walter+Eliza Hall Institutes Bioinformatics service and made avaialble for download.

The analysis is performed by:

commentary on GSEA. The article describing the original software is available here.

fgsea

The fgsea package (Sergushichev 2016) implements the same algorithm in R vignette “fast preranked gene set enrichment analysis (GSEA)”.

library(fgsea)
load("Robjects/Annotated_Results_LvV.RData")

Ranking Data

We need to provide fgsea a vector containing numeric data by which it should rank the genes. To start with we will simply use a rank based on their fold change. We do not need to rank the genes ourselves, fgsea will do this based on the data we provide.

We need to exclude genes for which we do not have Entrez IDs. Also, we should use the shrunk LFC values.

gseaDat <- filter(shrinkLvV, !is.na(Entrez))
rankData <- gseaDat$logFC
names(rankData) <- gseaDat$Entrez
head(rankData)
    497097      19888      20671      27395      18777      21399 
 0.5851914 -1.0498457 -0.5893634  0.4313540  0.6478714 -0.1110589 

Load pathways

load("Robjects/mouse_H_v5.RData")
pathwaysH <- Mm.H

Conduct analysis

fgseaRes <- fgsea(pathwaysH, 
                  rankData, 
                  minSize=15, 
                  maxSize = 500, 
                  nperm=1000)
There are ties in the preranked stats (0.05% of the list).
The order of those tied genes will be arbitrary, which may produce unexpected results.

The warning produced indicates that there are few genes that have the same fold change and so are ranked equally. fgsea with arbitrarily order determine which comes first in the ranked list. As long as this number is small it shouldn’t significantly effect the results. If the number is large something is suspicious about the fold change results.

Lets look at the top 10 results.

fgseaRes %>% 
    arrange(desc(abs(NES))) %>% 
    top_n(10, -padj)

Enrichment score plot

plotEnrichment(pathwaysH[["HALLMARK_ESTROGEN_RESPONSE_EARLY"]], rankData)

Remember to check the GSEA article for the complete explanation.

GSEA table plot

The function plotGseaTable allows us to plot a summary figue showing the results for multiple pathways.

topPathways <- fgseaRes %>% 
    top_n(20, wt=-padj) %>% 
    arrange(-NES) %>% 
    pull(pathway)
plotGseaTable(pathwaysH[topPathways], 
              rankData, 
              fgseaRes, 
              gseaParam = 0.5)

Challenge 1

Another common way to rank the genes is to order by pvalue, but also, sorting so that upregulated genes are at start and downregulated at the other - you can do this combining the sign of the fold change and the pvalue.
1. Rank the genes by statisical significance - you will need to create a new ranking value using -log10({p value}) * sign({Fold Change})
2. Load the “C2” pathways from the the data/mouse_c2_v5.RData file
3. Run fgsea using the new ranked genes and the C2 pathways
4. Run fgsea using the new ranked genes and the H pathways. How do these results differ from the ones we got when ranking by the fold change alone?

GO enrichment analysis

goseq

GOseq is a method to conduct Gene Ontology (GO) analysis suitable for RNA-seq data as it accounts for the gene length bias in detection of over-representation (Young et al. 2010).

From the GOseq vignette:

  • GOseq first needs to quantify the length bias present in the dataset under consideration.
  • This is done by calculating a Probability Weighting Function or PWF which can be thought of as a function which gives the probability that a gene will be differentially expressed (DE), based on its length alone.
  • The PWF is calculated by fitting a monotonic spline to the binary data series of differential expression (1=DE, 0=Not DE) as a function of gene length.
  • The PWF is used to weight the chance of selecting each gene when forming a null distribution for GO category membership.
  • The fact that the PWF is calculated directly from the dataset under consideration makes this approach robust, only correcting for the length bias present in the data.

“GO analysis of RNA-seq data requires the use of random sampling in order to generate a suitable null distribution for GO category membership and calculate each category’s significance for over representation amongst DE genes. … In most cases, the Wallenius distribution can be used to approximate the true null distribution, without any significant loss in accuracy. The goseq package implements this approximation as its default option.”

library(goseq)
supportedOrganisms() %>% filter(str_detect(Genome, "mm"))

Create a list of differentially expressed genes

The input for goseq is a vector that indicates, for each gene, whether or not it is significantly differentially expressed. This should be a named vector, where the names are the gene ids and the values are 1 if the gene is significant and a 0 if it is not.

In this case we can use the Ensembl gene IDs.

sigData <- as.integer( shrinkLvV$FDR < 0.01 & !is.na(shrinkLvV$FDR) )
names(sigData) <- shrinkLvV$GeneID

Fit the Probability Weighting Function (PWF)

pwf <- nullp(sigData, "mm10", "ensGene", bias.data = shrinkLvV$medianTxLength)
initial point very close to some inequality constraints

Conduct GO enrichment analysis

goResults <- goseq(pwf, "mm10","ensGene", test.cats=c("GO:BP"))
package ‘AnnotationDbi’ was built under R version 3.5.1

Plot the top 10

goResults %>% 
    top_n(10, wt=-over_represented_pvalue) %>% 
    mutate(hitsPerc=numDEInCat*100/numInCat) %>% 
    ggplot(aes(x=hitsPerc, 
               y=term, 
               colour=over_represented_pvalue, 
               size=numDEInCat)) +
        geom_point() +
        expand_limits(x=0) +
        labs(x="Hits (%)", y="GO term", colour="p value", size="Count")

Get the GO information for the GO accessions

library(GO.db)
GOTERM[[goResults$category[1]]]
GOID: GO:0044281
Term: small molecule metabolic process
Ontology: BP
Definition: The chemical reactions and pathways involving small molecules,
    any low molecular weight, monomeric, non-encoded molecule.
Synonym: small molecule metabolism

Challenge 2

  1. Create a vector showing genes that are statistically significant at FDR < 0.01 and that are up-regulated by at least 4x (logFC>2) in lactating mice
  2. Run a goseq analysis on this gene list
  3. Plot the results
  4. How is this result different to the previous GO analysis?

KEGG pathway enrichment analysis

clusterProfiler

We can analyse for enrichment of KEGG pathways in much the same way as for GO terms. We could also use goseq for this, but this time we’re going to use clusterProfiler (Yu et al. 2012). clusterprofiler supports direct online access of the current KEGG database, rather than relying on R annotation packages, it also provides some nice visualisation options.

library(clusterProfiler)
package ‘clusterProfiler’ was built under R version 3.5.1
search_kegg_organism('mmu', by='kegg_code')

KEGG enrichment analysis

The input for the KEGG enrichment is list of gene IDs for significant genes.

This time we’ll only use gene with an absolute fold change greater than 2.

For this tool we need to use Entrez IDs, so once again we will have to eliminate the missing values.

sigGenes <- shrinkLvV %>% 
    filter(FDR < 0.05 & !is.na(FDR) & 
               abs(logFC) > 1 & 
               !is.na(Entrez)) %>% 
    pull(Entrez)
kk <- enrichKEGG(gene = sigGenes, organism = 'mmu')
head(kk, n=10)

Visualise a pathway

In a browser

clusterProfile has a function browseKegg that allows you to view the Kegg pathway with the genes colours in in your browser.

browseKEGG(kk, 'mmu03320')

As a file

The package pathview (Luo et al. 2013) can be used to generate figures of KEGG pathways.

One advantage over clusterProfiles browser method is that the genes are coloured according to their fold change levels in our data. To do this we need to pass pathview a named vector of fold change data (actually you could colour by any numeric vector, e.g. p-value).

The package plots the KEGG pathway to a png file in the working directory.

library(pathview)
package ‘pathview’ was built under R version 3.5.1Loading required package: org.Hs.eg.db

##############################################################################
Pathview is an open source software package distributed under GNU General
Public License version 3 (GPLv3). Details of GPLv3 is available at
http://www.gnu.org/licenses/gpl-3.0.html. Particullary, users are required to
formally cite the original Pathview paper (not just mention it) in publications
or products. For details, do citation("pathview") within R.

The pathview downloads and uses KEGG data. Non-academic uses may require a KEGG
license agreement (details at http://www.kegg.jp/kegg/legal.html).
##############################################################################
logFC <- annotLvV$logFC
names(logFC) <- annotLvV$Entrez
pathview(gene.data = logFC, 
         pathway.id = "mmu03320", 
         species = "mmu", 
         limit = list(gene=5, cpd=1))
Info: Downloading xml files for mmu03320, 1/1 pathways..
Info: Downloading png files for mmu03320, 1/1 pathways..
'select()' returned 1:1 mapping between keys and columns
Info: Working in directory /Users/sawle01/Documents/RNAseq_course/RNAseq_March_2019/Course_Materials
Info: Writing image file mmu03320.pathview.png

mmu03320.pathview.png:

mmu03320 - PPAR signaling pathway

mmu03320 - PPAR signaling pathway

Challenge 3

  1. Use pathview to export a figure for “mmu04060”, but this time only use genes that are statistically significant at FDR < 0.01

References

Luo, Weijun, Brouwer, and Cory. 2013. “Pathview: An R/Bioconductor Package for Pathway-Based Data Integration and Visualization.” Bioinformatics 29 (14): 1830–1. doi:10.1093/bioinformatics/btt285.

Sergushichev, Alexey. 2016. “An Algorithm for Fast Preranked Gene Set Enrichment Analysis Using Cumulative Statistic Calculation.” bioRxiv. Cold Spring Harbor Labs Journals. doi:10.1101/060012.

Subramanian, Aravind, Pablo Tamayo, Vamsi K. Mootha, Sayan Mukherjee, Benjamin L. Ebert, Michael A. Gillette, Amanda Paulovich, et al. 2005. “Gene Set Enrichment Analysis: A Knowledge-Based Approach for Interpreting Genome-Wide Expression Profiles.” Proceedings of the National Academy of Sciences 102 (43). National Academy of Sciences: 15545–50. doi:10.1073/pnas.0506580102.

Young, Matthew D, Matthew J Wakefield, Gordon K Smyth, and Alicia Oshlack. 2010. “Gene Ontology Analysis for Rna-Seq: Accounting for Selection Bias.” Genome Biology 11: R14.

Yu, Guangchuang, Li-Gen Wang, Yanyan Han, and Qing-Yu He. 2012. “ClusterProfiler: An R Package for Comparing Biological Themes Among Gene Clusters.” OMICS: A Journal of Integrative Biology 16 (5): 284–87. doi:10.1089/omi.2011.0118.

LS0tCnRpdGxlOiAiUk5BLXNlcSBhbmFseXNpcyBpbiBSIgphdXRob3I6ICJTdGVwaGFuZSBCYWxsZXJlYXUsIE1hcmsgRHVubmluZywgQWJiaSBFZHdhcmRzLCBPc2NhciBSdWVkYSwgQXNobGV5IFNhd2xlIgpkYXRlOiAnYHIgZm9ybWF0KFN5cy50aW1lKCksICJMYXN0IG1vZGlmaWVkOiAlZCAlYiAlWSIpYCcKb3V0cHV0OgogIGh0bWxfbm90ZWJvb2s6CiAgICB0b2M6IHllcwogIGh0bWxfZG9jdW1lbnQ6CiAgICB0b2M6IHllcwptaW51dGVzOiAzMDAKbGF5b3V0OiBwYWdlCnN1YnRpdGxlOiBHZW5lIFNldCBUZXN0aW5nIGZvciBSTkEtc2VxCmJpYmxpb2dyYXBoeTogcmVmLmJpYgplZGl0b3Jfb3B0aW9uczogCiAgY2h1bmtfb3V0cHV0X3R5cGU6IGlubGluZQotLS0KCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKYGBgCgpUaGUgbGlzdCBvZiBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgZ2VuZXMgaXMgc29tZXRpbWVzIHNvIGxvbmcgdGhhdCBpdHMgCmludGVycHJldGF0aW9uIGJlY29tZXMgY3VtYmVyc29tZSBhbmQgdGltZSBjb25zdW1pbmcuIEEgY29tbW9uIGRvd25zdHJlYW0gCnByb2NlZHVyZSBpcyBnZW5lIHNldCB0ZXN0aW5nLiBJdCBhaW1zIHRvIHVuZGVyc3RhbmQgd2hpY2ggcGF0aHdheXMgb3IgZ2VuZSAKbmV0d29ya3MgdGhlIGRpZmZlcmVudGlhbGx5IGV4cHJlc3NlZCBnZW5lcyBhcmUgaW1wbGljYXRlZCBpbi4KClZhcmlvdXMgd2F5cyBleGlzdCB0byB0ZXN0IGZvciBlbnJpY2htZW50IG9mIGJpb2xvZ2ljYWwgcGF0aHdheXMuCgojIEdTRUEgYW5hbHlzaXMKCkdlbmUgU2V0IEVucmljaG1lbnQgQW5hbHlzaXMgR1NFQSB3YXMgdGVzdHMgd2hldGhlciBhIHNldCBvZiBnZW5lcyBvZiBpbnRlcmVzdCwgCmUuZy4gZ2VuZXMKW0BTdWJyYW1hbmlhbjE1NTQ1XS4gVGhlIHNvZnR3YXJlIGlzIGRpc3RyaWJ1dGVkIGJ5IHRoZSBbQnJvYWQgCkluc3RpdHV0ZV0oaHR0cDovL3NvZnR3YXJlLmJyb2FkaW5zdGl0dXRlLm9yZy9nc2VhL2luZGV4LmpzcCkgYW5kIGlzIGZyZWVseSAKYXZhaWxhYmxlIGZvciB1c2UgYnkgYWNhZGVtaWMgYW5kIG5vbi1wcm9maXQgb3JnYW5pc2F0aW9ucy4gCgpJbiBhZGRpdGlvbiB0byB0aGUgR1NFQSBzb2Z0d2FyZSB0aGUgQnJvYWQgYWxzbyBwcm92aWRlIGEgbnVtYmVyIG9mIHZlcnkgd2VsbCAKY3VyYXRlZCBgZ2VuZSBzZXRgcyBmb3IgdGVzdGluZyBhZ2FpbnN0IHlvdXIgZGF0YSAtIHRoZSBbTW9sZWN1bGFyIFNpZ25hdHVyZXMgCkRhdGFiYXNlIChNU2lnREIpXShodHRwOi8vc29mdHdhcmUuYnJvYWRpbnN0aXR1dGUub3JnL2dzZWEvbXNpZ2RiL2luZGV4LmpzcCkuIApVbmZvcnR1bmF0ZWx5LCB0aGVzZSBhcmUgY29sbGVjdGlvbnMgb2YgaHVtYW4gZ2VuZXMsIGhvd2V2ZXIsIHRoZXNlIGxpc3RzCmhhdmUgYmVlbiB0cmFuc2xhdGVkIHRvIG1vdXNlIGVxdWl2YWxlbnRzIGJ5IHRoZSBXYWx0ZXIrRWxpemEgSGFsbCBJbnN0aXR1dGVzCkJpb2luZm9ybWF0aWNzIHNlcnZpY2UgYW5kIG1hZGUgYXZhaWFsYmxlIGZvciBbZG93bmxvYWRdKGh0dHA6Ly9iaW9pbmYud2VoaS5lZHUuYXUvc29mdHdhcmUvTVNpZ0RCLykuCgoKVGhlIGFuYWx5c2lzIGlzIHBlcmZvcm1lZCBieToKCi0gKGkpIHJhbmtpbmcgYWxsIGdlbmVzIGluIHRoZSBgZGF0YSBzZXRgCi0gKGlpKSBpZGVudGlmeWluZyB0aGUgcmFuayBwb3NpdGlvbnMgb2YgYWxsIG1lbWJlcnMgb2YgdGhlIGBnZW5lIHNldGAgaW4gdGhlIApyYW5rZWQgYGRhdGEgc2V0YAotIChpaWkpIGNhbGN1bGF0aW5nIGFuIGVucmljaG1lbnQgc2NvcmUgKEVTKSB0aGF0IHJlcHJlc2VudHMgdGhlIGRpZmZlcmVuY2UgCmJldHdlZW4gdGhlIG9ic2VydmVkIHJhbmtpbmdzIGFuZCB0aGF0IHdoaWNoIHdvdWxkIGJlIGV4cGVjdGVkIGFzc3VtaW5nIGEgcmFuZG9tIApyYW5rIGRpc3RyaWJ1dGlvbi4KCltjb21tZW50YXJ5IG9uIEdTRUFdKGh0dHBzOi8vd3d3Lm5jYmkubmxtLm5paC5nb3YvcG1jL2FydGljbGVzL1BNQzEyNjYxMzEvKS4gClRoZSBhcnRpY2xlIGRlc2NyaWJpbmcgdGhlIG9yaWdpbmFsIHNvZnR3YXJlIGlzIGF2YWlsYWJsZSAKW2hlcmVdKGh0dHA6Ly93d3cucG5hcy5vcmcvY29udGVudC8xMDIvNDMvMTU1NDUubG9uZykuCgojIyBgZmdzZWFgCgpUaGUgYGZnc2VhYCBwYWNrYWdlIFtAU2VyZ3VzaGljaGV2MjAxNl0gaW1wbGVtZW50cyB0aGUgc2FtZSBhbGdvcml0aG0gaW4gUiBbdmlnbmV0dGVdKGh0dHA6Ly93d3cuYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvdmlnbmV0dGVzL2Znc2VhL2luc3QvZG9jL2Znc2VhLXR1dG9yaWFsLmh0bWwpICJmYXN0IHByZXJhbmtlZCBnZW5lIHNldCBlbnJpY2htZW50IGFuYWx5c2lzIChHU0VBKSIuCgpgYGB7ciBmZ3NlYX0KbGlicmFyeShmZ3NlYSkKYGBgCgpgYGB7ciBsb2FkRGF0YX0KbG9hZCgiUm9iamVjdHMvQW5ub3RhdGVkX1Jlc3VsdHNfTHZWLlJEYXRhIikKYGBgCgojIyBSYW5raW5nIERhdGEKCldlIG5lZWQgdG8gcHJvdmlkZSBgZmdzZWFgIGEgdmVjdG9yIGNvbnRhaW5pbmcgbnVtZXJpYyBkYXRhIGJ5IHdoaWNoIGl0IHNob3VsZCAKcmFuayB0aGUgZ2VuZXMuIFRvIHN0YXJ0IHdpdGggd2Ugd2lsbCBzaW1wbHkgdXNlIGEgcmFuayBiYXNlZCBvbiB0aGVpciBmb2xkIApjaGFuZ2UuIFdlIGRvIG5vdCBuZWVkIHRvIHJhbmsgdGhlIGdlbmVzIG91cnNlbHZlcywgYGZnc2VhYCB3aWxsIGRvIHRoaXMgYmFzZWQKb24gdGhlIGRhdGEgd2UgcHJvdmlkZS4KCldlIG5lZWQgdG8gZXhjbHVkZSBnZW5lcyBmb3Igd2hpY2ggd2UgZG8gbm90IGhhdmUgRW50cmV6IElEcy4gQWxzbywgd2Ugc2hvdWxkIAp1c2UgdGhlIHNocnVuayBMRkMgdmFsdWVzLgoKYGBge3IgcHJlcGFyZWRhdGF9CmdzZWFEYXQgPC0gZmlsdGVyKHNocmlua0x2ViwgIWlzLm5hKEVudHJleikpCgpyYW5rRGF0YSA8LSBnc2VhRGF0JGxvZ0ZDCm5hbWVzKHJhbmtEYXRhKSA8LSBnc2VhRGF0JEVudHJlegpoZWFkKHJhbmtEYXRhKQpgYGAKCiMjIExvYWQgcGF0aHdheXMKCmBgYHtyIGxvYWRQYXRod2F5c30KbG9hZCgiUm9iamVjdHMvbW91c2VfSF92NS5SRGF0YSIpCnBhdGh3YXlzSCA8LSBNbS5ICmBgYAoKIyMgQ29uZHVjdCBhbmFseXNpcwoKYGBge3IgcnVuRmdzZWF9CmZnc2VhUmVzIDwtIGZnc2VhKHBhdGh3YXlzSCwgCiAgICAgICAgICAgICAgICAgIHJhbmtEYXRhLCAKICAgICAgICAgICAgICAgICAgbWluU2l6ZT0xNSwgCiAgICAgICAgICAgICAgICAgIG1heFNpemUgPSA1MDAsIAogICAgICAgICAgICAgICAgICBucGVybT0xMDAwKQpgYGAKClRoZSB3YXJuaW5nIHByb2R1Y2VkIGluZGljYXRlcyB0aGF0IHRoZXJlIGFyZSBmZXcgZ2VuZXMgdGhhdCBoYXZlIHRoZSBzYW1lIGZvbGQKY2hhbmdlIGFuZCBzbyBhcmUgcmFua2VkIGVxdWFsbHkuIGBmZ3NlYWAgd2l0aCBhcmJpdHJhcmlseSBvcmRlciBkZXRlcm1pbmUgd2hpY2gKY29tZXMgZmlyc3QgaW4gdGhlIHJhbmtlZCBsaXN0LiBBcyBsb25nIGFzIHRoaXMgbnVtYmVyIGlzIHNtYWxsIGl0IHNob3VsZG4ndApzaWduaWZpY2FudGx5IGVmZmVjdCB0aGUgcmVzdWx0cy4gSWYgdGhlIG51bWJlciBpcyBsYXJnZSBzb21ldGhpbmcgaXMgCnN1c3BpY2lvdXMgYWJvdXQgdGhlIGZvbGQgY2hhbmdlIHJlc3VsdHMuCgpMZXRzIGxvb2sgYXQgdGhlIHRvcCAxMCByZXN1bHRzLgoKYGBge3IgdG9wMTBwYXRod2F5c30KZmdzZWFSZXMgJT4lIAogICAgYXJyYW5nZShkZXNjKGFicyhORVMpKSkgJT4lIAogICAgdG9wX24oMTAsIC1wYWRqKQpgYGAKCiMjIEVucmljaG1lbnQgc2NvcmUgcGxvdAoKYGBge3IgZ3NlYUVucmljaG1lbnRQbG90fQpwbG90RW5yaWNobWVudChwYXRod2F5c0hbWyJIQUxMTUFSS19FU1RST0dFTl9SRVNQT05TRV9FQVJMWSJdXSwgcmFua0RhdGEpCmBgYAoKUmVtZW1iZXIgdG8gY2hlY2sgdGhlIFtHU0VBIAphcnRpY2xlXShodHRwOi8vd3d3LnBuYXMub3JnL2NvbnRlbnQvMTAyLzQzLzE1NTQ1LmZ1bGwpIGZvciB0aGUgY29tcGxldGUgCmV4cGxhbmF0aW9uLgoKIyMgR1NFQSB0YWJsZSBwbG90CgpUaGUgZnVuY3Rpb24gYHBsb3RHc2VhVGFibGVgIGFsbG93cyB1cyB0byBwbG90IGEgc3VtbWFyeSBmaWd1ZSBzaG93aW5nIHRoZSAKcmVzdWx0cyBmb3IgbXVsdGlwbGUgcGF0aHdheXMuCgpgYGB7ciBnc2VhVGFibGVQbG90fQp0b3BQYXRod2F5cyA8LSBmZ3NlYVJlcyAlPiUgCiAgICB0b3BfbigyMCwgd3Q9LXBhZGopICU+JSAKICAgIGFycmFuZ2UoLU5FUykgJT4lIAogICAgcHVsbChwYXRod2F5KQoKcGxvdEdzZWFUYWJsZShwYXRod2F5c0hbdG9wUGF0aHdheXNdLCAKICAgICAgICAgICAgICByYW5rRGF0YSwgCiAgICAgICAgICAgICAgZmdzZWFSZXMsIAogICAgICAgICAgICAgIGdzZWFQYXJhbSA9IDAuNSkKYGBgCgo+ICMjIENoYWxsZW5nZSAxIHsuY2hhbGxlbmdlfQo+Cj4gQW5vdGhlciBjb21tb24gd2F5IHRvIHJhbmsgdGhlIGdlbmVzIGlzIHRvIG9yZGVyIGJ5IHB2YWx1ZSwgYnV0IGFsc28sIHNvcnRpbmcKPiBzbyB0aGF0IHVwcmVndWxhdGVkIGdlbmVzIGFyZSBhdCBzdGFydCBhbmQgZG93bnJlZ3VsYXRlZCBhdCB0aGUgb3RoZXIgLSAKPiB5b3UgY2FuIGRvIHRoaXMgY29tYmluaW5nIHRoZSBzaWduIG9mIHRoZSBmb2xkIGNoYW5nZSBhbmQgdGhlIHB2YWx1ZS4gIAo+IDEuIFJhbmsgdGhlIGdlbmVzIGJ5IHN0YXRpc2ljYWwgc2lnbmlmaWNhbmNlIC0geW91IHdpbGwgbmVlZCB0byBjcmVhdGUKPiBhIG5ldyByYW5raW5nIHZhbHVlIHVzaW5nIGAtbG9nMTAoe3AgdmFsdWV9KSAqIHNpZ24oe0ZvbGQgQ2hhbmdlfSlgICAKPiAyLiBMb2FkIHRoZSAiQzIiIHBhdGh3YXlzIGZyb20gdGhlIHRoZSBgZGF0YS9tb3VzZV9jMl92NS5SRGF0YWAgZmlsZSAgIAo+IDMuIFJ1biBgZmdzZWFgIHVzaW5nIHRoZSBuZXcgcmFua2VkIGdlbmVzIGFuZCB0aGUgQzIgcGF0aHdheXMgIAo+IDQuIFJ1biBgZmdzZWFgIHVzaW5nIHRoZSBuZXcgcmFua2VkIGdlbmVzIGFuZCB0aGUgSCBwYXRod2F5cy4gSG93IGRvIHRoZXNlIAo+IHJlc3VsdHMgZGlmZmVyIGZyb20gdGhlIG9uZXMgd2UgZ290IHdoZW4gcmFua2luZyBieSB0aGUgZm9sZCBjaGFuZ2UgYWxvbmU/ICAKCmBgYHtyIHNvbHV0aW9uMX0KCmBgYAoKIyBHTyBlbnJpY2htZW50IGFuYWx5c2lzCgojIyBgZ29zZXFgCgpHT3NlcSBpcyBhIG1ldGhvZCB0byBjb25kdWN0IEdlbmUgT250b2xvZ3kgKEdPKSBhbmFseXNpcyBzdWl0YWJsZSBmb3IgUk5BLXNlcSAKZGF0YSBhcyBpdCBhY2NvdW50cyBmb3IgdGhlIGdlbmUgbGVuZ3RoIGJpYXMgaW4gZGV0ZWN0aW9uIG9mIG92ZXItcmVwcmVzZW50YXRpb24gW1tAWW91bmcyMDEwXV0oaHR0cHM6Ly9nZW5vbWViaW9sb2d5LmJpb21lZGNlbnRyYWwuY29tL2FydGljbGVzLzEwLjExODYvZ2ItMjAxMC0xMS0yLXIxNCkuCgpGcm9tIHRoZSBbR09zZXEgCnZpZ25ldHRlXShodHRwczovL3d3dy5iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL3JlbGVhc2UvYmlvYy92aWduZXR0ZXMvZ29zZXEvaW5zdC9kb2MvZ29zZXEucGRmKToKCi0gR09zZXEgZmlyc3QgbmVlZHMgdG8gcXVhbnRpZnkgdGhlIGxlbmd0aCBiaWFzIHByZXNlbnQgaW4gdGhlIGRhdGFzZXQgdW5kZXIKY29uc2lkZXJhdGlvbi4KLSBUaGlzIGlzIGRvbmUgYnkgY2FsY3VsYXRpbmcgYSBQcm9iYWJpbGl0eSBXZWlnaHRpbmcgRnVuY3Rpb24gb3IgUFdGIHdoaWNoIGNhbiAKYmUgdGhvdWdodCBvZiBhcyBhIGZ1bmN0aW9uIHdoaWNoIGdpdmVzIHRoZSBwcm9iYWJpbGl0eSB0aGF0IGEgZ2VuZSB3aWxsIGJlCmRpZmZlcmVudGlhbGx5IGV4cHJlc3NlZCAoREUpLCBiYXNlZCBvbiBpdHMgbGVuZ3RoIGFsb25lLgotIFRoZSBQV0YgaXMgY2FsY3VsYXRlZCBieSBmaXR0aW5nIGEgbW9ub3RvbmljIHNwbGluZSB0byB0aGUgYmluYXJ5IGRhdGEgc2VyaWVzIApvZiBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiAoMT1ERSwgMD1Ob3QgREUpIGFzIGEgZnVuY3Rpb24gb2YgZ2VuZSBsZW5ndGguCi0gVGhlIFBXRiBpcyB1c2VkIHRvIHdlaWdodCB0aGUgY2hhbmNlIG9mIHNlbGVjdGluZyBlYWNoIGdlbmUgd2hlbiBmb3JtaW5nIGEgCm51bGwgZGlzdHJpYnV0aW9uIGZvciBHTyBjYXRlZ29yeSBtZW1iZXJzaGlwLgotIFRoZSBmYWN0IHRoYXQgdGhlIFBXRiBpcyBjYWxjdWxhdGVkIGRpcmVjdGx5IGZyb20gdGhlIGRhdGFzZXQgdW5kZXIgCmNvbnNpZGVyYXRpb24gbWFrZXMgdGhpcyBhcHByb2FjaCByb2J1c3QsIG9ubHkgY29ycmVjdGluZyBmb3IgdGhlIGxlbmd0aCBiaWFzIApwcmVzZW50IGluIHRoZSBkYXRhLgoKIkdPIGFuYWx5c2lzIG9mIFJOQS1zZXEgZGF0YSByZXF1aXJlcyB0aGUgdXNlIG9mIHJhbmRvbSBzYW1wbGluZyBpbiBvcmRlciB0byAKZ2VuZXJhdGUgYSBzdWl0YWJsZSBudWxsIGRpc3RyaWJ1dGlvbiBmb3IgR08gY2F0ZWdvcnkgbWVtYmVyc2hpcCBhbmQgY2FsY3VsYXRlIAplYWNoIGNhdGVnb3J5J3Mgc2lnbmlmaWNhbmNlIGZvciBvdmVyIHJlcHJlc2VudGF0aW9uIGFtb25nc3QgREUgZ2VuZXMuIC4uLiBJbiAKbW9zdCAgY2FzZXMsICB0aGUgIFdhbGxlbml1cyBkaXN0cmlidXRpb24gY2FuIGJlIHVzZWQgdG8gYXBwcm94aW1hdGUgdGhlIHRydWUgCm51bGwgZGlzdHJpYnV0aW9uLCB3aXRob3V0IGFueSBzaWduaWZpY2FudCBsb3NzIGluIGFjY3VyYWN5LiBUaGUgZ29zZXEgcGFja2FnZSAKaW1wbGVtZW50cyB0aGlzIGFwcHJveGltYXRpb24gYXMgaXRzIGRlZmF1bHQgb3B0aW9uLiIKCmBgYHtyIGdvU2VxUGFja2FnZSwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KbGlicmFyeShnb3NlcSkKc3VwcG9ydGVkT3JnYW5pc21zKCkgJT4lIGZpbHRlcihzdHJfZGV0ZWN0KEdlbm9tZSwgIm1tIikpCmBgYAoKIyMgQ3JlYXRlIGEgbGlzdCBvZiBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgZ2VuZXMKClRoZSBpbnB1dCBmb3IgYGdvc2VxYCBpcyBhIHZlY3RvciB0aGF0IGluZGljYXRlcywgZm9yIGVhY2ggZ2VuZSwgd2hldGhlciBvciBub3QKaXQgaXMgc2lnbmlmaWNhbnRseSBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQuIFRoaXMgc2hvdWxkIGJlIGEgbmFtZWQgdmVjdG9yLCAKd2hlcmUgdGhlIG5hbWVzIGFyZSB0aGUgZ2VuZSBpZHMgYW5kIHRoZSB2YWx1ZXMgYXJlICpgMWAqIGlmIHRoZSBnZW5lIGlzIApzaWduaWZpY2FudCBhbmQgYSAqYDBgKiBpZiBpdCBpcyBub3QuCgpJbiB0aGlzIGNhc2Ugd2UgY2FuIHVzZSB0aGUgRW5zZW1ibCBnZW5lIElEcy4KCmBgYHtyIGdldERFR3N9CnNpZ0RhdGEgPC0gYXMuaW50ZWdlciggc2hyaW5rTHZWJEZEUiA8IDAuMDEgJiAhaXMubmEoc2hyaW5rTHZWJEZEUikgKQpuYW1lcyhzaWdEYXRhKSA8LSBzaHJpbmtMdlYkR2VuZUlECmBgYAoKIyMgRml0IHRoZSBQcm9iYWJpbGl0eSBXZWlnaHRpbmcgRnVuY3Rpb24gKFBXRikKCmBgYHtyIHB3RnVuY3Rpb259CnB3ZiA8LSBudWxscChzaWdEYXRhLCAibW0xMCIsICJlbnNHZW5lIiwgYmlhcy5kYXRhID0gc2hyaW5rTHZWJG1lZGlhblR4TGVuZ3RoKQpgYGAKCiMjIENvbmR1Y3QgR08gZW5yaWNobWVudCBhbmFseXNpcwoKYGBge3IgcnVuR29zZXEsIG1lc3NhZ2U9RkFMU0V9CmdvUmVzdWx0cyA8LSBnb3NlcShwd2YsICJtbTEwIiwiZW5zR2VuZSIsIHRlc3QuY2F0cz1jKCJHTzpCUCIpKQpgYGAKCiMjIFBsb3QgdGhlIHRvcCAxMAoKYGBge3IgcGxvdEdPfQpnb1Jlc3VsdHMgJT4lIAogICAgdG9wX24oMTAsIHd0PS1vdmVyX3JlcHJlc2VudGVkX3B2YWx1ZSkgJT4lIAogICAgbXV0YXRlKGhpdHNQZXJjPW51bURFSW5DYXQqMTAwL251bUluQ2F0KSAlPiUgCiAgICBnZ3Bsb3QoYWVzKHg9aGl0c1BlcmMsIAogICAgICAgICAgICAgICB5PXRlcm0sIAogICAgICAgICAgICAgICBjb2xvdXI9b3Zlcl9yZXByZXNlbnRlZF9wdmFsdWUsIAogICAgICAgICAgICAgICBzaXplPW51bURFSW5DYXQpKSArCiAgICAgICAgZ2VvbV9wb2ludCgpICsKICAgICAgICBleHBhbmRfbGltaXRzKHg9MCkgKwogICAgICAgIGxhYnMoeD0iSGl0cyAoJSkiLCB5PSJHTyB0ZXJtIiwgY29sb3VyPSJwIHZhbHVlIiwgc2l6ZT0iQ291bnQiKQpgYGAKCiMjIEdldCB0aGUgR08gaW5mb3JtYXRpb24gZm9yIHRoZSBHTyBhY2Nlc3Npb25zCgpgYGB7ciBnZXRHT2luZm99CmxpYnJhcnkoR08uZGIpCkdPVEVSTVtbZ29SZXN1bHRzJGNhdGVnb3J5WzFdXV0KYGBgCgo+ICMjIENoYWxsZW5nZSAyIHsuY2hhbGxlbmdlfQo+Cj4gMS4gQ3JlYXRlIGEgdmVjdG9yIHNob3dpbmcgZ2VuZXMgdGhhdCBhcmUgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudCBhdCAKPiBGRFIgPCAwLjAxIGFuZCB0aGF0IGFyZSB1cC1yZWd1bGF0ZWQgYnkgYXQgbGVhc3QgNHggKGxvZ0ZDPjIpCj4gaW4gbGFjdGF0aW5nIG1pY2UKPiAyLiBSdW4gYSBgZ29zZXFgIGFuYWx5c2lzIG9uIHRoaXMgZ2VuZSBsaXN0Cj4gMy4gUGxvdCB0aGUgcmVzdWx0cwo+IDQuIEhvdyBpcyB0aGlzIHJlc3VsdCBkaWZmZXJlbnQgdG8gdGhlIHByZXZpb3VzIEdPIGFuYWx5c2lzPwoKYGBge3Igc29sdXRpb24yLCBldmFsPUZ9CgpgYGAKCiMgS0VHRyBwYXRod2F5IGVucmljaG1lbnQgYW5hbHlzaXMKCiMjIGBjbHVzdGVyUHJvZmlsZXJgCgpXZSBjYW4gYW5hbHlzZSBmb3IgZW5yaWNobWVudCBvZiBLRUdHIHBhdGh3YXlzIGluIG11Y2ggdGhlIHNhbWUgd2F5IGFzIGZvciBHTyAKdGVybXMuIFdlIGNvdWxkIGFsc28gdXNlIGBnb3NlcWAgZm9yIHRoaXMsIGJ1dCB0aGlzIHRpbWUgd2UncmUgZ29pbmcgdG8gdXNlIApgY2x1c3RlclByb2ZpbGVyYCBbQFl1MjAxMl0uIGBjbHVzdGVycHJvZmlsZXJgIHN1cHBvcnRzIGRpcmVjdCBvbmxpbmUgYWNjZXNzIG9mCnRoZSBjdXJyZW50IEtFR0cgZGF0YWJhc2UsIHJhdGhlciB0aGFuIHJlbHlpbmcgb24gUiBhbm5vdGF0aW9uIHBhY2thZ2VzLCBpdAphbHNvIHByb3ZpZGVzIHNvbWUgbmljZSB2aXN1YWxpc2F0aW9uIG9wdGlvbnMuCgpgYGB7ciBsb2FkQ2x1c3RlclByb2ZpbGVyLCBtZXNzYWdlPUZBTFNFfQpsaWJyYXJ5KGNsdXN0ZXJQcm9maWxlcikKc2VhcmNoX2tlZ2dfb3JnYW5pc20oJ21tdScsIGJ5PSdrZWdnX2NvZGUnKQpgYGAKCiMjIEtFR0cgZW5yaWNobWVudCBhbmFseXNpcwoKVGhlIGlucHV0IGZvciB0aGUgS0VHRyBlbnJpY2htZW50IGlzIGxpc3Qgb2YgZ2VuZSBJRHMgZm9yIHNpZ25pZmljYW50IGdlbmVzLiAKClRoaXMgdGltZSB3ZSdsbCBvbmx5IHVzZSBnZW5lIHdpdGggYW4gYWJzb2x1dGUgZm9sZCBjaGFuZ2UgZ3JlYXRlciB0aGFuIDIuCgpGb3IgdGhpcyB0b29sIHdlIG5lZWQgdG8gdXNlIEVudHJleiBJRHMsIHNvIG9uY2UgYWdhaW4gd2Ugd2lsbCBoYXZlIHRvIGVsaW1pbmF0ZQp0aGUgbWlzc2luZyB2YWx1ZXMuCgpgYGB7ciBlbnJpY2hLRUdHfQpzaWdHZW5lcyA8LSBzaHJpbmtMdlYgJT4lIAogICAgZmlsdGVyKEZEUiA8IDAuMDUgJiAhaXMubmEoRkRSKSAmIAogICAgICAgICAgICAgICBhYnMobG9nRkMpID4gMSAmIAogICAgICAgICAgICAgICAhaXMubmEoRW50cmV6KSkgJT4lIAogICAgcHVsbChFbnRyZXopCgprayA8LSBlbnJpY2hLRUdHKGdlbmUgPSBzaWdHZW5lcywgb3JnYW5pc20gPSAnbW11JykKaGVhZChraywgbj0xMCkKYGBgCgojIyBWaXN1YWxpc2UgYSBwYXRod2F5CgojIyMgSW4gYSBicm93c2VyCgpgY2x1c3RlclByb2ZpbGVgIGhhcyBhIGZ1bmN0aW9uIGBicm93c2VLZWdnYCB0aGF0IGFsbG93cyB5b3UgdG8gdmlldyB0aGUgS2VnZwpwYXRod2F5IHdpdGggdGhlIGdlbmVzIGNvbG91cnMgaW4gaW4geW91ciBicm93c2VyLgoKYGBge3IgYnJvd3NlS2VnZ30KYnJvd3NlS0VHRyhraywgJ21tdTAzMzIwJykKYGBgCgojIyMgQXMgYSBmaWxlCgpUaGUgcGFja2FnZSBgcGF0aHZpZXdgIFtATHVvMjAxM10gY2FuIGJlIHVzZWQgdG8gZ2VuZXJhdGUgZmlndXJlcyBvZiBLRUdHIApwYXRod2F5cy4gCgpPbmUgYWR2YW50YWdlIG92ZXIgYGNsdXN0ZXJQcm9maWxlc2AgYnJvd3NlciBtZXRob2QgaXMgdGhhdCB0aGUgZ2VuZXMgYXJlIApjb2xvdXJlZCBhY2NvcmRpbmcgdG8gdGhlaXIgZm9sZCBjaGFuZ2UgbGV2ZWxzIGluIG91ciBkYXRhLiBUbyBkbyB0aGlzIHdlIG5lZWQKdG8gcGFzcyBgcGF0aHZpZXdgIGEgbmFtZWQgdmVjdG9yIG9mIGZvbGQgY2hhbmdlIGRhdGEgKGFjdHVhbGx5IHlvdSBjb3VsZApjb2xvdXIgYnkgYW55IG51bWVyaWMgdmVjdG9yLCBlLmcuIHAtdmFsdWUpLgoKVGhlIHBhY2thZ2UgcGxvdHMgdGhlIEtFR0cgcGF0aHdheSB0byBhIGBwbmdgIGZpbGUgaW4gdGhlIHdvcmtpbmcgZGlyZWN0b3J5LgoKYGBge3IgcGF0aHZpZXcsIG1lc3NhZ2U9Rn0KbGlicmFyeShwYXRodmlldykKCmxvZ0ZDIDwtIGFubm90THZWJGxvZ0ZDCm5hbWVzKGxvZ0ZDKSA8LSBhbm5vdEx2ViRFbnRyZXoKCnBhdGh2aWV3KGdlbmUuZGF0YSA9IGxvZ0ZDLCAKICAgICAgICAgcGF0aHdheS5pZCA9ICJtbXUwMzMyMCIsIAogICAgICAgICBzcGVjaWVzID0gIm1tdSIsIAogICAgICAgICBsaW1pdCA9IGxpc3QoZ2VuZT01LCBjcGQ9MSkpCmBgYAoKbW11MDMzMjAucGF0aHZpZXcucG5nOgoKIVttbXUwMzMyMCAtIFBQQVIgc2lnbmFsaW5nIHBhdGh3YXldKC4uL2ltYWdlcy9tbXUwMzMyMC5wYXRodmlldy5wbmcpCgo+ICMjIENoYWxsZW5nZSAzIHsuY2hhbGxlbmdlfQo+Cj4gMS4gVXNlIGBwYXRodmlld2AgdG8gZXhwb3J0IGEgZmlndXJlIGZvciAibW11MDQwNjAiLCBidXQgdGhpcyB0aW1lIG9ubHkKPiB1c2UgZ2VuZXMgdGhhdCBhcmUgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudCBhdCBGRFIgPCAwLjAxCgpgYGB7ciBzb2x1dGlvbjMsIGV2YWw9Rn0KCmBgYAoKLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCgojIFJlZmVyZW5jZXM=