filtering.qmd

---
title: "Filtering in OpenCRAVAT"
format: 
  rst: default
  html: default
---

## Why Filter?

Filtering your annotated variants lets you query interesting subsets of your variants. 

A secondary purpose of filtering is when you want to view your results in the results viewer and you have more than 100K variants. 

## Learning Objectives

```{mermaid}
flowchart TD
  A["Results Viewer"] --> B
  B["Build Filter in GUI"] 
  B --> F["Explore Filtered Data"]
  F --> G["Save JSON filter"]
  G --> D["Apply filter to New Set"]
  G --> E["Apply Filter on CLI"]
```

After reading this vignette, you should be able to:

- **Explain** how to build simple filters using the OpenCRAVAT GUI
- **Compose** multiple filters using boolean logic to make complex filtering
- **Save** your filter and **export** it as a JSON file
- **Apply** your filter to a new set of variants

## Filtering in the GUI

Once you have your annotated results, you can filter variants in the results viewer. There are 4 kinds of filters:

- Samples
- Genes
- Sample Properties
- Variant Properties

We will go through each of these filters and their functionality after we talk about the common GUI elements of each.

## Sample Data for Demonstration Purposes

We will use a subset of a synthetic dataset that is a trio. There are 5 individuals in this dataset.

1. The Proband
2. Mother
3. Father
4. Sibling 1
5. Sibling 2

Some of the variants are shared by the Proband and Mother, and all other combinations. We will leverage these different combinations in our example below.

### Downloading the Sample Data

You can download the sample result file here: [Synthetic Trio Dataset](https://github.com/laderast/oc_vignettes/raw/main/data/Session2.filtered.sqlite)

On the command line, you can download it into your current directory with `wget`:

```bash
wget https://github.com/laderast/oc_vignettes/raw/main/data/Session2.filtered.sqlite
```

When you're ready, open up the file with `oc gui`:

```bash
oc gui Session2.filtered.sqlite
```


### Samples

You can remove variants associated with a set of sample IDs by clicking on the checkboxes here. For example, we want all of the variants that are associated with the mother. We'll click on the "mother" checkbox until it is a green check.

![](images/mother-filter1.png)
If we look at the Variant tab, we can see that our variants include multiple members:

![](images/mother-all-result.png)
Here's a visual summary of the results.

```{mermaid}
graph TD
  A["All Variants\n(n=1,738)"] --"Sample:mother"--> B
  B["Mother Variants\n(n=1187)"]
```

These are all of the variants that are associated with the mother, and include variants that are shared by other family members. What about variants that are *exclusive* to the mother?

For example, we can filter for 

To exclude samples, click on the checkbox until you see a red x.

![](images/mother-exclusive-filter.png)
After filtering, we find that 385 variants are exclusive to the mother. We can confirm this by going to the **Variant** tab:

![](images/mother-exclusive-result.png)
Here's the visual summary:

```{mermaid}
graph TD
  A["All Variants\n(n=1,738)"] --"Sample:mother\nExclude:others"--> B
  B["Mother Variants\n(n=385)"]
```

### Genes

Gene-level filtering can be done here. You can input a list of genes, separated by line-breaks. 

Clearing our previous filter, let's filter to only those variants that are in BRCA1 and BRCA2.

To filter, we take the following steps in the **Gene** filter section:

1. Enter a list of genes, 1 gene per line in the text box. You can also upload a list of genes as well.
2. Apply the filter
3. See the results. There are 63 variants associated with BRCA1/BRCA2.

![](images/gene-filter.png)
Going to the **Variant** tab, we can confirm that the variants belong to BRCA1 and BRCA2.

![](images/gene-filter-result.png)
Here's the visual summary of the filter:

```{mermaid}
graph TD
  A["All Variants\n(n=1,738)"] --"Gene:(BRCA1,BRCA2)"--> B
  B["BRCA1/BRCA2 Genes\n(n=63)"]
```

### Sample Properties

### Variant Properties

In variant properties, you can filter by variant type based on your annotations. For example, let's filter our dataset to missense variants. 

A. Under **Variant Properties**:

1. Click on "Query Builder".
2. Mouse into the left bottom corner of the query builder window, and click the "+" sign.

![](images/variant-filter.png)

B. Make your filter by selecting the values for the following dropdown boxes:

1. Variant Annotation
2. Sequence Ontology
3. One of
4. Missense checkbox

![](images/variant-filter2.png)
C. Finally, click the "Apply Filter" Button:

![](images/variant-apply-filter.png)

You will be left with 299 variants. Here's a visual summary of what we did:

```{mermaid}
graph TD
  A["All Variants\n(n=1,738)"] --"Sequence Ontology:\nMissense"--> B["Missense Variants\n(n=299)"]
```

## Advanced: Boolean Operations

You can build more sophisticated operations by combining each filter step using Boolean logic.

### Adding Another Filter

You can add another filter by clicking the **+** underneath your filter.

![](images/add-filter.png)

### AND Logic

By default, the filters are combined using **AND** logic, which are more restrictive, because they require variants to meet both filters. 

Here's an example of using **AND** logic. Here we are combining two filters: Missense Variants (from **Variant Annotation >> Sequence Ontology**) and Pathogenic variants (from **ClinVar >> Clinical Significance**).

![](images/and-filter.png)
Then do the following:

1. Select "ClinVar"
2. Select "Clinical Significance"
3. Select "One of"
4. Select "Pathogenic"
5. Click "Filter Result"

![](images/add-filter2.png )
If we look at the **Variant** tab, we can see that our combined filter selected both Pathogenic and Missense Variants. 

![](images/add-filter-result.png)

When we apply the filter, we get 9 variants that meet both criteria. Here's a visual summary of the filtering: 

```{mermaid}
graph TD
  A["All Variants\n(n=1,738)"] --"Sequence Ontology:\nmissense"--> B["Missense Variants\(n=299)"]
  A --"Clinical Significance:\nPathogenic"--> C["Pathogenic Variants\n(n=10)"]
  B --"AND"--> E["Pathogenic AND Missense\n(n=9)"]
  C --"AND"--> E
```

### OR Logic

These filters can also be combined using **OR** logic, which is more permissive (that is, these filters will return a greater number than the **AND** logic) we might want variants that are either missense **OR** pathogenic. 

We can do this by clicking the "and" that links our two filters, which will switch it to an "or":

![](images/or-filter.png)
When we apply the filter, we get 300 variants. The breakdown is below. 

```{mermaid}
graph TD
  A["All Variants\n(n=1,738)"] --"Sequence Ontology:\nmissense"--> B["Missense Variants\n(n=299)"]
  A --"Clinical Significance:\nPathogenic"--> C["Pathogenic Variants\n(n=10)"]
  B --"OR"--> E["Pathogenic OR Missense\n(n=300)"]
  C --"OR"--> E
```

### Grouping With Parentheses

We can use the parentheses to group filters together. This can be helpful when we combine different types of logic.

To make a grouped filter, click the **(** button next to the **+** button in the interface:

![](images/parentheses-filter1.png)

Then you can build a set of filters much like above. Notice that they are grouped within a set of parentheses.

![](images/parentheses-filter2.png)
Keep in mind that you can nest parentheses within parentheses to make even more complex filters. We'll use this to be able to combine three filters with **AND** and **OR** logic below.

### Combining **AND** / **OR** Logic

By default, when you click the "and" / "or" of one set of filters, all filters will be changed. If you want to combine **AND** / **OR** logic, you can group one of the logic operations using parentheses.

For example, say we want the above **OR** subset combined as an **AND** with those variants that have PS1 evidence. 

This is what our final filter looks like:

![](images/and-or-filter.png)
Here's a visual breakdown of this complex filter:

```{mermaid}
graph TD
  A["All Variants\n(n=1,738)"]--"Sequence Ontology:\nmissense"--> B
  A --"Clinical Significance:\nPathogenic"--> C
  subgraph OR
  B["Missense Variants\n(n=299)"]
  C["Pathogenic Variants\n(n=10)"]
  B --"OR"--> E["Pathogenic OR Missense\n(n=300)"]
  C --"OR"--> E
  end
  F["PS1 Variants\n(n=27)"]
  E --"AND"--> G["PS1 Variants AND\n(Missense OR Pathogenic)\n(n=27)"]
  A --"ClinVar ACMG\nPS1 variants"--> F
  F --"AND"--> G
  classDef White fill:#FFFFFF;
  class OR White
```

### NOT Logic

If we mouse over the top left of a filter or filter group, we'll see a faint "NOT". 

![](images/not-filter1.png)

Clicking this will change the logic to **NOT**. 

![](images/not-filter2.png)

Then we can hit **Apply Filter** again. In our case, we have 1,729 variants that do not meet our filter criterion. This makes sense because there were 9 variants total that met our grouped criterion, and 1,738 variants total, and 1738 - 9 = 1729.

### Deleting a Filter

Finally, if you need to delete a filter, you can mouse to the right of the filter. A faint "X" will appear. Clicking on it will delete that filter.

![](images/filter-delete.png)

## Exporting Filters as JSON

Filters can be exported and saved as JSON files for further reuse. They can be applied to a new set of variants in the GUI, or can be applied to result SQLite files on the command line. 

To export your filters, click on the save icon in the bottom right:

![](images/save-filter.png)
A modal box will pop up and ask you to name your filter set:

![](images/save-filter-name.png)

When you click **OK**, the filter will be saved on the top right:

![](images/saved-filter.png)

Finally, you can click on the download button for the individual filter to save it to disk:

![](images/saved-filter_download.png)

Here's an example of what a filter looks like in the JSON format:

```
{
  "sample": {
    "require": [],
    "reject": []
  },
  "genes": [],
  "variant": {
    "operator": "and",
    "rules": [
      {
        "operator": "and",
        "rules": [
          {
            "column": "clinvar__sig",
            "test": "hasData",
            "value": null,
            "negate": false
          },
          {
            "column": "clingen__disease",
            "test": "hasData",
            "value": null,
            "negate": false
          }
        ],
        "negate": false
      }
    ],
    "negate": false
  },
  "smartfilter": {},
  "samplefilter": {
    "operator": "and",
    "rules": [],
    "negate": false
  }
}
```

## Applying JSON Filters in the GUI

JSON filters can be applied by clicking on the upload icon:

![](images/upload-filter.png)

A file select box will pop up and you can select the JSON filter to load it:

![](images/upload-dialog.png)

Then you can hit **Apply Filter** as usual to apply it.

## Applying JSON Filters on the Command Line

JSON filters can also be applied on the command-line using the `oc util filtersqlite` command. [More information is here](https://open-cravat.readthedocs.io/en/latest/Filter-And-Merge-SQLite.html).

## What You Learned

- **Explain** how to build simple filters using the OpenCRAVAT GUI
- **Compose** multiple filters using boolean logic to make complex filtering
- **Save** your filter and **export** it as a JSON file
- **Apply** your filter to a new set of variants