-[](http://pepkit.github.io)
+[](https://pepkit.github.io)
-[](http://geofetch.databio.org/en/latest/)
+[](https://geofetch.databio.org/en/latest/)
[](https://pypi.org/project/geofetch)
[](https://github.com/psf/black)
-`geofetch` is a command-line tool that downloads sequencing data and metadata from GEO and SRA and creates [standard PEPs](http://pep.databio.org/). `geofetch` is hosted at [pypi](https://pypi.org/project/geofetch/) and documentation is hosted at [geofetch.databio.org](http://geofetch.databio.org) (source in the [/docs](/docs) folder).
+`geofetch` is a command-line tool that downloads sequencing data and metadata from GEO and SRA and creates [standard PEPs](https://pep.databio.org/). `geofetch` is hosted at [pypi](https://pypi.org/project/geofetch/). You can convert the result of geofetch into unmapped `bam` or `fastq` files with the included `sraconvert` command.
-You can convert the result of geofetch into unmapped `bam` or `fastq` files with the included `sraconvert` command.
+Key geofetch features:
+
+- Works with GEO and SRA metadata
+- Combines samples from different projects
+- Standardizes output metadata
+- Filters type and size of processed files (from GEO) before downloading them
+- Easy to use
+- Fast execution time
+- Can search GEO to find relevant data
+- Can be used either as a command-line tool or from within Python using an API
+
+For more information, see [complete documentation at geofetch.databio.org](http://geofetch.databio.org) (source in the [/docs](/docs) folder).
diff --git a/docs/README.md b/docs/README.md
index 1cc4626..7b87b32 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -12,6 +12,19 @@
- Produce a standardized [PEP](http://pepkit.github.io) sample table. This makes it really easy to run [looper](https://pepkit.github.io/docs/looper/)-compatible pipelines on public datasets by handling data acquisition and metadata formatting and standardization for you.
- Prepare a project to run with [sraconvert](sra_convert.md) to convert SRA files into FASTQ files.
+
+
+Key geofetch advantages:
+
+- Works with GEO and SRA metadata
+- Combines samples from different projects
+- Standardizes output metadata
+- Filters type and size of processed files (from GEO) before downloading them
+- Easy to use
+- Fast execution time
+- Can search GEO to find relevant data
+- Can be used either as a command-line tool or from within Python using an API
+
## Quick example
`geofetch` runs on the command line. This command will download the raw data and metadata for the given GSE number.
@@ -38,5 +51,37 @@ geofetch -i GSE95654 --just-metadata
geofetch -i GSE95654 --processed --just-metadata
```
+### Check out what exactly argument you want to use to download data:
+
+
+
+---
+### New features available in geofetch 0.11.0:
+1) Now geofetch is available as Python API package. Geofetch can initialize [peppy](http://peppy.databio.org/) projects without downloading any soft files. Example:
+
+```python
+from geofetch import Geofetcher
+
+# initiate Geofetcher with all necessary arguments:
+geof = Geofetcher(processed=True, acc_anno=True, discard_soft=True)
+
+# get projects by providing as input GSE or file with GSEs
+geof.get_projects("GSE160204")
+```
+
+2) Now to find GSEs and save them to file you can use `Finder` - GSE finder tool:
+
+```python
+from geofetch import Finder
+
+# initiate Finder (use filters if necessary)
+find_gse = Finder(filters='bed')
+
+# get all projects that were found:
+gse_list = find_gse.get_gse_all()
+```
+Find more information here: [GSE Finder](./gse_finder.md)
+
+
For more details, check out the [usage](usage.md) reference, [installation instructions](install.md), or head on over to the [tutorial for raw data](raw-data-downloading.md) and [tutorial for processed data](processed-data-downloading.md) for a detailed walkthrough.
diff --git a/docs/changelog.md b/docs/changelog.md
index 2adcd81..7f2ef68 100644
--- a/docs/changelog.md
+++ b/docs/changelog.md
@@ -1,5 +1,18 @@
# Changelog
+## [0.11.0] -- 2022-10-26
+- Added initialization of peppy Project without saving any files (from within Python using an API)
+- Added Finder (searching GSE tool)
+- Added progress bar
+- Switched way of saving soft files to request library
+- Improved documentation
+- Refactored code
+- Added `--add-convert-modifier` flag
+- fixed looper amendments in the config file
+- Fixed special character bug in the config file
+- Fixed None issue in config file
+- Fixed saving raw peps bug
+
## [0.10.1] -- 2022-08-04
- Updated metadata fetching requests from SRA database
diff --git a/docs/gse_finder.md b/docs/gse_finder.md
new file mode 100644
index 0000000..14a353f
--- /dev/null
+++ b/docs/gse_finder.md
@@ -0,0 +1,81 @@
+is a geofetch class that provides functions to find and retrieve a list of GSE ([GEO](https://www.ncbi.nlm.nih.gov/geo/) accession number) by using NCBI searching tool.
+
+
+### The main features of the geofetch Finder are:
+- Find GEO accession numbers (GSE) of the project that were uploaded or updated in certain period of time.
+- Use the same filter query as [GEO DataSets Advanced Search Builder](https://www.ncbi.nlm.nih.gov/gds/advanced) is using
+- Save list of the GSEs to file (This file with geo can be used later in **[geofetch](http://geofetch.databio.org/en/latest/)**)
+- Easier and faster to get GSEs using NCBI filter and certain period of time.
+
+
+___
+## Tutorial
+
+0) Initiale Finder object.
+```python
+from geofetch import Finder
+gse_obj = Finder()
+
+# Optionally: provide filter string and max number of retrieve elements
+gse_obj = Finder(filter="((bed) OR narrow peak) AND Homo sapiens[Organism]", retmax=10)
+```
+
+1) Get list of all GSE in GEO
+```python
+
+gse_list = gse_obj.get_gse_all()
+
+```
+
+2) Get list of GSE that were uploaded and updated last week
+```python
+
+gse_list = gse_obj.get_gse_last_week()
+
+```
+
+3) Get list of GSE that were uploaded and updated last 3 month
+```python
+
+gse_list = gse_obj.get_gse_last_3_month()
+
+```
+
+4) Get list of GSE that were uploaded and updated in las *number of days*
+```python
+
+# project that were uploaded in last 5 days:
+gse_list = gse_obj.get_gse_by_day_count(5)
+
+```
+
+5) Get list of GSE that were uploaded in certain period of time
+```python
+
+gse_list = gse_obj.get_gse_by_date(start_date="2015/05/05", end_date="2020/05/05")
+
+```
+
+6) Save last searched list of items to the file
+```python
+
+gse_obj.generate_file("path/to/the/file")
+
+# if you want to save different list of files you can provide it to the funciton
+gse_obj.generate_file("path/to/the/file", gse_list=["123", "124"])
+
+```
+
+7) Compare two lists:
+```python
+
+new_gse_list = gse_obj.find_differences(list1, list2)
+
+```
+
+----
+
+More information about gse and queries and id:
+- https://www.ncbi.nlm.nih.gov/geo/info/geo_paccess.html
+- https://newarkcaptain.com/how-to-retrieve-ncbi-geo-information-using-apis-part1/
+- https://www.ncbi.nlm.nih.gov/books/NBK3837/#EntrezHelp.Using_the_Advanced_Search_Pag
\ No newline at end of file
diff --git a/docs/img/arguments_outputs.svg b/docs/img/arguments_outputs.svg
new file mode 100644
index 0000000..89cb3c5
--- /dev/null
+++ b/docs/img/arguments_outputs.svg
@@ -0,0 +1,8186 @@
+
+
+
+
diff --git a/docs/img/pipeline.svg b/docs/img/pipeline.svg
new file mode 100644
index 0000000..ef4d501
--- /dev/null
+++ b/docs/img/pipeline.svg
@@ -0,0 +1,8072 @@
+
+
+
+
diff --git a/docs/install.md b/docs/install.md
index 355cd70..01c9c12 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -1,21 +1,5 @@
# Installing geofetch
-## Prerequisites
-
-You must have the [sratoolkit from NCBI](https://www.ncbi.nlm.nih.gov/books/NBK158900/) installed, with the tools in your PATH. Once it's installed, you should check to make sure you can run `prefetch`. Also, make sure it's configured to store SRA files where you want them. For more information, see how to change sratools download location.
-
-## Setting data download location for `sratools`
-
-`geofetch` is using the [sratoolkit](https://trace.ncbi.nlm.nih.gov/Traces/sra/?view=toolkit_doc&f=std) to download raw data from SRA -- which means it's stuck with the [default path for downloading SRA data](http://databio.org/posts/downloading_sra_data.html), which is in your home directory. So before you run `geofetch`, make sure you have set up your download location to the correct place. In our group, we use a shared group environment variable called `${SRARAW}`, which points to a shared folder (`${DATA}/sra`) where the whole group has access to downloaded SRA data. You can point the `sratoolkit` (and therefore `geofetch`) to use that location with this one-time configuration code:
-
-```
-echo "/repository/user/main/public/root = \"$DATA\"" > ${HOME}/.ncbi/user-settings.mkfg
-```
-
-Now `sratoolkit` will download data into an `/sra` folder in `${DATA}`, which is what `${SRARAW}` points to.
-
-If you are getting an error that the `.ncbi` folder does not exist in your home directory, you can just make a folder `.ncbi` with an empty file `user-settings.mkfg` and follow the same command above.
-
## Installing geofetch
Releases are posted as [GitHub releases](https://github.com/pepkit/geofetch/releases), or you can install from PyPI using `pip`:
@@ -35,3 +19,19 @@ If the executable in not in your $PATH, append this to your `.bashrc` or `.profi
```
export PATH=~/.local/bin:$PATH
```
+
+## Prerequisites for SRA data downloading
+
+To download **raw data** You must have the [sratoolkit from NCBI](https://www.ncbi.nlm.nih.gov/books/NBK158900/) installed, with the tools in your PATH. Once it's installed, you should check to make sure you can run `prefetch`. Also, make sure it's configured to store SRA files where you want them. For more information, see how to change sratools download location.
+
+## Setting data download location for `sratools`
+
+`geofetch` is using the [sratoolkit](https://trace.ncbi.nlm.nih.gov/Traces/sra/?view=toolkit_doc&f=std) to download raw data from SRA -- which means it's stuck with the [default path for downloading SRA data](http://databio.org/posts/downloading_sra_data.html), which is in your home directory. So before you run `geofetch`, make sure you have set up your download location to the correct place. In our group, we use a shared group environment variable called `${SRARAW}`, which points to a shared folder (`${DATA}/sra`) where the whole group has access to downloaded SRA data. You can point the `sratoolkit` (and therefore `geofetch`) to use that location with this one-time configuration code:
+
+```
+echo "/repository/user/main/public/root = \"$DATA\"" > ${HOME}/.ncbi/user-settings.mkfg
+```
+
+Now `sratoolkit` will download data into an `/sra` folder in `${DATA}`, which is what `${SRARAW}` points to.
+
+If you are getting an error that the `.ncbi` folder does not exist in your home directory, you can just make a folder `.ncbi` with an empty file `user-settings.mkfg` and follow the same command above.
\ No newline at end of file
diff --git a/docs/metadata_output.md b/docs/metadata_output.md
index 3c2eace..5763b2f 100644
--- a/docs/metadata_output.md
+++ b/docs/metadata_output.md
@@ -1,24 +1,46 @@
# Metadata output
-For each GSE input accession (ACC), `geofetch` produces:
+Geofetch produces [PEPs](http://pep.databio.org/) for either processed or raw data (including metadata from SRA).
+A project can be created either for a single combined (whole) input or for each project separately.
+(if `--acc-anno` is set). "combined" means that it will have rows for every sample in every GSE included
+in your input. So if you just gave a single GSE, then the combined file is the same as the GSE file.
+
+**For raw data**: a metadata file will be created including SRA and GSM annotation.
+
+**For processed data**: a metadata file will be created just for GSE and GSM annotation. User
+can choose which data should he download. There are 3 downloading options for processed: samples, series and both.
+
+### Single PEP will contain:
+- project_name.csv - all metadata for sample processed data
+- project_name_subannotation.csv (*just for raw data*) - for *merged* samples
+(samples for which there are multiple SRR Runs for a single SRX `Experiment`)
+- project_name.yaml - project config file that stores all project information + common samples metadata
+
+Storing common metadata in project file is an efficient way to reduce project size and complexity of csv files.
+To specify and manage common metadata (where and how it should be stored) you can use next arguments:
+`--const-limit-project`, `--const-limit-discard`, `--attr-limit-truncate`
+
+### Saving actual data:
+Actual data will be saved if `--just-metadata` argument is not set. User should specify path to the folder where this
+data should be downloaded.
+
+----
+Additionally, for each GSE input accession (ACC), `geofetch` produces (if discard-soft is not set):
- GSE_ACC####.soft a SOFT file (annotating the experiment itself)
- GSM_ACC####.soft a SOFT file (annotating the samples within the experiment)
- SRA_ACC####.soft a CSV file (annotating each SRA Run, retrieved from GSE->GSM->SRA)
-In addition, a single combined metadata file (.csv) for the whole input,
-including SRA and GSM annotations for each sample. Here, "combined" means that it will have
-rows for every sample in every GSE included in your input. So if you just gave a single GSE,
-then the combined file is the same as the GSE file. If any "merged" samples exist
-(samples for which there are multiple SRR Runs for a single SRX `Experiment`), the
-script will also produce a merge table CSV file with the relationships between
-SRX and SRR.
-
-The way this works: Starting from a GSE, select a subset of samples (GSM Accessions) provided,
-and then obtain the SRX identifier for each of these from GEO. Now, query SRA for these SRX
-accessions and get any associated SRR accessions. Finally, download all of these SRR data files.
-
-### The most important metadata in pep format will be stored in
-- NAME_annotation_sample_processed.csv - all metadata for sample processed data
-- NAME_annotation.csv - all metadata for series processed data
-- NAME_annotation_series_processed.csv file - all metadata for raw data
+____
+# geofetch - Geofetcher using Python
+
+user can use geofetch in Python without saving any files. All the geofetch projects will be automatically downloaded
+as peppy Project. It helps save time and processing work.
+
+THe output in this case will be dictionary of projects:
+```python
+{'key1': (some_project),
+ 'key2': (second_project)}
+```
+
+More information you can find in tutorial files.
\ No newline at end of file
diff --git a/docs/usage.md b/docs/usage.md
index 89b3fad..29ba6b1 100644
--- a/docs/usage.md
+++ b/docs/usage.md
@@ -1,19 +1,24 @@
-# 
-
-Usage reference
-
-`geofetch` command-line usage instructions:
+# Usage reference
+geofetch command-line usage instructions:
+`geofetch -V`
+```console
+geofetch 0.11.0
+```
`geofetch --help`
-```{console}
-usage: geofetch [-h] [-V] -i INPUT [-n NAME] [-m METADATA_ROOT] [-u METADATA_FOLDER] [--just-metadata] [-r] [--config-template CONFIG_TEMPLATE]
- [--pipeline_samples PIPELINE_SAMPLES] [--pipeline_project PIPELINE_PROJECT] [-k SKIP] [--acc-anno] [--discard-soft]
- [--const-limit-project CONST_LIMIT_PROJECT] [--const-limit-discard CONST_LIMIT_DISCARD]
- [--attr-limit-truncate ATTR_LIMIT_TRUNCATE] [-p] [--data-source {all,samples,series}] [--filter FILTER]
- [--filter-size FILTER_SIZE] [-g GEO_FOLDER] [-x] [-b BAM_FOLDER] [-f FQ_FOLDER] [--use-key-subset] [--silent] [--verbosity V]
- [--logdev]
+```console
+usage: geofetch [-h] [-V] -i INPUT [-n NAME] [-m METADATA_ROOT] [-u METADATA_FOLDER]
+ [--just-metadata] [-r] [--config-template CONFIG_TEMPLATE]
+ [--pipeline-samples PIPELINE_SAMPLES] [--pipeline-project PIPELINE_PROJECT]
+ [--disable-progressbar] [-k SKIP] [--acc-anno] [--discard-soft]
+ [--const-limit-project CONST_LIMIT_PROJECT]
+ [--const-limit-discard CONST_LIMIT_DISCARD]
+ [--attr-limit-truncate ATTR_LIMIT_TRUNCATE] [--add-dotfile] [-p]
+ [--data-source {all,samples,series}] [--filter FILTER]
+ [--filter-size FILTER_SIZE] [-g GEO_FOLDER] [-x] [-b BAM_FOLDER]
+ [-f FQ_FOLDER] [--use-key-subset] [--silent] [--verbosity V] [--logdev]
Automatic GEO and SRA data downloader
@@ -21,34 +26,47 @@ optional arguments:
-h, --help show this help message and exit
-V, --version show program's version number and exit
-i INPUT, --input INPUT
- required: a GEO (GSE) accession, or a file with a list of GSE numbers
+ required: a GEO (GSE) accession, or a file with a list of GSE
+ numbers
-n NAME, --name NAME Specify a project name. Defaults to GSE number
-m METADATA_ROOT, --metadata-root METADATA_ROOT
- Specify a parent folder location to store metadata. The project name will be added as a subfolder [Default: $SRAMETA:]
+ Specify a parent folder location to store metadata. The project name
+ will be added as a subfolder [Default: $SRAMETA:]
-u METADATA_FOLDER, --metadata-folder METADATA_FOLDER
- Specify an absolute folder location to store metadata. No subfolder will be added. Overrides value of --metadata-root
- [Default: Not used (--metadata-root is used by default)]
+ Specify an absolute folder location to store metadata. No subfolder
+ will be added. Overrides value of --metadata-root [Default: Not used
+ (--metadata-root is used by default)]
--just-metadata If set, don't actually run downloads, just create metadata
-r, --refresh-metadata
If set, re-download metadata even if it exists.
--config-template CONFIG_TEMPLATE
Project config yaml file template.
- --pipeline_samples PIPELINE_SAMPLES
- Optional: Specify one or more filepaths to SAMPLES pipeline interface yaml files. These will be added to the project
- config file to make it immediately compatible with looper. [Default: null]
- --pipeline_project PIPELINE_PROJECT
- Optional: Specify one or more filepaths to PROJECT pipeline interface yaml files. These will be added to the project
- config file to make it immediately compatible with looper. [Default: null]
+ --pipeline-samples PIPELINE_SAMPLES
+ Optional: Specify one or more filepaths to SAMPLES pipeline
+ interface yaml files. These will be added to the project config file
+ to make it immediately compatible with looper. [Default: null]
+ --pipeline-project PIPELINE_PROJECT
+ Optional: Specify one or more filepaths to PROJECT pipeline
+ interface yaml files. These will be added to the project config file
+ to make it immediately compatible with looper. [Default: null]
+ --disable-progressbar
+ Optional: Disable progressbar
-k SKIP, --skip SKIP Skip some accessions. [Default: no skip].
- --acc-anno Optional: Produce annotation sheets for each accession. Project combined PEP for the whole project won't be produced.
- --discard-soft Optional: After creation of PEP files, all soft and additional files will be deleted
+ --acc-anno Optional: Produce annotation sheets for each accession. Project
+ combined PEP for the whole project won't be produced.
+ --discard-soft Optional: After creation of PEP files, all soft and additional files
+ will be deleted
--const-limit-project CONST_LIMIT_PROJECT
- Optional: Limit of the number of the constant sample characters that should not be in project yaml. [Default: 50]
+ Optional: Limit of the number of the constant sample characters that
+ should not be in project yaml. [Default: 50]
--const-limit-discard CONST_LIMIT_DISCARD
- Optional: Limit of the number of the constant sample characters that should not be discarded [Default: 250]
+ Optional: Limit of the number of the constant sample characters that
+ should not be discarded [Default: 250]
--attr-limit-truncate ATTR_LIMIT_TRUNCATE
- Optional: Limit of the number of sample characters.Any attribute with more than X characters will truncate to the first
- X, where X is a number of characters [Default: 500]
+ Optional: Limit of the number of sample characters.Any attribute
+ with more than X characters will truncate to the first X, where X is
+ a number of characters [Default: 500]
+ --add-dotfile Optional: Add .pep.yaml file that points .yaml PEP file
--silent Silence logging. Overrides verbosity.
--verbosity V Set logging level (1-5 or logging module level name)
--logdev Expand content of logging message format.
@@ -56,27 +74,35 @@ optional arguments:
processed:
-p, --processed Download processed data [Default: download raw data].
--data-source {all,samples,series}
- Optional: Specifies the source of data on the GEO record to retrieve processed data, which may be attached to the
- collective series entity, or to individual samples. Allowable values are: samples, series or both (all). Ignored unless
- 'processed' flag is set. [Default: samples]
- --filter FILTER Optional: Filter regex for processed filenames [Default: None].Ignored unless 'processed' flag is set.
+ Optional: Specifies the source of data on the GEO record to retrieve
+ processed data, which may be attached to the collective series
+ entity, or to individual samples. Allowable values are: samples,
+ series or both (all). Ignored unless 'processed' flag is set.
+ [Default: samples]
+ --filter FILTER Optional: Filter regex for processed filenames [Default:
+ None].Ignored unless 'processed' flag is set.
--filter-size FILTER_SIZE
- Optional: Filter size for processed files that are stored as sample repository [Default: None]. Works only for sample
- data. Supported input formats : 12B, 12KB, 12MB, 12GB. Ignored unless 'processed' flag is set.
+ Optional: Filter size for processed files that are stored as sample
+ repository [Default: None]. Works only for sample data. Supported
+ input formats : 12B, 12KB, 12MB, 12GB. Ignored unless 'processed'
+ flag is set.
-g GEO_FOLDER, --geo-folder GEO_FOLDER
- Optional: Specify a location to store processed GEO files. Ignored unless 'processed' flag is set.[Default: $GEODATA:]
+ Optional: Specify a location to store processed GEO files. Ignored
+ unless 'processed' flag is set.[Default: $GEODATA:]
raw:
-x, --split-experiments
- Split SRR runs into individual samples. By default, SRX experiments with multiple SRR Runs will have a single entry in
- the annotation table, with each run as a separate row in the subannotation table. This setting instead treats each run as
- a separate sample
+ Split SRR runs into individual samples. By default, SRX experiments
+ with multiple SRR Runs will have a single entry in the annotation
+ table, with each run as a separate row in the subannotation table.
+ This setting instead treats each run as a separate sample
-b BAM_FOLDER, --bam-folder BAM_FOLDER
- Optional: Specify folder of bam files. Geofetch will not download sra files when corresponding bam files already exist.
- [Default: $SRABAM:]
+ Optional: Specify folder of bam files. Geofetch will not download
+ sra files when corresponding bam files already exist. [Default:
+ $SRABAM:]
-f FQ_FOLDER, --fq-folder FQ_FOLDER
- Optional: Specify folder of fastq files. Geofetch will not download sra files when corresponding fastq files already
- exist. [Default: $SRAFQ:]
+ Optional: Specify folder of fastq files. Geofetch will not download
+ sra files when corresponding fastq files already exist. [Default:
+ $SRAFQ:]
--use-key-subset Use just the keys defined in this module when writing out metadata.
-
```
diff --git a/docs_jupyter/build/processed-data-downloading.md b/docs_jupyter/build/processed-data-downloading.md
index 052d413..b851a61 100644
--- a/docs_jupyter/build/processed-data-downloading.md
+++ b/docs_jupyter/build/processed-data-downloading.md
@@ -15,126 +15,6 @@ geofetch 0.10.1
To see your CLI options, invoke `geofetch -h`:
-
-```bash
-geofetch -h
-```
-
-```.output
-usage: geofetch [-h] [-V] -i INPUT [-n NAME] [-m METADATA_ROOT]
- [-u METADATA_FOLDER] [--just-metadata] [-r]
- [--config-template CONFIG_TEMPLATE]
- [--pipeline-samples PIPELINE_SAMPLES]
- [--pipeline-project PIPELINE_PROJECT] [-k SKIP] [--acc-anno]
- [--discard-soft] [--const-limit-project CONST_LIMIT_PROJECT]
- [--const-limit-discard CONST_LIMIT_DISCARD]
- [--attr-limit-truncate ATTR_LIMIT_TRUNCATE] [--add-dotfile]
- [-p] [--data-source {all,samples,series}] [--filter FILTER]
- [--filter-size FILTER_SIZE] [-g GEO_FOLDER] [-x]
- [-b BAM_FOLDER] [-f FQ_FOLDER] [--use-key-subset] [--silent]
- [--verbosity V] [--logdev]
-
-Automatic GEO and SRA data downloader
-
-optional arguments:
- -h, --help show this help message and exit
- -V, --version show program's version number and exit
- -i INPUT, --input INPUT
- required: a GEO (GSE) accession, or a file with a list
- of GSE numbers
- -n NAME, --name NAME Specify a project name. Defaults to GSE number
- -m METADATA_ROOT, --metadata-root METADATA_ROOT
- Specify a parent folder location to store metadata.
- The project name will be added as a subfolder
- [Default: $SRAMETA:]
- -u METADATA_FOLDER, --metadata-folder METADATA_FOLDER
- Specify an absolute folder location to store metadata.
- No subfolder will be added. Overrides value of
- --metadata-root [Default: Not used (--metadata-root is
- used by default)]
- --just-metadata If set, don't actually run downloads, just create
- metadata
- -r, --refresh-metadata
- If set, re-download metadata even if it exists.
- --config-template CONFIG_TEMPLATE
- Project config yaml file template.
- --pipeline-samples PIPELINE_SAMPLES
- Optional: Specify one or more filepaths to SAMPLES
- pipeline interface yaml files. These will be added to
- the project config file to make it immediately
- compatible with looper. [Default: null]
- --pipeline-project PIPELINE_PROJECT
- Optional: Specify one or more filepaths to PROJECT
- pipeline interface yaml files. These will be added to
- the project config file to make it immediately
- compatible with looper. [Default: null]
- -k SKIP, --skip SKIP Skip some accessions. [Default: no skip].
- --acc-anno Optional: Produce annotation sheets for each
- accession. Project combined PEP for the whole project
- won't be produced.
- --discard-soft Optional: After creation of PEP files, all soft and
- additional files will be deleted
- --const-limit-project CONST_LIMIT_PROJECT
- Optional: Limit of the number of the constant sample
- characters that should not be in project yaml.
- [Default: 50]
- --const-limit-discard CONST_LIMIT_DISCARD
- Optional: Limit of the number of the constant sample
- characters that should not be discarded [Default: 250]
- --attr-limit-truncate ATTR_LIMIT_TRUNCATE
- Optional: Limit of the number of sample characters.Any
- attribute with more than X characters will truncate to
- the first X, where X is a number of characters
- [Default: 500]
- --add-dotfile Optional: Add .pep.yaml file that points .yaml PEP
- file
- --silent Silence logging. Overrides verbosity.
- --verbosity V Set logging level (1-5 or logging module level name)
- --logdev Expand content of logging message format.
-
-processed:
- -p, --processed Download processed data [Default: download raw data].
- --data-source {all,samples,series}
- Optional: Specifies the source of data on the GEO
- record to retrieve processed data, which may be
- attached to the collective series entity, or to
- individual samples. Allowable values are: samples,
- series or both (all). Ignored unless 'processed' flag
- is set. [Default: samples]
- --filter FILTER Optional: Filter regex for processed filenames
- [Default: None].Ignored unless 'processed' flag is
- set.
- --filter-size FILTER_SIZE
- Optional: Filter size for processed files that are
- stored as sample repository [Default: None]. Works
- only for sample data. Supported input formats : 12B,
- 12KB, 12MB, 12GB. Ignored unless 'processed' flag is
- set.
- -g GEO_FOLDER, --geo-folder GEO_FOLDER
- Optional: Specify a location to store processed GEO
- files. Ignored unless 'processed' flag is
- set.[Default: $GEODATA:]
-
-raw:
- -x, --split-experiments
- Split SRR runs into individual samples. By default,
- SRX experiments with multiple SRR Runs will have a
- single entry in the annotation table, with each run as
- a separate row in the subannotation table. This
- setting instead treats each run as a separate sample
- -b BAM_FOLDER, --bam-folder BAM_FOLDER
- Optional: Specify folder of bam files. Geofetch will
- not download sra files when corresponding bam files
- already exist. [Default: $SRABAM:]
- -f FQ_FOLDER, --fq-folder FQ_FOLDER
- Optional: Specify folder of fastq files. Geofetch will
- not download sra files when corresponding fastq files
- already exist. [Default: $SRAFQ:]
- --use-key-subset Use just the keys defined in this module when writing
- out metadata.
-
-```
-
Calling geofetch will do 4 tasks:
1. download all or filtered processed files from `GSE#####` into your geo folder.
@@ -144,6 +24,11 @@ Calling geofetch will do 4 tasks:
Complete details about geofetch outputs is cataloged in the [metadata outputs reference](metadata_output.md).
+from IPython.core.display import SVG
+SVG(filename='logo.svg')
+
+
+
## Download the data
First, create the metadata for processed data (by adding --processed and --just-metadata):
diff --git a/docs_jupyter/build/python-usage.md b/docs_jupyter/build/python-usage.md
new file mode 100644
index 0000000..13e98c6
--- /dev/null
+++ b/docs_jupyter/build/python-usage.md
@@ -0,0 +1,360 @@
+jupyter:True
+# Tutorial of usage geofetch as python package
+
+♪♫*•♪♪♫*•♪♪♫*•♪♪♫*•♪♪♫*
+
+Geofetch provides python fuctions to fetch metadata and metadata from GEO and SRA by using python language. `get_project` function returns dictionary of peppy projects that were found using filters and input you specified.
+ peppy is a Python package that provides an API for handling standardized project and sample metadata.
+
+More information you can get here:
+
+http://peppy.databio.org/en/latest/
+
+http://pep.databio.org/en/2.0.0/
+
+### First let's import geofetch
+
+
+```python
+from geofetch import Geofetcher
+```
+
+### Initiate Geofetch object by specifing parameters that you want to use for downloading metadata/data
+
+1) If you won't specify any parameters, defaul parameters will be used
+
+
+```python
+geof = Geofetcher()
+```
+
+```.output
+Metadata folder: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/project_name
+
+```
+
+2) To download processed data with samples and series specify this two arguments:
+
+
+```python
+geof = Geofetcher(processed=True, data_source="all")
+```
+
+```.output
+Metadata folder: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/project_name
+
+```
+
+3) To tune project parameter, where metadata should be stored use next parameters:
+
+
+```python
+geof = Geofetcher(processed=True, data_source="all", const_limit_project = 20, const_limit_discard = 500, attr_limit_truncate = 10000 )
+```
+
+```.output
+Metadata folder: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/project_name
+
+```
+
+4) To add more filter of other options see documentation
+
+## Run Geofetch
+
+### By default:
+1) No actual data will be downloaded (just_metadata=True)
+
+2) No soft files will be saved on the disc (discard_soft=True)
+
+
+```python
+projects = geof.get_projects("GSE95654")
+```
+
+```.output
+Trying GSE95654 (not a file) as accession...
+Trying GSE95654 (not a file) as accession...
+
+```
+
+
+ Output()
+
+
+```.output
+Skipped 0 accessions. Starting now.
+Processing accession 1 of 1: 'GSE95654'
+
+Total number of processed SAMPLES files found is: 40
+Total number of processed SERIES files found is: 0
+Expanding metadata list...
+Expanding metadata list...
+
+```
+
+
+
+
+
+
+
+++ + + +```.output +Finished processing 1 accession(s) +Cleaning soft files ... +Unifying and saving of metadata... + +``` + + + Output() + + + + + + + + +
++ + + + +
++ + + + + Output() + + + + + + + + +
++ + + + +
++ + + +```.output +No files found. No data to save. File /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/project_name/GSE95654_series/GSE95654_series.csv won't be created + +``` + +Check if projects were created by checking dict keys: + + +```python +projects.keys() +``` + + + + + dict_keys(['GSE95654_samples']) + + + +project for smaples was created! Now let's look into it. + +\* the values of the dictionary are peppy projects. More information about peppy Project you can find in the documentation: http://peppy.databio.org/en/latest/ + + +```python +len(projects['GSE95654_samples'].samples) +``` + + + + + 40 + + + +We got 40 samples from GSE95654 project. If you want to check if it's correct information go into: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE95654 + +Now let's see actuall data. first 15 project and 5 clolumns: + + +```python +projects['GSE95654_samples'].sample_table.iloc[:15 , :5] +``` + + + + +
| + | sample_name | +sample_library_strategy | +genome_build | +tissue | +sample_organism_ch1 | +
|---|---|---|---|---|---|
| sample_name | ++ | + | + | + | + |
| RRBS_on_CRC_patient_8 | +RRBS_on_CRC_patient_8 | +Bisulfite-Seq | +hg19 | +primary tumor | +Homo sapiens | +
| RRBS_on_adjacent_normal_colon_patient_8 | +RRBS_on_adjacent_normal_colon_patient_8 | +Bisulfite-Seq | +hg19 | +adjacent normal colon | +Homo sapiens | +
| RRBS_on_CRC_patient_32 | +RRBS_on_CRC_patient_32 | +Bisulfite-Seq | +hg19 | +primary tumor | +Homo sapiens | +
| RRBS_on_adjacent_normal_colon_patient_32 | +RRBS_on_adjacent_normal_colon_patient_32 | +Bisulfite-Seq | +hg19 | +adjacent normal colon | +Homo sapiens | +
| RRBS_on_CRC_patient_41 | +RRBS_on_CRC_patient_41 | +Bisulfite-Seq | +hg19 | +primary tumor | +Homo sapiens | +
| RRBS_on_adjacent_normal_colon_patient_41 | +RRBS_on_adjacent_normal_colon_patient_41 | +Bisulfite-Seq | +hg19 | +adjacent normal colon | +Homo sapiens | +
| RRBS_on_CRC_patient_42 | +RRBS_on_CRC_patient_42 | +Bisulfite-Seq | +hg19 | +primary tumor | +Homo sapiens | +
| RRBS_on_adjacent_normal_colon_patient_42 | +RRBS_on_adjacent_normal_colon_patient_42 | +Bisulfite-Seq | +hg19 | +adjacent normal colon | +Homo sapiens | +
| RRBS_on_ACF_patient_173 | +RRBS_on_ACF_patient_173 | +Bisulfite-Seq | +hg19 | +aberrant crypt foci | +Homo sapiens | +
| RRBS_on_ACF_patient_515 | +RRBS_on_ACF_patient_515 | +Bisulfite-Seq | +hg19 | +aberrant crypt foci | +Homo sapiens | +
| RRBS_on_normal_crypts_patient_139 | +RRBS_on_normal_crypts_patient_139 | +Bisulfite-Seq | +hg19 | +normal colonic crypt | +Homo sapiens | +
| RRBS_on_ACF_patient_143 | +RRBS_on_ACF_patient_143 | +Bisulfite-Seq | +hg19 | +aberrant crypt foci | +Homo sapiens | +
| RRBS_on_normal_crypts_patient_143 | +RRBS_on_normal_crypts_patient_143 | +Bisulfite-Seq | +hg19 | +normal colonic crypt | +Homo sapiens | +
| RRBS_on_normal_crypts_patient_165 | +RRBS_on_normal_crypts_patient_165 | +Bisulfite-Seq | +hg19 | +normal colonic crypt | +Homo sapiens | +
| RRBS_on_ACF_patient_165 | +RRBS_on_ACF_patient_165 | +Bisulfite-Seq | +hg19 | +aberrant crypt foci | +Homo sapiens | +
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Finished processing 1 accession(s)\n", + "Cleaning soft files ...\n", + "Unifying and saving of metadata... \n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "b2c4c738728b4b43938fa6e7f29615ef", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "02401b3d938a4a588052ba99af677f84", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n" + ], + "text/plain": [] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n" + ], + "text/plain": [ + "\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "No files found. No data to save. File /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/project_name/GSE95654_series/GSE95654_series.csv won't be created\n" + ] + } + ], + "source": [ + "projects = geof.get_projects(\"GSE95654\")" + ] + }, + { + "cell_type": "markdown", + "id": "bc198009", + "metadata": {}, + "source": [ + "Check if projects were created by checking dict keys:" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "95896f25", + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "data": { + "text/plain": [ + "dict_keys(['GSE95654_samples'])" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "projects.keys()" + ] + }, + { + "cell_type": "markdown", + "id": "4e27f971", + "metadata": {}, + "source": [ + "project for smaples was created! Now let's look into it." + ] + }, + { + "cell_type": "markdown", + "id": "fa2d0bda", + "metadata": {}, + "source": [ + "\\* the values of the dictionary are peppy projects. More information about peppy Project you can find in the documentation: http://peppy.databio.org/en/latest/" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "e8642711", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "40" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "len(projects['GSE95654_samples'].samples)" + ] + }, + { + "cell_type": "markdown", + "id": "a4d50082", + "metadata": {}, + "source": [ + "We got 40 samples from GSE95654 project. If you want to check if it's correct information go into: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE95654" + ] + }, + { + "cell_type": "markdown", + "id": "d0cd958a", + "metadata": {}, + "source": [ + "Now let's see actuall data. first 15 project and 5 clolumns:" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "ba7be762", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
| \n", + " | sample_name | \n", + "sample_library_strategy | \n", + "genome_build | \n", + "tissue | \n", + "sample_organism_ch1 | \n", + "
|---|---|---|---|---|---|
| sample_name | \n", + "\n", + " | \n", + " | \n", + " | \n", + " | \n", + " |
| RRBS_on_CRC_patient_8 | \n", + "RRBS_on_CRC_patient_8 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "primary tumor | \n", + "Homo sapiens | \n", + "
| RRBS_on_adjacent_normal_colon_patient_8 | \n", + "RRBS_on_adjacent_normal_colon_patient_8 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "adjacent normal colon | \n", + "Homo sapiens | \n", + "
| RRBS_on_CRC_patient_32 | \n", + "RRBS_on_CRC_patient_32 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "primary tumor | \n", + "Homo sapiens | \n", + "
| RRBS_on_adjacent_normal_colon_patient_32 | \n", + "RRBS_on_adjacent_normal_colon_patient_32 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "adjacent normal colon | \n", + "Homo sapiens | \n", + "
| RRBS_on_CRC_patient_41 | \n", + "RRBS_on_CRC_patient_41 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "primary tumor | \n", + "Homo sapiens | \n", + "
| RRBS_on_adjacent_normal_colon_patient_41 | \n", + "RRBS_on_adjacent_normal_colon_patient_41 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "adjacent normal colon | \n", + "Homo sapiens | \n", + "
| RRBS_on_CRC_patient_42 | \n", + "RRBS_on_CRC_patient_42 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "primary tumor | \n", + "Homo sapiens | \n", + "
| RRBS_on_adjacent_normal_colon_patient_42 | \n", + "RRBS_on_adjacent_normal_colon_patient_42 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "adjacent normal colon | \n", + "Homo sapiens | \n", + "
| RRBS_on_ACF_patient_173 | \n", + "RRBS_on_ACF_patient_173 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "aberrant crypt foci | \n", + "Homo sapiens | \n", + "
| RRBS_on_ACF_patient_515 | \n", + "RRBS_on_ACF_patient_515 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "aberrant crypt foci | \n", + "Homo sapiens | \n", + "
| RRBS_on_normal_crypts_patient_139 | \n", + "RRBS_on_normal_crypts_patient_139 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "normal colonic crypt | \n", + "Homo sapiens | \n", + "
| RRBS_on_ACF_patient_143 | \n", + "RRBS_on_ACF_patient_143 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "aberrant crypt foci | \n", + "Homo sapiens | \n", + "
| RRBS_on_normal_crypts_patient_143 | \n", + "RRBS_on_normal_crypts_patient_143 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "normal colonic crypt | \n", + "Homo sapiens | \n", + "
| RRBS_on_normal_crypts_patient_165 | \n", + "RRBS_on_normal_crypts_patient_165 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "normal colonic crypt | \n", + "Homo sapiens | \n", + "
| RRBS_on_ACF_patient_165 | \n", + "RRBS_on_ACF_patient_165 | \n", + "Bisulfite-Seq | \n", + "hg19 | \n", + "aberrant crypt foci | \n", + "Homo sapiens | \n", + "