Version 51.1 by robing on 2020/04/24 16:42
Hide last authors
robing 1.1 1 (% class="jumbotron" %)
2 (((
3 (% class="container" %)
4 (((
debonisg 50.1 5 = (% style="--darkreader-inline-color:inherit; color:inherit" %)Slow Wave Analysis Pipeline (SWAP)(%%) =
robing 1.1 6
denker 22.2 7 (% class="wikigeneratedid" id="HUseCaseSGA2-SP3-002:IntegratingmultiscaledataA0inareproducibleandadaptablepipeline" %)
debonisg 50.1 8 (% style="--darkreader-inline-color:inherit; color:inherit; font-size:24px" %)**Use Case SGA2-SP3-002 KR3.2: Integrating multi-scale data and the output of simulations in a reproducible and adaptable pipeline**
denker 25.1 9
debonisg 50.1 10 Robin Gutzen^^1^^, Giulia De Bonis^^2^^, Elena Pastorelli^^2,3^^, Cristiano Capone^^2^^,
denker 25.1 11
debonisg 50.1 12 Chiara De Luca^^2,3^^, Michael Denker^^1^^, Sonja Grün^^1^^,
denker 25.1 13
debonisg 50.1 14 Pier Stanislao Paolucci^^2^^, Andrew Davison^^4^^
denker 25.1 15
debonisg 50.1 16 Experiments: Anna Letizia Allegra Mascaro^^5,6^^, Francesco Resta^^5^^, Francesco Saverio Pavone^^5^^, Maria-Victoria Sanchez-Vives^^7,8^^
denker 41.2 17
denker 25.1 18 ,,1) Institute of Neuroscience and Medicine (INM-6) and Institute for Advanced Simulation (IAS-6) and JARA-Institute Brain Structure-Function Relationships (INM-10), Jülich Research Centre, Jülich, Germany,,
19
debonisg 50.1 20 ,,2) Istituto Nazionale di Fisica Nucleare (INFN), Sezione di Roma, Rome, Italy,,
denker 25.1 21
debonisg 50.1 22 ,,3) Ph.D. Program in Behavioural Neuroscience, “Sapienza” University of Rome, Rome, Italy,,
denker 41.2 23
debonisg 50.1 24 ,,4) Unité de Neurosciences, Information et Complexité, Neuroinformatics Group, CNRS FRE 3693, Gif-sur-Yvette, France,,
denker 41.2 25
debonisg 50.1 26 ,,5) European Laboratory for Non-linear Spectroscopy (LENS), (% style="color:inherit" %)University of Florence, Florence, Italy(%%),,
27
28 ,,6) Istituto di Neuroscienze, CNR, Pisa, Italy,,
29
30 ,,7) Institut d’Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain,,
31
32 ,,8) Institució Catalana de Recerca i Estudis Avanc ̨ats (ICREA), Barcelona, Spain,,
robing 1.1 33 )))
34 )))
35
36 (% class="row" %)
37 (((
38 (% class="col-xs-12 col-sm-8" %)
39 (((
denker 23.1 40 == Flexible workflows to generate multi-scale analysis scenarios ==
robing 1.1 41
robing 39.1 42 This Collab is aimed at experimental and computational neuroscientists interested in the usage of the [[Neo>>https://neo.readthedocs.io/en/stable/]] and [[Elephant>>https://elephant.readthedocs.io/en/latest/]] tools in performing data analysis of spiking data.
43 Here, the collab illustrates the tool usage with regards to KR3.2, investigating sleep, anesthesia, and the transition to wakefulness.
robing 1.1 44
robing 51.1 45 [[image:https://github.githubassets.com/images/modules/logos_page/GitHub-Mark.png||height="35" width="35"]][[INM-6/wavescalephant>>https://github.com/INM-6/wavescalephant]]
46
robing 36.1 47 == How the Pipeline works ==
48
robing 39.1 49 The design of the pipeline aims at interfacing a variety of general and specific analysis and processing steps in a flexible modular manner. Hence, it enables the pipeline to adapt to diverse types of data (e.g., electrical ECoG, or optical Calcium Imaging recordings) and to different analysis questions. This makes the analyses a) more reproducible and b) comparable amongst each other since they rely on the same stack of algorithms and any differences in the analysis are fully transparent.
50 The individual processing and analysis steps (**blocks**//, //see// //the arrow-connected elements below) are organized in sequential **stages**// (//see the columns below//). //Following along the stages the analysis becomes more specific but also allows to branch off at after any stage as each stage yields useful intermediate results is autonomous so that it can be reused and recombined. Within each stage, there is a collection of blocks from which the user can select and arrange the analysis via a config file. Thus, the pipeline can be thought of as a curated database of methods on which an analysis can be constructed by drawing a path along the blocks and stages.
robing 36.1 51
robing 37.1 52 (% class="wikigeneratedid" id="H" %)
53 [[image:pipeline_flowchart.png]]
robing 36.1 54
robing 10.1 55 == Executing the pipeline ==
robing 6.1 56
robing 38.1 57 There are two ways of getting started and testing the pipeline, i) online using the collab drive and jupyter hub, or ii) downloading the code and data from GitHub and the collab storage and running it locally.
robing 8.1 58
robing 38.1 59 === i) In the collab ===
60
robing 36.1 61 * (((
robing 38.1 62 **Copy the collab drive to your personal drive space**
robing 36.1 63
robing 38.1 64 * Open the Drive from the left menu
robing 40.1 65 * Select the folders //pipeline// and //datasets,//
denker 41.1 66 and the notebook// run_snakemake_in_collab.ipynb//
robing 36.1 67 * Select 'Copy', and then 'My Library' from the dropdown 'Other Libraries'
robing 19.1 68
robing 36.1 69 )))
robing 19.1 70 * **Start a Jupyter Hub instance **
robing 38.1 71 In another browser tab, open [[https:~~/~~/lab.ebrains.eu>>https://lab.ebrains.eu]]
robing 40.1 72
robing 10.1 73 * **Edit the config files**
robing 40.1 74 Each stage has a config file (//pipeline/<stage_name>/config.yaml//) to specify which analysis/processing blocks to execute and which parameters to use. General and specific information about the blocks and parameters can found in the README and config files of each stage. The default values are set for an example dataset (ECoG, anesthetized mouse, [[IDIBAPS>>https://kg.ebrains.eu/search/?facet_type[0]=Dataset&q=sanchez-vives#Dataset/2ead029b-bba5-4611-b957-bb6feb631396]]]).
robing 10.1 75
robing 20.1 76 * **Run the notebook**
robing 38.1 77 In the jupyter hub, navigate to //drive/My Libraries/My Library/pipeline/showcase_notebooks/run_snakemake_in_collab.ipynb//, or where you copied the //pipeline// folder to.
robing 40.1 78 Follow the notebook to install the required packages into your Python kernel, set the output path, and execute the pipeline with snakemake.
robing 20.1 79
robing 10.1 80 * **Coming soon**
81 ** Use of KnowledgeGraph API
82 ** Provenance Tracking
83 ** HPC support
84
robing 38.1 85 === ii) Local execution ===
robing 10.1 86
87 * **Get the code**
robing 38.1 88 The source code of the pipeline is available via Github: [[INM-6/wavescalephant>>https://github.com/INM-6/wavescalephant]] and can be cloned to your machine ([[how to Github>>https://guides.github.com/activities/hello-world/]]).
robing 10.1 89
robing 38.1 90 * (((
91 **Build the Python environment**
robing 51.1 92 In the wavescalephant git repository, there is an environment file ([[pipeline/envs/wavescalephant_env.yml>>https://drive.ebrains.eu/lib/905d7321-a16b-4147-8cca-31d710d1f946/file/pipeline/envs/wavescalephant_env.yml]]) specifying the required packages and versions. To build the environment, we recommend using conda ([[how to get started with conda>>https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html]]).
robing 38.1 93 ##conda env create ~-~-file /envs/wavescalephant_env.yml##
robing 10.1 94
robing 38.1 95 )))
robing 10.1 96 * **Edit the settings**
robing 51.1 97 The settings file specifies the path to the output folder, where results are saved to. Open the template file //[[pipeline/settings_template.py>>https://drive.ebrains.eu/lib/905d7321-a16b-4147-8cca-31d710d1f946/file/pipeline/settings_template.py]]//, set the ##output_path## to the desired path, and save it as //pipeline/settings.py//.
robing 10.1 98
99 * **Edit the config files**
robing 38.1 100 Each stage has a config file to specify which analysis/processing blocks to execute and which parameters to use. Edit the config template files //pipeline/stageXX_<stage_name>/config_template.yaml// according to your dataset and analysis goal, and save them as //pipeline/stageXX_<stage_name>/config.yaml//. A detailed description of the available parameter settings and their meaning is commented in the template files, and a more general description of the working mechanism of each stage can be found in the respective README file //pipeline/stageXX_<stage_name>/README.md//.
101 //Links are view-only//
robing 51.1 102 ** full pipeline: [[README.md>>https://drive.ebrains.eu/smart-link/896f8880-a7d1-4a30-adbf-98759860fed5/]], [[config.yaml>>https://drive.ebrains.eu/lib/905d7321-a16b-4147-8cca-31d710d1f946/file/pipeline/config_template.yaml]]
103 ** stage01_data_entry: [[README.md>>https://drive.ebrains.eu/smart-link/896f8880-a7d1-4a30-adbf-98759860fed5/]], [[config.yaml>>https://drive.ebrains.eu/f/8de751f48d7d47edaec1/]]
debonisg 50.1 104 ** stage02_processing: [[README.md>>https://drive.ebrains.eu/f/7f19d89913624425bf63/]], [[config.yaml>>https://drive.ebrains.eu/f/b1607671f6f2468aa43c/]]
105 ** stage03_trigger_detection: [[README.md>>https://drive.ebrains.eu/f/94d12860dde84bbab7b1/]], [[config.yaml>>https://drive.ebrains.eu/f/6dfb712d5fa24f4f9fcf/]]
106 ** stage04_wavefront_detection: [[README.md>>https://drive.ebrains.eu/d/9c53abd5eaf543b28615/]], [[config.yaml>>https://drive.ebrains.eu/f/9534e46c4fae41c78f17/]]
107 ** stage05_wave_characterization: [[README.md>>https://drive.ebrains.eu/f/4d79f3e314474c22a781/]], [[config.yaml>>https://drive.ebrains.eu/f/1689dda03be04251b85f/]]
robing 10.1 108
109 * **Enter a dataset**
robing 38.1 110 There are two test datasets in the collab drive (IDIBAPS and LENS) for which there are also corresponding config files and scripts in the data_entry stage. So, these datasets are ready to be used and analyzed.
debonisg 50.1 111 For adding new datasets see //[[pipeline/stage01_data_entry/README.md>>https://drive.ebrains.eu/f/b46ffe259b3a4a51a277/]]//
robing 10.1 112
113 * **Run the pipeline (-stages)**
robing 38.1 114 To run the pipeline with snakemake ([intro to snakemake]()) activate the Python environment ##conda activate wavescalephant_env,## make sure you are in the working directory `pipeline/`, and call ##snakemake## to run the entire pipeline.
115 To (re-)execute an individual stage, you can navigate to the corresponding stage folder and call the ##snakemake## command there. For running an individual stage, you may need to manually set the path for input file for the stage (i.e. the output file of the previous stage) in the config file ##INPUT: /path/to/file##.
robing 10.1 116
117 == Accessing and using the results ==
118
robing 38.1 119 All results are stored in the path specified in the //settings.py// file. The folder structure reflects the structuring of the pipeline into stages and blocks. All intermediate results are stored as //.nix// files using the [[Neo data format>>https://neo.readthedocs.io/en/stable/]] and can be loaded with ##neo.NixIO('/path/to/file.nix').read_block()##. Additionally, most blocks produce a figure, and each stage a report file, to give an overview of the execution log, parameters, intermediate results, and to help with debugging. The final stage (//stage05_wave_characterization//) stores the results as[[ //pandas.DataFrames//>>https://pandas.pydata.org/]] in //.csv// files, separately for each measure as well as in a combined dataframe for all measures.
robing 10.1 120
121 == References ==
122
123
denker 22.1 124 == License (to discuss) ==
125
126 All text and example data in this collab is licensed under Creative Commons CC-BY 4.0 license. Software code is licensed under a modified BSD license.
127
denker 29.1 128 [[image:https://i.creativecommons.org/l/by/4.0/88x31.png||style="float:left"]]
129
denker 33.1 130 == ==
131
robing 10.1 132 == Acknowledgments ==
133
denker 22.1 134 This open source software code was developed in part or in whole in the Human Brain Project, funded from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the Specific Grant Agreement No. 785907 (Human Brain Project SGA2).
denker 33.1 135
136
137 [[image:logos_sga2_sp3_uc002.png||alt="Logos SP3 Use Case 2"]]
robing 1.1 138 )))
139
140
robing 36.1 141 == Executing the pipeline ==
142
robing 1.1 143 (% class="col-xs-12 col-sm-4" %)
144 (((
145 {{box title="**Contents**"}}
146 {{toc/}}
147 {{/box}}
148
149
150 )))
151 )))