Last modified by robing on 2022/03/25 09:55
From version 77.1
edited by robing
on 2020/05/24 19:24
Change comment: There is no comment for this version
To version 60.1
edited by robing
on 2020/04/28 09:38
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -9,11 +9,11 @@
9 9  
10 10  Robin Gutzen^^1^^, Giulia De Bonis^^2^^, Elena Pastorelli^^2,3^^, Cristiano Capone^^2^^,
11 11  
12 -Chiara De Luca^^2,3^^, Michael Denker^^1^^, Sonja Grün^^1,4^^,
12 +Chiara De Luca^^2,3^^, Michael Denker^^1^^, Sonja Grün^^1^^,
13 13  
14 -Pier Stanislao Paolucci^^2^^, Andrew Davison^^5^^
14 +Pier Stanislao Paolucci^^2^^, Andrew Davison^^4^^
15 15  
16 -Experiments: Anna Letizia Allegra Mascaro^^6,7^^, Francesco Resta^^6^^, Francesco Saverio Pavone^^6^^, Maria-Victoria Sanchez-Vives^^8,9^^
16 +Experiments: Anna Letizia Allegra Mascaro^^5,6^^, Francesco Resta^^5^^, Francesco Saverio Pavone^^5^^, Maria-Victoria Sanchez-Vives^^7,8^^
17 17  
18 18  ,,1) Institute of Neuroscience and Medicine (INM-6) and Institute for Advanced Simulation (IAS-6) and JARA-Institute Brain Structure-Function Relationships (INM-10), Jülich Research Centre, Jülich, Germany,,
19 19  
... ... @@ -21,17 +21,15 @@
21 21  
22 22  ,,3) Ph.D. Program in Behavioural Neuroscience, “Sapienza” University of Rome, Rome, Italy,,
23 23  
24 -,,4) Theoretical Systems Neurobiology, RWTH Aachen University, Aachen, Germany,,
24 +,,4) Unité de Neurosciences, Information et Complexité, Neuroinformatics Group, CNRS FRE 3693, Gif-sur-Yvette, France,,
25 25  
26 -,,5) Unité de Neurosciences, Information et Complexité, Neuroinformatics Group, CNRS FRE 3693, Gif-sur-Yvette, France,,
26 +,,5) European Laboratory for Non-linear Spectroscopy (LENS), (% style="--darkreader-inline-color:inherit; color:inherit" %)University of Florence, Florence, Italy(%%),,
27 27  
28 -,,6) European Laboratory for Non-linear Spectroscopy (LENS), (% style="--darkreader-inline-color:inherit; color:inherit" %)University of Florence, Florence, Italy(%%),,
28 +,,6) Istituto di Neuroscienze, CNR, Pisa, Italy,,
29 29  
30 -,,7) Istituto di Neuroscienze, CNR, Pisa, Italy,,
30 +,,7) Institut d’Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain,,
31 31  
32 -,,8) Institut d’Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain,,
33 -
34 -,,9) Institució Catalana de Recerca i Estudis Avanc ̨ats (ICREA), Barcelona, Spain,,
32 +,,8) Institució Catalana de Recerca i Estudis Avanc ̨ats (ICREA), Barcelona, Spain,,
35 35  )))
36 36  )))
37 37  
... ... @@ -41,10 +41,13 @@
41 41  (((
42 42  == Flexible workflows to generate multi-scale analysis scenarios ==
43 43  
44 -This collab illustrates the usage of the [[Neo>>https://neo.readthedocs.io/en/stable/]] and [[Elephant>>https://elephant.readthedocs.io/en/latest/]] tools in performing data analysis with regards to the SGA2-SP3-UC002 KR3.2, investigating sleep, anesthesia, and the transition to wakefulness: see Chapter 1 and Figure 2  of SGA2[[ Deliverable D3.2.1.>>https://drive.ebrains.eu/smart-link/17ac0d6e-e050-4a49-8ca2-e223b70a3121/]], for an overview of the scientific motivations and a description of the UseCase workflow; Chapter 2 (same document) for an introduction to KR3.2; Chapter 3, for a description of the mice ECoG data sets; Chapter 5, about the Slow Wave Analysis Pipeline and Chapter 6 for the mice wide-field GECI data). For details on the datasets used in this collab, please see the References below.
42 +This Collab is aimed at experimental and computational neuroscientists interested in the usage of the [[Neo>>https://neo.readthedocs.io/en/stable/]] and [[Elephant>>https://elephant.readthedocs.io/en/latest/]] tools in performing data analysis of spiking data.
43 +Here, the collab illustrates the tool usage with regards to the SGA2-SP3-UC002 KR3.2, investigating sleep, anesthesia, and the transition to wakefulness: see Chapter 1 and Figure 2  of SGA2[[ Deliverable D3.2.1.>>https://drive.ebrains.eu/smart-link/17ac0d6e-e050-4a49-8ca2-e223b70a3121/]], for an overview of the scientific motivations and a description of the UseCase workflow; Chapter 2 (same document) for an introduction to KR3.2; Chapter 3, for a description of the mice ECoG data sets; Chapter 5, about the Slow Wave Analysis Pipeline and Chapter 6 for the mice wide-field GECI data)
45 45  
46 -== How the pipeline works ==
45 +[[image:https://github.githubassets.com/images/modules/logos_page/GitHub-Mark.png||height="35" width="35"]][[INM-6/wavescalephant>>https://github.com/INM-6/wavescalephant]]
47 47  
47 +== How the Pipeline works ==
48 +
48 48  The design of the pipeline aims at interfacing a variety of general and specific analysis and processing steps in a flexible modular manner. Hence, it enables the pipeline to adapt to diverse types of data (e.g., electrical ECoG, or optical Calcium Imaging recordings) and to different analysis questions. This makes the analyses a) more reproducible and b) comparable amongst each other since they rely on the same stack of algorithms and any differences in the analysis are fully transparent.
49 49  The individual processing and analysis steps (**blocks**//, //see// //the arrow-connected elements below) are organized in sequential **stages**// (//see the columns below//). //Following along the stages, the analysis becomes more specific but also allows to branch off at after any stage, as each stage yields useful intermediate results and is autonomous so that it can be reused and recombined. Within each stage, there is a collection of blocks from which the user can select and arrange the analysis via a config file. Thus, the pipeline can be thought of as a curated database of methods on which an analysis can be constructed by drawing a path along the blocks and stages.
50 50  
... ... @@ -70,24 +70,26 @@
70 70  In another browser tab, open [[https:~~/~~/lab.ebrains.eu>>https://lab.ebrains.eu]]
71 71  
72 72  * **Edit the config files**
73 -Each stage has config files (//pipeline/<stage_name>/configs/config_<profile>.yaml//) to specify which analysis/processing blocks to execute and which parameters to use. General and specific information about the blocks and parameters can be found in the README and config files of each stage. There are preset configuration profiles for the benchmark datasets IDIBAPS ([[ECoG, anesthetized mouse>>https://kg.ebrains.eu/search/?facet_type[0]=Dataset&q=sanchez-vives#Dataset/2ead029b-bba5-4611-b957-bb6feb631396]]) and LENS ([[Calcium Imaging, anesthetized mouse>>https://kg.ebrains.eu/search/instances/Dataset/71285966-8381-48f7-bd4d-f7a66afa9d79]]).
74 +Each stage has a config file (//pipeline/<stage_name>/config.yaml//) to specify which analysis/processing blocks to execute and which parameters to use. General and specific information about the blocks and parameters can be found in the README and config files of each stage. The default values are set for an example dataset (ECoG, anesthetized mouse, [[IDIBAPS>>https://kg.ebrains.eu/search/?facet_type[0]=Dataset&q=sanchez-vives#Dataset/2ead029b-bba5-4611-b957-bb6feb631396]]]).
74 74  
75 75  * **Run the notebook**
76 76  In the jupyter hub, navigate to //drive/My Libraries/My Library/pipeline/showcase_notebooks/run_snakemake_in_collab.ipynb//, or where you copied the //pipeline// folder to.
77 -Follow the notebook to install the required packages into your Python kernel, set the output path, and execute the pipeline with snakemake
78 +Follow the notebook to install the required packages into your Python kernel, set the output path, and execute the pipeline with snakemake.
79 +
80 +* **Coming soon**
81 +** Use of KnowledgeGraph API
82 +** Provenance Tracking
83 +** HPC support
78 78  
79 79  === ii) Local execution ===
80 80  
81 -//tested only with Linux!//
82 -
83 83  * **Get the code**
84 -The source code of the pipeline is available via Github: [[INM-6/wavescalephant>>https://github.com/INM-6/wavescalephant]] and can be cloned to your machine ([[how to get started with Github>>https://guides.github.com/activities/hello-world/]]).
88 +The source code of the pipeline is available via Github: [[INM-6/wavescalephant>>https://github.com/INM-6/wavescalephant]] and can be cloned to your machine ([[how to Github>>https://guides.github.com/activities/hello-world/]]).
85 85  
86 86  * (((
87 87  **Build the Python environment**
88 -In the wavescalephant git repository, there is an environment file ([[pipeline/environment.yaml>>https://drive.ebrains.eu/smart-link/1a0b15bb-be87-46ee-b838-4734bc320d20/]]) specifying the required packages and versions. To build the environment, we recommend using conda ([[how to get started with conda>>https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html]]).
89 -##conda env create ~-~-file environment.yaml
90 -conda activate wavescalephant_env##
92 +In the wavescalephant git repository, there is an environment file ([[pipeline/envs/wavescalephant_env.yml>>https://drive.ebrains.eu/lib/905d7321-a16b-4147-8cca-31d710d1f946/file/pipeline/envs/wavescalephant_env.yml]]) specifying the required packages and versions. To build the environment, we recommend using conda ([[how to get started with conda>>https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html]]).
93 +##conda env create ~-~-file /envs/wavescalephant_env.yml##
91 91  
92 92  )))
93 93  * **Edit the settings**
... ... @@ -94,14 +94,14 @@
94 94  The settings file specifies the path to the output folder, where results are saved to. Open the template file //[[pipeline/settings_template.py>>https://drive.ebrains.eu/lib/905d7321-a16b-4147-8cca-31d710d1f946/file/pipeline/settings_template.py]]//, set the ##output_path## to the desired path, and save it as //pipeline/settings.py//.
95 95  
96 96  * **Edit the config files**
97 -Each stage uses a config file to specify which analysis/processing blocks to execute and which parameters to use. Edit the config template files //pipeline/stageXX_<stage_name>/configs/config_template.yaml// according to your dataset and analysis goal, and save them as //pipeline/stageXX_<stage_name>/configs/config_<profile>.yaml//. A detailed description of the available parameter settings and their meaning is commented in the template files, and a more general description of the working mechanism of each stage can be found in the respective README file //pipeline/stageXX_<stage_name>/README.md//.
100 +Each stage has a config file to specify which analysis/processing blocks to execute and which parameters to use. Edit the config template files //pipeline/stageXX_<stage_name>/config_template.yaml// according to your dataset and analysis goal, and save them as //pipeline/stageXX_<stage_name>/config.yaml//. A detailed description of the available parameter settings and their meaning is commented in the template files, and a more general description of the working mechanism of each stage can be found in the respective README file //pipeline/stageXX_<stage_name>/README.md//.
98 98  //Links are view-only//
99 99  ** full pipeline: [[README.md>>https://drive.ebrains.eu/smart-link/d2e93a2a-09f6-4dce-982d-0370953a4da8/]], [[config.yaml>>https://drive.ebrains.eu/smart-link/7948fbb3-bf8a-4785-9b28-d5c15a1aafa7/]]
100 -** stage01_data_entry: [[README.md>>https://drive.ebrains.eu/smart-link/896f8880-a7d1-4a30-adbf-98759860fed5/]], [[config.yaml>>https://drive.ebrains.eu/smart-link/9bef8f59-1007-48c4-b5ba-30de4ff18f34/]]
101 -** stage02_processing: [[README.md>>https://drive.ebrains.eu/smart-link/01f21fa5-94f7-4883-8388-cc50957f9c81/]], [[config.yaml>>https://drive.ebrains.eu/smart-link/7e75caf6-e2d6-4393-a97c-4f481c908cf8/]]
102 -** stage03_trigger_detection: [[README.md>>https://drive.ebrains.eu/smart-link/18d276cd-a691-4ee1-81c6-7978cef9c1b4/]], [[config.yaml>>https://drive.ebrains.eu/smart-link/dfa375c0-cc80-4f95-b3ed-40140acbd96b/]]
103 -** stage04_wavefront_detection: [[README.md>>https://drive.ebrains.eu/smart-link/a8e80096-06a0-4ff4-b645-90e134e46ac5/]], [[config.yaml>>https://drive.ebrains.eu/smart-link/3a54be8c-b9f4-4698-a85d-6ad97990b40a/]]
104 -** stage05_wave_characterization: [[README.md>>https://drive.ebrains.eu/smart-link/3009a214-a11f-424c-8a6e-13e7506545eb/]], [[config.yaml>>https://drive.ebrains.eu/smart-link/83f68955-0ca8-4123-9734-6e93349ca3e3/]]
103 +** stage01_data_entry: [[README.md>>https://drive.ebrains.eu/smart-link/896f8880-a7d1-4a30-adbf-98759860fed5/]], [[config.yaml>>https://drive.ebrains.eu/smart-link/d429639d-b76e-4093-8fad-a25463d41edc/]]
104 +** stage02_processing: [[README.md>>https://drive.ebrains.eu/smart-link/01f21fa5-94f7-4883-8388-cc50957f9c81/]], [[config.yaml>>https://drive.ebrains.eu/f/b1607671f6f2468ahttps://drive.ebrains.eu/smart-link/02a3f92c-dc7d-4b33-94f5-91b00db060d5/a43c/]]
105 +** stage03_trigger_detection: [[README.md>>https://drive.ebrains.eu/smart-link/18d276cd-a691-4ee1-81c6-7978cef9c1b4/]], [[config.yaml>>https://drive.ebrains.eu/smart-link/76adbb12-7cb4-42df-9fd5-735927ea3ba8/]]
106 +** stage04_wavefront_detection: [[README.md>>https://drive.ebrains.eu/smart-link/a8e80096-06a0-4ff4-b645-90e134e46ac5/]], [[config.yaml>>https://drive.ebrains.eu/smart-link/6b0b233f-30b7-4bbd-8564-1abebd27ea6d/]]
107 +** stage05_wave_characterization: [[README.md>>https://drive.ebrains.eu/smart-link/3009a214-a11f-424c-8a6e-13e7506545eb/]], [[config.yaml>>https://drive.ebrains.eu/smart-link/471001d5-33f5-488e-a9a4-f03b190e3da7/]]
105 105  
106 106  * **Enter a dataset**
107 107  There are two test datasets in the collab drive (IDIBAPS and LENS) for which there are also corresponding config files and scripts in the data_entry stage. So, these datasets are ready to be used and analyzed.
... ... @@ -108,56 +108,28 @@
108 108  For adding new datasets see //[[pipeline/stage01_data_entry/README.md>>https://drive.ebrains.eu/smart-link/d2e93a2a-09f6-4dce-982d-0370953a4da8/]]//
109 109  
110 110  * **Run the pipeline (-stages)**
111 -To run the pipeline with [[snakemake>>https://snakemake.readthedocs.io/en/stable/]]), activate the Python environment ##conda activate wavescalephant_env,## make sure you are in the working directory (//pipeline/)//, and call ##snakemake## to run the entire pipeline.
112 -For a more detailed executed guide and how to execute individual stages and blocks see the pipeline [[Readme>>https://drive.ebrains.eu/smart-link/3009a214-a11f-424c-8a6e-13e7506545eb/]].
114 +To run the pipeline with [[snakemake>>https://snakemake.readthedocs.io/en/stable/]]), activate the Python environment ##conda activate wavescalephant_env,## make sure you are in the working directory (pipeline/), and call ##snakemake## to run the entire pipeline.
115 +To (re-)execute an individual stage, you can navigate to the corresponding stage folder and call the ##snakemake## command there. For running an individual stage, you may need to manually set the path for input file for the stage (i.e. the output file of the previous stage) in the config file ##INPUT: /path/to/file##.
113 113  
114 114  == Accessing and using the results ==
115 115  
116 116  All results are stored in the path specified in the //settings.py// file. The folder structure reflects the structuring of the pipeline into stages and blocks. All intermediate results are stored as //.nix// files using the [[Neo data format>>https://neo.readthedocs.io/en/stable/]] and can be loaded with ##neo.NixIO('/path/to/file.nix').read_block()##. Additionally, most blocks produce a figure, and each stage a report file, to give an overview of the execution log, parameters, intermediate results, and to help with debugging. The final stage (//stage05_wave_characterization//) stores the results as[[ //pandas.DataFrames//>>https://pandas.pydata.org/]] in //.csv// files, separately for each measure as well as in a combined dataframe for all measures.
117 117  
118 -**Examples of the output figures (for IDIBAPS dataset)**
119 -
120 -* Stage 01 - [[example signal traces and metadata>>https://drive.ebrains.eu/smart-link/cf2fa914-260d-4d61-a2da-03ea07b7f9be/]]
121 -* Stage 02 - [[background substraction>>https://drive.ebrains.eu/smart-link/586d2f3c-591b-4dfb-94ee-8c0e28050dc4/]]
122 -* Stage 02 - [[logMUA estimation>>https://drive.ebrains.eu/smart-link/c92e4b0c-0938-44e8-9f8d-00522796b2fd/]]
123 -* Stage 02 - [[processed signal trace>>https://drive.ebrains.eu/smart-link/26ed27c6-de56-4b48-a57b-f70aab629197/]]
124 -* Stage 03 - [[amplitude distribution>>https://drive.ebrains.eu/smart-link/8ba80293-ba75-4a37-8a8f-05d44cf6f65c/]]
125 -* Stage 03 - [[UP state detection>>https://drive.ebrains.eu/smart-link/ab172be0-178e-4153-a3e6-b4bace32dd50/]]
126 -* Stage 04 - [[trigger clustering>>https://drive.ebrains.eu/smart-link/4a1f0169-8b43-49ce-80c8-f2fa0f4d50d3/]]
127 -* Stage 05 - [[planar velocities>>https://drive.ebrains.eu/smart-link/f4de8073-cb40-47a7-bc82-f97d36dbae25/]]
128 -* Stage 05 - [[directionality>>https://drive.ebrains.eu/smart-link/5485032d-0121-4cde-9ea2-3e0af3f12178/]]
129 -
130 -=== Outlook ===
131 -
132 -* Using the **KnowledgeGraph API **to insert data directly from the Knowledge Graph into the pipeline and also register and store the corresponding results as Analysis Objects. Such Analysis Objects are to incorporate **Provenance Tracking, **using [[fairgraph>>https://github.com/HumanBrainProject/fairgraph]],** **to record the details of the processing and analysis steps.
133 -* Adding support for the pipeline to make use of **HPC** resources when running on the collab.
134 -* Further extending the available **methods** to address a wider variety of analysis objectives and support the processing of other datatypes. Additional documentation and guides should also make it easier for non-developers to contribute new method blocks.
135 -* Extending the **application** of the pipeline to the analysis of other types of activity waves and oscillations.
136 -* Integrating and co-developing new features of the underlying **software tools **[[Elephant>>https://elephant.readthedocs.io/en/latest/]], [[Neo>>https://neo.readthedocs.io/en/stable/]], [[Nix>>https://github.com/G-Node/nix]], [[Snakemake>>https://snakemake.readthedocs.io/en/stable/]].
137 -
138 138  == References ==
139 139  
140 140  * [[Celotto, Marco, et al. "Analysis and Model of Cortical Slow Waves Acquired with Optical Techniques." //Methods and Protocols// 3.1 (2020): 14.>>https://doi.org/10.3390/mps3010014]]
141 141  * [[De Bonis, Giulia, et al. "Analysis pipeline for extracting features of cortical slow oscillations." //Frontiers in Systems Neuroscience// 13 (2019): 70.>>https://doi.org/10.3389/fnsys.2019.00070]]
142 142  * [[Resta, F., Allegra Mascaro, A. L., & Pavone, F. (2020). "Study of Slow Waves (SWs) propagation through wide-field calcium imaging of the right cortical hemisphere of GCaMP6f mice" //EBRAINS//>>https://doi.org/10.25493/3E6Y-E8G]]// //
143 -* [[Sanchez-Vives, M. (2020). "Propagation modes of slow waves in mouse cortex".  //EBRAINS//>>https://doi.org/10.25493/WKA8-Q4T]]
144 -* [[Sanchez-Vives, M. (2019). "Cortical activity features in transgenic mouse models of cognitive deficits (Fragile X Syndrome).//" EBRAINS//>>https://doi.org/10.25493/ANF9-EG3]]
126 +* [[Sanchez-Vives, M. (2020). "Propagation modes of slow waves in mouse cortex".  //Human Brain Project Neuroinformatics Platform//>>https://doi.org/10.25493/WKA8-Q4T]]
145 145  
146 -Code developed at:
128 +== License (to discuss) ==
147 147  
148 -[[image:https://github.githubassets.com/images/modules/logos_page/GitHub-Mark.png||height="35" width="35"]][[INM-6/wavescalephant>>https://github.com/INM-6/wavescalephant]]
130 +All text and example data in this collab is licensed under Creative Commons CC-BY 4.0 license. Software code is licensed under a modified BSD license.
149 149  
150 -== License ==
151 -
152 -Text is licensed under the Creative Commons CC-BY 4.0 license. LENS data is licensed under the Creative Commons CC-BY-NC-ND 4.0 license. IDIBAPS data is licensed under the Creative Commons CC-BY-NC-SA 4.0 license. Software code is licensed under GNU General Public License v3.0.
153 -
154 154  [[image:https://i.creativecommons.org/l/by/4.0/88x31.png||style="float:left"]]
155 155  
156 -[[image:https://licensebuttons.net/l/by-nc-sa/4.0/88x31.png||alt="https://i.creativecommons.org/l/by/4.0/88x31.png" style="float:left"]]
134 +== ==
157 157  
158 -[[image:https://licensebuttons.net/l/by-nc-nd/4.0/88x31.png||alt="https://i.creativecommons.org/l/by/4.0/88x31.png" style="float:left"]]
159 -
160 -
161 161  == Acknowledgments ==
162 162  
163 163  This open source software code was developed in part or in whole in the Human Brain Project, funded from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the Specific Grant Agreement No. 785907 (Human Brain Project SGA2).
... ... @@ -167,12 +167,12 @@
167 167  )))
168 168  
169 169  
170 -== ==
145 +== Executing the pipeline ==
171 171  
172 172  (% class="col-xs-12 col-sm-4" %)
173 173  (((
174 174  {{box title="**Contents**"}}
175 -{{toc depth="3"/}}
150 +{{toc/}}
176 176  {{/box}}
177 177  
178 178  
Collaboratory.Apps.Collab.Code.CollabClass[0]
Description
... ... @@ -1,1 +1,1 @@
1 -Use Case SGA2-SP3-002 KR3.2: Integrating multi-scale data and the output of simulations in a reproducible and adaptable pipeline
1 +Space for developing and hosting a showcase pipeline for performing reproducible and adaptable analysis with the focus of slow cortical waves.
XWiki.XWikiRights[3]
Allow/Deny
... ... @@ -1,1 +1,0 @@
1 -Allow
Levels
... ... @@ -1,1 +1,0 @@
1 -view
Users
... ... @@ -1,1 +1,0 @@
1 -XWiki.XWikiGuest
XWiki.XWikiRights[4]
Allow/Deny
... ... @@ -1,1 +1,0 @@
1 -Allow
Groups
... ... @@ -1,1 +1,0 @@
1 -XWiki.XWikiAllGroup
Levels
... ... @@ -1,1 +1,0 @@
1 -view