Wiki source code of SGA2 SP3 UC002 KR3.2 - Slow Wave Analysis Pipeline

Version 36.1 by robing on 2020/04/17 10:33

version	line-number	content
1.1	1	(% class="jumbotron" %)
	2	(((
	3	(% class="container" %)
	4	(((
36.1	5	= (% style="--darkreader-inline-color:inherit; color:inherit" %)Slow Wave Analysis Pipeline(%%) =
1.1	6
22.2	7	(% class="wikigeneratedid" id="HUseCaseSGA2-SP3-002:IntegratingmultiscaledataA0inareproducibleandadaptablepipeline" %)
36.1	8	(% style="--darkreader-inline-color:inherit; color:inherit; font-size:24px" %)Use Case SGA2-SP3-002: Integrating multi-scale data in a reproducible and adaptable pipeline
25.1	9
30.1	10	To be discussed Author orders, contributions,...:
25.1	11
	12	Experiments: ...?
	13
30.1	14	Implementation: Robin Gutzen^^1^^, Elena Pastorelli^^2^^, ...
25.1	15
28.1	16	Lead: Michael Denker^^1^^, Sonja Grün^^1^^, Pier Stanislao Paolucci^^2^^, Andrew Davison?
25.1	17
	18	,,1) Institute of Neuroscience and Medicine (INM-6) and Institute for Advanced Simulation (IAS-6) and JARA-Institute Brain Structure-Function Relationships (INM-10), Jülich Research Centre, Jülich, Germany,,
	19
34.1	20	,,2) Dipartimento di Fisica, Università di Cagliari and INFN Sezione di Roma, Italy,,
25.1	21
	22
1.1	23	)))
	24	)))
	25
	26	(% class="row" %)
	27	(((
	28	(% class="col-xs-12 col-sm-8" %)
	29	(((
23.1	30	== Flexible workflows to generate multi-scale analysis scenarios ==
1.1	31
23.1	32	This Collab is aimed at experimental and computational neuroscientists interested in the usage of the Neo and Elephant tools in performing data analysis of spiking data.
1.1	33
36.1	34	== How the Pipeline works ==
	35
	36	The design of the pipeline aims at interfacing a variety of general and specific analysis and processing steps in a flexible modular manner. Hence, it enables the pipeline to adapt to diverse types of data (e.b. electrical EEG, or optical Calcium Imaging recordings) and to different analysis questions. This makes the analyses a) more reproducible and b) comparable amongst each other since they rely on the same stack of algorithms and any differences in the analysis are fully transparent.
	37	The individual processing and analysis steps (//blocks, //see// //the arrow-connected elements below) are organized in sequential //stages (//see the columns below//). //Following along the stages the analysis becomes more specific but also allows to branch off at after any stage as each stage yields useful intermediate results is autonomous so that it can be reused and recombined. Within each stage, there is a collection of blocks from which the user can select and arrange the analysis via a config file. Thus, the pipeline can be thought of as a curated database of methods on which an analysis can be constructed by drawing a path along the blocks and stages.
	38
	39	== [[image:pipeline_flowchart.png]] ==
	40
10.1	41	== Executing the pipeline ==
6.1	42
36.1	43	(% class="wikigeneratedid" %)
23.1	44	=== In the collab (beta) ===
8.1	45
36.1	46	* (((
	47	Copy the collab drive to your drive space
	48
	49	*
	50
	51	Open the Drive from the left menu
	52
	53	* Select the folders 'pipeline' and 'datasets'
	54	* Select 'Copy', and then 'My Library' from the dropdown 'Other Libraries'
19.1	55
36.1	56	)))
19.1	57	* Start a Jupyter Hub instance
23.2	58	In another browser, open [[https:~~/~~/lab.ebrains.eu>>https://lab.ebrains.eu]]
	59
10.1	60	* Edit the config files
	61	Each stage has a config file to specify which analysis/processing blocks to execute and which parameters to use. General and specific information about the blocks and parameters can found in the README and config files. The default values are set for an example dataset (ECoG, anesthetized mouse, IDIBAPS [ref]).
	62	** stage01_data_entry: [README.md](), [config.yaml]()
	63	** stage02_preprocessing: [README.md](), [config.yaml]()
	64	** stage03_trigger_detection: [README.md](), [config.yaml]()
	65	** stage04_wavefront_detection: [README.md](), [config.yaml]()
	66	** stage05_wave_characterization: [README.md](), [config.yaml]()
	67
20.1	68	* Run the notebook
	69	In the jupyter hub, navigate to `drive/My Libraries/My Library/pipeline/showcase_notebooks/run_snakemake_in_collab.ipynb`, or where you copied the 'pipeline' folder to.
19.1	70	* Follow the notebook to install the required packages into your Python kernel, set the output path, and execute the pipeline with snakemake.
20.1	71
10.1	72	* Coming soon
	73	** Use of KnowledgeGraph API
	74	** Provenance Tracking
	75	** HPC support
	76
23.1	77	=== Local execution ===
10.1	78
	79	* Get the code
	80	The source code of the pipeline is available via Github: [INM-6/wavescalephant]('https:~/~/github.com/INM-6/wavescalephant') and can be cloned to your machine ([how to use Github]()).
	81
	82	* Build the Python environment
	83	In the wavescalephant repository, there is an environment file (`pipeline/envs/wavescalephant_env.yaml`) specifying the required packages and versions. To build the environment, we recommend using conda ([how to get started with conda]()).
15.2	84	`conda env create ~-~-file /envs/wavescalephant_env.yml`.
10.1	85
	86	* Edit the settings
	87	The settings file specifies the path to the output folder, where results are saved to. Open the template file `pipeline/settings_template.py`, set the `output_path` to the desired path, and save it as `pipeline/settings.py`.
	88
	89	* Edit the config files
	90	Each stage has a config file to specify which analysis/processing blocks to execute and which parameters to use. Edit the config template files `pipeline/stageXX_<stage_name>/config_template.yaml` according to your dataset and analysis goal, and save them as `pipeline/stageXX_<stage_name>/config.yaml`. A detailed description of the available parameter settings and their meaning is commented in the template files, and a more general description of the working mechanism of each stage can be found in the respective README file `pipeline/stageXX_<stage_name>/README.md`.
	91
	92	* Enter a dataset
	93	see `pipeline/stage01_data_entry/README.md`
	94
	95	* Run the pipeline (-stages)
	96	To run the pipeline with snakemake ([intro to snakemake]()) activate the Python environment `conda activate wavescalephant_env`, make sure you are in the working directory `pipeline/`, and call `snakemake` to run the entire pipeline.
	97	To (re-)execute an individual stage, you can navigate to the corresponding stage folder and call the `snakemake` command there. For running an individual stage, you may need to manually set the path for input file for the stage (i.e. the output file of the previous stage) in the config file `INPUT: /path/to/file`.
	98
	99	== Accessing and using the results ==
	100
	101	All results are stored in the path specified in the `settings.py` file. The folder structure reflects the structuring of the pipeline into stages and blocks. All intermediate results are stored as `.nix` files using the Neo data format ([Neo]()) and can be loaded with `neo.NixIO('/path/to/file.nix').read_block()` ([documentation]()).
	102	Additionally, most blocks produce a figure, and each stage a report file, to give an overview of the execution log, parameters, intermediate results, and to help with debugging.
	103	The final stage (stage05_wave_characterization) stores the results as pandas.DataFrames ([pandas]()) in `.csv` files, separately for each measure as well as in a combined dataframe for all measures.
	104
	105	== References ==
	106
	107
22.1	108	== License (to discuss) ==
	109
	110	All text and example data in this collab is licensed under Creative Commons CC-BY 4.0 license. Software code is licensed under a modified BSD license.
	111
29.1	112	[[image:https://i.creativecommons.org/l/by/4.0/88x31.png\|\|style="float:left"]]
	113
33.1	114	== ==
	115
10.1	116	== Acknowledgments ==
	117
22.1	118	This open source software code was developed in part or in whole in the Human Brain Project, funded from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the Specific Grant Agreement No. 785907 (Human Brain Project SGA2).
33.1	119
	120
	121	[[image:logos_sga2_sp3_uc002.png\|\|alt="Logos SP3 Use Case 2"]]
1.1	122	)))
	123
	124
36.1	125	== Executing the pipeline ==
	126
1.1	127	(% class="col-xs-12 col-sm-4" %)
	128	(((
	129	{{box title="Contents"}}
	130	{{toc/}}
	131	{{/box}}
	132
	133
	134	)))
	135	)))