Wiki source code of Slow Wave Analysis Pipeline

Version 23.2 by denker on 2020/04/03 16:53
Hide last authors
robing 1.1 1 (% class="jumbotron" %)
2 (((
3 (% class="container" %)
4 (((
robing 9.1 5 = (% style="color:inherit" %)Slow Wave Analysis Pipeline(%%) =
robing 1.1 6
denker 22.2 7 (% class="wikigeneratedid" id="HUseCaseSGA2-SP3-002:IntegratingmultiscaledataA0inareproducibleandadaptablepipeline" %)
8 (% style="color:inherit; font-size:24px" %)Use Case SGA2-SP3-002: Integrating multi-scale data in a reproducible and adaptable pipeline
robing 1.1 9 )))
10 )))
11
12 (% class="row" %)
13 (((
14 (% class="col-xs-12 col-sm-8" %)
15 (((
denker 23.1 16 == Flexible workflows to generate multi-scale analysis scenarios ==
robing 1.1 17
denker 23.1 18 This Collab is aimed at experimental and computational neuroscientists interested in the usage of the Neo and Elephant tools in performing data analysis of spiking data.
robing 1.1 19
robing 10.1 20 == Executing the pipeline ==
robing 6.1 21
robing 10.1 22 [[image:pipeline_flowchart.png]]
robing 6.1 23
denker 23.1 24 === In the collab (beta) ===
robing 8.1 25
robing 19.1 26 * **Copy the collab drive to your drive space**
27 ** Open the Drive from the left menu
28 ** Select the folders 'pipeline' and 'datasets'
robing 20.1 29 ** Select 'Copy', and then 'My Library' from the dropdown 'Other Libraries'
robing 19.1 30
31 * **Start a Jupyter Hub instance **
denker 23.2 32 In another browser, open [[https:~~/~~/lab.ebrains.eu>>https://lab.ebrains.eu]]
33
robing 10.1 34 * **Edit the config files**
35 Each stage has a config file to specify which analysis/processing blocks to execute and which parameters to use. General and specific information about the blocks and parameters can found in the README and config files. The default values are set for an example dataset (ECoG, anesthetized mouse, IDIBAPS [ref]).
36 ** stage01_data_entry: [README.md](), [config.yaml]()
37 ** stage02_preprocessing: [README.md](), [config.yaml]()
38 ** stage03_trigger_detection: [README.md](), [config.yaml]()
39 ** stage04_wavefront_detection: [README.md](), [config.yaml]()
40 ** stage05_wave_characterization: [README.md](), [config.yaml]()
41
robing 20.1 42 * **Run the notebook**
43 In the jupyter hub, navigate to `drive/My Libraries/My Library/pipeline/showcase_notebooks/run_snakemake_in_collab.ipynb`, or where you copied the 'pipeline' folder to.
robing 19.1 44 * Follow the notebook to install the required packages into your Python kernel, set the output path, and execute the pipeline with snakemake.
robing 20.1 45
robing 10.1 46 * **Coming soon**
47 ** Use of KnowledgeGraph API
48 ** Provenance Tracking
49 ** HPC support
50
denker 23.1 51 === Local execution ===
robing 10.1 52
53 * **Get the code**
54 The source code of the pipeline is available via Github: [INM-6/wavescalephant]('https:~/~/github.com/INM-6/wavescalephant') and can be cloned to your machine ([how to use Github]()).
55
56 * **Build the Python environment**
57 In the wavescalephant repository, there is an environment file (`pipeline/envs/wavescalephant_env.yaml`) specifying the required packages and versions. To build the environment, we recommend using *conda* ([how to get started with conda]()).
epastorelli 15.2 58 `conda env create ~-~-file /envs/wavescalephant_env.yml`.
robing 10.1 59
60 * **Edit the settings**
61 The settings file specifies the path to the output folder, where results are saved to. Open the template file `pipeline/settings_template.py`, set the `output_path` to the desired path, and save it as `pipeline/settings.py`.
62
63 * **Edit the config files**
64 Each stage has a config file to specify which analysis/processing blocks to execute and which parameters to use. Edit the config template files `pipeline/stageXX_<stage_name>/config_template.yaml` according to your dataset and analysis goal, and save them as `pipeline/stageXX_<stage_name>/config.yaml`. A detailed description of the available parameter settings and their meaning is commented in the template files, and a more general description of the working mechanism of each stage can be found in the respective README file `pipeline/stageXX_<stage_name>/README.md`.
65
66 * **Enter a dataset**
67 see `pipeline/stage01_data_entry/README.md`
68
69 * **Run the pipeline (-stages)**
70 To run the pipeline with snakemake ([intro to snakemake]()) activate the Python environment `conda activate wavescalephant_env`, make sure you are in the working directory `pipeline/`, and call `snakemake` to run the entire pipeline.
71 To (re-)execute an individual stage, you can navigate to the corresponding stage folder and call the `snakemake` command there. For running an individual stage, you may need to manually set the path for input file for the stage (i.e. the output file of the previous stage) in the config file `INPUT: /path/to/file`.
72
73 == Accessing and using the results ==
74
75 All results are stored in the path specified in the `settings.py` file. The folder structure reflects the structuring of the pipeline into stages and blocks. All intermediate results are stored as `.nix` files using the Neo data format ([Neo]()) and can be loaded with `neo.NixIO('/path/to/file.nix').read_block()` ([documentation]()).
76 Additionally, most blocks produce a figure, and each stage a report file, to give an overview of the execution log, parameters, intermediate results, and to help with debugging.
77 The final stage (*stage05_wave_characterization*) stores the results as pandas.DataFrames ([pandas]()) in `.csv` files, separately for each measure as well as in a combined dataframe for all measures.
78
79 == References ==
80
81
denker 22.1 82 == License (to discuss) ==
83
84 All text and example data in this collab is licensed under Creative Commons CC-BY 4.0 license. Software code is licensed under a modified BSD license.
85
robing 10.1 86 == Acknowledgments ==
87
denker 22.1 88 This open source software code was developed in part or in whole in the Human Brain Project, funded from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the Specific Grant Agreement No. 785907 (Human Brain Project SGA2).
robing 1.1 89 )))
90
91
92 (% class="col-xs-12 col-sm-4" %)
93 (((
94 {{box title="**Contents**"}}
95 {{toc/}}
96 {{/box}}
97
98
99 )))
100 )))