Wiki source code of Provenance of simulation and data analysis workflows
Hide last authors
| author | version | line-number | content |
|---|---|---|---|
 |
4.2 | 1 | == Introduction == |
| |
1.1 | 2 | |
| |
7.1 | 3 | Computational provenance is a record of all the steps in a computational scientific workflow, including the code that was run, input data, the computational environment (hardware, OS, compiler versions, library version...), the person who performed each step, and output data. |
| |
1.1 | 4 | |
| |
4.2 | 5 | Capturing computational provenance facilitates: |
| |
1.1 | 6 | |
| |
4.2 | 7 | * reproducibility of results |
| 8 | * management and tracking of workflows/projects by the scientists/engineers involved | ||
| 9 | * evaluation/review by other scientists and engineers | ||
| |
1.1 | 10 | |
| 11 | |||
| |
4.2 | 12 | == Standards == |
| |
1.1 | 13 | |
| |
7.2 | 14 | The [[W3C PROV standard>>https://www.w3.org/TR/2013/NOTE-prov-overview-20130430/||rel="noopener noreferrer" target="_blank"]] provides a data model and related tools for provenance interchange on the web. The following diagram shows the three base classes of the PROV data model: Entity, Activity, and Agent. These three classes form the basis for the representation of provenance in the EBRAINS Knowledge Graph: every node in the KG has a type which is a subclass of one of these base classes. |
| |
1.1 | 15 | |
| |
7.1 | 16 | [[image:starting-points.svg||alt="The three Starting Point classes of the W3C PROV ontology and the properties that relate them."]] |
| |
1.1 | 17 | |
| |
4.2 | 18 | == Storage of provenance in the Knowledge Graph == |
| 19 | |||
| |
7.2 | 20 | We present here the current schemas for representing (a) data analysis and (b) simulations in the Knowledge Graph. These schemas will need to be extended to cover neurorobotics simulations, and probably a more explicit representation of pipelines/workflows (the chaining together of multiple analysis / simulation stages) will be needed. |
| |
4.2 | 21 | |
| |
10.2 | 22 | [[image:Workflow provenance in the EBRAINS KG.svg||alt="KG schema for data analysis"]][[image:Workflow provenance in the EBRAINS KG-2.png||alt="KG schema for data analysis"]] |
| 23 | |||
| 24 | [[image:Workflow provenance in the EBRAINS KG.png||alt="KG schema for simulation"]] | ||
| 25 | |||
| 26 | (note that the diagrams do not show Agents; the person who launched each analysis/simulation activity is linked to the activity with a ##wasAssociatedWith## connection). | ||
| 27 | |||
| |
4.2 | 28 | == Tools for automated capture of provenance == |
| 29 | |||
| 30 | * on different systems: | ||
| 31 | ** HPC systems | ||
| 32 | ** neuromorphic systems | ||
| 33 | ** Jupyter notebooks | ||
| 34 | ** users' own computers | ||
| 35 | * prospective/pre-emptive vs run-time provenance capture | ||
| 36 | * capture of metadata vs capture of artefacts | ||
| 37 | |||
| 38 | == Communication between computer systems and the KG == | ||
| 39 | |||
| 40 | * local cache and synchronization? | ||
| 41 | |||
| 42 | == User interfaces for browsing, visualizing, and searching provenance information == | ||
| 43 | |||
| |
1.1 | 44 |