Wiki source code of Provenance of simulation and data analysis workflows
Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | == Introduction == | ||
| 2 | |||
| 3 | Computational provenance is a record of all the steps in a computational scientific workflow, including the code that was run, input data, the computational environment (hardware, OS, compiler versions, library version...), the person who performed each step, and output data. | ||
| 4 | |||
| 5 | Capturing computational provenance facilitates: | ||
| 6 | |||
| 7 | * reproducibility of results | ||
| 8 | * management and tracking of workflows/projects by the scientists/engineers involved | ||
| 9 | * evaluation/review by other scientists and engineers | ||
| 10 | |||
| 11 | |||
| 12 | == Standards == | ||
| 13 | |||
| 14 | The [[W3C PROV standard>>https://www.w3.org/TR/2013/NOTE-prov-overview-20130430/||rel="noopener noreferrer" target="_blank"]] provides a data model and related tools for provenance interchange on the web. The following diagram shows the three base classes of the PROV data model: Entity, Activity, and Agent. These three classes form the basis for the representation of provenance in the EBRAINS Knowledge Graph: every node in the KG has a type which is a subclass of one of these base classes. | ||
| 15 | |||
| 16 | [[image:starting-points.svg||alt="The three Starting Point classes of the W3C PROV ontology and the properties that relate them."]] | ||
| 17 | |||
| 18 | == Storage of provenance in the Knowledge Graph == | ||
| 19 | |||
| 20 | We present here the current schemas for representing (a) data analysis and (b) simulations in the Knowledge Graph. These schemas will need to be extended to cover neurorobotics simulations, and probably a more explicit representation of pipelines/workflows (the chaining together of multiple analysis / simulation stages) will be needed. | ||
| 21 | |||
| 22 | [[image:Workflow provenance in the EBRAINS KG.svg||alt="KG schema for data analysis"]][[image:Workflow provenance in the EBRAINS KG-2.png||alt="KG schema for data analysis"]] | ||
| 23 | |||
| 24 | [[image:Workflow provenance in the EBRAINS KG.png||alt="KG schema for simulation"]] | ||
| 25 | |||
| 26 | (note that the diagrams do not show Agents; the person who launched each analysis/simulation activity is linked to the activity with a ##wasAssociatedWith## connection). | ||
| 27 | |||
| 28 | (% class="box warningmessage" %) | ||
| 29 | ((( | ||
| 30 | TODO: insert or link to the detailed schemas for each type | ||
| 31 | ))) | ||
| 32 | |||
| 33 | == Tools for automated capture of provenance == | ||
| 34 | |||
| 35 | * on different systems: | ||
| 36 | ** HPC systems | ||
| 37 | ** neuromorphic systems | ||
| 38 | ** Jupyter notebooks | ||
| 39 | ** users' own computers | ||
| 40 | * prospective/pre-emptive vs run-time provenance capture | ||
| 41 | * capture of metadata vs capture of artefacts | ||
| 42 | |||
| 43 | == Communication between computer systems and the KG == | ||
| 44 | |||
| 45 | Two issues arise: | ||
| 46 | |||
| 47 | (i) fine-grained provenance information may need to be obtained on compute nodes, which may not have network access; | ||
| 48 | |||
| 49 | (ii) failures of provenance upload should not cause the workflows to fail; | ||
| 50 | |||
| 51 | An overall solution for both of these issues would perhaps involve a local cache and later synchronization. | ||
| 52 | |||
| 53 | == User interfaces for browsing, visualizing, and searching provenance information == | ||
| 54 | |||
| 55 | (% class="box infomessage" %) | ||
| 56 | ((( | ||
| 57 | DISCUSSION NEEDED: integrate visualization of prov information into KG Search UI, and/or develop separate app? | ||
| 58 | ))) |