Automated testing of notebooks

Collaboratory Lab end-users develop Jupyter Notebooks to perform a wide variety of actions such as execute experiments, reproduce scientific/published results, and run workflows. These actions can also be linked to other core EBRAINS components (e.g., Knowledge Graph) or APIs which are necessary for their execution. Consequently, testing and determining the probability of successful execution of important public facing Jupyter notebooks forms an important part of the Quality Assurance (QA) in EBRAINS. Tests should be performed to ensure that all necessary resources are available for a successful execution while the visualization of test results can provide useful insights to Notebook owners. Furthermore, Notebook owners need to run the notebooks they have developed in advance to check that the desired results can indeed be produced before publishing it or sharing it with other users.

15

To satisfy the above requirements, a dedicated service for executing automated headless browser testing of Jupyter Notebooks has been developed during Phase 2 and a first version has already been released. The service is in production operation and available to all EBRAINS end-users.

16

In the present document, the steps required to register a notebook to the automated testing service and how the testing results can be accessed are documented.

17

(% id="HWhohasaccess3F" %)

18

== Testing Service ==

19

=== Current features ===

20

At the current state, the service is operational at this [[endpoint>>url:https://test-notebooks.apps.hbp.eu/get_notebooks]] and ready to be used by potential end-users. The pipeline that the service executes automatically once a week consists of the following actions:

21

**Collaboratory Lab container setup **

22

1. Connects to the [[Lab hub>>url:https://lab.ebrains.eu/hub/home]]

23

1. Selects one of the Lab Execution Sites (CSCS by default) and then performs login to the Collaboratory

24

1. Selects the official EBRAINS Docker image for the Collaboratory Lab and starts the server

25

**Testing Notebooks**

26

The following procedure is implemented for each of the notebooks that have been registered to be tested:

27

1. Opens the link of the notebook to be tested, loads the notebook itself and checks if the Collaboratory Lab container is available

28

1. Checks if the kernel is not busy and restarts the kernel in order to run the test in a clean environment.

29

1. Executes the notebook cells one-by-one.

30

If at least one cell raises an Exception or Traceback the result of the test is FAIL and the user is notified via email. The message body of the email includes a link to the artifacts produced by the service.

31

== Prerequisites ==

32

Before a notebook owner registers a notebook for the service the following actions must be taken:

33

* **The notebook owner requests access from TC to the following GitLab [[repository>>url:https://gitlab.ebrains.eu/technical-coordination/project-internal/testing/robot-development/clb-headless-browser-robot-tests]]:**

34

The tests are executed by a GitLab CI pipeline, coordinated by TC, in the above GitLab repository. Notebook owners receive a notification email in case of a failed execution with a url to the artifacts folder. The artifacts folder includes screenshots that depict the execution flow and what went wrong. Hence, the notebook owner needs to have access to this code repository.

35

* **The notebook owner provides access to the TC service account for the Collab where the notebook is located.**

36

The tests are managed and executed using a service account. Consequently, the service account must have at least //view// access rights to the collab in order to load the notebook. The service account’s username that must be provided with the proper access rights is “**tcsvacc**”

37

As soon as the notebook owner completes these actions, the following requirements should be considered. A notebook owner:

38

* **must not register notebooks with cells that wait for user interactions (e.g., user inputs) **

39

* **should use the service to test mainly core/critical public notebooks to ensure that notebook's end-users will not face issues when executing them.**

40

* **is encouraged to include notebooks with estimated execution time of (no more than) 20 minutes.**

41

\\The last two requirements arose because the aim is to test specific services or APIs that a notebook uses and not, for instance, notebooks that run long-time consuming experiments.

42

== Register a notebook to the testing service ==

43

There are two options for the notebook owner to register a notebook:

44

1. using a GUI widget (testing_widget.py)

45

1. programmatically, using a python wrapper (testing_wrapper.py).

46

A prerequisite here is to download the corresponding script that is needed to implement each option from the [[GitLab folder.>>url:https://gitlab.ebrains.eu/technical-coordination/project-internal/testing/robot-development/clb-headless-browser-robot-tests/-/tree/master/notebooks/registration_scripts]]

47

=== Register via widget ===

48

1. Open the following registration script: [[https:~~/~~/lab.ch.ebrains.eu/hub/user-redirect/lab/tree/shared/Collab%20Robot/front-interface-notebooks/interface.ipynb>>https://lab.ch.ebrains.eu/hub/user-redirect/lab/tree/shared/Collab%20Robot/front-interface-notebooks/interface.ipynb]]

49

2. Execute the first cell and the GUI appears with the respective parameters you need to fill:

50

* **Link**: The shareable link of the notebook which you can get by pressing:

51

//right_click// on the notebook’s filename ~-~-> //Copy Shareable Link//

52

like it is depicted below

53

(% style="text-align:center" %)

54

[[image:image-20220107164445-2.png||height="462" width="508"]]

55

(% class="box warningmessage" %)

56

(((

57

//Notebook link **must not** include:

**- Parenthesis

- Spaces**//

)))

* **notebook name**

(% class="box warningmessage" %)

63

(((

64

//Notebook name **must not** include:

65

**- Spaces**//

66

//**- Dots (.)**//

67

//and any other **non-alphanumeric **characters apart from dashes (-) //

68

)))

69

* **developer’s name/notebook owner **

70

* **email address(es)**

71

The address(es) where the notification email will be sent in case of failure. More than one email address can be registered

72

* **max runtime**

73

The estimation runtime in seconds.

74

(% class="box warningmessage" %)

75

(((

76

//The runtime must not exceed one hour.//

77

)))

78

* **execution frequency **

79

The frequency in days the notebook will be executed. E.g., 7 = once in 7 days (once a week)

80

**Example:**

81

(% style="text-align:center" %)

82

[[image:image-20220107165826-3.png]]

83

(% start="6" %)

84

1. Fill the fields and click “Submit”. One of the following messages will be returned:

85

* [[image:image-20220107170501-4.png||height="26" style="float:left" width="451"]]

86

\\In case the post request was successful, and the notebook saved in the database table that includes all the declared notebooks

87

* [[image:image-20220127120153-1.png||height="33" width="446"]](((

88

In case the post request was successful, but the notebook already exists in the database

89

)))

90

* [[image:image-20220127120237-2.png||height="26" width="254"]]

91

In case the post request was not successful the user should contact TC by opening a support ticket.

92

(% start="7" %)

93

1. Finally, delete the cell in which the widget was imported and executed, so the test will not execute it again.

94

=== Register programatically ===

95

==== Use a wrapper ====

96

To register a notebook programmatically, the notebook needs to be accessed via the shareable link. The notebook owner must create this URL before declaring the notebook to the service. Afterwards, the notebook owner must input the required parameters by writing them as environmental variables on the URL, based on [[this latest collab feature>>url:https://wiki.ebrains.eu/bin/view/Collabs/the-collaboratory/Documentation%20Lab/FAQ/Creating%20and%20accessing%20environment%20variables%20/]].

97

The steps to declare a notebook are the following:

98

1. Download the [[testing_wrapper.py script>>url:https://gitlab.ebrains.eu/technical-coordination/project-internal/testing/robot-development/clb-headless-browser-robot-tests/-/blob/master/notebooks/registration_scripts/wrapper/testing_wrapper.py]]:

99

(% start="2" %)

100

1. Create the shareable link for the notebook. The format of a shareable link is the following:

101

//https:~/~/lab.ebrains.eu/hub/user/user-redirect/lab/tree/shared/COLLAB/NOTEBOOK_NAME.ipy//

102

\\or you can get it by pressing:

103

\\//right_click// on the notebook’s filename ~-~-> //Copy Shareable Link//

104

\\like it is depicted below:

105

\\[[image:image-20220202232832-1.png||height="468" width="515"]]

106

(% start="3" %)

107

1. **Create the final URL **

108

The required parameters which can be provided as environmental variables are the following:

109

* (% id="cke_bm_24463S" style="display:none" %)** **(%%)**Link**: The shareable link of the notebook. You can get it by clicking //Copy Shareable Link //as described in step 2

110

111

(% class="box warningmessage" %)

112

(((

113

//Notebook link **must not** include~://

114

//**- Parenthesis**//

//**- Spaces**//

)))

* **notebook name**

(% class="box warningmessage" %)

119

(((

120

//Notebook name **must not** include: //

121

//- **Spaces**//

122

)))

123

* **developer’s name/notebook owner **

124

* **email address(es)**

125

The address(es) where the notification email will be sent in case of failure. More than one email address can be registered

126

* **max runtime**

127

The estimation runtime in seconds.

128

(% class="box warningmessage" %)

129

(((

130

//The runtime must not exceed one hour.//

131

)))

132

(% class="box infomessage" %)

133

(((

134

**Example:** The notebook owner has created the notebook **example.py** in the Lab. After opening the notebook and before running the wrapper, the final URL is created by using the corresponding environmental variables:

135

* //Shareable link~:// “LAB_NB_LINK=https:~/~/lab.ebrains.eu/hub/user/tc-service-account/lab/tree/shared/COLAB_NAME/NOTEBOOK_NAME.ipynb”

136

* //Notebook name~:// “LAB_NB_NAME=example”

137

* //Developer’s name~:///Notebook’s owner LAB_OWNER=aidono

138

* //Email~:// LAB_EMAIL=aidono@athenarc.gr

139

* //Max runtime //(20 seconds): “LAB_MAX_RUNTIME=20”

140

Therefore, the final URL will be:

141

"https:~/~/lab.ebrains.eu/hub/user/orfeasa/lab/tree/shared/Collab%20Robot/example.ipynb?LAB_NB_NAME=example&LAB_EXECUTION_SITE=ch&LAB_NB_LINK=https:~/~/lab.ebrains.eu/hub/user/tc-service-account/lab/tree/shared/COLAB_NAME/NOTEBOOK_NAME.ipynb&LAB_MAX_RUNTIME=200&LAB_EMAIL=aidono@athenarc.gr&LAB_OWNER=aidono"

142

(% class="box floatinginfobox" %)

143

(((

144

**NOTE//~://**// LAB_EXECUTION_SITE// is entered by default (CSCS)

)))

)))

(% start="4" %)

1. Open the final link created in step 3

149

1. Import the testing_wrapper.py script into the same folder the notebook lives

150

\\[[image:image-20220203000521-1.png]]

151

(% start="6" %)

152

1. Create a new cell and import the wrapper function as the line to call it as well

153

[[image:image-20220203001648-1.png]]

154

(% start="7" %)

155

1. Execute only this cell. One of the following messages will be returned:

156

* (% id="cke_bm_16643S" style="display:none" %)[[image:image-20220107170501-4.png||height="26" style="float:left" width="451"]][[image:image-20220107170501-4.png||height="26" style="float:left" width="451"]](%%)

157

\\In case the post request was successful, and the notebook saved in the database table that includes all the declared notebooks

158

* [[image:image-20220127120153-1.png||height="33" width="446"]](((

159

In case the post request was successful, but the notebook already exists in the database

160

)))

161

* [[image:image-20220127120237-2.png||height="26" width="253"]]

162

In case the post request was not successful the user should contact TC by opening a support ticket.