Summary

• Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

...	...	@@ -1,4 +1,40 @@
1		-(% class="wikigeneratedid" %)
	1	+== **Issue on the OpenShift server at CSCS (2021-11-24)** ==
	2	+
	3	+The EBRAINS OpenShift server running at CSCS was detected as having come down at 16:00 CEST. This issue affects all the services running on that OpenShift server including:
	4	+
	5	+* Collaboratory Lab,
	6	+* image service,
	7	+* atlas viewers,
	8	+* simulation services (NEST desktop, TVB, Brain Simulation Platform)
	9	+
	10	+We are working with the CSCS team and RedHat (provider of the OpenStack solution on which OpenShift runs) to reactivate the service ASAP.
	11	+
	12	+The issue affected the central VM node of the OpenShift service. Restarting the VM required that we resolve multiple issues including restarting at the same IP address and unlocking the storage volumes of that VM. The central node has now been restarted. The same issue has occurred on three other instances of OpenShift VMs. We have the procedure for restarting those and are working on implementing it. The root cause seems to be at the infrastructure level and RedHat is involved in analyzing that.
	13	+
	14	+We will keep updating this page with more information as it comes in.
	15	+
	16	+=== **Maintenance of multiple services (2021-11-11 to 2021-11-19)** ===
	17	+
	18	+We are migrating the last of our services away from an old cloud infrastructure. The production services were taken down on Wednesday November 17 and are now all back online. We will continue working on the non-production services, to return the services to the redundancy level prior to the migration, and various final tweaks.
	19	+
	20	+Please contact [[support>>https://ebrains.eu/support]] about any new issues you identify on our services.
	21	+
	22	+The services which were migrated during this maintenance include:
	23	+
	24	+* Collaboratory Bucket (% style="color:#2ecc71" %)**DONE**
	25	+* Knowledge Graph Search and API (% style="color:#2ecc71" %)**DONE**
	26	+* OpenShift servers which includes: (% style="color:#2ecc71" %)**DONE**
	27	+** Collaboratory Lab, (% style="color:#2ecc71" %)**DONE**
	28	+** image service, (% style="color:#2ecc71" %)**DONE**
	29	+** atlas viewers,
	30	+** simulation services (NEST desktop, TVB, Brain Simulation Platform)
	31	+
	32	+* EBRAINS Docker registry, (% style="color:#2ecc71" %)**DONE**
	33	+* the Forum linked from the Collaboratory Wiki, (% style="color:#2ecc71" %)**DONE** (may have login issues)
	34	+* the Education website. (% style="color:#2ecc71" %)**DONE**
	35	+
	36	+We apologize for the inconvenience and thank you for your understanding.
	37	+
2	2	=== **Technical difficulties on the Drive (2021-11-03)** ===
3	3
4	4	The Drive has experienced some technical difficulties over the past weeks caused by multiple issues.
...	...	@@ -12,8 +12,9 @@
12	12	* Review of the backup procedures on the Drive server which will be extended to all EBRAINS services which we manage.
13	13	* Request notification from the underlying infrastructure when service accounts reach the limit imposed by their storage quota.
14	14	* Improve the identification of production servers to reduce the risk of the operations team misidentifying which servers are running production service.
	51	+* Improve our procedure for communication about such incidents.
15	15
16		-We hope to have the Drive service up again sometime Thursday Nov 4.
	53	+(% style="color:#16a085" %)**The Drive service is up again. We have recovered all the data that we could. The Drive is working normally again. We will be performing a Drive upgrade ASAP. check the Wiki banner for more information.**
17	17
18	18	Additionally, the Collaboratory team will be looking into mirroring services onto more than one Fenix site to reduce down time when incidents occur. This task will be more challenging for some services than for others; it might not be materially feasible for some services, but we will be analysing possibilities for each Collaboratory service.
19	19