Summary

• Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

...	...	@@ -1,5 +1,57 @@
1		-== **Issue on the OpenShift server at CSCS (2021-11-24)** ==
	1	+(% class="wikigeneratedid" %)
	2	+=== **HBP and EBRAINS websites issue (2022-06-29)** ===
2	2
	4	+The HBP and EBRAINS main websites are down because of an issue with the .io top level domain DNS. This issue is preventing our websites from rendering images, CSS files, and other files referenced from the HTML of the web pages. The issue is not internal to our infrastructure nor to that of our cloud provider. We have decided to bring down the two websites for the time being. We are monitoring the situation and looking for solutions.
	5	+
	6	+=== **Issue with the Bucket service (2022-06-27)** ===
	7	+
	8	+The bucket service (aka data proxy) is down due to an issue with the Swift archive/object storage at the infrastructure level at CSCS. The issue has been resolved. It was caused by the high availability load balancers located in front of the Swift storage at CSCS.
	9	+
	10	+We apologize for the inconvenience.
	11	+
	12	+=== **Maintenance of the Drive (2022-04-14)** ===
	13	+
	14	+The drive service will be down for maintenance on Thursday April 14 from 18:00 CEST for an estimated 2 to 3 hours. We are performing a routine garbage collection on the storage. During the downtime, users will not have access to the Drive: no reading/downloading of files, no adding/modifying files. This in turn means the Lab will not be usable either.
	15	+
	16	+We apologize in advance for the interruption and thank you for your understanding.
	17	+
	18	+The maintenance is done.
	19	+
	20	+=== **Issue with the Drive (2022-02-05)** ===
	21	+
	22	+The Drive server had an issue starting Saturday Feb 5 from 5 AM CET. The service was brought back online and was operational on Saturday. The issue was related to a disk full on the server caused by logs which then had other issues rebooting. No data was lost and backups were not affected.
	23	+
	24	+=== **Issue with OpenShift (2022-01-17)** ===
	25	+
	26	+One of the NGINX servers running in front of our OpenShift service is down due to a problem at the infrastructure level. We are in communication with the infrastructure team to get the issue resolved as quickly as possible.
	27	+
	28	+The issue was resolved by the infrastructure team. The server at fault is running again. All services are up and accessible again.
	29	+
	30	+=== **Issue with the Collaboratory like/favourite feature (2022-01-13)** ===
	31	+
	32	+The feature to like/favourite a collab was down for a day due to a change in the deployment of the Wiki service. This issue has been resolved without data loss. We apologise for any inconvenience caused by this issue.
	33	+
	34	+=== **Issue on the OpenShift server at CSCS (2021-12-28)** ===
	35	+
	36	+The EBRAINS OpenShift server running at CSCS was detected as having come down. This issue affects all the services running on that OpenShift server including:
	37	+
	38	+* Collaboratory Lab,
	39	+* image service,
	40	+* atlas viewers,
	41	+* simulation services (NEST desktop, TVB, Brain Simulation Platform)
	42	+
	43	+We are working with the CSCS team to reactivate the service ASAP.
	44	+
	45	+The service was reestablished at around 14:15 CET. The infrastructure team will continue looking into the potential causes of the problem with RedHat. The suspected cause lies in network access to the storage.
	46	+
	47	+=== **Issue on the Forum service (2021-12-27)** ===
	48	+
	49	+The Forum service went down over the weekend along with one of the redundant OpenShift servers. The issue was due to a problem at the infrastructure level.
	50	+
	51	+The Forum was unavailable for a few hours on Monday. There was not perceptible effect on the services running on OpenShift.
	52	+
	53	+=== **Issue on the OpenShift server at CSCS (2021-11-24)** ===
	54	+
3	3	The EBRAINS OpenShift server running at CSCS was detected as having come down at 16:00 CEST. This issue affects all the services running on that OpenShift server including:
4	4
5	5	* Collaboratory Lab,
...	...	@@ -9,8 +9,10 @@
9	9
10	10	We are working with the CSCS team and RedHat (provider of the OpenStack solution on which OpenShift runs) to reactivate the service ASAP.
11	11
12		-The issue affected the central VM node of the OpenShift service. Restarting the VM required that we resolve multiple issues including restarting at the same IP address and unlocking the storage volumes of that VM. (% style="color:#16a085" %)The central node has now been restarted. The same issue has occurred on three other instances of OpenShift VMs. We have the procedure for restarting those and are working on implementing it. The root cause seems to be at the infrastructure level and RedHat is involved in analyzing that.
	64	+The issue affected the central VM node of the OpenShift service. Restarting the VM required that we resolve multiple issues including restarting at the same IP address and unlocking the storage volumes of that VM. (% style="color:#16a085" %)The central node has now been restarted. The same issue has occurred on three other instances of OpenShift VMs. They have been restarted too. OpenShift is up and running again. The DNS redirection(s) to the maintenance page is being removed. The services should be operational again.
13	13
	66	+(% style="color:#16a085" %)The root cause seems to be at the infrastructure level and RedHat is involved in analyzing that.
	67	+
14	14	We will keep updating this page with more information as it comes in.
15	15
16	16	=== **Maintenance of multiple services (2021-11-11 to 2021-11-19)** ===