Summary

• Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Author

...	...	@@ -1,1 +1,1 @@
1		-XWiki.mmorgan
	1	+XWiki.hbpadmin

Content

...	...	@@ -1,103 +1,35 @@
1		-=== **Maintenance of multiple services (2021-11-11 to 2021-11-19)** ===
	1	+(% class="wikigeneratedid" %)
	2	+=== Collaboratory Lab restart ( 15-06-2021) ===
2	2
3		-We are migrating the last of our services away from an old cloud infrastructure. The services were taken down on Wednesday November 17.
	4	+The Collaboratory lab will be restarted for the purpose of changing a few background configurations.
4	4
5		-We are facing issues restarting some services on the new cloud infrastructure. There is no data loss. The issues seem to be linked to network problems. We are actively trying to resolve them in collaboration with the infrastructure provider in order to restart the services as promptly as possible.
6	6
7		-The services which were migrated during this maintenance include:
	7	+=== Drive issue (Resolved) ===
8	8
9		-* Collaboratory Bucket (% style="color:#2ecc71" %)**DONE**
10		-* Knowledge Graph Search and API (% style="color:#e74c3c" %)**ONGOING**
11		-* OpenShift servers which includes: (% style="color:#e74c3c" %)**ONGOING**
12		-** Collaboratory Lab,
13		-** image service,
14		-** atlas viewers,
15		-** simulation services (NEST desktop, TVB, Brain Simulation Platform)
	9	+There is currently an issue that prevents the Drive option from appearing in new collabs. Although you can still create new collabs and can access all of the other functionality provided, you will not have access to the Drive by default. You can request a Drive be manually added to your collab via support.
	10	+\\We apologise for any inconvenience this causes.
16	16
17		-* EBRAINS Docker registry, (% style="color:#2ecc71" %)**DONE**
18		-* the Forum linked from the Collaboratory Wiki, (% style="color:#2ecc71" %)**DONE** (may have login issues)
19		-* the Education website. (% style="color:#2ecc71" %)**DONE**
20	20
21		-We apologize for the inconvenience. We strive to make the interruption of service as short as possible.
	13	+=== Maintenance Monday 7th June 8AM CEST ===
22	22
23		-=== **Technical difficulties on the Drive (2021-11-03)** ===
	15	+All services provided by virtual machines running on Openstack and Openshift at CSCS will be down for the preparation of the new updated Openstack server.
24	24
25		-The Drive has experienced some technical difficulties over the past weeks caused by multiple issues.
	17	+This will include all of Collaboratory 1 and 2, the Swift object storage, the EBRAINS authentication among many others. The following services will not be affected: HBP and EBRAINS web servers, supercomputing services, and services running on the 4 other Fenix sites (unless they depend on one of the above services).
26	26
27		-A reboot of the Drive server due to a hardware failure revealed an issue in the communication between the Lab and the Drive. A container is started for each Lab user who opens the Lab in their browser to run their notebooks. That container remains active by default well beyond the end of the execution of the notebook and the closing of the browser tab running the Lab, unless the users explicitly stop their container. Those containers were causing unusually high traffic to the Drive. We identified the problem and found a solution with an upgrade of the tool which the Drive is based on. We have had to shift the deployment date for this fix multiple times but we will be announcing the deployment date ASAP. In the meantime **we ask Lab users to kindly stop their servers** from the JupyterLab menu when they are not actively working in the Lab. This is done from the JupyterLab menu: //File > Hub control panel > Stop my server//.
	19	+The intervention is expected to fit within a time window of 3.5 hours. If it is shorter, the Collaboratory will come back up with a "Maintenance completed" banner. There is a small risk that the intervention may exceed 3.5 hours.
28	28
29		-The cloud infrastructure that the Drive is running on is being upgraded and the Drive service had to be moved to the newer servers. An incorrect management command of the server caused the migration of the data to be launched twice, with the second transfer overwriting production data dated Nov 2 with stale data from Oct 20. Several backup issues stacked on top of this problem. We have been able to recover a backup which was made on the night of Sunday Oct 31 to Monday Nov 1. We are verifying that we cannot identify issues with this data before we reopen access to the Drive. **Some data loss is expected** for data stored on the Drive after that backup and before we put up a banner asking users, as a precaution, not to add new data to the Drive on Tuesday Nov 2 around noon. Many countries had a bank holiday on Nov 1 so for many users, the data loss should be of a few hours on Nov 2.
	21	+=== EBRAINS services technical difficulties ===
30	30
31		-We sincerely apologize for the problems incurred. We appreciate that the service we provide must be trustworthy for our users. We are taking the following actions to ensure that similar problems do not happen again:
	23	+Most EBRAINS services are currently experiencing technical difficulties. Please bear with us as we work on resolving these issues. This involves Collaboratory 1, Collaboratory 2 and other EBRAINS services.
	24	+\\The issues occurring at the moment are the result of unexpected network issues due to network maintenance ongoing at CSCS.
32	32
33		-* Review of the backup procedures on the Drive server which will be extended to all EBRAINS services which we manage.
34		-* Request notification from the underlying infrastructure when service accounts reach the limit imposed by their storage quota.
35		-* Improve the identification of production servers to reduce the risk of the operations team misidentifying which servers are running production service.
36		-* Improve our procedure for communication about such incidents.
	26	+=== Collaboratory User Group (UG) meeting ===
37	37
38		-(% style="color:#16a085" %)**The Drive service is up again. We have recovered all the data that we could. The Drive is working normally again. We will be performing a Drive upgrade ASAP. check the Wiki banner for more information.**
39		-
40		-Additionally, the Collaboratory team will be looking into mirroring services onto more than one Fenix site to reduce down time when incidents occur. This task will be more challenging for some services than for others; it might not be materially feasible for some services, but we will be analysing possibilities for each Collaboratory service.
41		-
42		-=== **Maintenance on 2021-10-19** ===
43		-
44		-Due to migration to new hardware, various EBRAINS services will be down for up to 1 hour while these services and their data are migrated. Please See our [[Releases >>doc:Collabs.the-collaboratory.Releases.WebHome]]page for more information on our release.
45		-
46		-=== **Infrastructure failure on 2021-10-04** ===
47		-
48		-A failure on the Openstack infrastructure at the Fenix site which hosts most of our services has brought down many EBRAINS service this morning before 9 AM CEST. A second event later in the morning at the same Fenix site brought down most other EBRAINS services. The infrastructure was brought back online in the afternoon and we have been reactivating services as quickly as possible. Some issues are still being addressed at the infrastructure level [17:00 CEST].
49		-
50		-We apologize for the inconvenience and thank all users for their understanding.
51		-
52		-=== **SSL Certificates expiring September 30th** ===
53		-
54		-Our SSL certificates are produced by Let's Encrypt. One of their root certificates is expiring on September 30. This may cause issues for access to API services, especially from older browsers. You can find more information about these changes at the following links :
55		-
56		-* [[https:~~/~~/letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/>>url:https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/]]
57		-* [[https:~~/~~/letsencrypt.org/certificates/>>url:https://letsencrypt.org/certificates/]]
58		-
59		-As mentioned in the above links, if you are using one of our APIs, you should take the following steps to prevent issues:
60		-
61		-* You must trust ISRG Root X1 (not just DST Root CA X3)
62		-* If you use OpenSSL, you must have version 1.1.0 or later
63		-
64		-If you access our services via a browser, please ensure that you are using a modern browser,  you can find more information here : [[https:~~/~~/letsencrypt.org/docs/certificate-compatibility/>>url:https://letsencrypt.org/docs/certificate-compatibility/]]
65		-
66		-If you are using an older device, you may no longer be able to access our services correctly, for a list of non-compatible devices please see [[https:~~/~~/letsencrypt.org/docs/certificate-compatibility/>>url:https://letsencrypt.org/docs/certificate-compatibility/]].
67		-
68		-=== **Collaboratory 1 shutdown (2021-09-02)** ===
69		-
70		-Collaboratory 1 is shutting down Wednesday September 8th. As such, please ensure you have [[migrated>>https://wiki.ebrains.eu/bin/view/Collabs/collaboratory-migration/Tutorial/Migrate%20your%20data/]]  your data to Collaboratory 2. You can view our [[handy guide>>https://wiki.ebrains.eu/bin/view/Collabs/collaboratory-migration/Tutorial/Overview/]] on what data is migrated by our tools, as well as what is not covered by the semi-automated tools.
71		-
72		-If you follow the above links, you will also find recorded migrathons, where you can find a lot of information to help you migrate your data. If you encounter any issues or problems while migrating your data, please contact [[EBRAINS support>>https://ebrains.eu/support/]] and we will get back to you as soon as possible.
73		-
74		-NOTE: You are currently viewing Collaboratory 2, which will not be shut down.
75		-
76		-=== **Openshift issues (2021-07-09) (Resolved)** ===
77		-
78		-The Openshift service has gone down. As the Collaboratory Lab and the Image service both rely on this service, they are currently unavailable. We are investigating this issue and it will be resolved as soon as possible. We apologise for any inconvenience caused.
79		-
80		-**UPDATE 2021-07-12: **The issue seems to be with the Openstack servers. It is also affecting a few more VMs aside from the Openshift service. The infrastructure team is looking into it.
81		-
82		-**UPDATE 2021-07-12: **The Openstack issue has been fixed. Services should be back up and running. Thank you for your patience.
83		-
84		-=== **Collaboratory Lab restart (2021-06-15)** ===
85		-
86		-The Collaboratory lab will be restarted for the purpose of changing a few background configurations.
87		-
88		-=== **Collaboratory User Group (UG) meeting** ===
89		-
90	90	The UG meeting is a great way to stay informed about recent and upcoming changes to the Collaboratory as well as making your voice and ideas heard directly by the developers. If you are interested in attending this meeting, or know someone who is interested, please feel free[[ to join these meetings>>https://wiki.ebrains.eu/bin/view/Collabs/collaboratory-user-group/Joining%20the%20group/]]
91	91
92	92	If you would like to see new features or feel certain functionality is required to improve the Collaboratory, please [[join the Collaboratory user group>>https://wiki.ebrains.eu/bin/view/Collabs/collaboratory-user-group/Joining%20the%20group/]], once joined, please do not hesitate to [[make your suggestions>>https://drive.ebrains.eu/lib/aac38e36-4bd9-48ee-9ae6-b876a65028aa/file/Collaboratory%20User%20Group%20-%20Suggestion%20box.docx]].
93	93
94	94
95		-=== **Drive issue (Resolved 2021-06-14)** ===
96		-
97		-There is currently an issue that prevents the Drive option from appearing in new collabs. Although you can still create new collabs and can access all of the other functionality provided, you will not have access to the Drive by default. You can request a Drive be manually added to your collab via support.
98		-\\We apologise for any inconvenience this causes.
99		-
100		-
101	101	=== **Drive issue (Solved)** ===
102	102
103	103	Users have reported issues trying to access the drive. This is something we are aware of and trying to fix as soon as possible. This issue seems to be an Openshift problem.