Summary

• Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Author

...	...	@@ -1,1 +1,1 @@
1		-XWiki.hbpadmin
	1	+XWiki.mmorgan

Content

...	...	@@ -1,4 +1,22 @@
1	1	(% class="wikigeneratedid" %)
	2	+=== **Technical difficulties on the Drive (2021-11-03)** ===
	3	+
	4	+The Drive has experienced some technical difficulties over the past weeks caused by multiple issues.
	5	+
	6	+A reboot of the Drive server due to a hardware failure revealed an issue in the communication between the Lab and the Drive. A container is started for each Lab user who opens the Lab in their browser to run their notebooks. That container remains active by default well beyond the end of the execution of the notebook and the closing of the browser tab running the Lab, unless the users explicitly stop their container. Those containers were causing unusually high traffic to the Drive. We identified the problem and found a solution with an upgrade of the tool which the Drive is based on. We have had to shift the deployment date for this fix multiple times but we will be announcing the deployment date ASAP. In the meantime **we ask Lab users to kindly stop their servers** from the JupyterLab menu when they are not actively working in the Lab. This is done from the JupyterLab menu: //File > Hub control panel > Stop my server//.
	7	+
	8	+The cloud infrastructure that the Drive is running on is being upgraded and the Drive service had to be moved to the newer servers. An incorrect management command of the server caused the migration of the data to be launched twice, with the second transfer overwriting production data dated Nov 2 with stale data from Oct 20. Several backup issues stacked on top of this problem. We have been able to recover a backup which was made on the night of Sunday Oct 31 to Monday Nov 1. We are verifying that we cannot identify issues with this data before we reopen access to the Drive. **Some data loss is expected** for data stored on the Drive after that backup and before we put up a banner asking users, as a precaution, not to add new data to the Drive on Tuesday Nov 2 around noon. Many countries had a bank holiday on Nov 1 so for many users, the data loss should be of a few hours on Nov 2.
	9	+
	10	+We sincerely apologize for the problems incurred. We appreciate that the service we provide must be trustworthy for our users. We are taking the following actions to ensure that similar problems do not happen again:
	11	+
	12	+* Review of the backup procedures on the Drive server which will be extended to all EBRAINS services which we manage.
	13	+* Request notification from the underlying infrastructure when service accounts reach the limit imposed by their storage quota.
	14	+* Improve the identification of production servers to reduce the risk of the operations team misidentifying which servers are running production service.
	15	+
	16	+We hope to have the Drive service up again sometime Thursday Nov 4.
	17	+
	18	+Additionally, the Collaboratory team will be looking into mirroring services onto more than one Fenix site to reduce down time when incidents occur. This task will be more challenging for some services than for others; it might not be materially feasible for some services, but we will be analysing possibilities for each Collaboratory service.
	19	+
2	2	=== **Maintenance on 2021-10-19** ===
3	3
4	4	Due to migration to new hardware, various EBRAINS services will be down for up to 1 hour while these services and their data are migrated. Please See our [[Releases >>doc:Collabs.the-collaboratory.Releases.WebHome]]page for more information on our release.