Wiki source code of Data Proxy & Human Data Gateway
Last modified by alexisdurieux on 2022/03/25 08:38
Hide last authors
| author | version | line-number | content |
|---|---|---|---|
 |
1.1 | 1 | (% class="jumbotron" %) |
| 2 | ((( | ||
| 3 | (% class="container" %) | ||
| 4 | ((( | ||
| |
8.1 | 5 | (% style="text-align: justify;" %) |
| |
26.1 | 6 | = Data Proxy & HDG = |
| |
1.1 | 7 | |
| |
26.1 | 8 | The Data Proxy is an application that allows authenticated EBRAINS users to access Swift Object storage without a Fenix user account. |
| 9 | This application has 2 main use cases: | ||
| 10 | |||
| 11 | * ((( | ||
| 12 | It provides a dedicated optional swift container to every Collab. We call it the Collab Bucket. | ||
| |
1.1 | 13 | ))) |
| |
26.1 | 14 | * ((( |
| |
28.1 | 15 | It allows users to access and visualize Knowledge Graph datasets. It provides an additional layer to access datasets with more sensitive human data that has been strongly pseudonymized (e.g defaced brain scans): The **Human Data Gateway** |
| |
1.1 | 16 | ))) |
| |
26.1 | 17 | ))) |
| 18 | ))) | ||
| |
1.1 | 19 | |
| 20 | (% class="row" %) | ||
| 21 | ((( | ||
| 22 | ((( | ||
| |
8.1 | 23 | (% style="text-align: justify;" %) |
| |
20.1 | 24 | The data proxy core is the application that acts as a proxy to the object storage (Swift). |
| |
6.1 | 25 | |
| |
8.1 | 26 | (% style="text-align: justify;" %) |
| |
15.1 | 27 | === Object Storage === |
| |
6.1 | 28 | |
| |
20.1 | 29 | The documentation of Swift object storage can be found here: |
| |
15.1 | 30 | |
| |
20.1 | 31 | [[https:~~/~~/docs.openstack.org/swift/pike/admin/objectstorage-intro.html>>https://docs.openstack.org/swift/pike/admin/objectstorage-intro.html]] |
| |
15.1 | 32 | |
| |
8.1 | 33 | (% style="text-align: justify;" %) |
| |
15.1 | 34 | === Authentication === |
| 35 | |||
| 36 | (% style="text-align: justify;" %) | ||
| |
20.1 | 37 | The Data Proxy authenticates its users with the EBRAINS (Collaboratory) IAM service. |
| |
6.1 | 38 | |
| |
8.1 | 39 | (% style="text-align: justify;" %) |
| |
27.1 | 40 | Prior to the availability of the Data Proxy, EBRAINS users had to request a Fenix user account in order to access object storage capabilities on the Fenix infrastructure. With the Data Proxy, a user only needs to have an EBRAINS account to access object storage resources (effectively on the same Fenix object storage infrastructure). Data stored in this way is held in the name of the data proxy service account on Fenix, and the Data Proxy tracks who has access to which data. |
| |
6.1 | 41 | |
| |
8.1 | 42 | (% style="text-align: justify;" %) |
| |
20.1 | 43 | === Permissions === |
| |
6.1 | 44 | |
| |
8.1 | 45 | (% style="text-align: justify;" %) |
| |
20.1 | 46 | We use the Collaboratory authorization system to manage permissions in the Data Proxy. |
| |
6.1 | 47 | |
| |
20.1 | 48 | (% style="text-align: justify;" %) |
| 49 | A Swift object container can be associated to each collab. Object containers are also known as "buckets" to avoid confusion with other containers (e.g. Docker containers). An EBRAINS user can perform the following actions on a bucket depending on the user's permissions (as defined by the collab's Team) in the collab associated with the bucket. | ||
| 50 | |||
| |
6.1 | 51 | (% border="3" %) |
| |
20.1 | 52 | |=(% style="text-align: justify;" %)Team permissions of a collab|=(% style="text-align: justify;" %)Available actions on that collab's bucket |
| |
8.1 | 53 | |(% style="text-align:justify" %)Viewer|(% style="text-align:justify" %)Read |
| 54 | |(% style="text-align:justify" %)Editor|(% style="text-align:justify" %)Create, Read, Update, Delete | ||
| 55 | |(% style="text-align:justify" %)Admin|(% style="text-align:justify" %)Create, Read, Update, Delete | ||
| |
29.1 | 56 | |(% style="text-align:justify" %)Not a collab member|(% style="text-align:justify" %)Read access only if the collab is public |
| |
6.1 | 57 | |
| |
13.1 | 58 | (% class="wikigeneratedid" %) |
| |
20.1 | 59 | You can access the buckets in the "Bucket" navigation element in every collab. |
| |
6.1 | 60 | |
| |
20.1 | 61 | === Collaboratory bucket vs drive === |
| |
12.1 | 62 | |
| |
20.1 | 63 | A collab offers 2 main locations to store files: a drive and a bucket. The drive offers more advanced features like recognition of file formats (Office, Markdown, PDF) with applications specific to each, simplified version control, smart links. The bucket on the other hand offers larger storage capacity and better bandwidth. The bucket is recommended for datasets (brain scans, EEG, derived data) and videos (including for streaming). |
| 64 | |||
| 65 | === API === | ||
| 66 | |||
| 67 | The API is self-documented using Swagger UI. You can access it here: [[https:~~/~~/data-proxy.ebrains.eu/api/docs>>https://data-proxy.ebrains.eu/api/docs]] or in the [[API Documentation>>doc:API Documentation]] wiki page of this collab. | ||
| |
1.1 | 68 | ))) |
| |
12.1 | 69 | ))) |