Wiki source code of Storing data in user space

Version 7.1 by allan on 2019/11/26 16:17
Show last authors
1 This article describes a workflow that you can follow to use the Collaboratory.drive as a backend for your service to be able to store and read data inside a privatre user space.
2
3 == Solution description ==
4
5 Your Keycloak client can be setup to have a service account linked to it. This service account being seen as a user by Keycloak, it can log in the Collaboratory.drive to have its user account synchronised there.
6
7 From this point, everything is set up to let your service account create and share files and folders to existing users. This can be achieved by using the existing Seafile API (the tool behind the Collaboratory.drive).
8
9 == Creating a service account ==
10
11 If needed, follow the [[guide to create an OpenID Connect client>>doc:Collabs.collaboratory-community-apps.Community App Developer Guide.WebHome||anchor="HCreatingyourOpenIDConnectclient"]].
12
13 You will need to [[modify your client>>doc:Collabs.collaboratory-community-apps.Community App Developer Guide.WebHome||anchor="HModifyingyourclient"]] to allow service accounts:
14
15 {{code language="bash"}}
16 # Set your registration token
17 clb_reg_token=...
18
19 # Update the client
20 curl -X PUT https://iam.humanbrainproject.eu/auth/realms/hbp/clients-registrations/default/my-awesome-client \
21 -H "Authorization: Bearer ${clb_reg_token}" \
22 -H 'Content-Type: application/json' \
23 -d '{
24 "clientId": "my-awesome-client",
25 "serviceAccountsEnabled": true
26 }' |
27
28 # Prettify the JSON response
29 json_pp;
30 {{/code}}
31
32 == Creating a user account for the service account in the Collaboratory.drive ==
33
34 This step requires admin privileges. Please send a request to [[support@humanbrainproject.eu>>mailto:support@humanbrainproject.eu]] in order to get help.
35
36 The steps for the admins are described are the following:
37
38 1. get the service account sub
39 1. enable the service account user
40 1. impersonate the service account
41 1. log in Collaboratory.drive
42
43 === Getting the service account sub ===
44
45 One way to get the service account is to request a token with its credentials.
46
47 {{code language="bash"}}
48 # Set the client id and secret
49 clb_client_id=...
50 clb_client_secret=...
51
52 # Call the token endpoint
53 curl -X POST https://iam-dev.humanbrainproject.eu/auth/realms/hbp/protocol/openid-connect/token \
54 -d 'grant_type=client_credentials' \
55 -d "client_id=${clb_client_id}" \
56 -d "client_secret=${clb_client_secret}" |
57
58 # Prettify the JSON response
59 json_pp;
60 {{/code}}
61
62 Fetch the access token from the response and use a tool to decode its payload. [[https://jwt.io/]] is one option. Copy the sub from the payload.
63
64 === Enabling the service account user ===
65
66 Navigate to https://iam.humanbrainproject.eu/auth/admin/master/console/#/realms/hbp/users/$sub (replacing $sub with the value you got at the previous step).
67
68 Set the "Email Verified" value to "On" and remove any "Required User Actions".
69
70 You can now impersonate the user and log in the Collaboratory.drive, which will create the user account for the service account.
71
72 == Creating the user space ==
73
74 The general process is the following:
75
76 1. Fetch a token for your service account to be able to discuss with the drive API.
77 1. Your service needs to get its default library id. It is where it will create the users' spaces.
78 1. For a given user, your service should create a folder, using a unique identifier (either the sub, username or email).
79
80 From this point, your service can now store and read data linked to users accounts.
81
82 If your service needs the data to be available in notebooks, it will need to share it with the user:
83
84 1. Inside the user folder, your service should create a folder with a name that would be common across users.
85 1. Your service should now share the inside folder with the user.
86
87 This way, notebooks will be able to refer to your service data with a common path for every user.
88
89 === Getting an API token ===
90
91 The first step is to fetch an access token for your service account. This can be done with the following request:
92
93 {{code language="bash"}}
94 # Set the client id and secret
95 clb_client_id=...
96 clb_client_secret=...
97
98 # Call the token endpoint
99 curl -X POST https://iam-dev.humanbrainproject.eu/auth/realms/hbp/protocol/openid-connect/token \
100 -d 'grant_type=client_credentials' \
101 -d "client_id=${clb_client_id}" \
102 -d "client_secret=${clb_client_secret}" |
103
104 # Prettify the JSON response
105 json_pp;
106 {{/code}}
107
108 The response will be similar to:
109
110 {{code language="javascript"}}
111 {
112 "expires_in" : 108000,
113 "not-before-policy" : 1563261088,
114 "access_token" : "eyJhbGciOiJSU...",
115 "session_state" : "4882dbae-56dc-4a91-b8ae-8ad07117d4af",
116 "refresh_expires_in" : 14400,
117 "refresh_token" : "eyJhbGciOiJIUz...",
118 "token_type" : "bearer",
119 "scope" : "openid roles email"
120 }
121 {{/code}}
122
123 Fetch the access token from this response.
124
125 The next step is to get an API token with the access token:
126
127 {{code language="bash"}}
128 # Set the access token value
129 clb_access_token=...
130
131 # Call the token endpoint
132 curl -X GET https://drive.humanbrainproject.eu/api2/account/token/ \
133 -H "Authorization: Bearer ${clb_access_token}"
134 {{/code}}
135
136 The response will look like (% class="mark" %)##1c1345da8a99b36168afef92df7f83af8b4ca6f0##(%%). This is the API token that you will need to use in your ##Authorization## header to discuss with the Collaboratory.drive API.
137
138 {{warning}}
139 Note that, unlike access tokens , the API token needs an authorisation header in the form of "##Authorization: **Token** your-api-token"## (and not Bearer!).
140 {{/warning}}
141
142 === Fetching your service account default library ===
143
144 Note: from this point, you can refer to the [[documentation of Seafile>>https://download.seafile.com/published/web-api/v2.1-admin/]] to make API calls to the Collaboratory.drive.
145
146 {{code language="bash"}}
147 # Set the API token
148 clb_api_token=...
149
150 # Call the default-repo endpoint
151 curl -X GET https://drive.humanbrainproject.eu/api2/default-repo/ \
152 -H "Authorization: Token ${clb_api_token}" |
153
154 # Prettify the JSON response
155 json_pp;
156 {{/code}}
157
158 You will get a response in the form of:
159
160 {{code language="javascript"}}
161 {
162 "repo_id": "175a7b2e-f9f5-4f3c-a9e9-84c4e995f1ea",
163 "exists": true
164 }
165 {{/code}}
166
167 The `repo_id` is the identifier of the default library of your service account.
168
169 === Creating a folder for a given user ===
170
171 In order to separate your users data, you should create a folder for each user, based on a unique identifier of the user. The most secure identifier you can use is the `sub` as it is an internal unique identifier of IAM.
172
173 The `username` or `email` are also valid unique identifier but, in some rare cases, they might be claimed by a different user in the future.
174
175 Creating a folder is done with the following call:
176
177 {{code language="bash"}}
178 # Set the API token, your library id and the folder path
179 clb_api_token=...
180 clb_repo_id=...
181 clb_folder_path=/my/user/folder/path
182
183 # Call the default-repo endpoint
184 curl -X POST \
185 "https://drive.humanbrainproject.eu/api2/repos/${clb_repo_id}/dir/" \
186 -H "Authorization: Token ${clb_api_token}" \
187 -d "p=${clb_folder_path}"
188 -F 'operation=mkdir' |
189
190 # Prettify the JSON response
191 json_pp;
192 {{/code}}
193
194 Please note that you cannot create a path of folders all at once. You will need to make one call at each level of your path.
195
196 === Writing data in the user space ===
197
198 TODO
199
200 === Fetching data from the user space ===
201
202 TODO
203
204 == Sharing data with notebooks ==
205
206 The Collaboratory.drive is mounted in the Jupyter container of the user. This means that notebooks can access data stored in the Collaboratory.drive of the user.
207
208 Let's imagine your service generates data that should be ingested by a notebook. In order for the notebook to be usable by any user, the path to the data must be the same for each user.
209
210 When sharing a folder `/path/to/the/shared_folder` to a user, the folder gets reachable at the path `drive/share with me/shared_folder`. As you can see, only the name of the folder is shared with the user.
211
212 Imagine you want to generate data per user and you want notebooks to refer to this data in a folder named `my-app-data`. You will need to create a folder in your service account default library in the following form: ##/path/to/user/spaces/in/your/library/${user-id}/my-app-data##.
213
214 Once the folder is created, you will need to share it with the user.
215
216 === Sharing a folder with a user ===
217
218 {{code language="bash"}}
219 # Set the parameters
220 clb_api_token=...
221 clb_repo_id=...
222 clb_username=...
223 clb_folder_path="/path/to/user/spaces/in/your/library/${clb_username}/my-app-data"
224 clb_permission=rw # r=read, w=write
225
226 # Call the default-repo endpoint
227 curl -X PUT "https://drive.humanbrainproject.eu/api2/repos/${clb_repo_id}/dir/shared_items/" \
228 -H "Authorization: Token ${clb_api_token}" \
229 -d "p=${clb_folder_path}"
230 -F 'share_type=user' \
231 -F "permission=${clb_permission}" \
232 -F "username=${clb_username}@humanbrainproject.eu"
233 {{/code}}
234
235 You will get a response in the form of:
236
237 {{code language="javascript"}}
238 // TODO
239 {{/code}}