Version 6.2 by luehrs on 2023/04/12 07:19
Hide last authors
luehrs 1.1 1 == Acronyms ==
2
3 |(% style="width:78px" %)AAI|(% style="width:1257px" %)Authentication and Authorization Infrastructure
4 |(% style="width:78px" %)ARD|(% style="width:1257px" %)Archival Data Repository
5 |(% style="width:78px" %)CLI|(% style="width:1257px" %)Command Line Interface
6 |(% style="width:78px" %)FTS|(% style="width:1257px" %)File Transfer Service
7 |(% style="width:78px" %)FURMS|(% style="width:1257px" %)Fenix User and Resource Management Service
8 |(% style="width:78px" %)HBP|(% style="width:1257px" %)Human Brain Project
9 |(% style="width:78px" %)ICEI|(% style="width:1257px" %)Interactive Computing E-Infrastructure
10 |(% style="width:78px" %)IdP|(% style="width:1257px" %)Identity Provider
11 |(% style="width:78px" %)OIDC|(% style="width:1257px" %)OpenID Connect
12
luehrs 2.2 13 == Introduction ==
14
15 This user guide describes how to access and submit a transfer job to FTS, the Central Data Transfer service designed for the HBP community, at BSC via CLI. In case other servers are installed at other ICEI sites in the future, a few changes will be needed to this user guide. This document refers to Use Case 1 described in the “FTS3 - Service deployment strategy” document:
16
17 [[image:image-20230412090045-1.png]]
18
luehrs 3.2 19 == Requirements ==
20
21 To use the service correctly, the user needs
22
23 * valid credentials (to access both to source hosting site and destination hosting site);
24 * valid account(s) (defined both at the source hosting site and at the destination hosting site);
25 * at least one object storage as the source; and
26 * at least one object storage as the destination.
27
28 Additionally, the user needs:
29
30 * the FTS REST client; and
31 * a valid token to be mapped.
32
33 In case the user would like to check if something is missing, she/he can open an issue with the respective local User Support services of ICEI hosting sites.
34
35 **Object store URLs**
36
37 Presently, only three ICEI sites [[are providing object storages to the HBP community>>https://fenix-ri.eu/infrastructure/resources/available-resources]]: BSC (ES), CSCS (CH), and JSC (DE). As soon as other sites will start the provisioning of these resources, the list of URLs below will be updated.
38
39 * CSCS: [[https:~~/~~/object.cscs.ch>>https://object.cscs.ch]]
40 * BSC: [[https:~~/~~/swift.bsc.es>>https://swift.bsc.es]]
41 * JSC: [[https:~~/~~/just-object.fz-juelich.de:8080>>https://just-object.fz-juelich.de:8080]]
42
43 Presently the Central Data Transfer service, to transfer data between object storages, is provided at BSC. The endpoint of the service is reported below. In case other ICEI sites will host the service, the list below will be updated.
44
45 **FTS endpoint**: [[https:~~/~~/fts.bsc.es:8446>>https://fts.bsc.es:8446]]
46
luehrs 3.3 47 == How to obtain... ==
luehrs 3.2 48
luehrs 3.3 49 In this section, the user will find some useful instructions to solve the requirements.
50
51 === Valid credentials ===
52
53 To access the service, the user needs to be defined in the local user databases, and in FURMS user database. Valid credentials (username and password) are provided by ICEI local sites, according to Fenix and (possible) local policies applied, if the user is a Principal Investigator or Collaborator in an approved project. Please note that proposal submission to obtain resources is open every working day for the HBP community. More information is available at: [[https:~~/~~/wiki.ebrains.eu/bin/view/Collabs/fenix-icei/>>url:https://wiki.ebrains.eu/bin/view/Collabs/fenix-icei/]]
54
55 === Valid account(s) and object storages ===
56
57 To obtain a valid account on FURMS, the user needs to register at [[https:~~/~~/ebrains.eu/register/>>url:https://ebrains.eu/register/]]. After her/his proposal (sub-section above) has been approved, and the project has been defined at the hosting sites, the user will receive valid local credentials, too.
58
59 To obtain object storage resources, the user needs to request them explicitly in the submission form of the proposal, asking for ARD resources and specifying that object storage resources are needed. Please note that the submission form allows applicants to ask for resources distributed across ICEI sites with a single proposal.
60
61 === FTS REST client ===
62
63 To perform a data transfer through the FTS server, you need to install the fts-rest-client. The RPM package will provide the CLI tools for the FTS server. The tested version is available in the EBRAINS collaboratory bucket: [[https:~~/~~/data-proxy.ebrains.eu/api/v1/permalinks/4780cb34-8c71-497e-85bd-3bf58a9fc21e>>url:https://data-proxy.ebrains.eu/api/v1/permalinks/4780cb34-8c71-497e-85bd-3bf58a9fc21e]].
64
65 To install the RPM package one needs to run the following command:
66
luehrs 4.2 67 ##[promt]$ yum install fts-rest-client-1.0.0-%{_release}.el7.noarch##
luehrs 3.3 68
luehrs 4.2 69 Should the user prefer to build the fts-rest-client him/herself, the scripts can be found in the fts-rest-flask repository here: [[https:~~/~~/gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift>>url:https://gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift]]. The user can clone the repository, switch to the swift branch, and ##cd## into it. We advise creating a ##venv## for this project. Then run ##pip install .## or ##python setup.py## install to install the fts package. After that, the user may run the scripts in src/cli the same way he/she uses the FTS commands.
luehrs 3.3 70
71 === Valid OIDC token ===
72
73 To obtain a valid OIDC token, the user needs to login via her/his institute’s IdP or HBP user account, and she/he should be able to see her/his access token on the [[https:~~/~~/fts.bsc.es>>url:https://fts.bsc.es]] website. The user should copy the token to run the FTS commands. We advise all users to perform this action at least once to have their credentials registered with FTS properly. Please, note that this is a temporary solution for retrieving the access token.
74
luehrs 3.4 75 For CLI users, Device Code Flow ([[RFC 8628>>url:https://www.rfc-editor.org/rfc/rfc8628.html]]) is the recommended authorisation grant to obtain OIDC tokens. The user can install [[oidc-agent>>url:https://indigo-dc.gitbook.io/oidc-agent/]] to perform such workflows. To use oidc-agent with Fenix AAI, a development version 4.4.1 is needed; such packages can be found here: [[https:~~/~~/repo.data.kit.edu/devel/>>url:https://repo.data.kit.edu/devel/]].
luehrs 3.3 76
77 1. Add the following line to /etc/oidc-agent/pubclients.config (for the first time):
luehrs 4.2 78 ##APP-EF25A717-2059-461D-8275-A71CC098B82B@https:~/~/central-proxy.fenix-ri.eu##
79 1. Initiate oidc-agent:
80 ##[promt]$ eval $(oidc-agent)##
81 1. The user needs to run the oidc-gen command to generate account info for Fenix:
82 ##[promt]$ oidc-gen ~-~-pub ~-~-issuer https:~/~/central-proxy.fenix-ri.eu \
83 ~-~-configuration-endpoint https:~/~/central-webapp.fenix-ri.eu/oidc/.well-known/openid-configuration \
84 ~-~-flow device ~-~-no-save <define_an_account_name>##
85 Follow the instructions of the output, login to Fenix AAI.
86 1. Obtain an OIDC token:
87 ##[promt]$ oidc-token <defined_account_name>##
luehrs 3.3 88
luehrs 4.2 89 == How to perform a data transfer with FTS ==
luehrs 3.3 90
luehrs 4.2 91 In this section, the user can find useful commands and examples. More detailed commands are available at [[FTS3 Documentation - CERN>>url:https://fts3-docs.web.cern.ch/fts3-docs/]].
luehrs 3.3 92
luehrs 4.2 93 === FTS commands ===
luehrs 3.3 94
luehrs 4.2 95 The FTS transfers are managed by the following commands:
luehrs 3.3 96
luehrs 4.2 97 * fts-rest-transfer-submit             //to submit transfer job//
98 * fts-rest-transfer-status              //to query a transfer job//
99 * fts-rest-transfer-list                   //to list transfer jobs//
100 * fts-rest-transfer-cancel             //to cancel a transfer job//
luehrs 3.3 101
luehrs 4.2 102 === Source/destination URL format ===
luehrs 3.3 103
luehrs 4.2 104 FTS is a protocol-specific transfer service. If the user wants to transfer from/to a Swift endpoint, she/he should specify ##swifts:~/~/ + storage name## instead of ##https:~/~/ + storage name## in the source/destination URL. The ##swifts:~/~/## is to specify she/he is using Swift protocol with FTS.
luehrs 3.3 105
luehrs 4.2 106 The format of a Swift url to submit to FTS is ##swifts:~/~/ + storage name + container name + (folder name(s)) + object name##.
luehrs 3.3 107
luehrs 4.2 108 For example, ##swifts:~/~/swift.bsc.es/testcontainer/data.txt## locates the data.txt file in container testcontainer in the BSC object store.
luehrs 6.2 109
110 === Submit a Swift-to-Swift transfer ===
111
112 We give instructions in this manual for Swift-to-Swift data transfer only, because the current available Fenix archival repositories are Swift object stores. For other types of storage, users may refer to the official FTS guide [[FTS3 Documentation - CERN>>url:https://fts3-docs.web.cern.ch/fts3-docs/]].
113
114 For Swift-to-Swift transfer, the user has to specify OS project ids which indicate the data source and destinations.
115
116 Users can find the project id by either:
117
118 1. using the command ##openstack project list## with the Openstack CLI client.
119 1. login to the Openstack dashboard, go to //Identity -> Projects//.
120 For example, on [[https:~~/~~/ncloud.bsc.es>>url:https://ncloud.bsc.es]]:
121 [[image:image-20230412091641-1.png]]
122 and on [[https:~~/~~/castor.cscs.ch>>url:https://castor.cscs.ch]]:
123 [[image:image-20230412091702-2.png]]
124
125 To transfer from/to the object storages at JSC or CSCS, the user also needs to specify an OS token at the moment.
126
127 * For **JSC** users, please refer to the documentation [[here>>url:https://apps.fz-juelich.de/jsc/hps/just/object-storage.html#user-environment-preparation]] to create the ##rc## file, then do ##openstack token issue##, your OS token will be shown in the “id” field.
128 * For **CSCS** users, the easiest way to get an OS token is to follow the guide on this GitHub page: [[https:~~/~~/github.com/eth-cscs/openstack/tree/master/cli>>url:https://github.com/eth-cscs/openstack/tree/master/cli]]. When authenticated, the OS token is saved as environment variable ##OS_TOKEN##.
129 * If the user would like to specify the OS token for **BSC** her/himself (which is not necessary), guidance is provided here: [[https:~~/~~/bsc.es/supportkc/docs-ncloud/access#cli-access>>url:https://bsc.es/supportkc/docs-ncloud/access#cli-access]].
130
131 Below is an example of submitting the transfer with CLI:
132
133 ##$fts-rest-transfer-submit \
134 -s https:~/~/fts.bsc.es:8446/ \##
135 ##~-~-access-token $tok \##
136 ##~-~-os-project-id “<source_project_id>:<dest_project_id>” \##
137 ##~-~-os-token “<project_id>:<corresponding_os_token>” \##
138 ##swifts:~/~/object.cscs.ch/<container_name>/<object_name> \##
139 ##swifts:~/~/swift.bsc.es/<container_name>/<object_name>##
140
141 ### Using endpoint: https:~/~/fts.bsc.es:8446                                                        
142 # REST API version: 3.10.0  ##
143 ##Job successfully submitted.##
144 ##Job id: 532086a8-30a9-11eb-b0da-fa163e1833c1##
145
146 Users can use multiple ##~-~-os-token## to specify OS tokens for different projects.