Last modified by luehrs on 2023/04/12 09:26
From version 7.2
edited by luehrs
on 2023/04/12 07:25
Change comment: There is no comment for this version
To version 3.4
edited by luehrs
on 2023/04/12 07:11
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -64,9 +64,9 @@
64 64  
65 65  To install the RPM package one needs to run the following command:
66 66  
67 -##[promt]$ yum install fts-rest-client-1.0.0-%{_release}.el7.noarch##
67 +{{{[promt]$ yum install fts-rest-client-1.0.0-%{_release}.el7.noarch}}}
68 68  
69 -Should the user prefer to build the fts-rest-client him/herself, the scripts can be found in the fts-rest-flask repository here: [[https:~~/~~/gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift>>url:https://gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift]]. The user can clone the repository, switch to the swift branch, and ##cd## into it. We advise creating a ##venv## for this project. Then run ##pip install .## or ##python setup.py## install to install the fts package. After that, the user may run the scripts in src/cli the same way he/she uses the FTS commands.
69 +Should the user prefer to build the fts-rest-client him/herself, the scripts can be found in the fts-rest-flask repository here: [[https:~~/~~/gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift>>url:https://gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift]]. The user can clone the repository, switch to the swift branch, and cd into it. We advise creating a venv for this project. Then run pip install . or python setup.py install to install the fts package. After that, the user may run the scripts in src/cli the same way he/she uses the FTS commands.
70 70  
71 71  === Valid OIDC token ===
72 72  
... ... @@ -75,145 +75,21 @@
75 75  For CLI users, Device Code Flow ([[RFC 8628>>url:https://www.rfc-editor.org/rfc/rfc8628.html]]) is the recommended authorisation grant to obtain OIDC tokens. The user can install [[oidc-agent>>url:https://indigo-dc.gitbook.io/oidc-agent/]] to perform such workflows. To use oidc-agent with Fenix AAI, a development version 4.4.1 is needed; such packages can be found here: [[https:~~/~~/repo.data.kit.edu/devel/>>url:https://repo.data.kit.edu/devel/]].
76 76  
77 77  1. Add the following line to /etc/oidc-agent/pubclients.config (for the first time):
78 -##APP-EF25A717-2059-461D-8275-A71CC098B82B@https:~/~/central-proxy.fenix-ri.eu##
79 -1. Initiate oidc-agent:
80 -##[promt]$ eval $(oidc-agent)##
81 -1. The user needs to run the oidc-gen command to generate account info for Fenix:
82 -##[promt]$ oidc-gen ~-~-pub ~-~-issuer https:~/~/central-proxy.fenix-ri.eu \
83 -~-~-configuration-endpoint https:~/~/central-webapp.fenix-ri.eu/oidc/.well-known/openid-configuration \
84 -~-~-flow device ~-~-no-save <define_an_account_name>##
85 -Follow the instructions of the output, login to Fenix AAI.
86 -1. Obtain an OIDC token:
87 -##[promt]$ oidc-token <defined_account_name>##
88 88  
89 -== How to perform a data transfer with FTS ==
79 +{{{APP-EF25A717-2059-461D-8275-A71CC098B82B@https://central-proxy.fenix-ri.eu}}}
90 90  
91 -In this section, the user can find useful commands and examples. More detailed commands are available at [[FTS3 Documentation - CERN>>url:https://fts3-docs.web.cern.ch/fts3-docs/]].
81 +1. Initiate oidc-agent:
92 92  
93 -=== FTS commands ===
83 +[promt]$ eval $(oidc-agent)
94 94  
95 -The FTS transfers are managed by the following commands:
85 +1. The user needs to run the oidc-gen command to generate account info for Fenix:
96 96  
97 -* fts-rest-transfer-submit             //to submit transfer job//
98 -* fts-rest-transfer-status              //to query a transfer job//
99 -* fts-rest-transfer-list                   //to list transfer jobs//
100 -* fts-rest-transfer-cancel             //to cancel a transfer job//
87 +[promt]$ oidc-gen ~-~-pub ~-~-issuer https:~/~/central-proxy.fenix-ri.eu \ ~-~-configuration-endpoint https:~/~/central-webapp.fenix-ri.eu/oidc/.well-known/openid-configuration \
101 101  
102 -=== Source/destination URL format ===
89 +~-~-flow device ~-~-no-save <define_an_account_name>
103 103  
104 -FTS is a protocol-specific transfer service. If the user wants to transfer from/to a Swift endpoint, she/he should specify ##swifts:~/~/ + storage name## instead of ##https:~/~/ + storage name## in the source/destination URL. The ##swifts:~/~/## is to specify she/he is using Swift protocol with FTS.
91 +Follow the instructions of the output, login to Fenix AAI.
105 105  
106 -The format of a Swift url to submit to FTS is ##swifts:~/~/ + storage name + container name + (folder name(s)) + object name##.
93 +1. Obtain an OIDC token:
107 107  
108 -For example, ##swifts:~/~/swift.bsc.es/testcontainer/data.txt## locates the data.txt file in container testcontainer in the BSC object store.
109 -
110 -=== Submit a Swift-to-Swift transfer ===
111 -
112 -We give instructions in this manual for Swift-to-Swift data transfer only, because the current available Fenix archival repositories are Swift object stores. For other types of storage, users may refer to the official FTS guide [[FTS3 Documentation - CERN>>url:https://fts3-docs.web.cern.ch/fts3-docs/]].
113 -
114 -For Swift-to-Swift transfer, the user has to specify OS project ids which indicate the data source and destinations.
115 -
116 -Users can find the project id by either:
117 -
118 -1. using the command ##openstack project list## with the Openstack CLI client.
119 -1. login to the Openstack dashboard, go to //Identity -> Projects//.
120 -For example, on [[https:~~/~~/ncloud.bsc.es>>url:https://ncloud.bsc.es]]:
121 -[[image:image-20230412091641-1.png]]
122 -and on [[https:~~/~~/castor.cscs.ch>>url:https://castor.cscs.ch]]:
123 -[[image:image-20230412091702-2.png]]
124 -
125 -To transfer from/to the object storages at JSC or CSCS, the user also needs to specify an OS token at the moment.
126 -
127 -* For **JSC** users, please refer to the documentation [[here>>url:https://apps.fz-juelich.de/jsc/hps/just/object-storage.html#user-environment-preparation]] to create the ##rc## file, then do ##openstack token issue##, your OS token will be shown in the “id” field.
128 -* For **CSCS** users, the easiest way to get an OS token is to follow the guide on this GitHub page: [[https:~~/~~/github.com/eth-cscs/openstack/tree/master/cli>>url:https://github.com/eth-cscs/openstack/tree/master/cli]]. When authenticated, the OS token is saved as environment variable ##OS_TOKEN##.
129 -* If the user would like to specify the OS token for **BSC** her/himself (which is not necessary), guidance is provided here: [[https:~~/~~/bsc.es/supportkc/docs-ncloud/access#cli-access>>url:https://bsc.es/supportkc/docs-ncloud/access#cli-access]].
130 -
131 -Below is an example of submitting the transfer with CLI:
132 -
133 -##$fts-rest-transfer-submit \
134 --s https:~/~/fts.bsc.es:8446/ \##
135 -##~-~-access-token $tok \##
136 -##~-~-os-project-id “<source_project_id>:<dest_project_id>” \##
137 -##~-~-os-token “<project_id>:<corresponding_os_token>” \##
138 -##swifts:~/~/object.cscs.ch/<container_name>/<object_name> \##
139 -##swifts:~/~/swift.bsc.es/<container_name>/<object_name>##
140 -
141 -### Using endpoint: https:~/~/fts.bsc.es:8446                                                        
142 -# REST API version: 3.10.0  ##
143 -##Job successfully submitted.##
144 -##Job id: 532086a8-30a9-11eb-b0da-fa163e1833c1##
145 -
146 -Users can use multiple ##~-~-os-token## to specify OS tokens for different projects.
147 -
148 -==== Bulk transfer ====
149 -
150 -Users can use bulk transfer by using the ##~-~-file## option. Then they have to submit a JSON file.
151 -
152 -##$fts-rest-transfer-submit \
153 --s https:~/~/fts.bsc.es:8446/ \##
154 -##~-~-access-token $tok \##
155 -##~-~-file <your_file>.json##
156 -
157 -The format of such files is described here: [[https:~~/~~/fts3-docs.web.cern.ch/fts3-docs/fts-rest/docs/bulk.html>>url:https://fts3-docs.web.cern.ch/fts3-docs/fts-rest/docs/bulk.html]]. Note that the files the user transfers this way still need to be from and to the same object store, i.e., she/he can only transfer a list of files from project A at site 1 to project B at site 2 in one job.
158 -
159 -As it is the same for any Swift-to-Swift transfer, users have to supply OS project ids, and in some cases, OS tokens as well. Users can use the command line options to do so or put it in the ##params## section when using a JSON file. An example of the JSON file for a bulk transfer is shown below:
160 -
161 -##{
162 -"files": [
163 -{
164 - "sources": [
165 - "swifts:~/~/swift.bsc.es/testcontainer/cp1"
166 - ],
167 - "destinations": [
168 - "swifts:~/~/object.cscs.ch/testcontainer/bulk/cp1"
169 - ]
170 -},
171 -{
172 - "sources": [
173 - "swifts:~/~/swift.bsc.es/testcontainer/cp2"
174 - ],
175 - "destinations": [
176 - "swifts:~/~/object.cscs.ch/testcontainer/bulk/cp2"
177 - ]
178 -}
179 -],
180 -"params":{             "os_project_id":"c7ccbe370791496fa44632e6ccb10405:7cc304cc62d44ab180d4a28427e2b64b",
181 - "os_token":[ 
182 - "7cc304cc62d44ab180d4a28427e2b64b:gAAAAA…"
183 - ]
184 -}##
185 -
186 -=== Query a transfer ===
187 -
188 -This command checks the status of the transfer job; a detailed FTS job state machine can be found here: [[https:~~/~~/fts3-docs.web.cern.ch/fts3-docs/docs/state_machine.html>>url:https://fts3-docs.web.cern.ch/fts3-docs/docs/state_machine.html]].
189 -
190 -##$fts-rest-transfer-status \
191 --s https:~/~/fts.bsc.es:8446/ \
192 -~-~-access-token $tok \
193 -<your_job_id>##
194 -
195 -The job ID is given when the user submits the transfer.
196 -
197 -=== List transfers ===
198 -
199 -This command lists the ongoing transfers at the FTS endpoint.
200 -
201 -##$fts-rest-transfer-list \
202 --s https:~/~/fts.bsc.es:8446/ \
203 -~-~-access-token $tok##
204 -
205 -=== Cancel a transfer ===
206 -
207 -This command cancels the transfer job.
208 -
209 -##$fts-rest-transfer-cancel \
210 --s https:~/~/fts.bsc.es:8446/ \
211 -~-~-access-token $tok \
212 -<your_job_id>##
213 -
214 -== Notes ==
215 -
216 -The configuration of the service allows to:
217 -
218 -* transfer object files with a maximum size of 5GB, note that files larger than 512MB will be stored as large objects in Openstack, namely, files are split when larger than 512 MB; each chunk is 256 MB. Note that users may still view/download the files as a whole like normal objects after the upload is complete. Information on large object support can be found here: [[https:~~/~~/docs.openstack.org/swift/latest/overview_large_objects.html>>url:https://docs.openstack.org/swift/latest/overview_large_objects.html]].
219 -* manage up to 1000 concurrent requests. This means that up to 1000 object files, or chunks of an object file, can be sent to the storage system concurrently;
95 +[promt]$ oidc-token <defined_account_name>
image-20230412091641-1.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.luehrs
Size
... ... @@ -1,1 +1,0 @@
1 -24.0 KB
Content
image-20230412091702-2.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.luehrs
Size
... ... @@ -1,1 +1,0 @@
1 -36.3 KB
Content