Last modified by luehrs on 2023/04/12 09:26
From version 4.1
edited by luehrs
on 2023/04/12 07:11
Change comment: There is no comment for this version
To version 6.2
edited by luehrs
on 2023/04/12 07:19
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -64,9 +64,9 @@
64 64  
65 65  To install the RPM package one needs to run the following command:
66 66  
67 -{{{[promt]$ yum install fts-rest-client-1.0.0-%{_release}.el7.noarch}}}
67 +##[promt]$ yum install fts-rest-client-1.0.0-%{_release}.el7.noarch##
68 68  
69 -Should the user prefer to build the fts-rest-client him/herself, the scripts can be found in the fts-rest-flask repository here: [[https:~~/~~/gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift>>url:https://gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift]]. The user can clone the repository, switch to the swift branch, and cd into it. We advise creating a venv for this project. Then run pip install . or python setup.py install to install the fts package. After that, the user may run the scripts in src/cli the same way he/she uses the FTS commands.
69 +Should the user prefer to build the fts-rest-client him/herself, the scripts can be found in the fts-rest-flask repository here: [[https:~~/~~/gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift>>url:https://gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift]]. The user can clone the repository, switch to the swift branch, and ##cd## into it. We advise creating a ##venv## for this project. Then run ##pip install .## or ##python setup.py## install to install the fts package. After that, the user may run the scripts in src/cli the same way he/she uses the FTS commands.
70 70  
71 71  === Valid OIDC token ===
72 72  
... ... @@ -75,21 +75,72 @@
75 75  For CLI users, Device Code Flow ([[RFC 8628>>url:https://www.rfc-editor.org/rfc/rfc8628.html]]) is the recommended authorisation grant to obtain OIDC tokens. The user can install [[oidc-agent>>url:https://indigo-dc.gitbook.io/oidc-agent/]] to perform such workflows. To use oidc-agent with Fenix AAI, a development version 4.4.1 is needed; such packages can be found here: [[https:~~/~~/repo.data.kit.edu/devel/>>url:https://repo.data.kit.edu/devel/]].
76 76  
77 77  1. Add the following line to /etc/oidc-agent/pubclients.config (for the first time):
78 +##APP-EF25A717-2059-461D-8275-A71CC098B82B@https:~/~/central-proxy.fenix-ri.eu##
79 +1. Initiate oidc-agent:
80 +##[promt]$ eval $(oidc-agent)##
81 +1. The user needs to run the oidc-gen command to generate account info for Fenix:
82 +##[promt]$ oidc-gen ~-~-pub ~-~-issuer https:~/~/central-proxy.fenix-ri.eu \
83 +~-~-configuration-endpoint https:~/~/central-webapp.fenix-ri.eu/oidc/.well-known/openid-configuration \
84 +~-~-flow device ~-~-no-save <define_an_account_name>##
85 +Follow the instructions of the output, login to Fenix AAI.
86 +1. Obtain an OIDC token:
87 +##[promt]$ oidc-token <defined_account_name>##
78 78  
79 -{{{APP-EF25A717-2059-461D-8275-A71CC098B82B@https://central-proxy.fenix-ri.eu}}}
89 +== How to perform a data transfer with FTS ==
80 80  
81 -1. Initiate oidc-agent:
91 +In this section, the user can find useful commands and examples. More detailed commands are available at [[FTS3 Documentation - CERN>>url:https://fts3-docs.web.cern.ch/fts3-docs/]].
82 82  
83 -[promt]$ eval $(oidc-agent)
93 +=== FTS commands ===
84 84  
85 -1. The user needs to run the oidc-gen command to generate account info for Fenix:
95 +The FTS transfers are managed by the following commands:
86 86  
87 -[promt]$ oidc-gen ~-~-pub ~-~-issuer https:~/~/central-proxy.fenix-ri.eu \ ~-~-configuration-endpoint https:~/~/central-webapp.fenix-ri.eu/oidc/.well-known/openid-configuration \
97 +* fts-rest-transfer-submit             //to submit transfer job//
98 +* fts-rest-transfer-status              //to query a transfer job//
99 +* fts-rest-transfer-list                   //to list transfer jobs//
100 +* fts-rest-transfer-cancel             //to cancel a transfer job//
88 88  
89 -~-~-flow device ~-~-no-save <define_an_account_name>
102 +=== Source/destination URL format ===
90 90  
91 -Follow the instructions of the output, login to Fenix AAI.
104 +FTS is a protocol-specific transfer service. If the user wants to transfer from/to a Swift endpoint, she/he should specify ##swifts:~/~/ + storage name## instead of ##https:~/~/ + storage name## in the source/destination URL. The ##swifts:~/~/## is to specify she/he is using Swift protocol with FTS.
92 92  
93 -1. Obtain an OIDC token:
106 +The format of a Swift url to submit to FTS is ##swifts:~/~/ + storage name + container name + (folder name(s)) + object name##.
94 94  
95 -[promt]$ oidc-token <defined_account_name>
108 +For example, ##swifts:~/~/swift.bsc.es/testcontainer/data.txt## locates the data.txt file in container testcontainer in the BSC object store.
109 +
110 +=== Submit a Swift-to-Swift transfer ===
111 +
112 +We give instructions in this manual for Swift-to-Swift data transfer only, because the current available Fenix archival repositories are Swift object stores. For other types of storage, users may refer to the official FTS guide [[FTS3 Documentation - CERN>>url:https://fts3-docs.web.cern.ch/fts3-docs/]].
113 +
114 +For Swift-to-Swift transfer, the user has to specify OS project ids which indicate the data source and destinations.
115 +
116 +Users can find the project id by either:
117 +
118 +1. using the command ##openstack project list## with the Openstack CLI client.
119 +1. login to the Openstack dashboard, go to //Identity -> Projects//.
120 +For example, on [[https:~~/~~/ncloud.bsc.es>>url:https://ncloud.bsc.es]]:
121 +[[image:image-20230412091641-1.png]]
122 +and on [[https:~~/~~/castor.cscs.ch>>url:https://castor.cscs.ch]]:
123 +[[image:image-20230412091702-2.png]]
124 +
125 +To transfer from/to the object storages at JSC or CSCS, the user also needs to specify an OS token at the moment.
126 +
127 +* For **JSC** users, please refer to the documentation [[here>>url:https://apps.fz-juelich.de/jsc/hps/just/object-storage.html#user-environment-preparation]] to create the ##rc## file, then do ##openstack token issue##, your OS token will be shown in the “id” field.
128 +* For **CSCS** users, the easiest way to get an OS token is to follow the guide on this GitHub page: [[https:~~/~~/github.com/eth-cscs/openstack/tree/master/cli>>url:https://github.com/eth-cscs/openstack/tree/master/cli]]. When authenticated, the OS token is saved as environment variable ##OS_TOKEN##.
129 +* If the user would like to specify the OS token for **BSC** her/himself (which is not necessary), guidance is provided here: [[https:~~/~~/bsc.es/supportkc/docs-ncloud/access#cli-access>>url:https://bsc.es/supportkc/docs-ncloud/access#cli-access]].
130 +
131 +Below is an example of submitting the transfer with CLI:
132 +
133 +##$fts-rest-transfer-submit \
134 +-s https:~/~/fts.bsc.es:8446/ \##
135 +##~-~-access-token $tok \##
136 +##~-~-os-project-id “<source_project_id>:<dest_project_id>” \##
137 +##~-~-os-token “<project_id>:<corresponding_os_token>” \##
138 +##swifts:~/~/object.cscs.ch/<container_name>/<object_name> \##
139 +##swifts:~/~/swift.bsc.es/<container_name>/<object_name>##
140 +
141 +### Using endpoint: https:~/~/fts.bsc.es:8446                                                        
142 +# REST API version: 3.10.0  ##
143 +##Job successfully submitted.##
144 +##Job id: 532086a8-30a9-11eb-b0da-fa163e1833c1##
145 +
146 +Users can use multiple ##~-~-os-token## to specify OS tokens for different projects.
image-20230412091641-1.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.luehrs
Size
... ... @@ -1,0 +1,1 @@
1 +24.0 KB
Content
image-20230412091702-2.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.luehrs
Size
... ... @@ -1,0 +1,1 @@
1 +36.3 KB
Content