Last modified by luehrs on 2023/04/12 09:26
From version 6.2
edited by luehrs
on 2023/04/12 07:19
Change comment: There is no comment for this version
To version 4.1
edited by luehrs
on 2023/04/12 07:11
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -64,9 +64,9 @@
64 64  
65 65  To install the RPM package one needs to run the following command:
66 66  
67 -##[promt]$ yum install fts-rest-client-1.0.0-%{_release}.el7.noarch##
67 +{{{[promt]$ yum install fts-rest-client-1.0.0-%{_release}.el7.noarch}}}
68 68  
69 -Should the user prefer to build the fts-rest-client him/herself, the scripts can be found in the fts-rest-flask repository here: [[https:~~/~~/gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift>>url:https://gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift]]. The user can clone the repository, switch to the swift branch, and ##cd## into it. We advise creating a ##venv## for this project. Then run ##pip install .## or ##python setup.py## install to install the fts package. After that, the user may run the scripts in src/cli the same way he/she uses the FTS commands.
69 +Should the user prefer to build the fts-rest-client him/herself, the scripts can be found in the fts-rest-flask repository here: [[https:~~/~~/gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift>>url:https://gitlab.cern.ch/fts/fts-rest-flask/-/tree/swift]]. The user can clone the repository, switch to the swift branch, and cd into it. We advise creating a venv for this project. Then run pip install . or python setup.py install to install the fts package. After that, the user may run the scripts in src/cli the same way he/she uses the FTS commands.
70 70  
71 71  === Valid OIDC token ===
72 72  
... ... @@ -75,72 +75,21 @@
75 75  For CLI users, Device Code Flow ([[RFC 8628>>url:https://www.rfc-editor.org/rfc/rfc8628.html]]) is the recommended authorisation grant to obtain OIDC tokens. The user can install [[oidc-agent>>url:https://indigo-dc.gitbook.io/oidc-agent/]] to perform such workflows. To use oidc-agent with Fenix AAI, a development version 4.4.1 is needed; such packages can be found here: [[https:~~/~~/repo.data.kit.edu/devel/>>url:https://repo.data.kit.edu/devel/]].
76 76  
77 77  1. Add the following line to /etc/oidc-agent/pubclients.config (for the first time):
78 -##APP-EF25A717-2059-461D-8275-A71CC098B82B@https:~/~/central-proxy.fenix-ri.eu##
79 -1. Initiate oidc-agent:
80 -##[promt]$ eval $(oidc-agent)##
81 -1. The user needs to run the oidc-gen command to generate account info for Fenix:
82 -##[promt]$ oidc-gen ~-~-pub ~-~-issuer https:~/~/central-proxy.fenix-ri.eu \
83 -~-~-configuration-endpoint https:~/~/central-webapp.fenix-ri.eu/oidc/.well-known/openid-configuration \
84 -~-~-flow device ~-~-no-save <define_an_account_name>##
85 -Follow the instructions of the output, login to Fenix AAI.
86 -1. Obtain an OIDC token:
87 -##[promt]$ oidc-token <defined_account_name>##
88 88  
89 -== How to perform a data transfer with FTS ==
79 +{{{APP-EF25A717-2059-461D-8275-A71CC098B82B@https://central-proxy.fenix-ri.eu}}}
90 90  
91 -In this section, the user can find useful commands and examples. More detailed commands are available at [[FTS3 Documentation - CERN>>url:https://fts3-docs.web.cern.ch/fts3-docs/]].
81 +1. Initiate oidc-agent:
92 92  
93 -=== FTS commands ===
83 +[promt]$ eval $(oidc-agent)
94 94  
95 -The FTS transfers are managed by the following commands:
85 +1. The user needs to run the oidc-gen command to generate account info for Fenix:
96 96  
97 -* fts-rest-transfer-submit             //to submit transfer job//
98 -* fts-rest-transfer-status              //to query a transfer job//
99 -* fts-rest-transfer-list                   //to list transfer jobs//
100 -* fts-rest-transfer-cancel             //to cancel a transfer job//
87 +[promt]$ oidc-gen ~-~-pub ~-~-issuer https:~/~/central-proxy.fenix-ri.eu \ ~-~-configuration-endpoint https:~/~/central-webapp.fenix-ri.eu/oidc/.well-known/openid-configuration \
101 101  
102 -=== Source/destination URL format ===
89 +~-~-flow device ~-~-no-save <define_an_account_name>
103 103  
104 -FTS is a protocol-specific transfer service. If the user wants to transfer from/to a Swift endpoint, she/he should specify ##swifts:~/~/ + storage name## instead of ##https:~/~/ + storage name## in the source/destination URL. The ##swifts:~/~/## is to specify she/he is using Swift protocol with FTS.
91 +Follow the instructions of the output, login to Fenix AAI.
105 105  
106 -The format of a Swift url to submit to FTS is ##swifts:~/~/ + storage name + container name + (folder name(s)) + object name##.
93 +1. Obtain an OIDC token:
107 107  
108 -For example, ##swifts:~/~/swift.bsc.es/testcontainer/data.txt## locates the data.txt file in container testcontainer in the BSC object store.
109 -
110 -=== Submit a Swift-to-Swift transfer ===
111 -
112 -We give instructions in this manual for Swift-to-Swift data transfer only, because the current available Fenix archival repositories are Swift object stores. For other types of storage, users may refer to the official FTS guide [[FTS3 Documentation - CERN>>url:https://fts3-docs.web.cern.ch/fts3-docs/]].
113 -
114 -For Swift-to-Swift transfer, the user has to specify OS project ids which indicate the data source and destinations.
115 -
116 -Users can find the project id by either:
117 -
118 -1. using the command ##openstack project list## with the Openstack CLI client.
119 -1. login to the Openstack dashboard, go to //Identity -> Projects//.
120 -For example, on [[https:~~/~~/ncloud.bsc.es>>url:https://ncloud.bsc.es]]:
121 -[[image:image-20230412091641-1.png]]
122 -and on [[https:~~/~~/castor.cscs.ch>>url:https://castor.cscs.ch]]:
123 -[[image:image-20230412091702-2.png]]
124 -
125 -To transfer from/to the object storages at JSC or CSCS, the user also needs to specify an OS token at the moment.
126 -
127 -* For **JSC** users, please refer to the documentation [[here>>url:https://apps.fz-juelich.de/jsc/hps/just/object-storage.html#user-environment-preparation]] to create the ##rc## file, then do ##openstack token issue##, your OS token will be shown in the “id” field.
128 -* For **CSCS** users, the easiest way to get an OS token is to follow the guide on this GitHub page: [[https:~~/~~/github.com/eth-cscs/openstack/tree/master/cli>>url:https://github.com/eth-cscs/openstack/tree/master/cli]]. When authenticated, the OS token is saved as environment variable ##OS_TOKEN##.
129 -* If the user would like to specify the OS token for **BSC** her/himself (which is not necessary), guidance is provided here: [[https:~~/~~/bsc.es/supportkc/docs-ncloud/access#cli-access>>url:https://bsc.es/supportkc/docs-ncloud/access#cli-access]].
130 -
131 -Below is an example of submitting the transfer with CLI:
132 -
133 -##$fts-rest-transfer-submit \
134 --s https:~/~/fts.bsc.es:8446/ \##
135 -##~-~-access-token $tok \##
136 -##~-~-os-project-id “<source_project_id>:<dest_project_id>” \##
137 -##~-~-os-token “<project_id>:<corresponding_os_token>” \##
138 -##swifts:~/~/object.cscs.ch/<container_name>/<object_name> \##
139 -##swifts:~/~/swift.bsc.es/<container_name>/<object_name>##
140 -
141 -### Using endpoint: https:~/~/fts.bsc.es:8446                                                        
142 -# REST API version: 3.10.0  ##
143 -##Job successfully submitted.##
144 -##Job id: 532086a8-30a9-11eb-b0da-fa163e1833c1##
145 -
146 -Users can use multiple ##~-~-os-token## to specify OS tokens for different projects.
95 +[promt]$ oidc-token <defined_account_name>
image-20230412091641-1.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.luehrs
Size
... ... @@ -1,1 +1,0 @@
1 -24.0 KB
Content
image-20230412091702-2.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.luehrs
Size
... ... @@ -1,1 +1,0 @@
1 -36.3 KB
Content