Oneclient
Oneclient is the Onedata command line interface for mounting distributed virtual filesystem on local machines.
Install / Use
/learn @onedata/OneclientREADME
oneclient
Latest stable release:
oneclient is a command line Onedata client. It provides a POSIX interface to user's files in Onedata system.
User Guide
Building
Dependencies
An up-to-date list of oneclient build dependencies for Ubuntu and Fedora is available in control and oneclient.spec files respectively. For building on Ubuntu Bionic, detailed dependecy list can be found in Travis configuration.
Compilation
export ONEDATA_GIT_URL=https://github.com/onedata
git clone https://github.com/onedata/oneclient.git && cd oneclient
# Initialize submodules
make submodules
# To build debug version
mkdir debug && cd debug
cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DCODE_COVERAGE=ON -DWITH_CEPH=ON -DWITH_SWIFT=ON -DWITH_S3=ON -DWITH_GLUSTERFS=ON -DWITH_WEBDAV=ON -DWITH_XROOTD=ON -DWITH_ONEDATAFS=ON ..
cd ..
cmake --build release
oneclient by default compiles with built-in support for Ceph, S3, OpenStack SWIFT, GlusterFS, WebDAV and XRootD. These drivers can be disabled during compilation by providing the following flags:
- WITH_CEPH=OFF - disables Ceph support
- WITH_S3=OFF - disables S3 support
- WITH_SWIFT=OFF - disables Swift support
- WITH_GLUSTERFS=OFF - disables GlusterFS support
- WITH_WEBDAV=OFF - disables WebDAV support
- WITH_XROOTD=OFF - disables XRootD support
- WITH_NFS=OFF - disable NFS support
The compiled binary oneclient will be created on path release/oneclient.
Installation
Linux
Oneclient is supported on several major Linux platforms including Ubuntu and CentoOS. To install oneclient using packages simply use the following command:
curl -sS http://get.onedata.org/oneclient.sh | bash
Oneclient is packaged into self-contained packages, i.e. it has to be installed into it's default prefix
/opt/oneclient. The provided packages will do that by default and create symlinks in the/usrprefix to theoneclientbinary as well as man pages, configuration file and auto-completion scripts.
Usage
oneclient can be called directly from command line to mount Onedata virtual filesystem on the machine. For most cases basic usage should be sufficient:
oneclient -t <ACCESS_TOKEN> -H <PROVIDER_IP> <MOUNT_PATH>
When connecting to a Oneprovider instance without a valid trusted SSL certificate, -i option must be added.
Mounting selected spaces
oneclient will present under the specified mountpoint all spaces available to the user whose access token was passed on the command line.
It is however possible to limit the spaces which are visible, by providing a white list of the spaces on the command line. This can be achieved using 2 options:
--space <name>- every occurence of this option followed by the name of a space will limit the mounted spaces to the specified spaces (e.g.--space Space1 --space Space2)--space-id <id>- every occurence of this option followed by the id of a space will limit the mounted spaces to the specified spaces (e.g.--space-id a58a461875b59988bd16eca960d8130b --space-id bd16eca960d8130ba58a461875b53451)
Direct IO and Proxy IO modes
By default oneclient will automatically try to detect if it can access storage supporting mounted spaces directly, which significantly improves IO performance as all read and write operations go directly to the storage and not via the Oneprovider service.
This feature can be controlled using 2 command line options:
--force-proxy-io- disables Direct IO mode, all data transfers will go via Oneprovider service--force-direct-io- forces Direct IO mode, if it is not available for any of mounted spaces,oneclientwill fail to mount
In direct io mode, Oneclient will attempt to access the target storage directly on first attempt to read/write a file. This means that very often the first operation will fail with warning
Resource temporarily unavailable. However if the storage access is detected, the consecutive operations should work as expected.
Buffering
oneclient employs an in-memory buffer for input and output data blocks, which can significantly improve performance for various types of storages, in particular object based storages such as S3.
If for some reason this local cache is undesired, it can be disabled using --no-buffer option.
Force full block read mode
By default, POSIX read request can return less bytes than requested, especially on network filesystem which can return partial data range which is immediately available and request the remaining bytes assuming the application will run another read request with adjusted offset and size. However, some applications assume that the read always return the requested range or error. In order to enable this behavior in oneclient it necessary to provide the --force-fullblock-read on the command line.
Overriding storage helper parameters
Oneclient allows to override certain storage helper parameters in order to customize direct access to storage from a Oneclient host to the storage. Use cases for this feature include specifying custom mounpoint for POSIX storages, alternate IP addresses for network storages (e.g. available over local network from Oneclient host), etc.
For example, to tell Oneclient that storage with a NFS storage is mounted at /home/user1/nfs the following option should be added to the Oneclient command line: --override 2bede2623303bc2a19696e5817e13c0b:mountPoint:/home/user/nfs. 2bede2623303bc2a19696e5817e13c0b is the storage Id of this storage.
The --override option takes 3 arguments separated by ::
- storade ID - this is Onedata internal storage Id, which can be obtained from Onepanel administrator interface or using REST API
- parameter name - this is the name of the storage helper parameter, these are specific to particular type of storage
- parameter value - a value which should override the value specified in the Oneprovider when registering the storage
Logging
In order to enable a verbose log, oneclient provides a -v flag which takes a single integer argument which determines the log verbosity:
-v 0- (default) only serious errors-v 1- warnings and errors which are not fatal-v 2- verbose information on requests and their handling-v 3- trace function calls along with their arguments-v 4- binary messages between Oneclient and Oneprovider
Please note that above level 2, the size of the logs can be substantial thus it is necessary to monitor free disk space.
All options
The list of all options can be accessed using:
$ oneclient -h
Usage: oneclient [options] mountpoint
A Onedata command line client.
General options:
-h [ --help ] Show this help and exit.
-V [ --version ] Show current Oneclient version and
exit.
-u [ --unmount ] Unmount Oneclient and exit.
-c [ --config ] <path> (=/etc/oneclient.conf)
Specify path to config file.
--ignore-env Ignore options from environment
variables.
-H [ --host ] <host> Specify the hostname of the Oneprovider
instance to which the Oneclient should
connect.
-P [ --port ] <port> (=443) Specify the port to which the Oneclient
should connect on the Oneprovider.
-i [ --insecure ] Disable verification of server
certificate, allows to connect to
servers without valid certificate.
-t [ --token ] <token> Specify Onedata access token for
authentication and authorization.
--space <name> Allows to specify which space should be
mounted, where the value of the
argument is space name. Specify
multiple times for multiple spaces. If
not specified, all users spaces will be
mounted.
--space-id <id> Allows to specify which space should be
mounted, where the value of the
argument is space id. Specify multiple
times for multiple spaces. If not
specified, all users spaces will be
mounted.
-l [ --log-dir ] <path> (=/tmp/oneclient/0)
Specify custom path for Oneclient logs.
--custom-ca-dir <path> Path to directory with custom CA
certificates in PEM format.
-v [ --verbose-log-level ] <level> (=0)
Specify the verbosity level (0-3) for
verbose logs (only available in debug
builds).
--disable-log-buffering Disable log buffering.
Advanced options:
--io-trace-log Enable detailed IO trace log
(experimental).
--log-read-write-perf Enable read write
