SwiftOnHPSS Preliminary Deployment Guide ======================================== Phil Bridges :Author Initials: PGB :toc: :icons: :numbered: :website: http://www.hpss-collaboration.org [red]*Disclaimer: This interface is not certified for use in an HPSS production environment to store/retrieve production (valued) data! + It has not been extensively tested, and is still early in development. + Only for use on a dedicated SwiftOnHPSS development system!* This document is meant to provide general instruction on the installation and configuration of a stand-alone Swift node running SwiftOnHPSS. [[X1]] Working Assumptions ------------------- This guide assumes a good working knowledge of: * OpenStack Swift * OpenStack Keystone * HPSS administration * Basic Linux system administration This guide also makes assumptions on your: * Client machine: running RHEL 7.1 on a reasonably modern x86-64 processor, with Internet access for the duration of the install and LAN connectivity to the HPSS core and movers at all times. * HPSS installation: HPSS 7.5 or newer, using UNIX authentication with local or system passwd files * OpenStack infrastructure: that there exists a Keystone server already, configured with a service tenant and 'swift' service account. [[X2]] Initial Considerations ---------------------- Install the HPSS client. This machine will run a stand-alone Swift node with SwiftOnHPSS. Add a 'swift' service user in HPSS and on the client machine: ---- # adduser swift # /opt/hpss/bin/hpss_unix_user add swift ---- Also, make sure that the /swift folder in HPSS is owned by the 'swift' service user. ---- # /opt/hpss/bin/scrub scrub> chown /swift swift recurse top scrub> quit ---- [[X3]] Install Prerequisites for SwiftOnHPSS ------------------------------------- It's recommended to install 'pip', which is a Python package management tool. That can be found at https://pip.pypa.io: ---- # wget https://pip.pypa.io/get-pip.py # python get-pip.py ---- Install memcached, start the service and make it run on startup. ---- # yum install memcached python-memcached # service memcached start # systemctl enable memcached ---- [[X4]] Install Keystone (if not already existing) ------------------------------------------ Install OpenStack Keystone, and deploy it following this guide: https://docs.openstack.org/kilo/install-guide/yum/content/keystone-install.html This does not have to be on the same machine that SwiftOnHPSS is on, but it does need to be installed on an HPSS client machine if the HPSS Keystone driver is desired to use HPSS identities with Swift. After installing Keystone and verifying that it works using that guide, create the service entity and API endpoint, but don't create any users or projects yet if the HPSS-specific Keystone backend is desired. If it is desired, install the HPSS Keystone backend, and configure /etc/keystone/keystone.conf to use it as the sole identity provider. ---- # cd /opt/openstack # git clone https://github.com/hpss-collaboration/hpss-keystone-driver # cd hpss-keystone-driver # pip install -r requirements.txt # python setup.py develop ---- ./etc/keystone/keystone.conf ---- [identity] driver = hpss_keystone.identity.Identity ---- [[X5]] Install and Configure SwiftOnHPSS --------------------------------- We have tested SwiftOnHPSS against the Liberty stable release of Swift. Download the Liberty release of Swift (https://launchpad.net/swift/liberty/2.5.0, or more optimally from https://github.com/openstack/swift) and install it. Additionally, install the 'keystonemiddleware' library using Pip if you are using Keystone. ---- # mkdir /opt/openstack # cd /opt/openstack # git clone https://github.com/openstack/swift stable/liberty # git clone https://github.com/openstack/keystone stable/liberty # git clone https://github.com/hpss-collaboration/swiftonhpss # pip install keystonemiddleware # pip install -r swift/requirements.txt # pip install -r swift/setup_requirements.txt # pip install -r keystone/requirements.txt # cd swiftonhpss # python setup.py develop ---- Contact your HPSS support representative for the 'hpss' Python library appropriate for your site's version of HPSS. Upon receipt of those libraries, install them. Assuming the libraries are in /opt/hpss_lib: ---- # cd /opt/hpss_lib # python hpssfs/setup.py install # python hpsspy/setup.py install # pip install -r swiftonhpss/prereqs.txt # python swiftonhpss/setup.py develop ---- Copy the sample configuration files over from the 'etc' folder of the Swift installation over to /etc/swift, and remove the extra account and object server configurations. ---- # cp -r doc/saio/etc/ /etc/swift # rm /etc/swift/account-server/2.conf # rm /etc/swift/account-server/3.conf # rm /etc/swift/account-server/4.conf # rm /etc/swift/container-server/2.conf # rm /etc/swift/container-server/3.conf # rm /etc/swift/container-server/4.conf # rm /etc/swift/object-server/2.conf # rm /etc/swift/object-server/3.conf # rm /etc/swift/object-server/4.conf ---- Then go through the object server config for SwiftOnHPSS and alter it like this: ./etc/swift/object-server/1.conf: [source,conf] ---- [DEFAULT] hpss_swift_dir = /swift mount_check = false disable_fallocate = true bind_ip = 127.0.0.1 bind_port = 6010 workers = 1 user = swift log_facility = LOG_LOCAL2 recon_cache_path = /var/cache/swift eventlet_debug = true [pipeline:main] pipeline = recon object-server [app:object-server] use = egg:swiftonhpss#object [filter:recon] use = egg:swift#recon ### Specifically comment these out, because we don't want to use any of these ### extra servers. #[object-replicator] #vm_test_mode = yes #[object-reconstructor] #[object-updater] #[object-auditor] ---- ./etc/swift/proxy-server.conf: [source,conf] ---- [DEFAULT] bind_ip = 127.0.0.1 # Replace this with your node's IP address! bind_port = 8080 workers = 1 user = swift log_facility = LOG_LOCAL1 eventlet_debug = true [pipeline:main] # Yes, proxy-logging appears twice. This is so that # middleware-originated requests get logged too. pipeline = catch_errors gatekeeper healthcheck proxy-logging cache bulk tempurl ratelimit crossdomain authtoken keystoneauth staticweb container-quotas account-quotas slo dlo proxy-logging proxy-server [filter:catch_errors] use = egg:swift#catch_errors [filter:healthcheck] use = egg:swift#healthcheck [filter:proxy-logging] use = egg:swift#proxy_logging [filter:bulk] use = egg:swift#bulk [filter:ratelimit] use = egg:swift#ratelimit [filter:crossdomain] use = egg:swift#crossdomain [filter:dlo] use = egg:swift#dlo [filter:slo] use = egg:swift#slo [filter:tempurl] use = egg:swift#tempurl #[filter:tempauth] #use = egg:swift#tempauth #user_admin_admin = admin .admin .reseller_admin #user_test_tester = testing .admin #user_test2_tester2 = testing2 .admin #user_test_tester3 = testing3 ### Replace all of this with your Keystone server's configuration! [filter:authtoken] paste.filter_factory = keystonemiddleware.auth_token:filter_factory identity_uri = http://127.0.0.1:35357/ # Replace with Keystone server IP admin_tenant_name = service admin_user = swift admin_password = password auth_uri = http://127.0.0.1:5000/ # Replace with Keystone server IP #cache = swift.cache include_service_catalog = False delay_auth_decision = True [filter:keystoneauth] use = egg:swift#keystoneauth operator_roles = admin, swiftoperator reseller_prefix = KEY [filter:staticweb] use = egg:swift#staticweb [filter:account-quotas] use = egg:swift#account_quotas [filter:container-quotas] use = egg:swift#container_quotas [filter:cache] use = egg:swift#memcache [filter:gatekeeper] use = egg:swift#gatekeeper [app:proxy-server] use = egg:swift#proxy allow_account_management = true account_autocreate = true ---- After editing the configuration scripts, it is time to generate the Swift server rings. There is a script to automate doing this in doc/saio/bin/remakerings that could be used, but it needs some modification to fit the recommended configuration. ---- # cp doc/saio/bin/remakerings ./remakerings-for-HPSS # vim remakerings-for-HPSS ---- ../remakerings-for-HPSS: [source,shell] ---- #!/bin/bash cd /etc/swift rm -f *.builder *.ring.gz backups/*.builder backups/*.ring.gz swift-ring-builder object.builder create 0 1 1 swift-ring-builder object.builder add r1z1-127.0.0.1:6010/ 1 swift-ring-builder object.builder rebalance swift-ring-builder container.builder create 0 1 1 swift-ring-builder container.builder add r1z1-127.0.0.1:6011/metadata 1 swift-ring-builder container.builder rebalance swift-ring-builder account.builder create 0 1 1 swift-ring-builder account.builder add r1z1-127.0.0.1:6012/metadata 1 swift-ring-builder account.builder rebalance ---- ---- # ./remakerings-for-HPSS ---- Next, configure rsyslog for Swift. Swift comes with a perfectly usable example configuration, so just copy it over. ---- # cp doc/saio/rsyslog.d/10-swift.conf /etc/rsyslog.d/10-swift.conf # service syslog restart ---- Next, in order to work around a known linking bug, it's necessary to preload the libtirpc library when starting up the Swift object server, or else SwiftOnHPSS will hang and then fail to connect to HPSS. One could just throw LD_PRELOAD=libtirpc.so in front of it and it'll probably work So, start up the Swift server, and make sure that it is accessible and works properly. ---- # LD_PRELOAD=libtirpc.so swift-init main start ---- [[X6]] Where To Go From Here --------------------- At this point, the SwiftOnHPSS archive machine is fully configured and ready to develop on and use for experimental purposes. A command-line client can be had by installing the "python-openstackclient" package with 'pip'. If Keystone is in use, then in order to log into Swift it is necessary to create a Swift service user in Keystone and grant it the 'admin' role in the 'service' project in the default domain, and also to add the Swift URL to the service endpoint catalog. SwiftOnHPSS now has a tool to synchronize changes made through other HPSS interfaces with the Swift metadata databases, called 'swiftonhpss-nstool'. It is included in the SwiftOnHPSS Git repository, and should have been automatically installed along with the rest of the system. For further reading on how the Swift object storage system works and how to do further customization, see: https://docs.openstack.org/developer/swift/index.html