From e6604b36bccf00707a47aa3d2fa120f254f930af Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Tue, 27 May 2014 17:35:35 +0530 Subject: [PATCH 01/14] Draft for Quick Start guide The attempt was to make it more FS neutral, However certain references are made to glusterFS in particular. --- doc/markdown/quick_start_guide.md | 254 ++++++++++++------------------ 1 file changed, 100 insertions(+), 154 deletions(-) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index 4ebabe6..d1f07e9 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -3,155 +3,92 @@ ## Contents * [Overview](#overview) * [System Setup](#system_setup) -* [Gluster For Swift Setup](#swift_setup) -* [Using Gluster for Swift](#using_swift) +* [SwiftOnFile Setup](#swift_setup) +* [Using SwiftOnFile](#using_swift) * [What now?](#what_now) ## Overview -Gluster-swift allows GlusterFS to be used as the backend to the object -store OpenStack Swift. +SwiftOnFile allows any POSIX complaint filesystem to be used as the +backend to the object store OpenStack Swift. -The following guide will get you quickly started with a gluster-swift +The following guide will get you quickly started with a SwiftOnFile environment on a Fedora or RHEL/CentOS system. This guide is a -great way to begin using gluster-swift, and can be easily deployed on -a single virtual machine. The final result will be a single gluster-swift +great way to begin using SwiftOnFile, and can be easily deployed on +a single virtual machine. The final result will be a single SwiftOnFile node. -> NOTE: In Gluster-Swift, accounts must be GlusterFS volumes. +> NOTE: In SwiftOnFile a swift account is a mounted FS under path mentioned +in configuration parameter.It is assumed you have two xattrr supporting FS mounted +under certain paths.We suggest you to start with two xfs formatted FS then you can +move on to other FS that supports xattr.For setting up gluster volume in particular +you can look here [GlusterFS Quick Start Guide][] ## System Setup ### Prerequisites on CentOS/RHEL -On CentOS/RHEL you will need to setup GlusterFS and EPEL repos. +On CentOS/RHEL you may need to EPEL repo.Please refer to +[EPEL][] for more information on how to setup the EPEL repo. -#### GlusterFS CentOS/RHEL Repo +SwiftOnfile requires corresponding OpenStack Swift release packages.There are two +possible ways to get OpenStack Swift packages. -* CentOS +1. Get & build the OpenStack swift source from github.(This should work for all linux flavors) -~~~ -wget -O /etc/yum.repos.d/glusterfs-epel.repo \ - http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo -~~~ + a.) Git clone the required branch (assume icehouse) + ~~~ + git clone -b icehouse-stable https://github.com/openstack/swift.git + ~~~ + b.)Install the prerequisite + ~~~ + python-pip install -r requirements.txt + python-pip install -r test-requirements.txt + ~~~ + c.)Install the packages + ~~~ + python setup.py install + ~~~ + d.) Please refer to the OpenStack swift SAIO guide, + if you face any difficulty in doing above. -* RHEL +2. Use the Stable RDO release (Fedora/RHEL/CentOS) -~~~ -wget -O /etc/yum.repos.d/glusterfs-epel.repo \ - http://download.gluster.org/pub/gluster/glusterfs/LATEST/RHEL/glusterfs-epel.repo -~~~ + a.) Please setup corresponding Red Hat RDO release repo (assume icehouse) + ~~~ + yum install -y http://repos.fedorapeople.org/repos/openstack/openstack-icehouse/rdo-release-icehouse-3.noarch.rpm + ~~~ + b.) Install required rpms: + ~~~ + yum install -y openstack-swift-proxy openstack-swift-account openstack-swift-container\ + openstack-swift-object memcached python-swiftclient python-keystoneclient + ~~~ -#### EPEL CentOS/RHEL Repo -Please refer to [EPEL][] for more information on how to setup the EPEL repo. +### Install SwiftOnFile +1. Install from source + a.) Git clone the required branch (assume icehouse) + ~~~ + git clone -b icehouse https://github.com/swiftonfile/swiftonfile.git + ~~~ + b.)Install the prerequisite + ~~~ + python-pip install -r requirements.txt + python-pip install -r test-requirements.txt + ~~~ + c.)Install the packages + ~~~ + python setup.py install + ~~~ + +2. Using RPMs -### Required Package Installation -Install and start the required packages on your system to create a GlusterFS volume. - -~~~ -yum install glusterfs glusterfs-server glusterfs-fuse memcached xfsprogs -~~~ - -#### Start services - -Type the following to start `memcached` and `glusterfs` services: - -~~~ -service memcached start -service glusterd start -~~~ - -Type the following to start the services automatically on system startup: - -~~~ -chkconfig memcached on -chkconfig glusterd on -~~~ - -### Gluster Volume Setup -Now you need to determine whether you are going to use a partition or a loopback device -for storage. - -#### Partition Storage Setup -If you are using a separate disk partition, please execute the following instructions -to create a GlusterFS brick: - -~~~ -mkfs.xfs -i size=512 /dev/ -mkdir -p /export/brick -~~~ - -Add the following line to `/etc/fstab` to mount the storage automatically on system -startup: - -~~~ -/dev/ /export/brick xfs inode64,noatime,nodiratime 0 0 -~~~ - -Now type the following to mount the storage: - -~~~ -mount -a -~~~ - -#### Loopback Storage Setup -If you do not have a separate partition, please execute the following instructions -to create a disk image as a file: - -~~~ -truncate -s 5GB /srv/swift-disk -mkfs.xfs -i size=512 /srv/swift-disk -mkdir -p /export/brick -~~~ - -Add the following line to `/etc/fstab` to mount the storage automatically on system -startup: - -~~~ -/srv/swift-disk /export/brick xfs loop,inode64,noatime,nodiratime 0 0 -~~~ - -Now type the following to mount the storage: - -~~~ -mount -a -~~~ - -### Create a GlusterFS Volume -You now need to create a GlusterFS volume (make sure your hostname is in /etc/hosts or is DNS-resolvable) - -~~~ -mkdir /export/brick/b1 -gluster volume create myvolume `hostname`:/export/brick/b1 -gluster volume start myvolume -~~~ - - -## Gluster-Swift Setup - -### Repository Setup on RHEL/CentOS -Gluster-Swift requires OpenStack Swift's Havana release, which -may not be available on some older operating systems. For RHEL/CentOS -systems, please setup Red Hat RDO's repo by executing the following command: - -~~~ -yum install -y http://rdo.fedorapeople.org/rdo-release.rpm -~~~ - -### Download -Download the latest Havana release RPMs from [launchpad.net downloads][]: - -### Install -Install the RPM by executing the following: - -~~~ -yum install -y -~~~ - -### Enabling gluster-swift accross reboots -Type the following to make sure gluster-swift is enabled at -system startup: + a.) Download the rpms from [Jenkins CI][] + b.)Install the RPM by executing the following: + ~~~ + yum install -y + ~~~ +### Enabling Swift Service available accross reboots ~~~ chkconfig openstack-swift-proxy on chkconfig openstack-swift-account on @@ -160,7 +97,7 @@ chkconfig openstack-swift-object on ~~~ #### Fedora 19 Adjustment -Currently gluster-swift requires its processes to be run as `root`. You need to +Currently SwiftOnFile requires its processes to be run as `root`. You need to edit the `openstack-swift-*.service` files in `/etc/systemd/system/multi-user.target.wants` and change the `User` entry value to `root`. @@ -172,9 +109,9 @@ systemctl --system daemon-reload ~~~ ### Configuration -As with OpenStack Swift, gluster-swift uses `/etc/swift` as the +As with OpenStack Swift, SwiftOnFile uses `/etc/swift` as the directory containing the configuration files. You will need to base -the configuration files on the template files provided. On new +the configuration files on the template files provided. On new RPM based installations, the simplest way is to copy the `*.conf-gluster` files to `*.conf` files as follows: @@ -183,28 +120,30 @@ cd /etc/swift for tmpl in *.conf-gluster ; do cp ${tmpl} ${tmpl%.*}.conf; done ~~~ +Else you can base your config files on [test code ][]. + #### Generate Ring Files -You now need to generate the ring files, which inform gluster-swift -which GlusterFS volumes are accessible over the object -storage interface. The format is +You now need to generate the ring files, which inform SwiftOnFile +which FS volumes are accessible over the object storage interface. +This is a borrowed legacy from gluster-swift and it will soon change. +This script uses OpenStack Swift ring builder with the fundamental +assumption that the replication/sync/HA/etc are provided by underlying FS +(gluster in this case).The format is ~~~ -gluster-swift-gen-builders [VOLUME] [VOLUME...] +gluster-swift-gen-builders [mount-point-name] [mount-point-name...] ~~~ -Where *VOLUME* is the name of the GlusterFS volume which you would -like to access over gluster-swift. - -Let's now expose the GlusterFS volume called `myvolume` you created above -by executing the following command: - +Where *mount-point-name* is the name of the a directory in the path mentioned in +/etc/swift{account,object,container}.conf under the section [DEFAULT] +for parameter 'devices'.For ex: If 'device' parameter has the value '/mnt/FS-objects' +and you mounted two gluster/xfs volumes on /mnt/FS-objects/gfs-vol1 & +/mnt/FS-objects/gfs-vol2 then the command would look like this: ~~~ -cd /etc/swift -/usr/bin/gluster-swift-gen-builders myvolume +gluster-swift-gen-builders gfs-vol1 gfs-vol2 ~~~ -### Start gluster-swift -Use the following commands to start gluster-swift: +### Start swift services using the following commands: ~~~ service openstack-swift-object start @@ -213,22 +152,26 @@ service openstack-swift-account start service openstack-swift-proxy start ~~~ +Or using +~~~ +swift-init main start +~~~ + -## Using gluster-swift +## Using SwiftOnFile ### Create a container Create a container using the following command: ~~~ -curl -v -X PUT http://localhost:8080/v1/AUTH_myvolume/mycontainer +curl -v -X PUT http://localhost:8080/v1/AUTH_gfs-vol1/mycontainer ~~~ It should return `HTTP/1.1 201 Created` on a successful creation. You can -also confirm that the container has been created by inspecting the GlusterFS -volume: +also confirm that the container has been created by inspecting the FS: ~~~ -ls /mnt/gluster-object/myvolume +ls /mnt/FS-object/gfs-vol1 ~~~ #### Create an object @@ -236,14 +179,14 @@ You can now place an object in the container you have just created: ~~~ echo "Hello World" > mytestfile -curl -v -X PUT -T mytestfile http://localhost:8080/v1/AUTH_myvolume/mycontainer/mytestfile +curl -v -X PUT -T mytestfile http://localhost:8080/v1/AUTH_gfs-vol1/mycontainer/mytestfile ~~~ To confirm that the object has been written correctly, you can compare the test file with the object you created: ~~~ -cat /mnt/gluster-object/myvolume/mycontainer/mytestfile +cat /mnt/FS-object/gf-vol1/mycontainer/mytestfile ~~~ #### Request the object @@ -251,10 +194,13 @@ Now you can retreive the object and inspect its contents using the following commands: ~~~ -curl -v -X GET -o newfile http://localhost:8080/v1/AUTH_myvolume/mycontainer/mytestfile +curl -v -X GET -o newfile http://localhost:8080/v1/AUTH_gfs-vol1/mycontainer/mytestfile cat newfile ~~~ +You can also use etag information provided while you do HEAD on object +and compare it with md5sum of the file on your FS. + ## What now? For more information, please visit the following links: @@ -268,5 +214,5 @@ For more information, please visit the following links: [Jenkins]: http://jenkins-ci.org [Authentication Services Start Guide]: auth_guide.md [EPEL]: https://fedoraproject.org/wiki/EPEL -[launchpad.net downloads]: http://launchpad.net/gluster-swift/havana/1.10.0-2 +[Jenkins CI]: http://build.gluster.org/job/swiftonfile-builds/lastSuccessfulBuild/artifact/build/ From a9cef12ca4ffe958790db85c3a970cde3c10b27b Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Tue, 27 May 2014 17:50:02 +0530 Subject: [PATCH 02/14] Minor Update --- doc/markdown/quick_start_guide.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index d1f07e9..77cf63a 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -65,21 +65,23 @@ possible ways to get OpenStack Swift packages. ~~~ ### Install SwiftOnFile + 1. Install from source - a.) Git clone the required branch (assume icehouse) + + a.) Git clone the required branch (assume icehouse) ~~~ git clone -b icehouse https://github.com/swiftonfile/swiftonfile.git ~~~ - b.)Install the prerequisite - ~~~ + b.)Install the prerequisite + ~~~ python-pip install -r requirements.txt python-pip install -r test-requirements.txt ~~~ - c.)Install the packages + c.)Install the packages ~~~ python setup.py install ~~~ - + 2. Using RPMs a.) Download the rpms from [Jenkins CI][] @@ -186,7 +188,7 @@ To confirm that the object has been written correctly, you can compare the test file with the object you created: ~~~ -cat /mnt/FS-object/gf-vol1/mycontainer/mytestfile +cat /mnt/FS-object/gfs-vol1/mycontainer/mytestfile ~~~ #### Request the object @@ -215,4 +217,4 @@ For more information, please visit the following links: [Authentication Services Start Guide]: auth_guide.md [EPEL]: https://fedoraproject.org/wiki/EPEL [Jenkins CI]: http://build.gluster.org/job/swiftonfile-builds/lastSuccessfulBuild/artifact/build/ - +[test code ] : https://github.com/swiftonfile/swiftonfile/tree/master/test/functional_auth/tempauth/conf/ From d78dfa15c68a810508c67cc8d0ad0921ffde52b7 Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Tue, 27 May 2014 17:54:24 +0530 Subject: [PATCH 03/14] Minor Update --- doc/markdown/quick_start_guide.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index 77cf63a..691d7f5 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -69,22 +69,23 @@ possible ways to get OpenStack Swift packages. 1. Install from source a.) Git clone the required branch (assume icehouse) - ~~~ + ~~~ git clone -b icehouse https://github.com/swiftonfile/swiftonfile.git ~~~ b.)Install the prerequisite ~~~ - python-pip install -r requirements.txt - python-pip install -r test-requirements.txt - ~~~ + python-pip install -r requirements.txt + python-pip install -r test-requirements.txt + ~~~ c.)Install the packages - ~~~ - python setup.py install - ~~~ + ~~~ + python setup.py install + ~~~ 2. Using RPMs a.) Download the rpms from [Jenkins CI][] + b.)Install the RPM by executing the following: ~~~ yum install -y @@ -217,4 +218,4 @@ For more information, please visit the following links: [Authentication Services Start Guide]: auth_guide.md [EPEL]: https://fedoraproject.org/wiki/EPEL [Jenkins CI]: http://build.gluster.org/job/swiftonfile-builds/lastSuccessfulBuild/artifact/build/ -[test code ] : https://github.com/swiftonfile/swiftonfile/tree/master/test/functional_auth/tempauth/conf/ +[test code] : https://github.com/swiftonfile/swiftonfile/tree/master/test/functional_auth/tempauth/conf/ From 117199a5cd450e5ac4e3797c48fa1a4c16e271e1 Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Tue, 27 May 2014 18:13:12 +0530 Subject: [PATCH 04/14] Minor Update --- doc/markdown/quick_start_guide.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index 691d7f5..d46eebc 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -72,11 +72,13 @@ possible ways to get OpenStack Swift packages. ~~~ git clone -b icehouse https://github.com/swiftonfile/swiftonfile.git ~~~ + b.)Install the prerequisite ~~~ python-pip install -r requirements.txt python-pip install -r test-requirements.txt ~~~ + c.)Install the packages ~~~ python setup.py install From 35cc268b656504c9953954652672478922157e09 Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Tue, 27 May 2014 18:20:18 +0530 Subject: [PATCH 05/14] Fixing Markdown format --- doc/markdown/quick_start_guide.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index d46eebc..691d7f5 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -72,13 +72,11 @@ possible ways to get OpenStack Swift packages. ~~~ git clone -b icehouse https://github.com/swiftonfile/swiftonfile.git ~~~ - b.)Install the prerequisite ~~~ python-pip install -r requirements.txt python-pip install -r test-requirements.txt ~~~ - c.)Install the packages ~~~ python setup.py install From 09039648919c706b38005643736e8355a2664703 Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Tue, 27 May 2014 18:27:40 +0530 Subject: [PATCH 06/14] Fixing Markdown --- doc/markdown/quick_start_guide.md | 36 +++++++++++++------------------ 1 file changed, 15 insertions(+), 21 deletions(-) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index 691d7f5..8e92797 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -63,33 +63,27 @@ possible ways to get OpenStack Swift packages. yum install -y openstack-swift-proxy openstack-swift-account openstack-swift-container\ openstack-swift-object memcached python-swiftclient python-keystoneclient ~~~ - ### Install SwiftOnFile - 1. Install from source - a.) Git clone the required branch (assume icehouse) - ~~~ + a.) Git clone the required branch (assume icehouse) git clone -b icehouse https://github.com/swiftonfile/swiftonfile.git - ~~~ - b.)Install the prerequisite - ~~~ - python-pip install -r requirements.txt - python-pip install -r test-requirements.txt - ~~~ - c.)Install the packages - ~~~ - python setup.py install - ~~~ - + + b.)Install the prerequisite + + python-pip install -r requirements.txt + python-pip install -r test-requirements.txt + + c.)Install the packages + + python setup.py install + 2. Using RPMs - a.) Download the rpms from [Jenkins CI][] - - b.)Install the RPM by executing the following: - ~~~ - yum install -y - ~~~ + a.) Download the rpms from [Jenkins CI][] + + b.)Install the RPM by executing the following: + yum install -y ### Enabling Swift Service available accross reboots ~~~ From 40d713f886618d61c108cc855ed3827b91bba4e7 Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Tue, 27 May 2014 18:30:35 +0530 Subject: [PATCH 07/14] Markdown Fix --- doc/markdown/quick_start_guide.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index 8e92797..a2bbb4f 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -63,6 +63,7 @@ possible ways to get OpenStack Swift packages. yum install -y openstack-swift-proxy openstack-swift-account openstack-swift-container\ openstack-swift-object memcached python-swiftclient python-keystoneclient ~~~ + ### Install SwiftOnFile 1. Install from source From a07bfd42c48a9bfac381629af17e0e69491654c1 Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Tue, 27 May 2014 18:32:27 +0530 Subject: [PATCH 08/14] Markdown Again --- doc/markdown/quick_start_guide.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index a2bbb4f..13c5d31 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -68,7 +68,8 @@ possible ways to get OpenStack Swift packages. 1. Install from source a.) Git clone the required branch (assume icehouse) - git clone -b icehouse https://github.com/swiftonfile/swiftonfile.git + + git clone -b icehouse https://github.com/swiftonfile/swiftonfile.git b.)Install the prerequisite From 71228826b9ab31a38b276ddf776c5c93ff985d42 Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Thu, 26 Jun 2014 16:28:51 +0530 Subject: [PATCH 09/14] SwiftOnFile as StoragePolicy of OpenStack Swift Done necessary changes to quick start guide.This guide suggest easy steps to extend swift SAIO setup to use swiftonfile as storage policy. --- doc/markdown/quick_start_guide.md | 194 ++++++++++-------------------- 1 file changed, 64 insertions(+), 130 deletions(-) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index 13c5d31..765ec22 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -9,183 +9,116 @@ ## Overview -SwiftOnFile allows any POSIX complaint filesystem to be used as the -backend to the object store OpenStack Swift. +SwiftOnFile allows any POSIX complaint filesystem to be used as the backend to the object store OpenStack Swift. -The following guide will get you quickly started with a SwiftOnFile -environment on a Fedora or RHEL/CentOS system. This guide is a -great way to begin using SwiftOnFile, and can be easily deployed on -a single virtual machine. The final result will be a single SwiftOnFile -node. +The following guide assumes you have a running [OpenStack Swift SAIO setup][], and you want to extend this setup to try SwiftOnFile as Storage Policy on a XFS/gluster volume. This will get you quickly started with a SwiftOnFile environment on a Fedora or RHEL/CentOS system. -> NOTE: In SwiftOnFile a swift account is a mounted FS under path mentioned -in configuration parameter.It is assumed you have two xattrr supporting FS mounted -under certain paths.We suggest you to start with two xfs formatted FS then you can -move on to other FS that supports xattr.For setting up gluster volume in particular -you can look here [GlusterFS Quick Start Guide][] +This guide will not go on detail on how to prepare a Swift SAIO setup or how to create a gluster volume (or other FS).This guide assumes you know about these technologies, if you require any help in setting those please refer to the link provided. ## System Setup ### Prerequisites on CentOS/RHEL -On CentOS/RHEL you may need to EPEL repo.Please refer to -[EPEL][] for more information on how to setup the EPEL repo. -SwiftOnfile requires corresponding OpenStack Swift release packages.There are two -possible ways to get OpenStack Swift packages. +1. OpenStack SAIO deployment on Fedora20 onwards +2. One xfs/glusterfs volume - named vol -1. Get & build the OpenStack swift source from github.(This should work for all linux flavors) +>Note: Swift SAIO deployment should contain Storage Policy code changes. Initialy Storage Policy feature was developed seprately in openstack swift feature/ec branch, and it is now merged in master branch. The latest OpenStack Swift2.0 release also contain storage policy code. - a.) Git clone the required branch (assume icehouse) - ~~~ - git clone -b icehouse-stable https://github.com/openstack/swift.git - ~~~ - b.)Install the prerequisite - ~~~ - python-pip install -r requirements.txt - python-pip install -r test-requirements.txt - ~~~ - c.)Install the packages - ~~~ - python setup.py install - ~~~ - d.) Please refer to the OpenStack swift SAIO guide, - if you face any difficulty in doing above. +Each xfs/glusterfs volume will be defined as a separate storage policy. -2. Use the Stable RDO release (Fedora/RHEL/CentOS) +### Install SwiftOnfile +1. Before you begin swiftonfile setup please ensure you have OpenStack Swift SAIO setup up & running. Please refer to the SAIO guide for this. +2. cd ~; git clone https://github.com/swiftonfile/swiftonfile.git +3. cd ~/swiftonfile;python setup.py develop;cd ~ - a.) Please setup corresponding Red Hat RDO release repo (assume icehouse) - ~~~ - yum install -y http://repos.fedorapeople.org/repos/openstack/openstack-icehouse/rdo-release-icehouse-3.noarch.rpm - ~~~ - b.) Install required rpms: - ~~~ - yum install -y openstack-swift-proxy openstack-swift-account openstack-swift-container\ - openstack-swift-object memcached python-swiftclient python-keystoneclient - ~~~ - -### Install SwiftOnFile -1. Install from source - - a.) Git clone the required branch (assume icehouse) - - git clone -b icehouse https://github.com/swiftonfile/swiftonfile.git - - b.)Install the prerequisite - - python-pip install -r requirements.txt - python-pip install -r test-requirements.txt - - c.)Install the packages - - python setup.py install - -2. Using RPMs - - a.) Download the rpms from [Jenkins CI][] - - b.)Install the RPM by executing the following: - yum install -y - -### Enabling Swift Service available accross reboots -~~~ -chkconfig openstack-swift-proxy on -chkconfig openstack-swift-account on -chkconfig openstack-swift-container on -chkconfig openstack-swift-object on -~~~ - -#### Fedora 19 Adjustment -Currently SwiftOnFile requires its processes to be run as `root`. You need to -edit the `openstack-swift-*.service` files in -`/etc/systemd/system/multi-user.target.wants` and change the `User` entry value -to `root`. - -Then run the following command to reload the configuration: +### Configure SwiftOnFile as Storage Policy +1. A SAIO setup mimic a four node swift setup and should have four object server running.Add another object server for SwiftOnFile by setting the following configurations in the file /etc/swift/object-server/5.conf: ~~~ -systemctl --system daemon-reload +[DEFAULT] +devices = /mnt/xfsvols/ +mount_check = false +bind_port = 6050 +max_clients = 1024 +workers = 1 +disable_fallocate = true + +[pipeline:main] +pipeline = object-server + +[app:object-server] +use = egg:gluster_swift#object +user = root +log_facility = LOG_LOCAL2 +log_level = DEBUG +log_requests = on +disk_chunk_size = 65536 ~~~ -### Configuration -As with OpenStack Swift, SwiftOnFile uses `/etc/swift` as the -directory containing the configuration files. You will need to base -the configuration files on the template files provided. On new RPM based -installations, the simplest way is to copy the `*.conf-gluster` -files to `*.conf` files as follows: +>Note: The parameter 'devices' tells about the path where your xfs/glusterfs volume is mounted. The sub directory under which your volume is mounted will be called volume name. For ex: You have a xfs formated partition /dev/sdb1, and you mounted it under /mnt/xfsvols/vol, then your volume name would be 'vol'& and the parameter 'devices' would contain value '/mnt/xfsvols'. + +2. Edit /etc/swift.conf to add swiftonfile as a storage policy: ~~~ -cd /etc/swift -for tmpl in *.conf-gluster ; do cp ${tmpl} ${tmpl%.*}.conf; done +[storage-policy:0] +name = swift + +[storage-policy:1] +name = swiftonfile-test +default = yes ~~~ -Else you can base your config files on [test code ][]. - -#### Generate Ring Files -You now need to generate the ring files, which inform SwiftOnFile -which FS volumes are accessible over the object storage interface. -This is a borrowed legacy from gluster-swift and it will soon change. -This script uses OpenStack Swift ring builder with the fundamental -assumption that the replication/sync/HA/etc are provided by underlying FS -(gluster in this case).The format is +3. Edit the remakerings script to prepare rings for this new storage policy: ~~~ -gluster-swift-gen-builders [mount-point-name] [mount-point-name...] +swift-ring-builder object-1.builder create 1 1 1 +swift-ring-builder object-1.builder add r1z1-127.0.0.1:6050/vol 1 +swift-ring-builder object-1.builder rebalance ~~~ -Where *mount-point-name* is the name of the a directory in the path mentioned in -/etc/swift{account,object,container}.conf under the section [DEFAULT] -for parameter 'devices'.For ex: If 'device' parameter has the value '/mnt/FS-objects' -and you mounted two gluster/xfs volumes on /mnt/FS-objects/gfs-vol1 & -/mnt/FS-objects/gfs-vol2 then the command would look like this: -~~~ -gluster-swift-gen-builders gfs-vol1 gfs-vol2 -~~~ - -### Start swift services using the following commands: +4. Restart swift services to reflect new changes: ~~~ -service openstack-swift-object start -service openstack-swift-container start -service openstack-swift-account start -service openstack-swift-proxy start +swift-init all restart ~~~ -Or using -~~~ -swift-init main start -~~~ + +5. Running functional tests: TBD + ## Using SwiftOnFile +It is assumed that you are still using 'tempauth' as authnetication method, which is default in SAIO deployment. + +### Get the token +~~~ +curl -v -H 'X-Auth-User: test:tester' -H "X-Auth-key: testing" -k http://localhost:8080/auth/v1.0 +~~~ +Use 'X-Auth-Token' & 'X-Storage-Url' returned in above request for all sucequent request. ### Create a container Create a container using the following command: ~~~ -curl -v -X PUT http://localhost:8080/v1/AUTH_gfs-vol1/mycontainer +curl -v -X PUT -H 'X-Auth-Token: AUTH_XXXX' http://localhost:8080/v1/AUTH_test/mycontainer ~~~ -It should return `HTTP/1.1 201 Created` on a successful creation. You can -also confirm that the container has been created by inspecting the FS: - -~~~ -ls /mnt/FS-object/gfs-vol1 -~~~ +It should return `HTTP/1.1 201 Created` on a successful creation. #### Create an object -You can now place an object in the container you have just created: +You can now place an object in the container you have just dreated: ~~~ echo "Hello World" > mytestfile -curl -v -X PUT -T mytestfile http://localhost:8080/v1/AUTH_gfs-vol1/mycontainer/mytestfile +curl -v -X PUT -T mytestfile 'X-Auth-Token: AUTH_XXXX' http://localhost:8080/v1/AUTH_test/mycontainer/mytestfile ~~~ To confirm that the object has been written correctly, you can compare the test file with the object you created: ~~~ -cat /mnt/FS-object/gfs-vol1/mycontainer/mytestfile +cat /mnt/xfsvols/vol/test/mycontainer/mytestfile ~~~ #### Request the object @@ -193,7 +126,7 @@ Now you can retreive the object and inspect its contents using the following commands: ~~~ -curl -v -X GET -o newfile http://localhost:8080/v1/AUTH_gfs-vol1/mycontainer/mytestfile +curl -v -X GET -o newfile http://localhost:8080/v1/AUTH_test/mycontainer/mytestfile cat newfile ~~~ @@ -214,4 +147,5 @@ For more information, please visit the following links: [Authentication Services Start Guide]: auth_guide.md [EPEL]: https://fedoraproject.org/wiki/EPEL [Jenkins CI]: http://build.gluster.org/job/swiftonfile-builds/lastSuccessfulBuild/artifact/build/ -[test code] : https://github.com/swiftonfile/swiftonfile/tree/master/test/functional_auth/tempauth/conf/ +[test code]: https://github.com/swiftonfile/swiftonfile/tree/master/test/functional_auth/tempauth/conf/ +[OpenStack Swift SAIO setup]: http://docs.openstack.org/developer/swift/development_saio.html From 873ff534c522ff826469fe0c99e0e295c5eb4b55 Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Thu, 26 Jun 2014 16:56:38 +0530 Subject: [PATCH 10/14] Minor update --- doc/markdown/quick_start_guide.md | 32 ++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index 765ec22..5768ed7 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -33,7 +33,9 @@ Each xfs/glusterfs volume will be defined as a separate storage policy. 3. cd ~/swiftonfile;python setup.py develop;cd ~ ### Configure SwiftOnFile as Storage Policy -1. A SAIO setup mimic a four node swift setup and should have four object server running.Add another object server for SwiftOnFile by setting the following configurations in the file /etc/swift/object-server/5.conf: + +#### Object Server Configuration +A SAIO setup mimic a four node swift setup and should have four object server running.Add another object server for SwiftOnFile by setting the following configurations in the file /etc/swift/object-server/5.conf: ~~~ [DEFAULT] @@ -55,29 +57,36 @@ log_level = DEBUG log_requests = on disk_chunk_size = 65536 ~~~ - >Note: The parameter 'devices' tells about the path where your xfs/glusterfs volume is mounted. The sub directory under which your volume is mounted will be called volume name. For ex: You have a xfs formated partition /dev/sdb1, and you mounted it under /mnt/xfsvols/vol, then your volume name would be 'vol'& and the parameter 'devices' would contain value '/mnt/xfsvols'. -2. Edit /etc/swift.conf to add swiftonfile as a storage policy: +#### Setting SwiftOnFile as storage policy +Edit /etc/swift.conf to add swiftonfile as a storage policy: ~~~ [storage-policy:0] name = swift +default = yes [storage-policy:1] name = swiftonfile-test -default = yes ~~~ +You can also make "swiftonfile-test" the default storage policy by using the 'default' parameter. -3. Edit the remakerings script to prepare rings for this new storage policy: +#### Prepare rings +Edit the remakerings script to prepare rings for this new storage policy: ~~~ swift-ring-builder object-1.builder create 1 1 1 swift-ring-builder object-1.builder add r1z1-127.0.0.1:6050/vol 1 swift-ring-builder object-1.builder rebalance ~~~ +Execute the remakerings script to prepare new rings files. +In a SAIO setup remakerings scipt is usually situated at ~/bin/remakerings.It you can also run above rings builder commands manually. -4. Restart swift services to reflect new changes: +Notice the mapping between SP index (1) defined in conf file above and the object ring builder command. + +#### Load the new configurations +Restart swift services to reflect new changes: ~~~ swift-init all restart @@ -86,28 +95,29 @@ swift-init all restart -5. Running functional tests: TBD +#### Running functional tests +TBD ## Using SwiftOnFile It is assumed that you are still using 'tempauth' as authnetication method, which is default in SAIO deployment. -### Get the token +#### Get the token ~~~ curl -v -H 'X-Auth-User: test:tester' -H "X-Auth-key: testing" -k http://localhost:8080/auth/v1.0 ~~~ Use 'X-Auth-Token' & 'X-Storage-Url' returned in above request for all sucequent request. -### Create a container +#### Create a container Create a container using the following command: ~~~ -curl -v -X PUT -H 'X-Auth-Token: AUTH_XXXX' http://localhost:8080/v1/AUTH_test/mycontainer +curl -v -X PUT -H 'X-Auth-Token: AUTH_XXXX' -H 'X-Storage-Policy: swiftonfile-test' http://localhost:8080/v1/AUTH_test/mycontainer ~~~ It should return `HTTP/1.1 201 Created` on a successful creation. #### Create an object -You can now place an object in the container you have just dreated: +You can now place an object in the container you have just created: ~~~ echo "Hello World" > mytestfile From 1cdc18e93d2c61e269d79ddab23e7a0a0267eacd Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Tue, 15 Jul 2014 16:43:37 +0530 Subject: [PATCH 11/14] Included suggestions --- doc/markdown/quick_start_guide.md | 45 ++++++++++++++++--------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index 5768ed7..19be758 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -9,33 +9,33 @@ ## Overview -SwiftOnFile allows any POSIX complaint filesystem to be used as the backend to the object store OpenStack Swift. +SwiftOnFile allows any POSIX complaint filesystem (which supports extended attributes) to be used as the backend to the OpenStack Swift(Object Store). -The following guide assumes you have a running [OpenStack Swift SAIO setup][], and you want to extend this setup to try SwiftOnFile as Storage Policy on a XFS/gluster volume. This will get you quickly started with a SwiftOnFile environment on a Fedora or RHEL/CentOS system. +The following guide assumes you have a running [OpenStack Swift SAIO setup][], and you want to extend this setup to try SwiftOnFile as Storage Policy on a with XFS partition or GlusterFS volume. This will get you quickly started with a SwiftOnFile deployment on a Fedora or RHEL/CentOS system. -This guide will not go on detail on how to prepare a Swift SAIO setup or how to create a gluster volume (or other FS).This guide assumes you know about these technologies, if you require any help in setting those please refer to the link provided. +This guide will not go on detail on how to prepare a Swift SAIO setup or how to create a gluster volume (or other FS).This guide assumes you know about these technologies, if you require any help in setting those please refer to the links provided. ## System Setup ### Prerequisites on CentOS/RHEL -1. OpenStack SAIO deployment on Fedora20 onwards -2. One xfs/glusterfs volume - named vol +1. OpenStack SAIO deployment(this guide uses SAIO on Fedora20) +2. One xfs partition /glusterfs volume - named vol ->Note: Swift SAIO deployment should contain Storage Policy code changes. Initialy Storage Policy feature was developed seprately in openstack swift feature/ec branch, and it is now merged in master branch. The latest OpenStack Swift2.0 release also contain storage policy code. +>Note: Swift SAIO deployment should contain Storage Policy code changes. SwiftOnFile should work with OpenStack Swift2.0.0 release and onwards. Icehouse and prior release of OpenStack Swift does not have Storage Policy code. -Each xfs/glusterfs volume will be defined as a separate storage policy. +Each xfs partition/glusterfs volume will be defined as a separate storage policy. ### Install SwiftOnfile 1. Before you begin swiftonfile setup please ensure you have OpenStack Swift SAIO setup up & running. Please refer to the SAIO guide for this. -2. cd ~; git clone https://github.com/swiftonfile/swiftonfile.git -3. cd ~/swiftonfile;python setup.py develop;cd ~ +2. cd $HOME; git clone https://github.com/swiftonfile/swiftonfile.git +3. cd $HOME/swiftonfile;python setup.py develop;cd $HOME ### Configure SwiftOnFile as Storage Policy #### Object Server Configuration -A SAIO setup mimic a four node swift setup and should have four object server running.Add another object server for SwiftOnFile by setting the following configurations in the file /etc/swift/object-server/5.conf: +A SAIO setup emulates a four node swift setup and should have four object server running.Add another object server for SwiftOnFile DiskFile API implementation by setting the following configurations in the file /etc/swift/object-server/5.conf: ~~~ [DEFAULT] @@ -57,7 +57,7 @@ log_level = DEBUG log_requests = on disk_chunk_size = 65536 ~~~ ->Note: The parameter 'devices' tells about the path where your xfs/glusterfs volume is mounted. The sub directory under which your volume is mounted will be called volume name. For ex: You have a xfs formated partition /dev/sdb1, and you mounted it under /mnt/xfsvols/vol, then your volume name would be 'vol'& and the parameter 'devices' would contain value '/mnt/xfsvols'. +>Note: The parameter 'devices' tells about the path where your xfs partition /glusterfs volume is mounted. The sub directory under which your xfs partition/glusterfs volume is mounted will be called device name. For ex: You have a xfs formated partition /dev/sdb1, and you mounted it under /mnt/xfsvols/vol, then your device name would be 'vol' & and the parameter 'devices' would contain value '/mnt/xfsvols'. #### Setting SwiftOnFile as storage policy Edit /etc/swift.conf to add swiftonfile as a storage policy: @@ -81,7 +81,7 @@ swift-ring-builder object-1.builder add r1z1-127.0.0.1:6050/vol 1 swift-ring-builder object-1.builder rebalance ~~~ Execute the remakerings script to prepare new rings files. -In a SAIO setup remakerings scipt is usually situated at ~/bin/remakerings.It you can also run above rings builder commands manually. +In a SAIO setup remakerings scipt is usually situated at ~/bin/remakerings.You can also run above rings builder commands manually. Notice the mapping between SP index (1) defined in conf file above and the object ring builder command. @@ -89,7 +89,7 @@ Notice the mapping between SP index (1) defined in conf file above and the objec Restart swift services to reflect new changes: ~~~ -swift-init all restart +swift-init main restart ~~~ @@ -99,7 +99,7 @@ swift-init all restart TBD ## Using SwiftOnFile -It is assumed that you are still using 'tempauth' as authnetication method, which is default in SAIO deployment. +It is assumed that you are still using 'tempauth' as authentication method, which is default in SAIO deployment. #### Get the token ~~~ @@ -128,7 +128,7 @@ To confirm that the object has been written correctly, you can compare the test file with the object you created: ~~~ -cat /mnt/xfsvols/vol/test/mycontainer/mytestfile +cat /mnt/xfsvols/vol/AUTH_test/mycontainer/mytestfile ~~~ #### Request the object @@ -145,17 +145,18 @@ and compare it with md5sum of the file on your FS. ## What now? +You now have a single node SwiftOnFile setup ready, next sane step is a multinode swift and SwiftOnFile setup. It is recomended to have a look at [OpenStack Swift deployment guide][] & [Multiple Server Swift Installation][].If you now consider yourself familiar with a typical 4-5 node swift setup, you are good to extent this setup further and add SwiftOnFile DiskFile implementation as a Storage Policy to it. If you want to use SwiftOnFile on a gluster volume, it would be good to have a seprate gluster cluster. We would love to hear about any deployment scenarios involving SOF. + For more information, please visit the following links: - -* [Authentication Services Start Guide][] +* [OpenStack Swift deployment guide][] +* [Multiple Server Swift Installation][] +* [OpenStack Swift Storage Policy][] * [GlusterFS Quick Start Guide][] * [OpenStack Swift API][] [GlusterFS Quick Start Guide]: http://www.gluster.org/community/documentation/index.php/QuickStart [OpenStack Swift API]: http://docs.openstack.org/api/openstack-object-storage/1.0/content/ -[Jenkins]: http://jenkins-ci.org -[Authentication Services Start Guide]: auth_guide.md -[EPEL]: https://fedoraproject.org/wiki/EPEL -[Jenkins CI]: http://build.gluster.org/job/swiftonfile-builds/lastSuccessfulBuild/artifact/build/ -[test code]: https://github.com/swiftonfile/swiftonfile/tree/master/test/functional_auth/tempauth/conf/ +[OpenStack Swift Storage Policy]: http://docs.openstack.org/developer/swift/overview_policies.html [OpenStack Swift SAIO setup]: http://docs.openstack.org/developer/swift/development_saio.html +[OpenStack Swift deployment guide][http://docs.openstack.org/developer/swift/deployment_guide.html] +[Multiple Server Swift Installation][http://docs.openstack.org/developer/swift/howto_installmultinode.html] From 01abe7dc6846ac09bb31741407723973162d82bd Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Fri, 18 Jul 2014 13:10:24 +0530 Subject: [PATCH 12/14] Minor updates --- doc/markdown/quick_start_guide.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index 19be758..781cddf 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -35,7 +35,7 @@ Each xfs partition/glusterfs volume will be defined as a separate storage policy ### Configure SwiftOnFile as Storage Policy #### Object Server Configuration -A SAIO setup emulates a four node swift setup and should have four object server running.Add another object server for SwiftOnFile DiskFile API implementation by setting the following configurations in the file /etc/swift/object-server/5.conf: +An SAIO setup emulates a four node swift setup and should have four object server running. Add another object server for SwiftOnFile DiskFile API implementation by setting the following configurations in the file /etc/swift/object-server/5.conf: ~~~ [DEFAULT] @@ -57,7 +57,9 @@ log_level = DEBUG log_requests = on disk_chunk_size = 65536 ~~~ ->Note: The parameter 'devices' tells about the path where your xfs partition /glusterfs volume is mounted. The sub directory under which your xfs partition/glusterfs volume is mounted will be called device name. For ex: You have a xfs formated partition /dev/sdb1, and you mounted it under /mnt/xfsvols/vol, then your device name would be 'vol' & and the parameter 'devices' would contain value '/mnt/xfsvols'. +>Note: The parameter 'devices' tells about the path where your xfs partition or glusterfs volume is mounted. The sub directory under which your xfs partition or glusterfs volume is mounted will be called device name. + +For example: You have a xfs formated partition /dev/sdb1, and you mounted it under /mnt/xfsvols/vol, then your device name would be 'vol' & and the parameter 'devices' would contain value '/mnt/xfsvols'. #### Setting SwiftOnFile as storage policy Edit /etc/swift.conf to add swiftonfile as a storage policy: @@ -158,5 +160,5 @@ For more information, please visit the following links: [OpenStack Swift API]: http://docs.openstack.org/api/openstack-object-storage/1.0/content/ [OpenStack Swift Storage Policy]: http://docs.openstack.org/developer/swift/overview_policies.html [OpenStack Swift SAIO setup]: http://docs.openstack.org/developer/swift/development_saio.html -[OpenStack Swift deployment guide][http://docs.openstack.org/developer/swift/deployment_guide.html] -[Multiple Server Swift Installation][http://docs.openstack.org/developer/swift/howto_installmultinode.html] +[OpenStack Swift deployment guide]: [http://docs.openstack.org/developer/swift/deployment_guide.html] +[Multiple Server Swift Installation]: [http://docs.openstack.org/developer/swift/howto_installmultinode.html] From 581e6145f018389d7038114305976c3c09220684 Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Fri, 18 Jul 2014 13:46:00 +0530 Subject: [PATCH 13/14] Minor Update --- doc/markdown/quick_start_guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index 781cddf..9ea0046 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -59,7 +59,7 @@ disk_chunk_size = 65536 ~~~ >Note: The parameter 'devices' tells about the path where your xfs partition or glusterfs volume is mounted. The sub directory under which your xfs partition or glusterfs volume is mounted will be called device name. -For example: You have a xfs formated partition /dev/sdb1, and you mounted it under /mnt/xfsvols/vol, then your device name would be 'vol' & and the parameter 'devices' would contain value '/mnt/xfsvols'. +>For example: You have a xfs formated partition /dev/sdb1, and you mounted it under /mnt/xfsvols/vol, then your device name would be 'vol' & and the parameter 'devices' would contain value '/mnt/xfsvols'. #### Setting SwiftOnFile as storage policy Edit /etc/swift.conf to add swiftonfile as a storage policy: From 8bf1d6bb69c11e2713ddfc26172a2308a9fcc14a Mon Sep 17 00:00:00 2001 From: Pushpesh Sharma Date: Fri, 18 Jul 2014 14:09:21 +0530 Subject: [PATCH 14/14] Minor Update --- doc/markdown/quick_start_guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/markdown/quick_start_guide.md b/doc/markdown/quick_start_guide.md index 9ea0046..c37975e 100644 --- a/doc/markdown/quick_start_guide.md +++ b/doc/markdown/quick_start_guide.md @@ -160,5 +160,5 @@ For more information, please visit the following links: [OpenStack Swift API]: http://docs.openstack.org/api/openstack-object-storage/1.0/content/ [OpenStack Swift Storage Policy]: http://docs.openstack.org/developer/swift/overview_policies.html [OpenStack Swift SAIO setup]: http://docs.openstack.org/developer/swift/development_saio.html -[OpenStack Swift deployment guide]: [http://docs.openstack.org/developer/swift/deployment_guide.html] -[Multiple Server Swift Installation]: [http://docs.openstack.org/developer/swift/howto_installmultinode.html] +[OpenStack Swift deployment guide]: http://docs.openstack.org/developer/swift/deployment_guide.html +[Multiple Server Swift Installation]: http://docs.openstack.org/developer/swift/howto_installmultinode.html