NOTE:
The previous rebase was to Swift 2.1.0 and this rebase is to
Swift 2.2.1 (first release in kilo series). There was a
Swift 2.2.0 (last release in juno series) release in between.
Change-Id: Ibce2e299935e165db89a91a6fe8c4c5c027db098
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Used vulture utility to detect unused code in swiftonfile.
https://pypi.python.org/pypi/vulture
Change-Id: I045f5a96a48d2384718ad0f993540caa3a866309
Signed-off-by: Prashanth Pai <ppai@redhat.com>
SOF can support object names of upto 1024 if it conforms to the
following constraints:
Object names can have forward slashes ('/') in them. Each segment in
between these slashes cannot exceed 255 characters with exception of
the last segment which cannot exceed 221 characters.
This constraint arises from the fact that each segment except the last
one in object name is a directory, the last segment being an actual
file on the filesystem.
Also, restored default constraint values in swift.conf since the SOF
constraints middleware (always in proxy pipeline) will take care of it.
Change-Id: Ia7dc44671a87911c092fecd0344eace92f5c225b
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Adding a constraints middleware that allows different constraints for a
Swift-on-File storage policy.
Currently, Swift allows only one set of object naming and metadata rules
per cluster. As new storage policies are implemented, it is possible that
different storage back-ends have different set of rules on how objects should
be created for that specific storage system. Swift-on-File has different rules
as objects name are mapped to a POSIX filesystem directory structure.
For example: no names with double slashes or ending with slashes are allowed.
At first a solution was proposed to include a generic patch in the upstream
swift code, but after discussing with the Swift community it became clear that
it would be better to have a global set of constraints that covers the whole
cluster and if a specific storage policy has specific constraints than it
should have its own middleware.
Link to patch for reference: https://review.openstack.org/#/c/113325/
Change-Id: I323ead5d98bf5c087930ccf446d3e8d83075e584
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Current tests from upstream swift leave containers in the
database after they are done running.
This causes an issue with sof tests because we clean up the
tests directory /mnt/swiftonfile. When running the
functional tests a second time always fails because the
setup functions tries to delete existing containers,
but the sof object servers returns 404, thus causing
the whole test to fail.
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Current tests from upstream swift leave containers in the
database after they are done running.
This causes an issue with sof tests because we clean up the
tests directory /mnt/swiftonfile. When running the
functional tests a second time always fails because the
setup functions tries to delete existing containers,
but the sof object servers returns 404, thus causing
the whole test to fail.
Signed-off-by: Thiago da Silva <thiago@redhat.com>
These files are no long in use by swiftonfile.
- functional_auth/tempauth files are no longer required because
tests now rely on a saio setup where tempauth is already used
by default.
- tools/functional_tests.sh has been replaced by .functests.
- tools/tox_run.sh is no longer required, but swiftonfile now requires
a saio setup and swiftonfile to be configured as a storage policy
for testing.
- test/unit/common/data/* files are also no longer in use.
Signed-off-by: Thiago da Silva <thiago@redhat.com>
* Rename references of '/mnt/gluster-object' to '/mnt/swiftonfile'.
* Rename swiftonfile SP name from 'sof' to 'swiftonfile'.
* Retain reseller_prefix in account dir created. This is to comply
with Swift's behavior.
* Cleanup mountpoint after running swiftonfile specific tests.
Signed-off-by: Prashanth Pai <ppai@redhat.com>
* Sync to Swift 2.0.0rc1 release.
* Use separate conf files for functional tests
* Define swiftonfile as policy with index 2
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Added Swift On File specific functional tests that checks SoF's
object naming convention by comparing names of objects PUT and
actual files found over mountpoint.
Remove old reference to keystone functional tests.
Signed-off-by: Prashanth Pai <ppai@redhat.com>
* Functional tests now run in SAIO+SoF setup.
* Fix and update various constraints.
* Update object-server.conf, swift.conf files. Add test.conf file.
* Added ./.functests script
TODO:
* Reintroduce gluster_swift_tests.py with SoF
specific functional tests.
Signed-off-by: Prashanth Pai <ppai@redhat.com>
* Fix API inconsistencies
* Remove hard-coded reseller_prefix = AUTH_
* Remove unused methods from gluster/swift/common/utils.py
These used to be called from DiskDir earlier.
* Remove hidden configuration option that were related to account
and container listings and metadata.
* Remove unused and redundant attributes
Signed-off-by: Prashanth Pai <ppai@redhat.com>
* Inherit DiskFileMananger class and few other methods from Swift
and reuse them.
* Unit tests for write_pickle method.
* Added policy_idx args, wherever missing.
Signed-off-by: Prashanth Pai <ppai@redhat.com>
* The file path on mountpoint was /container/obj and has been changed to
/account/container/obj. This structure is not created on account and
container creation but rather during the first object PUT to that
container.
* Rename OnDiskManager class to DiskFileManager
* Remove overridden container_update() method and use Swift's default
implementation that updates container DB.
* Add support for async_dir in storage policies - failed database
updates are stored in these directories to be later picked up by
object-updater daemon. This change reintroduced write_pickle and
pickle_async_update methods which are slightly modified versions
of Swift's methods of the same name.
* Updated tox.ini dependency on Swift to point to feature/ec branch of
Swift's github repo.
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This commit only removes code that is not required when swiftonfile
will be used as Storage Policy in Swift.
This commit does NOT:
* include code changes to support SP
* fix functional tests
* update documentation related to SP
This commit removes the following:
* Gen builders script and its man page.
* Authentication guide, user guide and other docs.
* Swiftkerbauth code, unit tests, func tests and doc.
* gswauth code, unit tests, func tests and doc.
* Object expirer code modifications, unit tests and doc.
* Conf files of account, container, proxy.
* Account, container and proxy server code - server.py of each.
* Account, container and proxy unit and functional tests.
* DiskDir class and corresponding unit tests.
* Our overridden ring.py (enforces account = volume = device) and tests.
* Functional tests for authentication filters.
* modules/swift and test/deploy directories.
* Proxy base controllers unit tests.
NOTE: We may have to reintroduce some of the above functional and unit
tests after SP related code changes - during fixing functional tests.
This commit modifies:
* setup.py to reflect the above code removals.
* constraints.py to remove references to ring.py
* object server.py to remove object-expirer changes
* tox.ini to remove ksfunctest and swfunctest
Signed-off-by: Prashanth Pai <ppai@redhat.com>
The difference between these tests and those under the functional
directory, is that these tests do not run under tox, and so
require a full OpenStack Swift + swiftonfile deployment.
The conf files where copies of the gswauth test, but I modified them
to be able to do a mount check.
The purpose of this tests is to confirm the quality of the software
in a deployed system.
The tests require the following:
- GlusterFS Volumes: test, test2, gsmetadata
- Directory /mnt/gluster-object
- OpenStack Swift + swiftonfile to be installed.
Signed-off-by: Luis Pabon <lpabon@redhat.com>
functional test writes and read a large object to exercise
code that calls drop_cache after reading 1MB of data
Signed-off-by: Thiago da Silva <thiago@redhat.com>
This chunk of code is only executed when reading a large file
(> 1014 * 1024), so the issue was never caught in our unit tests
refer to issue #13
Signed-off-by: Thiago da Silva <thiago@redhat.com>
* Updated tox.ini
* Created .unittests and .functests
* Added tempurl to pipeline in conf files so that tempurl tests are not
skipped
* Updated .gitignore file
Signed-off-by: Prashanth Pai <ppai@redhat.com>
With the Icehouse release of openstack-swift functional
case classified as fucntionalnose tests have been moved
to functional cases.Although we copied test suites from
test/fucntionalnose/ to test/functional/ but we still
have same cases under test/functionalnose.This might
cause duplicate tests run.
Change-Id: I025206467aad364debd9050b3186e1379d89ffaf
Signed-off-by: Pushpesh Sharma psharma@redhat.com
Reviewed-on: http://review.gluster.org/7516
Reviewed-by: Prashanth Pai ppai@redhat.com
Tested-by: Prashanth Pai ppai@redhat.com
Reviewed-by: Chetan Risbud crisbud@redhat.com
Reviewed-by: Thiago da Silva thiago@redhat.com
This change is being done to prepare the code to always call
fs_utils for all filesystem calls on the data being stored.
By creating this interface (i.e., fs_utils), we can then
make a seamless change to use the current method of 'os.'
calls over FUSE _or_ libgfapi. This new method will be introduced
in a new separate patch.
Change-Id: Ic768fa7352d7672b1f060230bb4486f0ec228152
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Reviewed-on: http://review.gluster.org/7333
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
This change is being made to allow callers to handle
any exception thrown by os.mkdir.
This function is currently never called anywhere in the code.
It was introduced as part of the first commit to this project
but it was never used.
This patch (http://review.gluster.org/#/c/5304/) removed the
early version of this function, and this patch
(http://review.gluster.org/#/c/5305/) added it back with new
exception handling.
Change-Id: I71325660cb47594b0804da3da21920e26d2055f2
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Reviewed-on: http://review.gluster.org/7350
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Chetan Risbud <crisbud@redhat.com>
Tested-by: Chetan Risbud <crisbud@redhat.com>
The file descriptor is not being closed because
it is self._fd is None
Change-Id: I7edc8a78b09bdd76d59ac8f3dbc809af652f9b0e
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Reviewed-on: http://review.gluster.org/7315
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
This function is no longer used, the code that called
this function was removed as part of this patch:
http://review.gluster.com/#/c/3390/
Change-Id: I6e81e4b763ce302289e2be5467dc4776f0750c15
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Reviewed-on: http://review.gluster.org/7336
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
Preventing access to expired objects
------------------------------------
Re-enabled accepting X-Delete-At and X-Delete-After headers. During a GET on
an expired object, DiskFileExpired is raised by DiskFile class. This will
result in object-server returning HTTPNotFound (404) to the client.
Tracking objects to be deleted
------------------------------
Objects to be deleted are tracked using "tracker objects". These are PUT into
a special account(a volume, for now). These zero size "tracker objects" have
names that contain:
* Expiration timestamp
* Path of the actual object to be deleted
Deleting actual objects from GlusterFS volume
---------------------------------------------
The object-expirer daemon runs a pass once every X seconds. For every pass it
makes, it queries the special account for "tracker objects". Based on
(timestamp, path) present in name of "tracker objects", object-expirer then
deletes the actual object and the corresponding tracker object.
To run object-expirer forever:
swift-init object-expirer start
To run just once:
swift-object-expirer -o -v /etc/swift/object-expirer.conf
Caveat/Limitation: Object-expirer needs a separate account(volume) that
is not used by other services like gswauth. By default, this volume is
named "gsexpiring" and is configurable.
More info about object expiration:
http://docs.openstack.org/developer/swift/overview_expiring_objects.html
Change-Id: I876995bf4f16ef4bfdff901561e0558ecf1dc38f
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Reviewed-on: http://review.gluster.org/6891
Tested-by: Chetan Risbud <crisbud@redhat.com>
Reviewed-by: pushpesh sharma <psharma@redhat.com>
Tested-by: pushpesh sharma <psharma@redhat.com>
Reviewed-by: Chetan Risbud <crisbud@redhat.com>
This provides an infrastructure for swiftkerbauth
related functional test cases.
More test cases will be added later.
Added a section in swiftkerbauth guide about how to run
functional tests.
test/functional_auth/swiftkerbauth
----------------------------------
A new authentication filter related functional
tests and configuration to reside here.
The configuration would help setup the
environment. All the generic functional tests
should run fine with PASSIVE mode of swiftkerbatuh.
Please refere to swiftkerbatuh documentation for
ACTIVE/PASSIVE mode of working.
swiftkerbauth/test_swkrbath_active.py
-------------------------------------
This file has all the testcases of active mode of
swiftkerbauth. More test cases to be added later.
SwiftKerbAuth related test cases are meant to run
on the setup where SwiftKerbAuth is setup and
installed.
Change-Id: Ibc2a3945f5c9b6714475fcec0ee9d153debb48e3
Signed-off-by: Chetan Risbud <crisbud@redhat.com>
Reviewed-on: http://review.gluster.org/6925
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
Also, bumped version of gluster-swift to v1.13.0
Change-Id: I797dc704c9523540cba847b1e8ff3da97b79630c
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Reviewed-on: http://review.gluster.org/7229
Reviewed-by: Chetan Risbud <crisbud@redhat.com>
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
PUT of a directory fails with gluster-swift when there exists
object of a same name. This is because the objects and directories are placed on
the glusterfs. And hence the filesystem semantics are applicable there.
Exceptions raised in such situation are needed to be handled and reported
correctly back to proxy-server with HTTPConflict as a error code. Although
swift still continues to choose 503 as the best respose in such cases. No
tracebacks reported.
Fix covers the case when there exists a directory and object of the same
name is PUT.
Code changes fixes both the failure cases mentioned in the bug.
Examples of failing PUT requests:
1) curl -v -X PUT http://127.0.0.1:8080/v1/AUTH_test/c1/dir1/obj2/anotherobject
-d'asdasdsadA'
-- obj2 was already an object.
2)curl -v -H 'Content-Length: 0' -H 'Content-Type: application/directory' -X PUT
http://127.0.0.1:8080/v1/AUTH_test/c1/obj1
-- obj1 was already and object
Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1071021
Change-Id: Id3042d920e3f99e740d4042ef5907ac8c59e04db
Signed-off-by: Chetan Risbud <crisbud@redhat.com>
Reviewed-on: http://review.gluster.org/7181
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Tested-by: Prashanth Pai <ppai@redhat.com>
* Bumped the version of gluster-swift to v1.12.0.
* Added document on how to do a sync
Change-Id: I676e110759b0bce3007a2a38f3b384b4ca9d710d
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/6977
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Chetan Risbud <crisbud@redhat.com>
Tested-by: Chetan Risbud <crisbud@redhat.com>
added input validation for a couple of options and
error handling in case a non-existing account name
is provided
Change-Id: I6d703d584552fc7b7574f34e79ed25a2982b6d5e
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Reviewed-on: http://review.gluster.org/6767
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Tested-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: pushpesh sharma <psharma@redhat.com>
Tested-by: pushpesh sharma <psharma@redhat.com>
Reviewed-by: Chetan Risbud <crisbud@redhat.com>
For some scenarios although the expected response as per the
REST spec is 403, but due to technical limitation/performance
improvements it is decided to go with 401.EX:An admin/regular
user tries to add a reseller admin user.
Change-Id: Ibc280a0be6e5cb81d4d2c46d9a758915ff60ae8f
Reviewed-on: http://review.gluster.org/6833
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Chetan Risbud <crisbud@redhat.com>
Tested-by: Chetan Risbud <crisbud@redhat.com>
This commit creates a new class to validate the cleanup-token
& set-account-service CLI utility.This test tries to cover all the psossible values
that can be passed as user input valid/invalid.
Change-Id: Ia8ca40ca9e63a296b70cb999c6507b7b1ffbdf61
Signed-off-by: Pushpesh Sharma <psharma@redhat.com>
Reviewed-on: http://review.gluster.org/6759
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
CLI test case were failing on RHEL systems as the error string
on Fedora19 and RHEL6.4 systems are different however a more gereric
string is there on both of the systems and this is about that only.
It replaces the 'ECONNREFUSED' & 'No address associated with hostname'
with 'Check that the admin_url is valid'.Now the TCs pass on both of
the OS.
Change-Id: Iff8b2719e8917864ed8f80d961c55a7f9f6fc77b
Signed-off-by: Pushpesh Sharma <psharma@redhat.com>
Reviewed-on: http://review.gluster.org/6718
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
