The method assertEquals has been deprecated since python 2.7.
http://docs.python.org/2/library/unittest.html#deprecated-aliases
Also in Python 3, a deprecated warning is raised when using assertEquals
therefore we should use assertEqual instead.
This is part of blueprint assertequal
Change-Id: Ie8887ba468bee10ddeb5b7a1d095821f898b9302
fallocate() allows for reserving disk space for a particular inode and
fd. Hence, a client can be sure that he won't see a ENOSPC (eventually
a 507 HTTP response) later during writes. Swift's object server has
had fallocate() support from a long time.
P.S: Older versions of glusterfs (<3.6) did not support fallocate
because FUSE did not support it earlier.
http://review.gluster.org/4969http://fuse.996288.n3.nabble.com/fallocate-support-in-FUSE-td10668.html
Change-Id: Ida4b16357901707d624f92bf1b2dc8f07da4f1ad
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This will optimize the first GET on files added from file interface.
More info: https://gist.github.com/prashanthpai/62e0bec770421561ea79
Change-Id: I3f0fd897eedf1413c3e7d5dca0f6196c62549fcb
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This change:
* Simplifies read_metadata() method.
* Validates pickle header before attempting to unpickle.
Change-Id: I08d4187f7f4cc963d095b2cd2bcee3039a7dc858
Signed-off-by: Prashanth Pai <ppai@redhat.com>
There was a corner case where Etag returned would be incorrect. This
happens when the object is changed from file interface but with object
size remaining the same.
This change introduces an additional metadata that stores the mtime of
object during PUT. This stored mtime is compared with actual mtime
during GET to determine if the file has changed or not. Etag is
recalculated if the file has changed.
The scope of this fix in addressing the above mentioned corner case is
limited to new objects only. Also, refactoring the code further by
moving some methods from utils.py to classes in diskfile.py should
prevent some redundant (f)stat syscalls. These minor optimizations
will be addressed in a separate change.
Change-Id: If724697ef2b17a3c569b60bd81ebf98dab283da6
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This is a manual forward-port of the following change merged into
icehouse branch: https://review.openstack.org/215119
When content of an object is modified from filesystem interface, a GET
on the object will return inconsistent or incomplete content because the
content length originally stored as metadata no longer reflects the
actual length of the file after modification.
The complete fix will have two parts:
(1) Return the entire content of object as is to the client
(2) The Etag returned should reflect the actual md5sum of object content
This change only fixes (1) mentioned above. This means, the client will
always get the complete content of the file.
Fix (2) is not part of this change. This means, if content length of the
object remains same even after modification, the Etag returned would be
incorrect. Fixing (2) involves more invasive changes in the code. So
that is deferred for now and will be sent as a separate change later.
Reference:
https://bugs.launchpad.net/swiftonfile/+bug/1416720https://review.openstack.org/151897
Change-Id: I28d0ec33c59eb520be7d15a60adb968692226e3e
Closes-Bug: #1416720
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This is a manual forward-port if the following change in icehouse
branch: https://review.openstack.org/211866
Object server never closes file descriptor when DiskFile.open()
raises an exception. This is fixed by catching exceptions thrown
by code block inside DiskFile.open() and manually closing the
file descriptor.
Change-Id: Ie3e047443f4893e21b9cbdea691867f069363d7e
Signed-off-by: Prashanth Pai <ppai@redhat.com>
The xattr module raises IOError which is what the call to
read_metadata() in rmobjdir() should be catching.
Change-Id: I2983ad0be453647f80b862eea93a5fc16fcef04a
Signed-off-by: Prashanth Pai <ppai@redhat.com>
These are old remnant code that is no longer in use.
Change-Id: Ic0afe1f16884efec49bfa2f64345129e1421daa6
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This change does the following:
* Update all functional tests, unit tests and conf files based on
Swift 2.3.0 release.
* Fix "test_versioning_dlo" test from Swift to be SOF compatible.
* Bump swiftonfile release version to 2.3.0
* Update swiftonfile with DiskFile API changes
Change-Id: I75e322724efa4b946007b6d2786d92f73d5ae313
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Allow a slash at the end of object file name but ONLY if the PUT
request also has a "Content-type: application/directory" header.
Change-Id: Ic775de052ee3635e95f5e32ca6e2038909fe1005
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Override Swift's REPLICATE request handler method with a no-op which
returns HTTPNotImplemented (501).
Change-Id: Iaa52ca7e66ddaeda55e73841cd06b64e3e8a3369
Signed-off-by: Prashanth Pai <ppai@redhat.com>
NOTE:
The previous rebase was to Swift 2.1.0 and this rebase is to
Swift 2.2.1 (first release in kilo series). There was a
Swift 2.2.0 (last release in juno series) release in between.
Change-Id: Ibce2e299935e165db89a91a6fe8c4c5c027db098
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Used vulture utility to detect unused code in swiftonfile.
https://pypi.python.org/pypi/vulture
Change-Id: I045f5a96a48d2384718ad0f993540caa3a866309
Signed-off-by: Prashanth Pai <ppai@redhat.com>
SOF can support object names of upto 1024 if it conforms to the
following constraints:
Object names can have forward slashes ('/') in them. Each segment in
between these slashes cannot exceed 255 characters with exception of
the last segment which cannot exceed 221 characters.
This constraint arises from the fact that each segment except the last
one in object name is a directory, the last segment being an actual
file on the filesystem.
Also, restored default constraint values in swift.conf since the SOF
constraints middleware (always in proxy pipeline) will take care of it.
Change-Id: Ia7dc44671a87911c092fecd0344eace92f5c225b
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Adding a constraints middleware that allows different constraints for a
Swift-on-File storage policy.
Currently, Swift allows only one set of object naming and metadata rules
per cluster. As new storage policies are implemented, it is possible that
different storage back-ends have different set of rules on how objects should
be created for that specific storage system. Swift-on-File has different rules
as objects name are mapped to a POSIX filesystem directory structure.
For example: no names with double slashes or ending with slashes are allowed.
At first a solution was proposed to include a generic patch in the upstream
swift code, but after discussing with the Swift community it became clear that
it would be better to have a global set of constraints that covers the whole
cluster and if a specific storage policy has specific constraints than it
should have its own middleware.
Link to patch for reference: https://review.openstack.org/#/c/113325/
Change-Id: I323ead5d98bf5c087930ccf446d3e8d83075e584
Signed-off-by: Thiago da Silva <thiago@redhat.com>
These files are no long in use by swiftonfile.
- functional_auth/tempauth files are no longer required because
tests now rely on a saio setup where tempauth is already used
by default.
- tools/functional_tests.sh has been replaced by .functests.
- tools/tox_run.sh is no longer required, but swiftonfile now requires
a saio setup and swiftonfile to be configured as a storage policy
for testing.
- test/unit/common/data/* files are also no longer in use.
Signed-off-by: Thiago da Silva <thiago@redhat.com>
* Functional tests now run in SAIO+SoF setup.
* Fix and update various constraints.
* Update object-server.conf, swift.conf files. Add test.conf file.
* Added ./.functests script
TODO:
* Reintroduce gluster_swift_tests.py with SoF
specific functional tests.
Signed-off-by: Prashanth Pai <ppai@redhat.com>
* Fix API inconsistencies
* Remove hard-coded reseller_prefix = AUTH_
* Remove unused methods from gluster/swift/common/utils.py
These used to be called from DiskDir earlier.
* Remove hidden configuration option that were related to account
and container listings and metadata.
* Remove unused and redundant attributes
Signed-off-by: Prashanth Pai <ppai@redhat.com>
* Inherit DiskFileMananger class and few other methods from Swift
and reuse them.
* Unit tests for write_pickle method.
* Added policy_idx args, wherever missing.
Signed-off-by: Prashanth Pai <ppai@redhat.com>
* The file path on mountpoint was /container/obj and has been changed to
/account/container/obj. This structure is not created on account and
container creation but rather during the first object PUT to that
container.
* Rename OnDiskManager class to DiskFileManager
* Remove overridden container_update() method and use Swift's default
implementation that updates container DB.
* Add support for async_dir in storage policies - failed database
updates are stored in these directories to be later picked up by
object-updater daemon. This change reintroduced write_pickle and
pickle_async_update methods which are slightly modified versions
of Swift's methods of the same name.
* Updated tox.ini dependency on Swift to point to feature/ec branch of
Swift's github repo.
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This commit only removes code that is not required when swiftonfile
will be used as Storage Policy in Swift.
This commit does NOT:
* include code changes to support SP
* fix functional tests
* update documentation related to SP
This commit removes the following:
* Gen builders script and its man page.
* Authentication guide, user guide and other docs.
* Swiftkerbauth code, unit tests, func tests and doc.
* gswauth code, unit tests, func tests and doc.
* Object expirer code modifications, unit tests and doc.
* Conf files of account, container, proxy.
* Account, container and proxy server code - server.py of each.
* Account, container and proxy unit and functional tests.
* DiskDir class and corresponding unit tests.
* Our overridden ring.py (enforces account = volume = device) and tests.
* Functional tests for authentication filters.
* modules/swift and test/deploy directories.
* Proxy base controllers unit tests.
NOTE: We may have to reintroduce some of the above functional and unit
tests after SP related code changes - during fixing functional tests.
This commit modifies:
* setup.py to reflect the above code removals.
* constraints.py to remove references to ring.py
* object server.py to remove object-expirer changes
* tox.ini to remove ksfunctest and swfunctest
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This chunk of code is only executed when reading a large file
(> 1014 * 1024), so the issue was never caught in our unit tests
refer to issue #13
Signed-off-by: Thiago da Silva <thiago@redhat.com>
This change is being done to prepare the code to always call
fs_utils for all filesystem calls on the data being stored.
By creating this interface (i.e., fs_utils), we can then
make a seamless change to use the current method of 'os.'
calls over FUSE _or_ libgfapi. This new method will be introduced
in a new separate patch.
Change-Id: Ic768fa7352d7672b1f060230bb4486f0ec228152
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Reviewed-on: http://review.gluster.org/7333
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
This change is being made to allow callers to handle
any exception thrown by os.mkdir.
This function is currently never called anywhere in the code.
It was introduced as part of the first commit to this project
but it was never used.
This patch (http://review.gluster.org/#/c/5304/) removed the
early version of this function, and this patch
(http://review.gluster.org/#/c/5305/) added it back with new
exception handling.
Change-Id: I71325660cb47594b0804da3da21920e26d2055f2
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Reviewed-on: http://review.gluster.org/7350
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Chetan Risbud <crisbud@redhat.com>
Tested-by: Chetan Risbud <crisbud@redhat.com>
The file descriptor is not being closed because
it is self._fd is None
Change-Id: I7edc8a78b09bdd76d59ac8f3dbc809af652f9b0e
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Reviewed-on: http://review.gluster.org/7315
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
This function is no longer used, the code that called
this function was removed as part of this patch:
http://review.gluster.com/#/c/3390/
Change-Id: I6e81e4b763ce302289e2be5467dc4776f0750c15
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Reviewed-on: http://review.gluster.org/7336
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
Preventing access to expired objects
------------------------------------
Re-enabled accepting X-Delete-At and X-Delete-After headers. During a GET on
an expired object, DiskFileExpired is raised by DiskFile class. This will
result in object-server returning HTTPNotFound (404) to the client.
Tracking objects to be deleted
------------------------------
Objects to be deleted are tracked using "tracker objects". These are PUT into
a special account(a volume, for now). These zero size "tracker objects" have
names that contain:
* Expiration timestamp
* Path of the actual object to be deleted
Deleting actual objects from GlusterFS volume
---------------------------------------------
The object-expirer daemon runs a pass once every X seconds. For every pass it
makes, it queries the special account for "tracker objects". Based on
(timestamp, path) present in name of "tracker objects", object-expirer then
deletes the actual object and the corresponding tracker object.
To run object-expirer forever:
swift-init object-expirer start
To run just once:
swift-object-expirer -o -v /etc/swift/object-expirer.conf
Caveat/Limitation: Object-expirer needs a separate account(volume) that
is not used by other services like gswauth. By default, this volume is
named "gsexpiring" and is configurable.
More info about object expiration:
http://docs.openstack.org/developer/swift/overview_expiring_objects.html
Change-Id: I876995bf4f16ef4bfdff901561e0558ecf1dc38f
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Reviewed-on: http://review.gluster.org/6891
Tested-by: Chetan Risbud <crisbud@redhat.com>
Reviewed-by: pushpesh sharma <psharma@redhat.com>
Tested-by: pushpesh sharma <psharma@redhat.com>
Reviewed-by: Chetan Risbud <crisbud@redhat.com>
Also, bumped version of gluster-swift to v1.13.0
Change-Id: I797dc704c9523540cba847b1e8ff3da97b79630c
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Reviewed-on: http://review.gluster.org/7229
Reviewed-by: Chetan Risbud <crisbud@redhat.com>
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
PUT of a directory fails with gluster-swift when there exists
object of a same name. This is because the objects and directories are placed on
the glusterfs. And hence the filesystem semantics are applicable there.
Exceptions raised in such situation are needed to be handled and reported
correctly back to proxy-server with HTTPConflict as a error code. Although
swift still continues to choose 503 as the best respose in such cases. No
tracebacks reported.
Fix covers the case when there exists a directory and object of the same
name is PUT.
Code changes fixes both the failure cases mentioned in the bug.
Examples of failing PUT requests:
1) curl -v -X PUT http://127.0.0.1:8080/v1/AUTH_test/c1/dir1/obj2/anotherobject
-d'asdasdsadA'
-- obj2 was already an object.
2)curl -v -H 'Content-Length: 0' -H 'Content-Type: application/directory' -X PUT
http://127.0.0.1:8080/v1/AUTH_test/c1/obj1
-- obj1 was already and object
Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1071021
Change-Id: Id3042d920e3f99e740d4042ef5907ac8c59e04db
Signed-off-by: Chetan Risbud <crisbud@redhat.com>
Reviewed-on: http://review.gluster.org/7181
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Tested-by: Prashanth Pai <ppai@redhat.com>
* Bumped the version of gluster-swift to v1.12.0.
* Added document on how to do a sync
Change-Id: I676e110759b0bce3007a2a38f3b384b4ca9d710d
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/6977
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Chetan Risbud <crisbud@redhat.com>
Tested-by: Chetan Risbud <crisbud@redhat.com>
Imported code till commit f64a3354185f32928e2568d9ece4a52fa4746c05
Changed a code bit to import correct definitions.
kerbauth unit tests do run along with gluster-swift.
Install script does install swiftkerbauth.
import swiftkerbauth from http://review.gluster.org/swiftkrbauth.git
Change-Id: Ia89f2b77cc68df10dee2f41ce074f3381ac3c408
Signed-off-by: Chetan Risbud <crisbud@redhat.com>
Reviewed-on: http://review.gluster.org/6597
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
Changed the code to only allow the super_admin to delete a reseller_admin.
This follows the same logic of user creation, where only the super_admin
can create a reseller_admin.
Also, took the opportunity to refactor some code and implemented
get_user_detail method to remove duplicated code
Bug 1260239: https://bugs.launchpad.net/gluster-swift/+bug/1260239
Change-Id: I9e4866cd7ad08698f427846be566ab2364ad4850
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Reviewed-on: http://review.gluster.org/6516
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
Users were not able to update their own password/key
with the update operation resulting in 403 (HTTPForbidden).
EXAMPLES:
Command to update password/key of regular user:
gswauth-add-user -U account1:user1 -K old_pass account1 user1 new_pass
Command to update password/key of account admin:
gswauth-add-user -U account1:admin -K old_pass -a account1 admin new_pass
Command to update password/key of reseller_admin:
gswauth-add-user -U account1:radmin -K old_pass -r account1 radmin new_pass
BUG: https://bugs.launchpad.net/gluster-swift/+bug/1262227
Change-Id: I604da5aee67099b29541eb7e51a040a041f1961b
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Reviewed-on: http://review.gluster.org/6650
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
