Open()ing an object is necessarry only in two cases:
* Serving a GET request
* Recalculating etag when metadata is stale
(can be triggered by any type of request)
This change ensures that for requests other than GET, a file is not
opened if the metadata is valid (size and etag accurate). Note that
if metadata is stale, the file is still opened and read to compute etag.
This patch does not change the behaviour of triggering metadata
validation and regeneration for non-GET requests.
This is a port of following change:
http://review.gluster.org/#/c/13684/
Change-Id: I54caf2242dfe94c1feb1060abfba971f862587fa
Signed-off-by: Prashanth Pai <ppai@redhat.com>
DiskFile._finalize_put() will now retry renames if it fails with EBUSY
or ESTALE. This is required because for a brief period of time, rename
operation in glusterfs was non-blocking.
Reference: http://review.gluster.org/#/c/13366/
This change also does the following:
* Updates comments to add clarity for operations done and exceptions
caught in DiskFile.create()
* Handles race between container existance check (memcache) and
object creation a little more gracefully by logging what really
happened.
This is port of following change:
http://review.gluster.org/#/c/14118
Change-Id: I1e993f5fed5c75bb77bee0620e1e7104f7f69deb
Signed-off-by: Prashanth Pai <ppai@redhat.com>
During process of POST requests which updates object metadata (xattrs),
the following (ordered) sequence of syscalls were being made twice:
open(), fstat(), fgetxattr(), close()
Intuitively, one may assume that a getxattr() and setxattr() is enough
to fulfil the POST request as it is only supposed to update metadata.
But this isn't the case. The above series of syscalls is made first
during disk_file.open(). This will trigger an update of all stale
metadata (outdated size/etag) and the result is retained in a diskfile
class attribute named 'self._metadata'
Instead of using this pre-fetched metadata, the POST path was internally
invoking disk_file.open() again in disk_file.write_metadata(). This is
redundant and serves no purpose. self._metadata was being erased during
the context manager cleanup of disk_file.open()
This change is simple and does the following:
* Don't erase fetched metadata during context manager exit of open()
* Use a different internal variable to detect and raise DiskFileNotOpen
* Re-use self._metadata if available in disk_file.write_metadata()
Here's comparing syscalls made (POST path) with and without this fix:
https://bugzilla.redhat.com/show_bug.cgi?id=1314171#c4
This is a port of this change:
http://review.gluster.org/#/c/13668/
Change-Id: I74e6f849f4415f3816bb367e1900924a76bf19c1
Signed-off-by: Prashanth Pai <ppai@redhat.com>
A fd was not being closed after it was duplicated. This code path can
be easily hit when doing a GET on a file that needs Etag (md5sum) to
be recalculated.
Change-Id: I3ea5ae5b30f38797d4367a65239716832e344431
Signed-off-by: Prashanth Pai <ppai@redhat.com>
fallocate() allows for reserving disk space for a particular inode and
fd. Hence, a client can be sure that he won't see a ENOSPC (eventually
a 507 HTTP response) later during writes. Swift's object server has
had fallocate() support from a long time.
P.S: Older versions of glusterfs (<3.6) did not support fallocate
because FUSE did not support it earlier.
http://review.gluster.org/4969http://fuse.996288.n3.nabble.com/fallocate-support-in-FUSE-td10668.html
Change-Id: Ida4b16357901707d624f92bf1b2dc8f07da4f1ad
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Running `tox -e functest` when swift services are not running
is incorrectly reported as success! The fix for this borrowed
from here: https://review.openstack.org/235933
Also, sphinx package is not needed in swiftonfile as we do not
have docs to build.
Change-Id: I675a438367497bc9a3c4aca21a0e48458673ab04
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This will optimize the first GET on files added from file interface.
More info: https://gist.github.com/prashanthpai/62e0bec770421561ea79
Change-Id: I3f0fd897eedf1413c3e7d5dca0f6196c62549fcb
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This change:
* Simplifies read_metadata() method.
* Validates pickle header before attempting to unpickle.
Change-Id: I08d4187f7f4cc963d095b2cd2bcee3039a7dc858
Signed-off-by: Prashanth Pai <ppai@redhat.com>
The open() method of OS module don't have any keyword arguments and would fail
with exceptions if **kwargs is passed.
Instead, I have added mode as named argument with 0o777 as default mode for
os.open() call
Change-Id: I5628883f4fb5ef7f08944673b0e5cc09bc166540
Signed-off-by: Varun Mittal <varun.mittal@in.ibm.com>
There was a corner case where Etag returned would be incorrect. This
happens when the object is changed from file interface but with object
size remaining the same.
This change introduces an additional metadata that stores the mtime of
object during PUT. This stored mtime is compared with actual mtime
during GET to determine if the file has changed or not. Etag is
recalculated if the file has changed.
The scope of this fix in addressing the above mentioned corner case is
limited to new objects only. Also, refactoring the code further by
moving some methods from utils.py to classes in diskfile.py should
prevent some redundant (f)stat syscalls. These minor optimizations
will be addressed in a separate change.
Change-Id: If724697ef2b17a3c569b60bd81ebf98dab283da6
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This is a manual forward-port of the following change merged into
icehouse branch: https://review.openstack.org/215119
When content of an object is modified from filesystem interface, a GET
on the object will return inconsistent or incomplete content because the
content length originally stored as metadata no longer reflects the
actual length of the file after modification.
The complete fix will have two parts:
(1) Return the entire content of object as is to the client
(2) The Etag returned should reflect the actual md5sum of object content
This change only fixes (1) mentioned above. This means, the client will
always get the complete content of the file.
Fix (2) is not part of this change. This means, if content length of the
object remains same even after modification, the Etag returned would be
incorrect. Fixing (2) involves more invasive changes in the code. So
that is deferred for now and will be sent as a separate change later.
Reference:
https://bugs.launchpad.net/swiftonfile/+bug/1416720https://review.openstack.org/151897
Change-Id: I28d0ec33c59eb520be7d15a60adb968692226e3e
Closes-Bug: #1416720
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This is a manual forward-port if the following change in icehouse
branch: https://review.openstack.org/211866
Object server never closes file descriptor when DiskFile.open()
raises an exception. This is fixed by catching exceptions thrown
by code block inside DiskFile.open() and manually closing the
file descriptor.
Change-Id: Ie3e047443f4893e21b9cbdea691867f069363d7e
Signed-off-by: Prashanth Pai <ppai@redhat.com>
The xattr module raises IOError which is what the call to
read_metadata() in rmobjdir() should be catching.
Change-Id: I2983ad0be453647f80b862eea93a5fc16fcef04a
Signed-off-by: Prashanth Pai <ppai@redhat.com>
restrict pyeclib version to 1.0.7, similar to swift
Change-Id: Ia69b6e1ac82a60d5cca9bbac8bb355124dc75cd6
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Signed-off-by: Prashanth Pai <ppai@redhat.com>
The intent was (and still is) to devise an unique name for the tempfile.
I can't think of any reason why uuid wouldn't work.
Also, added a comment which has a link to the document explaining why
the specific naming convention is used for temp file name.
Change-Id: I5f0d2b631e276c47c88c554485caaec767703695
Signed-off-by: Prashanth Pai <ppai@redhat.com>
These are old remnant code that is no longer in use.
Change-Id: Ic0afe1f16884efec49bfa2f64345129e1421daa6
Signed-off-by: Prashanth Pai <ppai@redhat.com>
This change does the following:
* Update all functional tests, unit tests and conf files based on
Swift 2.3.0 release.
* Fix "test_versioning_dlo" test from Swift to be SOF compatible.
* Bump swiftonfile release version to 2.3.0
* Update swiftonfile with DiskFile API changes
Change-Id: I75e322724efa4b946007b6d2786d92f73d5ae313
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Add IOERROR.strerror to exception messages in utils.py
Correct some types in exception messages in fs_utils.py
Change-Id: Iac81860b08daf8cdbb7f8e8950f4ab6104b75bd4
Closes-Bug: #1445237
Allow a slash at the end of object file name but ONLY if the PUT
request also has a "Content-type: application/directory" header.
Change-Id: Ic775de052ee3635e95f5e32ca6e2038909fe1005
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Override Swift's REPLICATE request handler method with a no-op which
returns HTTPNotImplemented (501).
Change-Id: Iaa52ca7e66ddaeda55e73841cd06b64e3e8a3369
Signed-off-by: Prashanth Pai <ppai@redhat.com>
NOTE:
The previous rebase was to Swift 2.1.0 and this rebase is to
Swift 2.2.1 (first release in kilo series). There was a
Swift 2.2.0 (last release in juno series) release in between.
Change-Id: Ibce2e299935e165db89a91a6fe8c4c5c027db098
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Add note about swift geo-replication not being supported for
Swift on File containers.
Also add reference and links to Swift on File presentation from
Paris summit.
Change-Id: If8b4608d7b20c16b4e8890728eb9714db01becc9
Used vulture utility to detect unused code in swiftonfile.
https://pypi.python.org/pypi/vulture
Change-Id: I045f5a96a48d2384718ad0f993540caa3a866309
Signed-off-by: Prashanth Pai <ppai@redhat.com>
