Added a fake_file class that implements minimal set of functions that
are invoked by the code in GET.
BUG: 987841
https://bugzilla.redhat.com/show_bug.cgi?id=987841
Change-Id: I5bdf5be1c0c4c8231f34c9be529e6edc83774f2e
Signed-off-by: Chetan Risbud <crisbud@redhat.com>
Reviewed-on: http://review.gluster.org/5511
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
While it is very improbable, it seemed fairly straight forward to add
a small bit of code to ensure the temporary file name is random and
won't clash with another host, pid or thread potentially working on
the same object.
BUG: XXXXXX (https://bugzilla.redhat.com/show_bug.cgi?id=XXXXXX)
Change-Id: I862021b725efbfe58b98754c4470aef4882eb8f7
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5671
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
For whatever reason, it appears that GlusterFS, or perhaps FUSE can
continuously return ENOENT on a rename system call even when we have
double checked that there is no reason to do so. That is a bug for
that sub system. However, our response to that bug can result in an
infinite loop, which is bad.
This code reduces that to 10 attempts.
In addition, we restructed the open retry loop to match, providing
module constants for the upper bounds of both retry loops.
BUG: XXXXXX (https://bugzilla.redhat.com/show_bug.cgi?id=XXXXXX)
Change-Id: Ia2d6dd427daba3ea0461863c5ffe3aef27c88f9b
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5670
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
The code changes are basically:
* Apply refactoring in the DiskFile class to use the new DiskWriter
abstraction
* Move and rename our diskfile module to match upstream
* ThreadPools allow us to remove the tpool usage around fsync
* Update the Ring subclass to support the get_part() method
* Update to use the 1.9.1 proxy server unit tests
* Move the DebugLogger class to test.unit
* Rebuild the Rings to use the new layout
* Remove backup ring builder files
* Update spec files to 1.9.1, and tox to use swift 1.9.1
* Updated version to 1.9.0-0
Change-Id: Ica12cac8b351627d67500723f1dbd8a54d45f7c8
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5331
Our initial implementation only required Jenkins
to export binary RPMs, but as we move foward, we really
need to also export SRPMs. To support SRPMs, the spec
file in the RPM has to have the correct NAME, VERSION,
and RELEASE information.
Change-Id: Icd7132b4aafdbe7a1f02a35d0be7ad63b2e7c056
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5669
Reviewed-by: Peter Portante <pportant@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
Reviewed-on: http://review.gluster.org/5679
* Container and accounts performance increase by removing
the need to update either the object count or the
container count, respectively. New hidden
configuratoins added to re-enable the functionality.
* object_only configuratoin removed from fs.conf and
replaced with a hidden configuration. The new hidden
configuration reports gratuituosly crated directories
as objects, to support a compatibility behavior with
previous version 1.4.8 (even though it was incorrect).
BUG 988969: https://bugzilla.redhat.com/show_bug.cgi?id=988969
Change-Id: Idca20b8629ec38606ff5692fe62bc2cadabffc86
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5403
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
Reviewed-on: http://review.gluster.org/5504
A gluster volume could yield an ENOSPC condition seeing
that a volume is full. This needed to handled correctly.
Added error handling.
BUG: 985253
https://bugzilla.redhat.com/show_bug.cgi?id=985253
Change-Id: I85472c0a81a354a2796327fead606da3a938d4bf
Signed-off-by: Chetan Risbud <crisbud@redhat.com>
Reviewed-on: http://review.gluster.org/5362
Reviewed-by: Peter Portante <pportant@redhat.com>
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
Copyrights are updated across the code to reflect 2013.
Copyrights message was added to couple of scripts.
Change-Id: I2cfc7bdc6adaa5c9ab0075aab49021a5f0dea7a1
Signed-off-by: Chetan Risbud <crisbud@redhat.com>
Reviewed-on: http://review.gluster.org/5371
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
Still need to change unit tests to use Havana
Change-Id: I7847e7916043933a47b3553115c79c5d6b28a836
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5312
Remove a number of stat() system calls that were redundant.
BUG 903396 (https://bugzilla.redhat.com/show_bug.cgi?id=903396)
Change-Id: I2e89ce8a023650cd745f136f5c697902cf2462ea
Signed-off-by: Mohammed Junaid <junaid@redhat.com>
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5293
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
Change-Id: Ibcef5d817390a11d85cf125bb9addfbfded1b019
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5301
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
This commit has the following changes:
* G4S no longer accepts URLs that end in /. A HTTP code
of 400 is returned when a / at the end of the object
is detected.
* Directories can be created as objects setting the
content-type to application/directory and content-length
to 0.
* Functional tests have been adjusted to work with
G4S constraints
Change-Id: I31038a59699a8e3eeaba902db322218c6400093e
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5246
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
It was painfully slow to gather a list of all the files in a directory tree
when there were as many directories as files, since we also stat'd all of the
directories as well. We only did that to cache using memcache the result, to
know when it changed. However, there was no way for a memcache object to be
passed down to this code. So we have removed the memcache support entirely,
and removed the stat of the directories as a result.
See BZ 911448 (https://bugzilla.redhat.com/show_bug.cgi?id=911448).
Change-Id: I2625f82eca36c31eeffa84dc76ef7f3b48aafec5
Signed-off-by: Mohammed Junaid <junaid@redhat.com>
Reviewed-on: http://review.gluster.org/5252
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
This is a step towards making fd based system calls where ever possible to avoid
path lookups.
Signed-off-by: Mohammed Junaid <junaid@redhat.com>
Change-Id: I482ea29ebe0859d0a5307ff25ecb5945d54bc7ca
Reviewed-on: http://review.gluster.org/5251
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
The code was raising an exception when the container (which
happens to be a directory) did not exist. To be compatible
with OpenStack Swift, we need to handle an object which its
container/directory does not exist.
BUG: 960944 (https://bugzilla.redhat.com/show_bug.cgi?id=960944)
Change-Id: Ibb2db354a655e040fb70ebbe6a7d8f815d33dc0f
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5201
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
We have refactored DiskDir and DiskAccount so that truly common,
shared functions live in the base class DiskCommon, and each of these
then sub-class that. This allows us to isolate unique behaviors.
We have also enabled all the skipped unit tests in test_diskdir.py
that we could, removing a few that we don't need right now.
Change-Id: I48d9c915108df8cc92c3e9a764563e1d10c50050
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5148
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
By using a generator pattern, we avoid creating whole new lists each
time, instead we iterate through the original list once (after it is
sorted), constructing the final list only once.
We also address the behavioral differences between the swift filtering
results and our code so that ported unit tests work the same
(non-slash objects, that is).
Change-Id: If32c1987f24781ff81ab4c28c9ddfff17c2e7787
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5145
Tested-by: Luis Pabon <lpabon@redhat.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Reviewed-by: Luis Pabon <lpabon@redhat.com>
The various filter_* methods of the DiskCommon class don't operate on
the DiskCommon object instance itself, but always on the list of
objects passed in to it. This change does not modify those functions'
behaviors but simply refactors them to live at the module level.
Change-Id: Ic0f578d94aab65a524470278f10d1fdc8b1d6392
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5138
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
This is a simple sub-classing of the proper update methods to
just no-op their behavior.
Change-Id: Ib1ae5234d372cbce572da34cfe702235b78f2310
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5088
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
This is a copy of the proxy unit test from OpenStack
Swift. The file has minor changes to make it compatible
with Gluster For Swift.
Change-Id: I64a0f8d274461eb2a2c38524c6282e0d3d3d1457
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5113
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
Instead we save the account in the a list, where the
index to the account is the partition number.
Change-Id: Ie4abefee48a3b237306a1e301ffa798e24e3f1db
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5120
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
This is quite ugly. Sorry. We ported a set of test from OpenStack
Swift's test/unit/commont/test_db.py, the testAccountBroker and
testContainerBroker, but because of the divergent nature of the
current attempt UFO (Unified File and Object) implementation, it was
not possible to use the interface calls directly.
Instead, we are using these tests to make sure most of the code paths
are exercised, and to define much of the expected behavior. Further
refactoring, unit tests and functional test work will help us bring
the code base and these tests closer to the originals in upstream
OpenStack Swift (as of Havana).
Change-Id: I095bb03619de6e7e1378b5252913e39b1ea8bf27
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5135
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
We need the code to pass pep8 tests so that we can
enable pep8 in Jenkins
Change-Id: I31e4bd10fad65735fabceaedcb1a3bd10e120153
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5114
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
Change-Id: I67f695b57405465c7bec22c437c2169403fa59dd
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5089
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Tested-by: Mohammed Junaid <junaid@redhat.com>
Reviewed-by: Luis Pabon <lpabon@redhat.com>
To bring the DiskFile module coverage to 100%, we first recognize that
do_fsync() will invoke fsync() in a separate thread, which gives
coverage fits (see the commit history at
https://github.com/portante/coverage/commits/master). To avoid that,
we mock out do_fsync to make it a no-op and avoid the problem.
The second thing we recognize is that mkstemp() relies on do_unlink
from the fs_utils module, which already consumes ENOENT errors, so we
don't need that code path in mkstemp() itself. The unused mock routine
for do_unlink was removed as well, and we renamed the other do_unlink
mock routine to os_unlink since it was mocking out os.unlink directly.
Lastly, we rejigger the error on close test for mkstemp() to
prematurely close the fd to cause an error which should just be
squelched, completing the full coverage.
Change-Id: I98283c17cf139f92282f8afd7083d567d3dd9a79
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5082
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
The new DiskDir unit test skeleton is quite incomplete, but gets the
DiskDir module on the board for modules covered, explicitly exposing
the fact that we need to get test coverage. This is a first step.
At the same time, we also update all the modules we have applying the
fix for pep8 errors now run under tox. We can then add a Jenkins
pre-commit job to fail on pep8 errors. This brings our code to parity
with what they are doing in OpenStack Swift.
Change-Id: Ia0565606512efda6e73f67bd00269177b89db858
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5080
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
This is the first step. I have confirmed that I can install,
update, and remove. I have also setup the ability for Jenkins
to pass the BuildNumber and use that value as the release
number for the RPM.
The RPM depends on Grizzly(1.8.0) Swift from OpenStack. To verify
you may need to add the appropiate repo file to your Fedora/RHEL system:
http://repos.fedorapeople.org/repos/openstack/openstack-grizzly
I have not had the opportunity to test that G4S itself works
once installed, but I plan on doing that as the next phase.
Change-Id: Ib90f335f5e1e4fc552c32e00ff29b6e8a680c42a
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5006
Reviewed-by: Peter Portante <pportant@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Additionally, we drop the "ufo" references from setup.py, spec file
and README, and add the HISTORY file describing how the repo was
initially created. We also update the RPM spec file to use the name
"gluster-for-swift" to avoid colliding with existing RPM names from
RHS 2.0 (the spec file's description was also updated, along with
the version number).
Change-Id: If804224a94208d57896e4189c63736ffc9e01d5e
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4966
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
