66 Commits

Author SHA1 Message Date
Lisa Zangrando
f8ca81d631 Added queue usage to project
This commit adds some info about the queue usage to
each project

Change-Id: If14ca606148b0dbd2b22bfa470196b2db7c47d07
Sem-Ver: bugfix
2017-08-03 14:52:51 +02:00
Lisa Zangrando
5f28805ca7 Added support for OpenStack Ocata to NovaManager
This commit updates NovaManager for supporting the Ocata version

BUG: #1704801
Change-Id: Icad9c14b14f34d16861caf82845b0e847085556c
Sem-Ver: feature
2017-07-31 15:01:39 +02:00
Lisa Zangrando
94456be8fc Partition Director requires configuring Synergy through RESTful API
Partition Director, the INDIGO-DataCloud service, requires the setting
of the Synergy configuration (e.g. the list of projects allowed the
access the shared quota, their values, etc) through the RESTful API.
This implies some changes at configuration level of Synergy: some
parameters must be drop out from the synergy_scheduler.conf file and
the stored into a database. The use of the database for storing
configuration parameters will be useful even for configuring Synergy
in high availability mode.

Bug: #1690123
Change-Id: Id8f9c6b0e0a2804b43984f7353dc3fc0882cd651
Sem-Ver: feature
2017-07-19 10:54:10 +00:00
Lisa Zangrando
23e9382c7d Added support for notifications to KeystoneManager
This update enables KeystoneManager to receive asynchronous
notifications about the users and projects status from Keystone
service

Change-Id: Ia75b89c825bb8536e0462eff7a873699b219d781
Sem-Ver: bugfix
2017-07-18 10:13:31 +00:00
Lisa Zangrando
fc16dbd695 MessagingAPI should be implemented as common module
nova_manager.py implements the NovaManager which interacts with Nova
service but even provides the MessagingAPI class which interacts with
the AMQP system. This fix moves the MessagingAPI in common/messaging.py.

Bug: 1690133
Change-Id: Ifed11126a1af227950c03ccffc48d577fb631ded
Sem-Ver: bugfix
2017-05-11 14:42:06 +02:00
Lisa Zangrando
0e5e7b46de The SchedulerManager doesn't receive all the compute.instance.delete.end notifications
Sometimes the SchedulerManager doesn't receive the "compute.instance.delete.end"
notifications of VMs destroyed before to be active.

Bug: #1690126
Change-Id: If874d436b825e2f430e246ccc7663df1bebd461c
Sem-Ver: bugfix
2017-05-11 12:59:08 +02:00
Lisa Zangrando
56061a4aaf Add new "synergy_topic" parameter
The new "synergy_topic" parameter simplifies the Synergy integration with Nova.

Bug: #1672409
Change-Id: I00f7e6ce7b6c4b22d9267d2b956c599fb71c68a2
Sem-Ver: feature
2017-03-13 15:27:21 +01:00
Lisa Zangrando
b16ab75ca6 Fix a possible shared quota consistency issue
It may happen that a VM instance goes in error during the building phase.
Although the server is deleted by Synergy, the compute node may not send
the notification about the cancellation by leaving the shared quota not
consistent. This fix addresses this issue.

Bug: #1668199
Change-Id: I9ff9eed7b234117e24ae629fdd3e00ae7ce528cd
Sem-Ver: bugfix
2017-02-27 10:17:11 +01:00
Lisa Zangrando
92892640d7 Fix the bug concerning the user which is still showed after deletion
Bug: #1663639
Change-Id: Ifc0278076913475282c15d2092caa57354145487
Sem-Ver: bugfix
2017-02-23 15:31:12 +01:00
Lisa Zangrando
e45ce144b4 Fix the share percentage
This fix solves the problem of the calculation of the user
share which keeps memory of deleted users.

bug: #1663644
Change-Id: Ie591b60a413ab16c90fa6adc5a1f5be3046e0122
Sem-Ver: bugfix
2017-02-20 17:34:24 +01:00
Jenkins
a00088c413 Merge "Fix the error: Arguments already parsed" 2017-02-20 11:18:49 +00:00
Anh Tran
61a80c47a7 Remove unused logging import
Change-Id: I801b5a4991e2678ea45c99b836638b05e842e003
2017-02-20 11:27:05 +07:00
Lisa Zangrando
b41121390f Fix the error: Arguments already parsed
On Newton, the startup of Synergy fails and the reported message in synergy.log is:
"ERROR - manager 'NovaManager' instantiation error: arguments already parsed: cannot
register CLI option". We fix this bug in this commit.

- removed unuseful conductor interfaces

bug: #1664567
Change-Id: I501196c40efd6fbd0e3e3669005441e7a8c23364
Sem-Ver: bugfix
2017-02-16 15:32:43 +01:00
Lisa Zangrando
667d6cd6e3 Fix the TypeError: not all arguments converted during string formatting
This commit fixes the bug concerning the use of a wrong logging format
into the SchedulerManager.

Bug: #1664608
Change-Id: Id39fab710571e21be1e4580621a2f698febf2a7a
Sem-Ver: bugfix
2017-02-14 16:27:40 +01:00
Jenkins
f21b883247 Merge "Removed logging message" 2017-01-30 09:30:30 +00:00
Lisa Zangrando
f9619e8797 Fix a bug related to NUMA topology
Previously, one could encounter the following error:
"NumaTopologyNotFound: Instance does not specify a NUMA topology"

Bug: #1659794
Change-Id: I065e4ed0cfe56c3026b9eefc473fe6baf0889baf
Sem-Ver: bugfix
2017-01-30 09:15:13 +00:00
Lisa Zangrando
5887b41fdd Removed logging message
The logging message is not needed anymore.

Change-Id: Id83a4752415785bc1108f0787cf2e2ee940a535f
2017-01-30 10:08:27 +01:00
Jenkins
b3504c00db Merge "Retry mechanism improved and fixed" 2017-01-27 16:43:18 +00:00
Lisa Zangrando
e107158e9b Time data does not match format '%Y-%m-%dT%H:%M:%S.%fZ'
Bug: #1659807
Change-Id: Iec6bf294562d9df8d2db7dbd212b7d35e91ad43a
Sem-Ver: bugfix
2017-01-27 13:47:33 +01:00
Lisa Zangrando
89537e9f99 Retry mechanism improved and fixed
Bug: #1659780
Change-Id: Ifabc4f0ce576fc5c0348fe591d0ca92abe2f7b64
Sem-Ver: bugfix
2017-01-27 11:28:27 +01:00
Jenkins
80ef20c334 Merge "Add support for Keystone domains" 2017-01-25 16:29:51 +00:00
Jenkins
2180a51e92 Merge "Heat and Synergy interfere on the management of the users trust" 2017-01-25 15:35:43 +00:00
Lisa Zangrando
e7e36a1fbd Add support for Keystone domains
This feature fixes even the bug #1659246

Change-Id: I189455f5b1c546304e4b9b3ce9f6c17ebdf0d133
Sem-Ver: feature
2017-01-25 15:24:34 +01:00
Lisa Zangrando
75696d2e6b Heat and Synergy interfere on the management of the users trust
Heat and Synergy use the Trust mechanism for making service delegations.
Synergy must be able to distinguish the trusts created by Heat from the
ones created by itself.

Change-Id: I5fdc04fdf94397f8f3b5f336bbaf22781e3ed467
Sem-Ver: bugfix
2017-01-25 13:48:03 +00:00
Lisa Zangrando
ea89a0f8d2 Synergy and Ceilometer compete for consuming AMQP notifications
Synergy and Ceilometer compete for consuming the notifications about
the VM state changes because both listen on the "notifications" AMQP topic.

- added a new configuration parameter for SchedulerMaanger: "notification_topic"

Change-Id: I444aeac5539bc4b6994f8722637e7aba7d82dc78
Sem-Ver: bugfix
2017-01-25 13:40:32 +00:00
Lisa Zangrando
a858311eb9 Add support for the automatic recycle of DB connections
This solves the error 'Lost connection to MySQL server during query'
due to MySQL, which removes connections after a timeout.

- added a new configuration parameter for QueueManager: 'db_pool_recycle'

Change-Id: I2974f414afa6cb3d44e85dd3e05600cfd49cb905
Sem-Ver: bugfix
2017-01-25 10:11:46 +01:00
Jenkins
358d5d453d Merge "Add support for AMQP HA to NovaManager" 2017-01-20 14:44:28 +00:00
Jenkins
5bc82944c4 Merge "Enable SSL for OpenStack Trust" 2017-01-20 14:44:23 +00:00
Jenkins
ebe95a1761 Merge "QuotaManager: private quota shrinking must not be always allowed" 2017-01-20 14:42:38 +00:00
Lisa Zangrando
87c83deaaa Enable SSL for OpenStack Trust
Added support to SSL connections

Change-Id: I03cfb4fef50b6fee805369883f0d4436f9d6567f
Sem-Ver: feature
2017-01-20 14:37:13 +00:00
Lisa Zangrando
7926137bb5 Add support for AMQP HA to NovaManager
- AMQP HA cluster support added
- added the new 'amqp_hosts' configuration attribute

Change-Id: I20973b3a66540814b86970628270dc83adab8018
Sem-Ver: feature
2017-01-20 14:34:44 +00:00
Lisa Zangrando
12d2842596 use API v3 for KeystoneManager.getUsers()
Before, KeystoneManager.getUsers() used the OLD v2.0 API version.


Change-Id: I49c10c8a3452b84c02de441f28d23c60ae505db4
Sem-Ver: feature
2017-01-20 13:26:09 +00:00
Lisa Zangrando
449d35439d QuotaManager: private quota shrinking must not be always allowed
QuotaManager not allows the private quota shrinking if
the current resource usage exceeds the new size.

Change-Id: Ie86314d137991a755ec178b82d3716b3fc35bc0e
Sem-Ver: bugfix
2017-01-20 12:50:28 +01:00
Lisa Zangrando
447f11d315 NovaManager and KeystoneManager are not SSL-enabled
NovaManager and KeystoneManager MUST support the HTTPS connections
for interacting with the OpenStack services.

- added two new manager config options: ssl_ca_file and ssl_cert_file

Change-Id: Ibacbf3504badd65a792c72c9134e2c7c13adea69
Sem-Ver: feature
2017-01-16 12:59:30 +01:00
Lisa Zangrando
90e6d14aab Make SchedulerManager handle ERROR notifications.
SchedulerManager receives all VMs status changes through AMQP but it doesn't
handle the ERROR notifications (not used before to Liberty version)

- updated test unit (test_scheduler_manager.TestNotifications.test_info_quota)

Change-Id: Ia1e4db66a743f2d42f01241505310cbed37b625e
Sem-Ver: bugfix
Closes-bug: #1648057
2016-12-12 16:30:56 +00:00
Jenkins
8ff83d7f9d Merge "Queue.updatePriority() takes much time if the queue is large" 2016-12-12 10:00:31 +00:00
Lisa Zangrando
8451e011a7 Synergy releases in advance the VMs that are going to be destroyed
Synergy releases in advance the VMs that are going to be destroyed,
before the confirmation notification is arrived.

Change-Id: Ia0ba1c8c715ff831f5d3579e23fc324488570937
Sem-Ver: bugfix
Closes-bug: #1646529
2016-12-08 08:59:24 +00:00
Lisa Zangrando
3cc99f5c14 Queue.updatePriority() takes much time if the queue is large
The method updates the priority of all user requests in the queue, one by one.
The priority of each request is updated in memory and into the DB table.
If the queue is too large (>5000), the updatePriority() takes much time for
its execution.

Change-Id: I982f98be94a8d23a159320887f51e7393fecb4a9
Sem-Ver: bugfix
Closes-bug: #1646466
2016-12-08 08:58:58 +00:00
Lisa Zangrando
4db25a4f5b Add the new clock_skew parameter for KeystoneManager
A clock skew (e.g. clock_skew (time in seconds)) parameter is needed to set the
maximum difference allowed between the Synergy and Keystone clocks.

Change-Id: Ia347524472e86101a47240e7b1b1cb0ee05939c0
Sem-Ver: feature
Closes-bug: #1646783
2016-12-07 11:49:39 +01:00
Lisa Zangrando
91919336b8 Invalid input for field/attribute quota_class_set
The optional input attribute 'force' for os-quota-class-sets is no more
supported in Mitaka and so NovaManager throws the following exception
reported in the log file:

ERROR - error on updating the quota info (id=<built-in function id>):
{u'badRequest': {u'message': u"Invalid input for field/attribute
quota_class_set. Value: {u'cores': 2, u'ram': 1024, u'force': True,
u'instances': 10}. Additional properties are not allowed
(u'force' was unexpected)", u'code': 400}}

- fixed method: NovaManager.updateQuota()
- moved some not useful logging info from LOG.info() to LOG.debug()

Change-Id: I72a401beee24b24bd2f8fc6dc8a3784813af0864
Sem-Ver: bugfix
Closes-bug: #1645397
2016-11-28 17:39:28 +00:00
Lisa Zangrando
0309dd1cc6 KeystoneManager.authenticate() uses a wrong domain attribute
The request body for /v3/auth/tokens (in KeystoneManager.authenticate())
includes a payload that specifies a wrong domain attribute (i.e.
"id"="default"). The proper attribute to use is "name"="default".
This bug affects the OpenStack Mitaka version while in Liberty seems to
be accepted or ignored.
- added two new KeystoneManager configuration attributes:
    user_domain_name and project_domain_name

Change-Id: I0b9e56260d2e425399fe5a0c18a10af250a9f022
Sem-Ver: bugfix
Closes-bug: #1645318
2016-11-28 13:22:40 +00:00
Lisa Zangrando
1a36a1db90 Add a backfill_depth parameter
Allow Synergy to not check the whole queue when looking for jobs to
start.

Change-Id: Idf8b92d3d91c0beadabea09f31e042c39320e4f0
Sem-Ver: feature
Closes-bug: #1644188
DocImpact
2016-11-24 16:32:13 +00:00
Jenkins
3b74b591fe Merge "Item readjustment fixed in PriorityQueue" 2016-11-24 16:28:20 +00:00
Lisa Zangrando
314e9f4ab8 Item readjustment fixed in PriorityQueue
Allow the PriorityQueue to re-insert an item by guaranteeing the proper priority ordering
- fixed methods: put(), items(), smallest(), largest()
- fixed unit test (test_queue.py)

Change-Id: I12b461a3cbc84d51877a455fca29589491ceb7fd
Sem-Ver: bugfix
Closes-bug: #1644257
2016-11-24 17:06:02 +01:00
Lisa Zangrando
9e16739c8e The FairshareManager should not use the Manager.condition var
The Manager.condition (i.e. threading.Condition()) variable is used by Synergy for scheduling the periodic manager's tasks at fixed rate. FairshareManager inherits the same variable from Manager and wrongly uses it for different purposes. The problem is that every time the FairshareManager calls the condition.notifyAll() method, triggers unwittingly the execution of the task() method.

Change-Id: I910a50c566e076439de78d0f4e4df0d9a893aadb
Sem-Ver: bugfix
Closes-bug: #1644496
2016-11-24 11:18:16 +01:00
Lisa Zangrando
98221a2a6e Fix scheduling when shared quota is disabled
- quota.py: now the method allocate() handles the case of quota disabled
  and blocking=True
- quota_manager.py: fixed the method updateDynamicQuota() which set the
  shared quota disabled or enabled
- scheduler_manager.py: in case the shared quota is disabled, the
  scheduling process is stopped

Change-Id: I80dd04505f8ce4cb65b04d2ee46819fe078a363f
Sem-Ver: bugfix
Closes-bug: #1643833
2016-11-23 13:48:25 +00:00
Lisa Zangrando
d6e2010abd CLI command "synergy usage show" enhanced
- now it is possible to show the usage by project and user
- removed unuseful check in QueueCommand
- removed unused var in UsageCommand
- removed unused import: 'from synergy_scheduler_manager.common.queue import Queue'

Sem-Ver: feature

Please enter the commit message for your changes. Lines starting

Change-Id: I2ed6304328dcb628fe4662eb6977ce4b988b8a15
2016-11-09 15:12:43 +01:00
Vincent Llorens
3f700fe681 Clean up oslo imports
We used to put `try / except ImportError` around oslo lib imports
to support both the `oslo_lib` and `oslo.lib` name patterns.

It is now useless since the newer versions of these libs all support
the `import oslo_lib` pattern.

Tested on CentOS 7 with the OpenStack Liberty repository.
Tested on Ubuntu 14.04 with OpenStack Liberty CloudArchive repository.

Change-Id: I91a14baea4df0d66db4818bdcaf895b0e2a62245
Sem-Ver: bugfix
2016-11-08 11:10:56 +01:00
Jenkins
b5768baef3 Merge "Synergy doesn't rely anymore on nova.conf" 2016-11-07 17:28:15 +00:00
Jenkins
f00097756f Merge "Method deserialize() fixed" 2016-11-07 14:07:57 +00:00