This change implements a collector which does an HTTP GET via
python requests to fetch the metadata.
It should work with any GET-able URL, however it is designed to
work with Swift TempURLs.
Swift objects are not consistent, so the Last-Modified header is
checked for each poll and metadata is not fetched if the last
modified is not newer than the previous successful poll.
This collector will be enabled for OS::Nova::Server
software_config_transport: POLL_TEMP_URL which is available
in the Juno release of Heat. Using POLL_TEMP_URL will result
in no metadata polling load on heat, which has historically been
an issue with tripleo scalability.
Change-Id: I22155c22bdcc3c81a5e945ca5436a8f29f196528
When we detect a failed command we log ERROR but we do not return an
error status. This makes it difficult for programs which may run
os-collect-config to detect whether a run was sucessful.
This only applies to runs which are performed with --one-time argument
as this is a straightforward case.
Change-Id: I168862e8c75c15d1ea405a417908d1284feb7b32
A popen call is made in the Python 3 platform module which causes our
assertions for our mock'd popen to fail. Also fixing a locale issue to
use defaultlocale() rather than LC_ALL.
Co-Authored-By: Steve Kowalik <steven@wedontsleep.org>
Change-Id: I45cdf7d921fbca5f09e5f65bd69fcee83fb7c8e4
This patch uses keystone client discover to find correct V3
auth_url which is expected by heat collector.
Change-Id: I2b8b5a768bdeb1cd1d20fab3b4234306c1429d6d
Closes-Bug: #1341936
Make changes pretty much all over the code base with respect to
encoding strings and fixing imports to support Python 3.
Change-Id: Id1920129001b8e223474c1b2faf8bd9d527fe7e7
Update the copy of oslo's log to oslo-incubator
0533894d6bfacd82a371bccd628b61d5016032cd.
This is the first step towards supporting Python 3 in
os-collect-config.
Change-Id: If282091d7f0cbd4e2ce90d70c066bb11cbf56d38
The local collector is not in DEFAULT_COLLECTORS, but should be usable
explicitly. It, however, suffers from a bug where only
DEFAULT_COLLECTORS are allowed through.
Change-Id: Ia42d1acd39638b448e2e2bfa26aff1c7ae415b71
This collector will collect data from the local system, allowing image
builds or simple processes to influence the metadata.
implements bp tripleo-juno-occ-localdatasource
Change-Id: I0e58e8c631ffe8b63e8b4117df2c9ce2f413044f
The configuration will dictate whether or not something is configured.
If it is not, this is a normal state and should not be logged as a
warning.
Change-Id: I479f0aed5837871009bc69fa028f5eb64a060c53
Closes-Bug: #1321551
This reverts commit 6b478e9d90bce18b03f5411822897740f68c3dd0.
We will break anybody who is expecting CFN to be tried in all
circumstances with this. We probably just need to base which collectors
to try on what configuration we have, and not log warnings on
unconfigured collectors.
Change-Id: I4bf7d6f9af9487bf9d2c0942381c0ba68fc03ee9
Previously we were relying on the CFN compatibility API. This makes the
native Heat version the default.
Note that we want to keep full coverage, which is why we are explicitly
adding cfn back in during tests.
Change-Id: I5adedd052827e176e2f39071c719600df62019d7
Closes-Bug: #1321551
Auth_ref is documented as a serializable object which we can save off
and use to reinitialize keystoneclient. That should in turn be passable
to heatclient directly. With this code, we will avoid authorizing with
Heat if we have a token in the cache.
Change-Id: I3dc6c3726054e77449cc9be46229e6d3521d4ef2
Closes-Bug: #1321437
In later commits we will use this cache to memoize access to the
authentication details.
Change-Id: I389f78fe1eb176e37c90a1a87a4ba5fde3b33f05
Related-Bug: #1321437
This will help segregate the changes as we abstract away details of
keystone such as caching the tokens.
Change-Id: I81ccb17658e5c9fa349fb183240f901d61b1fa93
Related-Bug: #1321437
It turns out sometimes json will serialize differently given the same
python structure. A check of the two parsed data structures will detect
changes more reliably.
Change-Id: Id165b36c0fa0fa89730c3507444a41c68bd70fb3
Closes-Bug: #1320262
Only config which contains os-apply-config data should be written
to the top level config structure. This change will only top-level
configs which are in the group os-apply-config or the default group
Heat::Ungrouped.
This tripleo-heat-templates change will specify a group for all
config resources https://review.openstack.org/#/c/91731/ however
the expected top-levels will still be written out without this
change.
This change is to fix heat software config when the config is not
json.
Change-Id: I16e42825121cf4968d4b0f0d266f4f6daa962921
Closes-Bug: #1299109
This collector uses keystoneclient and heatclient to poll for the
configured resource metadata.
Changes were required to test_collect to allow collectors which needed
to fake something other than requests.
Change-Id: I3e93fe38b15f71193a4c024b24e6260d6adcf1b3
Before this, the exploded deployments that the cfn collector produced
would not ever be committed, and thus would always appear to have been
changed. This resulted in os-collect-config running the command
endlessly.
This requires some refactoring so that we commit changes to the cache
based on what was actually written, rather than just the static list of
collectors.
Closes-Bug: #1307153
Change-Id: I618ef5d752ed6519e8b7bfc090de03f2f24e73ce
This improves the test coverage for this very common case and sets up
for a new test which will expose that we fail in this case for exploded
deployments collected by the cfn collector.
Change-Id: I74807107824f8795619207389c064e83580c1c87
Related-Bug: #1307153
When cfn explodes deployments we end up with many things stored per
collector, so some assumptions around 1:1 collector to cached content
cannot be made anymore.
Change-Id: I43c5fb759bc99690e0def1c6ecb672ab7f087df8
In recent changes support was added to explode the deployment key into
multiple json files for merging by os-apply-config. But this breaks
other users who want direct access to the full deployments structure.
There seems to be no real reason to delete the key other than tidiness,
so we will stop deleting it from the main cfn structure.
Change-Id: Icf9479376e32c3ad8f9be29359e6100c1aeda2a5
Closes-Bug: #1299110
Will result in multiple cache files to be merged per
OS::Heat::StructuredDeployment. This is needed as the new features for
software configuration break things up a bit differently in Metadata.
Change-Id: Iec0fd947bac674f6b6f36e8c0789d10580c325fd
Closes-Bug: #1295787
With the new OS::Heat::StructuredDeployment resource, each Metadata
section may have multiple "deployments" in it. With this, we will return
a list with tuples of key and content to write to the cache.
Change-Id: I9f4272b0761e1dfd850bc5a5c6b27a78f126281f
Related-Bug: #1295787
While this is called a "cache", it is important for it to survive. On
reboot, servers may need what was in the cfn config to restore complex
network configurations.
We introduce a new command line option, --backup-cachedir, that will
default to the old path, /var/run/os-collect-config. This will keep
things working for any tools that have been hard coded to use the old
path.
Change-Id: I78b3851b35addfc16913e3cd53c9d0e7eb3d191a
We pass the list of json files containing the collected metadata to
os-refresh-config using the OS_CONFIG_FILES env variable, so it's a
pretty useful piece of information to log.
Change-Id: Id09e3f352e6a5a09e4183c0743a6e99a2783a888
With new hacking>=0.8.0,<0.9 requirements, some functions are
now deprecated.
assertEquals -> assertEqual
assertNotEquals -> assertNotEqual
Change-Id: I70287b0c8561a901069bddcf3007751c34a4e0b5
The initial value of 300 seconds was a conservative estimate. However,
the requests and responses are somewhat small, so we can drop the polling
interval significantly and still maintain a high degree of network
scalability. After measuring the responses from the ec2 and cfn servers
with typical workloads, at 30 second intervals 100 servers will generate
around 26kB/s of requests, with about 66kB/s of responses.
Change-Id: Iaa99ae405ba7c72ef8afc11c946400a2d0db5206
Without this change, if a user runs os-collect-config --force, it will
lock the user in an infinite loop running the command over and over with
very little chance to cancel. There are no compelling use cases for that
behavior, but it is extremely inconvenient, so implying --one-time
improves usability of os-collect-config for users.
Change-Id: Ia8c9bf0bf97ab9e40e465c947c2f0cbeb981c08e
The use case for --print is an administrator wanting to view the
metadata that os-collect-config sees without running any commands.
Fixes bug #1213195
Change-Id: I0251f2c70574aeaa79997ce822d2a5ffbe08e345
This is a useful debugging and/or system fixer tool for instances where
metadata has not changed but one needs to re-run the configuration.
Fixes bug #1223693
Change-Id: I62b097bafa339fefcf6e03d11636f5ab622fb71a
This will allow tools like os-apply-config to read the list even when
they are run out of band from os-collect-config.
Change-Id: Ic4eaf649e234f4a1367d20c7ec52e93e787a7bb3
The option allows other programs to find the cache directory and files
without having access to OS_CONFIG_FILES.
Change-Id: Iad87efb65ea4db387e94160376c9eaf956fff413
Keep a hash of the config file for os-collect-config and if it changes
during a failed run then rerun immediately(without sleep), effectively
causing new nodes to be ready 5 minutes earlier.
Because the cfn credentials are placed into os-collect-config.conf by
os-apply-config and are not in place the first time os-collect-config is
run, the first run of os-collect-config results in error, o-c-c then
sleeps for 5 minutes before running successfully the second time.
Fixes bug #1219186
Change-Id: I090de7a3d84e0ea342f1a422646c0c455eb37f4a
On a system with o-c-c installed by pip the binary generated by PBR
calls __main__() directly, the code that sets up logging sould be placed
here otherwise it will be bypassed. Resulting in missing log messages.
Change-Id: I94ba4f61be9595a6ddee134d806e5f99ae4adf73
__main__ is called directly during tests, but was resetting the
logging environment within it, which prevented tests from capturing
the log events.
Change-Id: If710e11091723144c97c88aab4aa5e6126844d2b
The point of delaying the commit of data to the cache is that we want to
make sure the command succeeds before giving up on the data changes.
This will ensure that we keep trying the command with any given change
to the metadata until it succeeds.
Change-Id: Idf3a09686b4bbf0e16a9bc9f3359ee9937fcc627
Smart hooks may want to perform delta analysis on metadata, and tests
want to be able to tell if the cache is updated it's previous state,
so we should make the presence and name of the old versions of the
files a stable interface.
Change-Id: Ia29c1ef9e10308d6461dae7823c5f497f6b90122
FakePopen will make the migration to check_call easier, and narrowly
replaces the engire rather than replacing the subprocess entry point.
Change-Id: Ic868ce4cdbef79f256ef01a9a030767522b855a3
The default order results in less-dynamic heat_local and ec2 overriding
the more dynamic cfn source. That is the opposite of what is desired.
Change-Id: I7e1feb2e6869b4f076200668dd204219ecc4224e
