It seems that it is possible for pods to go into 'MatchNodeSelector'
status after a hard reboot of the node due to reasons mentioned in
[0]. The current preflight check opertor will fail the health checks
even when the entire cluster goes back to normal after the hard reboot
as it will flag any pod(s) that are not in 'Succeeded' or 'Running'
state. This means that our workflow will stop and go into failed state.
This patch set is meant to take care of such scenario and to relax the
health check requirements for the k8s cluster by logging the information
of such pods instead of failing the workflow (note that the status of such
pods will resemble [1]).
[0] https://github.com/kubernetes/kubernetes/issues/52902
[1]
'status': {'conditions': None,
'container_statuses': None,
'host_ip': None,
'init_container_statuses': None,
'message': 'Pod Predicate MatchNodeSelector failed',
'phase': 'Failed',
'pod_ip': None,
'qos_class': None,
'reason': 'MatchNodeSelector',
'start_time': datetime.datetime(2018, 3, 30, 15, 49, 39, tzinfo=tzlocal())}}
Change-Id: Idb1208d93cddc01cd0375a5ac2e6e73dd3dfad61
017faba69f