Merge branch 'master' into v1
This commit is contained in:
f3flight
commit
e65b00060b
@ -9,7 +9,7 @@ An example of a configuration file is ``config.yaml``.
|
||||
Some of the parameters available in configuration file:
|
||||
|
||||
* **ssh_opts** parameters to send to ssh command directly (recommended to leave at default), such as connection timeout, etc. See ``timmy/conf.py`` to review defaults.
|
||||
* **env_vars** environment variables to pass to the commands and scripts - you can use this to expand variables in commands or scripts
|
||||
* **env_vars** environment variables to pass to the commands and scripts - you can use these to expand variables in commands or scripts
|
||||
* **fuel_ip** the IP address of the master node in the environment
|
||||
* **fuel_user** username to use for accessing Nailgun API
|
||||
* **fuel_pass** password to access Nailgun API
|
||||
@ -22,24 +22,24 @@ Some of the parameters available in configuration file:
|
||||
Configuring actions
|
||||
===================
|
||||
|
||||
Actions can be configured in a separate yaml file (by default ``rq.yaml`` is used) and / or defind in the main config file or passed via command line options ``-P``, ``-C``, ``-S``, ``-G``.
|
||||
Actions can be configured in a separate yaml file (by default ``rq.yaml`` is used) and / or defined in the main config file or passed via command line options ``-P``, ``-C``, ``-S``, ``-G``.
|
||||
|
||||
The following actions are available for definition:
|
||||
|
||||
* **put** - a list of tuples / 2-element lists: [source, destination]. Passed to ``scp`` like so ``scp source <node-ip>:destination``. Wildcards supported for source.
|
||||
* **cmds** - a list of dicts: {'command-name':'command-string'}. Example: {'command-1': 'uptime'}. Command string is a bash string. Commands are executed in a sorted order of their names.
|
||||
* **scripts** - a list of elements, each of which can be a string or a dict:
|
||||
* string - represents a script filename located on a local system. If filename does not contain a path separator, the script is expected ot be located inside ``rqdir/scripts``. Otherwise the provided path is used to access the script. Example: ``'./my-test-script.sh'``
|
||||
* dict - use this option if you need to pass variables to your script. Script parameters are not supported, but instead you can use env variables. A dict should only contain one key which is the script filename (read above), and the value is a Bash space-separated variable assignment string. Example: ``'./my-test-script.sh': 'var1=123 var2="HELLO WORLD"'``
|
||||
* **LIMITATION**: if you use a script with the same name more than once for a given node, the collected output will only contain the last execution's result.
|
||||
* **INFO**: Scripts are not copied to the destination system - script code is passed as stdin to `bash -s` executed via ssh or locally. Therefore passing parameters to scripts is not supported (unlike cmds where you can write any Bash string). You can use variables in your scripts instead. Scripts are executed in the following order: all scripts without variables, sorted by their full filename, then all scripts with variables, also sorted by full filename. Therefore if the order matters, it's better to put all scripts into the same folder and name them according to the order in which you want them executed on the same node, and mind that scripts with variables are executed after all scripts without variables. If you need to mix scripts with variables and without and maintain order, just use dict structure for all scripts, and set `null` as the value for those which do not need variables.
|
||||
* string - represents a script filename located on a local system. If filename does not contain a path separator, the script is expected to be located inside ``rqdir/scripts``. Otherwise the provided path is used to access the script. Example: ``'./my-test-script.sh'``
|
||||
* dict - use this option if you need to pass variables to your script. Script parameters are not supported, but you can use env variables instead. A dict should only contain one key which is the script filename (read above), and the value is a Bash space-separated variable assignment string. Example: ``'./my-test-script.sh': 'var1=123 var2="HELLO WORLD"'``
|
||||
* **LIMITATION**: if you use a script with the same name more than once for a given node, the collected output will only contain the result of the last execution.
|
||||
* **INFO**: Scripts are not copied to the destination system - script code is passed as stdin to `bash -s` executed via ssh or locally. Therefore passing parameters to scripts is not supported (unlike cmds where you can write any Bash string). You can use variables in your scripts instead. Scripts are executed in the following order: all scripts without variables, sorted by their full filename, then all scripts with variables, also sorted by full filename. Therefore if the order matters, it's better to put all scripts into the same folder and name them according to the order in which you want them executed on the same node. Mind that scripts with variables are executed after all scripts without variables. If you need to mix scripts with variables and without and maintain order, just use dict structure for all scripts, and set `null` as the value for those which do not need variables.
|
||||
* **files** - a list of filenames to collect. passed to ``scp``. Supports wildcards.
|
||||
* **filelists** - a list of filelist filenames located on a local system. Filelist is a text file containing files and directories to collect, passed to rsync. Does not support wildcards. If filename does not contain path separator, the filelist is expected to be located inside ``rqdir/filelists``. Otherwise the provided path is used to read the filelist.
|
||||
* **filelists** - a list of filelist filenames located on a local system. Filelist is a text file containing files and directories to collect, passed to rsync. Does not support wildcards. If the filename does not contain path separator, the filelist is expected to be located inside ``rqdir/filelists``. Otherwise the provided path is used to read the filelist.
|
||||
* **log_files**
|
||||
* **path** - base path to scan for logs
|
||||
* **include** - regexp string to match log files against for inclusion (if not set = include all)
|
||||
* **exclude** - regexp string to match log files against. Excludes matched files from collection.
|
||||
* **start** - date or datetime string to collect only files modified on or after the specified time. Format - ``YYYY-MM-DD`` or ``YYYY-MM-DD HH:MM:SS``
|
||||
* **start** - date or datetime string to collect only files modified on or after the specified time. Format - ``YYYY-MM-DD`` or ``YYYY-MM-DD HH:MM:SS`` or ``N`` where N = integer number of days (meaning last N days).
|
||||
|
||||
===============
|
||||
Filtering nodes
|
||||
@ -55,18 +55,18 @@ Nodes can be filtered by the following parameters defined inside **soft_filter**
|
||||
* **ids** - the list of ids, ex. **[0,5,6]**
|
||||
* any other attribute of Node object which is a simple type (int, float, str, etc.) or a list containing simple types
|
||||
|
||||
Lists match **any**, meaning that if any element of the filter list matches node value (if value is a list - any element in it), node passes.
|
||||
Lists match **any**, meaning that if any element of the filter list matches node value (if value is a list - any element in it), the node passes.
|
||||
|
||||
Negative filters are possible by pretending filter parameter with **no_**, example: **no_id = [0]** will filter out Fuel.
|
||||
Negative filters are possible by prefacing filter parameter with **no_**, example: **no_id = [0]** will filter out Fuel.
|
||||
|
||||
Negative lists also match **any** - if any match / collision found, node is skipped.
|
||||
Negative lists also match **any** - if any match / collision found, the node is skipped.
|
||||
|
||||
You can combine any number of positive and negative filters as long as their names differ (since this is a dict).
|
||||
|
||||
You can use both positive and negative parameters to match the same node parameter (though it does not make much sense):
|
||||
**roles = ['controller', 'compute']**
|
||||
**no_roles = ['compute']**
|
||||
This will skip computes and run only on controllers. Like stated above, does not make much sense :)
|
||||
This will skip computes and run only on controllers. As already said, does not make much sense :)
|
||||
|
||||
=============================
|
||||
Parameter-based configuration
|
||||
@ -82,7 +82,7 @@ It is possible to define special **by_<parameter-name>** dicts in config to (re)
|
||||
|
||||
In this example for any controller node, cmds setting will be reset to the value above. For nodes without controller role, default (none) values will be used.
|
||||
|
||||
It is also possible to define a special **once_by_<parameter-name>** which works similarly, but will only result in attributes being assigned to single (first in the list) matching node. Example:
|
||||
It is also possible to define a special **once_by_<parameter-name>** which works similarly, but will only result in attributes being assigned to a single (first in the list) matching node. Example:
|
||||
|
||||
::
|
||||
|
||||
@ -116,7 +116,7 @@ rqfile format
|
||||
by_roles:
|
||||
compute: [d, e, f]
|
||||
|
||||
The **config** and **rqfile** definitions presented above are equivalent. It is possible to define config in a config file using the **config** format, or in an **rqfile** using **rqfile** format, linking to the **rqfile** in config with ``rqfile`` setting. It is also possible to define part here and part there. Mixing identical parameters in both places is not recommended - results may be unpredictable (such scenario was not thoroughly tested). In general **rqfile** is good for fewer settings with more parameter-based variations (``by_``), and main config for more different settings with less such variations.
|
||||
The **config** and **rqfile** definitions presented above are equivalent. It is possible to define config in a config file using the **config** format, or in an **rqfile** using **rqfile** format, linking to the **rqfile** in config with ``rqfile`` setting. It is also possible to define part here and part there. Mixing identical parameters in both places is not recommended - the results may be unpredictable (such a scenario has not been thoroughly tested). In general, **rqfile** is good for fewer settings with more parameter-based variations (``by_``), and main config for more different settings with less such variations.
|
||||
|
||||
===============================
|
||||
Configuration application order
|
||||
@ -131,6 +131,6 @@ Configuration is assembled and applied in a specific order:
|
||||
1. first the top-level attributes are set
|
||||
2. then ``by_<attribute-name>`` parameters except ``by_id`` are iterated to override or append(accumulate) the attributes
|
||||
3. then ``by_id`` is iterated to override any matching attributes, redefining what was set before
|
||||
5. finally ``once_by_`<attribute-name>`` parameters are applied - only for one matching node for any set of matching values. This is useful for example if you want a specific file or command from only a single node matching a specific role, like running ``nova list`` only on one controller.
|
||||
5. finally ``once_by_`<attribute-name>`` parameters are applied - only for one matching node for any set of matching values. This is useful, for example, if you want a specific file or command from only a single node matching a specific role, like running ``nova list`` only on one controller.
|
||||
|
||||
Once you are done with the configuration, you might want to familiarize yourself with :doc:`Usage </usage>`.
|
||||
|
||||
@ -2,15 +2,17 @@
|
||||
Specification
|
||||
=============
|
||||
|
||||
Mirantis OpenStack Ansible-like tool for parallel node operations: two-way data transfer, log collection, remote command execution
|
||||
|
||||
|
||||
* The tool is based on https://etherpad.openstack.org/p/openstack-diagnostics
|
||||
* Should work fine on the following environments that were tested: 4.x, 5.x, 6.x, 7.0, 8.0
|
||||
* Should work fine in the following environments that have been tested: 4.x, 5.x, 6.x, 7.0, 8.0, 9.0
|
||||
* Operates non-destructively.
|
||||
* Can be launched on any host within admin network, provided the fuel node IP is specified and access is possible to Fuel and other nodes via ssh from local system.
|
||||
* Parallel launch, only on the nodes that are 'online'. Some filters for nodes are also available.
|
||||
* Can be launched on any host within admin network, provided the fuel node IP is specified and access to Fuel and other nodes is possible via ssh from the local system.
|
||||
* Parallel launch - only on the nodes that are 'online'. Some filters for nodes are also available.
|
||||
* Commands (from ./cmds directory) are separated according to roles (detected automatically) by the symlinks. Thus, the command list may depend on release, roles and OS. In addition, there can be some commands that run everywhere. There are also commands that are executed only on one node according to its role, using the first node of this type they encounter.
|
||||
* Modular: possible to create a special package that contains only certain required commands.
|
||||
* Collects log files from the nodes using filters
|
||||
* Some archives are created - general.tar.bz2 and logs-*
|
||||
* Checks are implemented to prevent filesystem filling due to log collection, appropriate error shown.
|
||||
* Can be imported in other python scripts (ex. https://github.com/f3flight/timmy-customtest) and used as a transport and structure to access node parameters known to Fuel, run commands on nodes, collect outputs, etc. with ease.
|
||||
* Checks are implemented to prevent filesystem overfilling due to log collection, appropriate error shown.
|
||||
* Can be imported into other python scripts (ex. https://github.com/f3flight/timmy-customtest) and used as a transport and structure to access node parameters known to Fuel, run commands on nodes, collect outputs, etc. with ease.
|
||||
|
||||
@ -20,17 +20,17 @@ Basic parameters:
|
||||
* ``-v``, ``--verbose`` verbose(INFO) logging
|
||||
* ``-d``, ``--debug`` debug(DEBUG) logging
|
||||
|
||||
**Shell Mode** - a mode of exectution which does the following changes:
|
||||
**Shell Mode** - a mode of execution which makes the following changes:
|
||||
|
||||
* rqfile (``rq.yaml`` by default) is skipped
|
||||
* Fuel node is skipped
|
||||
* outputs of commands (specified with ``-C`` options) and scripts (specified with ``-S``) are printed on screen
|
||||
* any actions (cmds, scripts, files, filelists, put, **except** logs) and Parameter Based configuration defined in config are ignored.
|
||||
|
||||
The following parameters ("actions") are available, using any of them enables **Shell Mode**:
|
||||
The following parameters ("actions") are available, the usage of any of them enables **Shell Mode**:
|
||||
|
||||
* ``-C <command>`` - Bash command (string) to execute on nodes. Using multiple ``-C`` statements will give the same result as using one with several commands separated by ``;`` (traditional Shell syntax), but for each ``-C`` statement a new SSH connection is established
|
||||
* ``-S <script>`` - name of the Bash script file to execute on nodes (if you do not have a path separator in the filename, you need to put the file into ``scripts`` folder inside a path specified by ``rqdir`` config parameter, defaults to ``rq``. If path separator is present, a given filename will be used directly as provided)
|
||||
* ``-C <command>`` - Bash command (string) to execute on nodes. Using multiple ``-C`` statements will produce the same result as using one with several commands separated by ``;`` (traditional Shell syntax), but for each ``-C`` statement a new SSH connection is established
|
||||
* ``-S <script>`` - name of the Bash script file to execute on nodes (if you do not have a path separator in the filename, you need to put the file into ``scripts`` folder inside a path specified by ``rqdir`` config parameter, defaults to ``rq``. If a path separator is present, the given filename will be used directly as provided)
|
||||
* ``-P <file/path> <dest>`` - upload local data to nodes (wildcards supported). You must specify 2 values for each ``-P`` switch.
|
||||
* ``-G <file/path>`` - download (collect) data from nodes
|
||||
|
||||
@ -44,13 +44,13 @@ Examples
|
||||
* ``timmy -C 'uptime; free -m'`` - check uptime and memory on all nodes
|
||||
* ``timmy -G /etc/nova/nova.conf`` - get nova.conf from all nodes
|
||||
* ``timmy -R controller -P package.deb '' -C 'dpkg -i package.deb' -C 'rm package.deb' -C 'dpkg -l | grep [p]ackage'`` - push a package to all nodes, install it, remove the file and check that it is installed
|
||||
* ``timmy -с myconf.yaml`` - use a custom config file and run according to it. Custom config can specify any actions, log setup, and other settings. See configuration doc for more details.
|
||||
* ``timmy -с myconf.yaml`` - use a custom config file and run the program according to it. Custom config can specify any actions, log setup, and other settings. See configuration doc for more details.
|
||||
|
||||
===============================
|
||||
Using custom configuration file
|
||||
===============================
|
||||
|
||||
If you want to do a set of actions on the nodes and you do not want to write a long command line (or you want to use options only available in config), you may want to set up config file instead. An example config structure would be:
|
||||
If you want to perform a set of actions on the nodes without writing a long command line (or if you want to use the options only available in config), you may want to set up config file instead. An example config structure would be:
|
||||
|
||||
::
|
||||
|
||||
@ -60,7 +60,7 @@ If you want to do a set of actions on the nodes and you do not want to write a l
|
||||
roles: # only execute on Fuel and controllers
|
||||
- fuel
|
||||
- controller
|
||||
cmds: # some commands to run on all nodes (after filtering). cmds syntax is {name: value, ...}. cmds are executed in alphabetical order of names.
|
||||
cmds: # some commands to run on all nodes (after filtering). cmds syntax is {name: value, ...}. cmds are executed in alphabetical order.
|
||||
01-my-first-command: 'uptime'
|
||||
02-disk-check: 'df -h'
|
||||
and-also-ram: 'free -m'
|
||||
@ -68,7 +68,7 @@ If you want to do a set of actions on the nodes and you do not want to write a l
|
||||
exclude: '.*' # exclude all logs by default
|
||||
by_roles:
|
||||
controller:
|
||||
scripts: # I use script here to not overwrite cmds we have already defined for all nodes earlier
|
||||
scripts: # I use script here to not overwrite the cmds we have already defined for all nodes
|
||||
- pacemaker-debug.sh # the name of the file inside 'scripts' folder inside 'rqdir' path, which will be executed (by default) on all nodes
|
||||
files:
|
||||
- '/etc/coros*' # get all files from /etc/coros* wildcard path
|
||||
@ -78,7 +78,7 @@ If you want to do a set of actions on the nodes and you do not want to write a l
|
||||
|
||||
Then you would run ``timmy -l -c my-config.yaml`` to execute timmy with such config.
|
||||
|
||||
Instead of setting all structure in a config file you can move actions (cmds, files, filelists, scripts, logs) to an rqfile, and specify ``rqfile`` path in config (although with this example the config-way is more compact). ``rqfile`` structure is a bit different:
|
||||
Instead of putting all structure in a config file you can move actions (cmds, files, filelists, scripts, logs) to an rqfile, and specify ``rqfile`` path in config (although in this example the config-way is more compact). ``rqfile`` structure is a bit different:
|
||||
|
||||
::
|
||||
|
||||
@ -88,7 +88,7 @@ Instead of setting all structure in a config file you can move actions (cmds, fi
|
||||
- 02-disk-check: 'df -h'
|
||||
- and-also-ram: 'free -m'
|
||||
scripts:
|
||||
by_roles: # all non "__default" keys should be matches, "by_<parameter>"
|
||||
by_roles: # all non "__default" keys should match, "by_<parameter>"
|
||||
controller:
|
||||
- pacemaker-debug.sh
|
||||
files:
|
||||
@ -102,7 +102,7 @@ Instead of setting all structure in a config file you can move actions (cmds, fi
|
||||
__default:
|
||||
exclude: '.*'
|
||||
|
||||
Then the config should look like:
|
||||
Then the config should look like this:
|
||||
|
||||
::
|
||||
|
||||
|
||||
1
requirements-doc.txt
Normal file
1
requirements-doc.txt
Normal file
@ -0,0 +1 @@
|
||||
sphinx-argparse
|
||||
11
setup.py
11
setup.py
@ -25,6 +25,12 @@ dtm = os.path.join(os.path.abspath(os.sep), 'usr', 'share', pname)
|
||||
rqfiles = [(os.path.join(dtm, root), [os.path.join(root, f) for f in files])
|
||||
for root, dirs, files in os.walk('rq')]
|
||||
rqfiles.append((os.path.join(dtm, 'configs'), ['config.yaml', 'rq.yaml']))
|
||||
package_data = True
|
||||
|
||||
if os.environ.get("READTHEDOCS", False):
|
||||
rqfiles = None
|
||||
package_data = False
|
||||
|
||||
|
||||
setup(name=pname,
|
||||
version=version,
|
||||
@ -32,10 +38,13 @@ setup(name=pname,
|
||||
author_email='dobdin@gmail.com',
|
||||
license='Apache2',
|
||||
url='https://github.com/adobdin/timmy',
|
||||
description = ('Mirantis OpenStack Ansible-like tool for parallel node '
|
||||
'operations: two-way data transfer, log collection, '
|
||||
'remote command execution'),
|
||||
long_description=open('README.md').read(),
|
||||
packages=[pname],
|
||||
install_requires=['pyyaml'],
|
||||
data_files=rqfiles,
|
||||
include_package_data=True,
|
||||
include_package_data=package_data,
|
||||
entry_points={'console_scripts': ['%s=%s.cli:main' % (pname, pname)]},
|
||||
)
|
||||
|
||||
52
specs/python-timmy.spec
Normal file
52
specs/python-timmy.spec
Normal file
@ -0,0 +1,52 @@
|
||||
%{!?__python2: %global __python2 /usr/bin/python2}
|
||||
%{!?python2_sitelib: %global python2_sitelib %(%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")}
|
||||
%{!?python2_sitearch: %global python2_sitearch %(%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")}
|
||||
|
||||
%define name python-timmy
|
||||
%{!?version: %define version 1.8.1}
|
||||
%{!?release: %define release 1}
|
||||
|
||||
Summary: Console utility for collecting cluster information
|
||||
Name: %{name}
|
||||
Version: %{version}
|
||||
Release: %{release}
|
||||
Source0: %{name}-%{version}.tar.gz
|
||||
License: Apache
|
||||
Group: Support/Tools
|
||||
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-buildroot
|
||||
Prefix: %{_prefix}
|
||||
|
||||
BuildArch: noarch
|
||||
|
||||
BuildRequires: python-setuptools
|
||||
|
||||
%if 0%{!?rhel:0} == 6
|
||||
Requires: python-argparse
|
||||
%endif
|
||||
|
||||
Requires: PyYAML
|
||||
|
||||
%description
|
||||
Summary: Console utility for collecting cluster information
|
||||
|
||||
%prep
|
||||
%setup -cq -n %{name}-%{version}
|
||||
|
||||
%build
|
||||
cd %{_builddir}/%{name}-%{version} && %{__python2} setup.py build
|
||||
|
||||
%install
|
||||
rm -rf $RPM_BUILD_ROOT
|
||||
cd %{_builddir}/%{name}-%{version} && %{__python2} setup.py install --single-version-externally-managed -O1 --root=$RPM_BUILD_ROOT
|
||||
|
||||
%clean
|
||||
rm -rf $RPM_BUILD_ROOT
|
||||
|
||||
%files -n python-timmy
|
||||
/usr/bin/timmy
|
||||
/usr/share/timmy/*
|
||||
%{python2_sitelib}/timmy*
|
||||
|
||||
%changelog
|
||||
* Thu Jun 30 2016 Aleksandr Dobdin <adobdin@mirantis.com> - 1.8.1
|
||||
- Created spec
|
||||
10
timmy/cli.py
10
timmy/cli.py
@ -57,6 +57,11 @@ def parse_args():
|
||||
parser.add_argument('-r', '--role', action='append',
|
||||
help=('Can be specified multiple times.'
|
||||
' Run only on the specified role.'))
|
||||
parser.add_argument('-d', '--days', type=int,
|
||||
help=('Define log collection period in days.'
|
||||
' Timmy will collect only logs updated on or'
|
||||
' more recently then today minus the given'
|
||||
' number of days. Default - 30.'))
|
||||
parser.add_argument('-G', '--get', action='append',
|
||||
help=('Enables shell mode. Can be specified multiple'
|
||||
' times. Filemask to collect via "scp -r".'
|
||||
@ -178,6 +183,8 @@ def main(argv=None):
|
||||
conf['clean'] = False
|
||||
if args.rqfile:
|
||||
conf['rqfile'] = args.rqfile
|
||||
if args.days:
|
||||
conf['logs']['start'] = -args.days
|
||||
if conf['shell_mode']:
|
||||
filter = conf['hard_filter']
|
||||
# config cleanup for shell mode
|
||||
@ -242,7 +249,8 @@ def main(argv=None):
|
||||
enough = pretty_run(args.quiet, 'Checking free space',
|
||||
nm.is_enough_space)
|
||||
if enough:
|
||||
pretty_run(args.quiet, 'Collecting and packing logs', nm.get_logs,
|
||||
msg = 'Collecting and packing %dMB of logs' % (nm.alogsize / 1024)
|
||||
pretty_run(args.quiet, msg, nm.get_logs,
|
||||
args=(conf['compress_timeout'],),
|
||||
kwargs={'maxthreads': args.logs_maxthreads,
|
||||
'fake': args.fake_logs})
|
||||
|
||||
@ -60,7 +60,8 @@ def load_conf(filename):
|
||||
conf['files'] = []
|
||||
conf['filelists'] = []
|
||||
conf['logs'] = {'path': '/var/log',
|
||||
'exclude': '[-_]\d{8}$|atop[-_]|\.gz$'}
|
||||
'exclude': '\.[^12]\.gz$|\.\d{2,}\.gz$',
|
||||
'start': '30'}
|
||||
'''Shell mode - only run what was specified via command line.
|
||||
Skip actionable conf fields (see timmy/nodes.py -> Node.conf_actionable);
|
||||
Skip rqfile import;
|
||||
|
||||
@ -16,7 +16,7 @@
|
||||
# under the License.
|
||||
|
||||
project_name = 'timmy'
|
||||
version = '1.8.1'
|
||||
version = '1.9.0'
|
||||
|
||||
if __name__ == '__main__':
|
||||
exit(0)
|
||||
|
||||
@ -25,7 +25,7 @@ import shutil
|
||||
import logging
|
||||
import sys
|
||||
import re
|
||||
from datetime import datetime
|
||||
from datetime import datetime, date, timedelta
|
||||
import tools
|
||||
from tools import w_list, run_with_lock
|
||||
from copy import deepcopy
|
||||
@ -323,12 +323,32 @@ class Node(object):
|
||||
re.search(item['exclude'], string)))
|
||||
|
||||
for item in self.logs:
|
||||
start_str = ''
|
||||
if 'start' in item:
|
||||
start = ' -newermt \\"$(date -d \'%s\')\\"' % item['start']
|
||||
start = item['start']
|
||||
if any([type(start) is str and re.match(r'-?\d+', start),
|
||||
type(start) is int]):
|
||||
days = abs(int(str(start)))
|
||||
start_str = str(date.today() - timedelta(days=days))
|
||||
else:
|
||||
for format in ['%Y-%m-%d', '%Y-%m-%d %H:%M:%S']:
|
||||
try:
|
||||
if datetime.strptime(start, format):
|
||||
start_str = start
|
||||
break
|
||||
except ValueError:
|
||||
pass
|
||||
if not start_str:
|
||||
self.logger.warning(('incorrect value of "start"'
|
||||
' parameter in "logs": "%s" -'
|
||||
' ignoring...')
|
||||
% start)
|
||||
if start_str:
|
||||
start_param = ' -newermt "$(date -d \'%s\')"' % start_str
|
||||
else:
|
||||
start = ''
|
||||
start_param = ''
|
||||
cmd = ("find '%s' -type f%s -exec du -b {} +" % (item['path'],
|
||||
start))
|
||||
start_param))
|
||||
self.logger.info('node: %s, logs du-cmd: %s' %
|
||||
(self.id, cmd))
|
||||
outs, errs, code = tools.ssh_node(ip=self.ip,
|
||||
@ -349,6 +369,8 @@ class Node(object):
|
||||
size, f = line.split('\t')
|
||||
if filter_by_re(item, f):
|
||||
item['files'][f] = int(size)
|
||||
else:
|
||||
self.logger.debug('log file "%s" excluded' % f)
|
||||
self.logger.debug('logs: %s' % (item['files']))
|
||||
return self.logs
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user