x/compass-adapters
Code Issues Proposed changes

Add rsyslog cookbook and sync openstack logs

Browse Source 
Modified the client recipe to sync openstack related
logs if a test indicator is enabled.

Change-Id: Ia41c28ccecb4ef298221494e6444d4909d917c56
This commit is contained in:
Xinyu Zhao 2014-04-11 04:10:41 +00:00
parent a0753a946e
commit 1f13960afb
14 changed files with 666 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

177
chef/cookbooks/rsyslog/README.md Normal file
View File

@ -0,0 +1,177 @@
Description
===========
Installs rsyslog to replace sysklogd for client and/or server use. By default, server will be set up to log to files.
Changes
=======
## v1.0.0:
* [COOK-836] - use an attribute to specify the role to search for
instead of relying on the rsyslog['server'] attribute.
* Clean up attribute usage to use strings instead of symbols.
* Update this README.
* Better handling for chef-solo.
Requirements
============
Platform
--------
Tested on Ubuntu 8.04, 9.10, 10.04.
For Ubuntu 8.04, the rsyslog package will be installed from a PPA via the default.rb recipe in order to get 4.2.0 backported from 10.04.
* https://launchpad.net/~a.bono/+archive/rsyslog
Ubuntu 8.10 and 9.04 are no longer supported releases and have not been tested with this cookbook.
Cookbooks
---------
* cron (http://community.opscode.com/cookbooks/cron)
Other
-----
To use the `recipe[rsyslog::client]` recipe, you'll need to set up a
role to search for. See the __Recipes__, and __Examples__ sections below.
Attributes
==========
See `attributes/default.rb` for default values.
* `node['rsyslog']['log_dir']` - If the node is an rsyslog server,
this specifies the directory where the logs should be stored.
* `node['rsyslog']['server']` - Used to indicate whether the node
running Chef is an rsyslog server. As of cookbook v1.0.0, this is
determined automatically through search. The server recipe will set
this to true. It is otherwise unused in the current version.
* `node['rsyslog']['protocol']` - Specify whether to use `udp` or
`tcp` for remote loghost.
* `node['rsyslog']['port']` - Specify the port which rsyslog should
connect to a remote loghost.
* `node['rsyslog']['server_role']` - Role applied to a remote
loghost. Used by `recipe[rsyslog::client]` to search for the
loghost.
Recipes
=======
default
-------
Installs the rsyslog package, manages the rsyslog service and sets up
basic configuration for a standalone machine.
client
------
Includes `recipe[rsyslog]`.
Uses Chef search to find a remote loghost node with the role specified
by `node['rsyslog']['server_role']` and uses its `ipaddress` attribute
to send log messages. If the node itself has the `server_role` in the
expanded roles, then the configuration is skipped. If the node had an
`/etc/rsyslog.d/server.conf` file previously configured, this file
gets removed to prevent duplicate logging. Any previous logs are not
cleaned up from the `log_dir`.
server
------
Configures the node to be an rsyslog loghost. The node should have the
role specified by `node['rsyslog']['server_role']` applied so client
nodes can find it with search. This recipe will create the logs in
`node['rsyslog']['log_dir']`, and the configuration is in
`/etc/rsyslog.d/server.conf`. This recipe also removes any previous
configuration to a remote server by removing the
`/etc/rsyslog.d/remote.conf` file. Finally, a cron job is set up to
compress logs in the `log_dir` that are older than one day.
The server configuration will set up `log_dir` for each client, by
date. Directory structure:
<%= @log_dir %>/YEAR/MONTH/DAY/HOSTNAME/"logfile"
For example:
/srv/rsyslog/2011/11/19/www/messages
At this time, the server can only listen on UDP *or* TCP.
Usage
=====
Use `recipe[rsyslog]` to install and start rsyslog as a basic
configured service for standalone systems.
Use `recipe[rsyslog::client]` to have nodes search for the loghost
automatically to configure remote [r]syslog.
Use `recipe[rsyslog::server]` to set up a loghost. It will listen on
`node['rsyslog']['port']` protocol `node['rsyslog']['protocol']`.
If you set up a different kind of centralized loghost (syslog-ng,
graylog2, logstash, etc), you can still send log messages to it as
long as the port and protocol match up with the server
software. See __Examples__
Examples
--------
A `base` role (e.g., roles/base.rb), applied to all nodes so they are syslog clients:
name "base"
description "Base role applied to all nodes
run_list("recipe[rsyslog::client]")
Then, a role for the loghost (should only be one):
name "loghost"
description "Central syslog server"
run_list("recipe[rsyslog::server]")
By default this will set up the clients search for a node with the
`loghost` role to talk to the server on TCP port 514. Change the
`protocol` and `port` rsyslog attributes to modify this.
If you're using another log server software on your loghost, such as
graylog2, you can use the role for that loghost for the search
instead. For example, if the role of your graylog2 server is
`graylog2_server`, then modify the base role for the server role:
name "base"
description "Base role applied to all nodes
run_list("recipe[rsyslog::client]")
default_attributes(
"rsyslog" => {
"server_role" => "graylog2_server"
}
)
Then make sure you have a role named `graylog2_server` applied to some
node, and `recipe[rsyslog::client]` will configure the local system to
send logs to graylog2.
License and Author
==================
Author:: Joshua Timberman (<joshua@opscode.com>)
Copyright:: 2009-2011, Opscode, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

50
chef/cookbooks/rsyslog/attributes/default.rb Normal file
View File

@ -0,0 +1,50 @@
#
# Cookbook Name:: rsyslog
# Attributes:: rsyslog
#
# Copyright 2009, Opscode, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
default["rsyslog"]["log_dir"] = "/srv/rsyslog"
default["rsyslog"]["server"] = false
default["rsyslog"]["protocol"] = "tcp"
default["rsyslog"]["port"] = "514"
default["rsyslog"]["server_role"] = "loghost"
default["rsyslog"]["openstacklog"] = {:"keystone"=>"/var/log/keystone/keystone.log",
:"glance-api"=>"/var/log/glance/api.log",
:"glance-reg"=>"/var/log/glance/registry.log",
:"nova-api"=>"/var/log/nova/api.log",
:"nova-cert"=>"/var/log/nova/cert.log",
:"nova-conductor"=>"/var/log/nova/conductor.log",
:"nova-consoleauth"=>"/var/log/nova/consoleauth.log",
:"nova-console"=>"/var/log/nova/console.log",
:"nova-manage"=>"/var/log/nova/nova-manange.log",
:"nova-compute"=>"/var/log/nova/compute.log",
:"nova-scheduler"=>"/var/log/nova/scheduler.log",
:"cinder-api"=>"/var/log/cinder/api.log",
:"cinder-scheduler"=>"/var/log/cinder/scheduler.log",
:"cinder-volume"=>"/var/log/cinder/volume.log",
:"quantum-server"=>"/var/log/quantum/server.log",
:"quantum-dhcp"=>"/var/log/quantum/dhcp-agent.log",
:"quantum-l3agent"=>"/var/log/quantum/l3-agent.log",
:"quantum-ovsagent"=>"/var/log/quantum/openvswitch-agent.log",
:"dashboard-access"=>"/var/log/httpd/openstack-dashboard-access.log",
:"dashboard-error"=>"/var/log/httpd/openstack-dashboard-error.log",
:"mysql"=>"/var/log/mysqld.log",
:"rabbitmq"=>"/var/log/rabbitmq/rabbit\@#{node['hostname']}.log",
:"ovs-vswitchd"=>"/var/log/openvswitch/ovs-vswitchd.log",
:"ovs-dbserver"=>"/var/log/openvswitch/ovs-dbserver.log",
:"libvirtd"=>"/var/log/libvirt/libvirtd.log"}

9
chef/cookbooks/rsyslog/files/default/rsyslog.default Normal file
View File

@ -0,0 +1,9 @@
# Generated by Chef
#
# Use v3 native mode, rather than compatibility mode by specifying -c3
# here. Compatibility mode for older versions is not recommended as
# custom configuration may get messy.
#
# See rsyslogd(8) for more details
RSYSLOGD_OPTIONS="-c3"

9
chef/cookbooks/rsyslog/files/ubuntu-10.04/rsyslog.default Normal file
View File

@ -0,0 +1,9 @@
# Generated by Chef
#
# Options for rsyslogd
# -m 0 disables 'MARK' messages (deprecated, only used in compat mode < 3)
# -r enables logging from remote machines (deprecated, only used in compat mode < 3)
# -x disables DNS lookups on messages received with -r
# -c compatibility mode
# See rsyslogd(8) for more details
RSYSLOGD_OPTIONS="-c4"

9
chef/cookbooks/rsyslog/files/ubuntu-9.10/rsyslog.default Normal file
View File

@ -0,0 +1,9 @@
# Generated by Chef
#
# Options for rsyslogd
# -m 0 disables 'MARK' messages (deprecated, only used in compat mode < 3)
# -r enables logging from remote machines (deprecated, only used in compat mode < 3)
# -x disables DNS lookups on messages received with -r
# -c compatibility mode
# See rsyslogd(8) for more details
RSYSLOGD_OPTIONS="-c4"

39
chef/cookbooks/rsyslog/metadata.rb Normal file
View File

@ -0,0 +1,39 @@
maintainer "Opscode, Inc."
maintainer_email "cookbooks@opscode.com"
license "Apache 2.0"
description "Installs and configures rsyslog"
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version "1.0.0"
recipe "rsyslog", "Installs rsyslog"
recipe "rsyslog::client", "Sets up a client to log to a remote rsyslog server"
recipe "rsyslog::server", "Sets up an rsyslog server"
supports "ubuntu", ">= 8.04"
supports "debian", ">= 5.0"
attribute "rsyslog",
:display_name => "Rsyslog",
:description => "Hash of Rsyslog attributes",
:type => "hash"
attribute "rsyslog/log_dir",
:display_name => "Rsyslog Log Directory",
:description => "Filesystem location of logs from clients",
:default => "/srv/rsyslog"
attribute "rsyslog/server",
:display_name => "Rsyslog Server?",
:description => "Is this node an rsyslog server?",
:default => "false"
attribute "rsyslog/protocol",
:display_name => "Rsyslog Protocol",
:description => "Set which network protocol to use for rsyslog",
:default => "tcp"
attribute "rsyslog/port",
:display_name => "Rsyslog Port",
:description => "Port that Rsyslog listens for incoming connections",
:default => "514"

35
chef/cookbooks/rsyslog/recipes/client.rb Normal file
View File

@ -0,0 +1,35 @@
#
# Cookbook Name:: rsyslog
# Recipe:: client
#
# Copyright 2009-2011, Opscode, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
include_recipe "rsyslog"
template "/etc/rsyslog.d/openstack.conf" do
source "openstack.conf.erb"
backup false
owner "root"
group "root"
mode 0644
variables :loglist => node['rsyslog']['openstacklog']
notifies :restart, "service[rsyslog]"
end
file "/etc/rsyslog.d/server.conf" do
action :delete
notifies :reload, "service[rsyslog]"
only_if do ::File.exists?("/etc/rsyslog.d/server.conf") end
end

73
chef/cookbooks/rsyslog/recipes/default.rb Normal file
View File

@ -0,0 +1,73 @@
#
# Cookbook Name:: rsyslog
# Recipe:: default
#
# Copyright 2009-2011, Opscode, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
if platform?("ubuntu") && node['platform_version'].to_f == 8.04
apt_repository "hardy-rsyslog-ppa" do
uri "http://ppa.launchpad.net/a.bono/rsyslog/ubuntu"
distribution "hardy"
components ["main"]
keyserver "keyserver.ubuntu.com"
key "C0061A4A"
action :add
notifies :run, "execute[apt-get update]", :immediately
end
end
package "rsyslog" do
action :install
end
service "rsyslog" do
service_name "rsyslogd" if platform?("arch")
supports :restart => true, :reload => true
action [:enable, :start]
end
if platform?("ubuntu")
cookbook_file "/etc/default/rsyslog" do
source "rsyslog.default"
owner "root"
group "root"
mode 0644
end
end
directory "/etc/rsyslog.d" do
owner "root"
group "root"
mode 0755
end
#template "/etc/rsyslog.conf" do
# source "rsyslog.conf.erb"
# owner "root"
# group "root"
# mode 0644
# notifies :restart, "service[rsyslog]"
#end
if platform?("ubuntu")
template "/etc/rsyslog.d/50-default.conf" do
source "50-default.conf.erb"
backup false
owner "root"
group "root"
mode 0644
end
end

12
chef/cookbooks/rsyslog/templates/default/openstack.conf.erb Normal file
View File

@ -0,0 +1,12 @@
$Modload imfile
<% @loglist.each do |service, path| -%>
# <%= service %>
$InputFileName <%= path %>
$InputFileReadMode 0
$InputFileTag <%= service %>
$InputFileSeverity notice
$InputFileFacility local4
$InputFileStateFile
$InputRunFileMonitor
$InputFilePollInterval 1
<% end -%>

6
chef/cookbooks/rsyslog/templates/default/remote.conf.erb Normal file
View File

@ -0,0 +1,6 @@
<% case @protocol -%>
<% when "tcp" -%>
*.* @@<%= @server %>:<%= node['rsyslog']['port'] %>
<% when "udp" -%>
*.* @<%= @server %>:<%= node['rsyslog']['port'] %>
<% end -%>

115
chef/cookbooks/rsyslog/templates/default/rsyslog.conf.erb Normal file
View File

@ -0,0 +1,115 @@
# /etc/rsyslog.conf Configuration file for rsyslog v3.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#################
#### MODULES ####
#################
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability
# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun <%= node['rsyslog']['port'] %>
# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun <%= node['rsyslog']['port'] %>
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use default timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
<% unless node[:platform] == 'ubuntu' -%>
###############
#### RULES ####
###############
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Logging for INN news system.
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
#
# Some "catch-all" log files.
#
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg *
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8
# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole
<% end -%>

58
chef/cookbooks/rsyslog/templates/default/server.conf.erb Normal file
View File

@ -0,0 +1,58 @@
# Generated by Chef.
# Local modifications will be overwritten.
<% case @protocol -%>
<% when "tcp" -%>
$ModLoad imtcp
$InputTCPServerRun <%= node['rsyslog']['port'] %>
<% when "udp" -%>
$ModLoad imudp
$UDPServerRun <%= node['rsyslog']['port'] %>
<% end -%>
$DirGroup root
$DirCreateMode 0755
$FileGroup root
$template PerHostAuth,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/auth.log"
$template PerHostCron,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/cron.log"
$template PerHostSyslog,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/syslog"
$template PerHostDaemon,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/daemon.log"
$template PerHostKern,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/kern.log"
$template PerHostLpr,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/lpr.log"
$template PerHostUser,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/user.log"
$template PerHostMail,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/mail.log"
$template PerHostMailInfo,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/mail.info"
$template PerHostMailWarn,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/mail.warn"
$template PerHostMailErr,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/mail.err"
$template PerHostNewsCrit,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/news.crit"
$template PerHostNewsErr,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/news.err"
$template PerHostNewsNotice,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/news.notice"
$template PerHostDebug,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug"
$template PerHostMessages,"<%= @log_dir %>/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/messages"
auth,authpriv.* ?PerHostAuth
*.*;auth,authpriv.none -?PerHostSyslog
cron.* ?PerHostCron
daemon.* -?PerHostDaemon
kern.* -?PerHostKern
lpr.* -?PerHostLpr
mail.* -?PerHostMail
user.* -?PerHostUser
mail.info -?PerHostMailInfo
mail.warn ?PerHostMailWarn
mail.err ?PerHostMailErr
news.crit ?PerHostNewsCrit
news.err ?PerHostNewsErr
news.notice -?PerHostNewsNotice
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -?PerHostDebug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -?PerHostMessages

69
chef/cookbooks/rsyslog/templates/ubuntu/50-default.conf.erb Normal file
View File

@ -0,0 +1,69 @@
# Default rules for rsyslog.
#
# For more information see rsyslog.conf(5) and /etc/rsyslog.conf
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Logging for INN news system.
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
#
# Some "catch-all" log files.
#
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg *
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8
# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole

5
chef/roles/test-synclog.rb Normal file
View File

@ -0,0 +1,5 @@
name "test-synclog"
description "Sync application related logs for debugging"
run_list(
"recipe[rsyslog::client]"
)