I’m using the new drive space and systems to basically mirror the work environment. Part of it is in order to have a playground or sandbox where I can try new things and learn how to use the tools we have and part is just “Because It’s There” 🙂 There’s satisfaction in being able to recreate the basic work setup at home.
As noted in previous posts, I have a pretty decent computer network now and I’ve created four environments.
Site 1. CentOS 7 based and hosts my personal, more live stuff like a movie and music server, development environment (2 servers), and backups. I also have a couple of Windows Workstation installations and Server installs for Jeanne. Plus of course the firewall. 13 Servers in total.
Site 2. CentOS 5, 6, and 7 based and hosts the Ansible and Kubernetes/Docker environments. In addition, there’s now an Ansible Tower server and a Spacewalk server. 24 Servers in total.
Site 3. Red Hat 6 and 7 based for Ansible testing. 11 Servers in total.
Site 4. Miscellaneous operating systems for further Ansible testing. 16 Servers in total.
16 Servers on the main ESX host.
48 Servers on the sandbox ESX host.
Total Servers: 64 Servers.
Red Hat
One of the nice things is Red Hat has a Developer network which provides self-support for Red Hat Enterprise Linux (RHEL) to someone who’s signed up. The little known bit though is you can have unlimited copies of RHEL if you’re running them virtually. Sign up is simple. Go to Red Hat and sign up to the Developer Network. Then download RHEL and install it. Run the following command to register a server:
# subscription-manager register --auto-attach
Note that you will need to renew your registration every year.
Spacewalk
Spacewalk is the freely available tool used for managing your servers. Red Hat’s paid version is Satellite. For ours at work, it’s $10,000 a year for a license. So Spacewalk it is 🙂
I use Satellite at work and it works pretty well. We have about 300 servers registered since the start of the year and are working to add more. I am finding Spacewalk, even though it’s older, to be quite a bit easier to use compared to Satellite. It’s quicker and the tasks are more obvious. Not perfect of course but it seems to be a simpler system to use. I set up CentOS 5, 6, and 7 repositories (repos) to sync and download updates each week.
Before you can connect a client, you need to create a channel for the operation system.
1. You need to create a Channel to provide an anchor for any underlying repos. I created a ‘hcs-centos54’, ‘hcs-centos65′, and hcs-centos7’ channel. Create a Channel: Channels -> Manage Software Channels -> Create Channel
2. You need to create repositories. You can create a one-for-one relationship or add multiple repos to a channel. I did mine one-for-one for now. I had to locate URLs for repositories. For the ‘centos7_mirror’, I used the mirror.centos.org site. For older versions, I had to use the vault.centos.org site. Create a Repository: Channels -> Manage Software Channels -> Manage Repositories
3. Now associate the repo with a channel. Simply go to the channel and click on the Repositories tab. Check the appropriate repo(s) and click the Update Repositories button.
The command to associate a server requires an activation key. This lets you auto-register clients so you don’t have to pop into Spacewalk to manually associate servers. The only thing needed is a name, I used ‘centos5-base’ for one, and associate a channel. The key is created automatically once you click the button. Create an Activation Key: Systems -> Activation Keys -> Create Key -> Description, Base Channel, click Update Activation Key
You’ll need the ‘1-‘ at the beginning of the key to activate a client.
There’s a set of tools needed in order to support the activation and what gets installed depends on the OS version. For my purposes, the following are needed:
RHEL5/CentOS 5
rpm -Uvh http://yum.spacewalkproject.org/2.5-client/RHEL/5/x86_64/spacewalk-client-repo-2.5-3.el5.noarch.rpm BASEARCH=$(uname -i) rpm -Uvh http://dl.fedoraproject.org/pub/epel/epel-release-latest-5.noarch.rpm yum install -y rhn-client-tools rhn-check rhn-setup rhnsd m2crypto yum-rhn-plugin
RHEL6/CentOS 6
rpm -Uvh http://yum.spacewalkproject.org/2.5-client/RHEL/6/x86_64/spacewalk-client-repo-2.5-3.el6.noarch.rpm BASEARCH=$(uname -i) rpm -Uvh http://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm yum install -y rhn-client-tools rhn-check rhn-setup rhnsd m2crypto yum-rhn-plugin
RHEL7/CentOS 7
rpm -Uvh http://yum.spacewalkproject.org/2.5-client/RHEL/7/x86_64/spacewalk-client-repo-2.5-3.el7.noarch.rpm BASEARCH=$(uname -i) rpm -Uvh http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm yum install -y rhn-client-tools rhn-check rhn-setup rhnsd m2crypto yum-rhn-plugin
Once that’s installed (if there’s an error, you’ll need to install the epel-release package and try again), register the system.
rpm -Uvh http://192.168.1.5/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm rhnreg_ks --serverUrl=http://192.168.1.5/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=1-[key]
Once done, log in to Spacewalk and click on Systems -> Systems to see the newly registered system. If you’re running different OSs under different channels, you’ll need different keys for the various OSs.
In order to activate kickstarts, you need to sync a kickstart with the Spacewalk server. It’s not complicated but it’s not obvious 🙂 Get the channel name for the kickstart repo you want to create and run the following command:
# spacewalk-repo-sync -c [Channel Name] --sync-kickstart
My channel name is hcs-centos7 so the command on my system would be:
# spacewalk-repo-sync -c hcs-centos65 --sync-kickstart
I plan on taking the kickstart configurations I built for the servers and adding them to Spacewalk to see how that works and maybe kickstart some systems to play with kickstarting.
Configuration
I also have the scripts I wrote for work and have them deployed on all the servers plus adding accounts. I needed to update the times to be Mountain Time as the times for the kick off of scheduled nightly or weekly tasks were going off in early evening and slowing down access to the ‘net for Jeanne and me. This involved updating the timezones and starting the ntp daemon.
RHEL7/CentOS 7
# timedatectl set-timezone America/Denver
RHEL6/CentOS 6
You link because if there’s an update that changes the zone information, such as a day change, the system is automatically correct.
# rm /etc/localtime # ln -s /usr/share/zoneinfo/America/Denver /etc/localtime
RHEL5/CentOS 5
# rm /etc/localtime # ln -s /usr/share/zoneinfo/America/Denver /etc/localtime
Time
And related to time, I need to ensure either ntp or chrony is properly configured and started. Kubernetes especially requires consistent time.
The chronyd and chronyc are the replacements for ntpd and ntpq. The configuration is similar though and with the same understanding about how it works. As I have a time server running on pfSense, I’m ensuring the servers all are enable and are pointing to the local time server. No point in generating a bunch of unnecessary traffic through comcast and just keep pfSense updated.
chronyd
Edit /etc/chrony.conf, comment out the existing pool servers and add in this line:
server 192.168.1.1 iburst
Enable and start chronyd if it’s not running and restart it if it’s already up. Then run the chronyc command to verify the change.
# systemctl enable chronyd # systemctl start chronyd
or
# systemctl restart chronyd
Results:
# chronyc sources 210 Number of sources = 1 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* pfSense.internal.pri 3 6 17 1 -13us[ -192us] +/- 109ms
ntpd
Edit /etc/ntp.conf, comment out the existing pool servers and add in this line:
server 192.168.1.1
Enable and start ntpd if it’s not running and restart it if it’s already up. Then run the ntpq command to verify the change.
# service ntpd start # chkconfig ntpd on
or
# service ntpd restart
Results:
# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
pfSense.interna 42.91.213.246 3 u 11 64 1 0.461 -5.037 0.001
LOCAL(0) .LOCL. 10 l 10 64 1 0.000 0.000 0.001
Nagios
I started setting a nagios server. Nagios is a tool used to monitor various aspect of servers. At work we’re using it as a basic ping test just to make sure we know servers are up with a quick look. Other bits are being added in as time permits. Here I did install net-snmp and net-snmp-utils in order to build the check_snmp plugin. This gives me lots and lots of options on what to check and might let me replace some of the scripts I have in place.
SNMP Configuration
#### # First, map the community name "public" into a "security name" # sec.name source community com2sec AllUser default CHANGEME #### # Second, map the security name into a group name: # groupName securityModel securityName group notConfigGroup v1 notConfigUser group AllGroup v2c AllUser #### # Third, create a view for us to let the group have rights to: # Make at least snmpwalk -v 1 localhost -c public system fast again. # name incl/excl subtree mask(optional) view systemview included .1.3.6.1.2.1.1 view systemview included .1.3.6.1.2.1.25.1.1 view AllView included .1 #### # Finally, grant the group read-only access to the systemview view. # group context sec.model sec.level prefix read write notif access AllGroup "" any noauth exact AllView none none
Unfortunately the default check_snmp command in the commands.cfg file was a bit off.
Old:
# 'check_snmp' command definition
define command{
command_name check_snmp
command_line $USER1$/check_snmp -H $HOSTADDRESS$ $ARG1$
In running the program with a -h command, I found the correct options for what I needed to do:
New:
# 'check_snmp' command definition
define command{
command_name check_snmp
command_line $USER1$/check_snmp -H $HOSTADDRESS$ -C $ARG1$ -o $ARG2$ -P 2c
Per the configuration, I’m using snmp version 2. Other than that, just pass the appropriate community string (-C) and object id (-o) to do the check you want to check.
Uptime:
In the linux.cfg file I created, I added the following check_snmp block:
define service{
use local-service ; Name of service template to use
host_name [comma separated list of hosts]
service_description Uptime
check_command check_snmp!CHANGEME!.1.3.6.1.2.1.1.3.0!
}
Possibly Interesting OIDs:
Network Interface Statistics
- List NIC names: .1.3.6.1.2.1.2.2.1.2
- Get Bytes IN: .1.3.6.1.2.1.2.2.1.10
- Get Bytes IN for NIC 4: .1.3.6.1.2.1.2.2.1.10.4
- Get Bytes OUT: .1.3.6.1.2.1.2.2.1.16
- Get Bytes OUT for NIC 4: .1.3.6.1.2.1.2.2.1.16.4
Load
- 1 minute Load: .1.3.6.1.4.1.2021.10.1.3.1
- 5 minute Load: .1.3.6.1.4.1.2021.10.1.3.2
- 15 minute Load: .1.3.6.1.4.1.2021.10.1.3.3
CPU times
- percentages of user CPU time: .1.3.6.1.4.1.2021.11.9.0
- percentages of system CPU time: .1.3.6.1.4.1.2021.11.10.0
- percentages of idle CPU time: .1.3.6.1.4.1.2021.11.11.0
- raw user cpu time: .1.3.6.1.4.1.2021.11.50.0
- raw system cpu time: .1.3.6.1.4.1.2021.11.52.0
- raw idle cpu time: .1.3.6.1.4.1.2021.11.53.0
- raw nice cpu time: .1.3.6.1.4.1.2021.11.51.0
Memory Statistics
- Total Swap Size: .1.3.6.1.4.1.2021.4.3.0
- Available Swap Space: .1.3.6.1.4.1.2021.4.4.0
- Total RAM in machine: .1.3.6.1.4.1.2021.4.5.0
- Total RAM used: .1.3.6.1.4.1.2021.4.6.0
- Total RAM Free: .1.3.6.1.4.1.2021.4.11.0
- Total RAM Shared: .1.3.6.1.4.1.2021.4.13.0
- Total RAM Buffered: .1.3.6.1.4.1.2021.4.14.0
- Total Cached Memory: .1.3.6.1.4.1.2021.4.15.0
Disk Statistics
- Path where the disk is mounted: .1.3.6.1.4.1.2021.9.1.2.1
- Path of the device for the partition: .1.3.6.1.4.1.2021.9.1.3.1
- Total size of the disk/partion (kBytes): .1.3.6.1.4.1.2021.9.1.6.1
- Available space on the disk: .1.3.6.1.4.1.2021.9.1.7.1
- Used space on the disk: .1.3.6.1.4.1.2021.9.1.8.1
- Percentage of space used on disk: .1.3.6.1.4.1.2021.9.1.9.1
- Percentage of inodes used on disk: .1.3.6.1.4.1.2021.9.1.10.1
System Uptime OID’s
- .1.3.6.1.2.1.1.3.0
One problem with the OIDs are they’re statistics and not much use without a trigger. They’re really more useful with MRTG where you can see what things look like over a period of time. What you really want to do is check to see when stats exceed expected norms.
MRTG
This is primarily a network traffic monitoring type tool but I’ve configured it to track other system statistics regarding disk space, swap, memory, and whatnot. It’s not configured just yet but that’s my next configuration task.