Saturday, September 16, 2017

Updating system time using chrony on CentOS 7

HOW TO: If the current system time is completely off by several months or years, using chrony to update the time can be a bit tricky.  The online documentation does not clearly explain how to handle this type of scenario, and probably with good reason.  Why would you use an NTP system to update a device that has gone off the correct time by months or years?  That's not what ntpd or chronyd are designed for; rather they are designed to prevent time-drift by seconds, minutes or at worse, hours.

The normal way to update the time on a RHEL/CentOS 7 that has gone WAY off, for whatever reason, is to set it manually and only then use an NTP daemon to maintain it.

Setting the time and date manually to the current time:

# timedatectl set-time "2017-09-16 11:51:58"

Next, simply make sure chronyd or ntpd is running - not both:

# systemctl status chronyd

-- How to do this using chrony? --

It is tricky to set the system time using chrony when it has gone way off; chronyd works by increasing the clock speed or reducing the clock speed to catch up.  Obviously this is not efficient, hence why you should set it manually as demonstrated above.

The first step is to restart the chronyd service:

# systemctl restart chronyd

In most cases, this will not immediately update the system time, but at least the service will be aware of the correct time internally.  However, using timedatectl will show that the system time has not been changed.

Next you will need to force update the time by "making the step":

# chronyc -a makestep

At this point the system time should be updated and you can verify this with:

# timedatectl

Finally you can update the hardware clock with:

# hwclock --systohc

Wednesday, September 13, 2017

Rescue a CentOS 7 system with a deleted /boot directory

HOW TO: Your /boot directory is missing or deleted on CentOS 7, you can't boot!  Imagine this type of situation happening on a real production system.  It is unlikely to happen, but it is always good to know how to recover from such a disastrous failure.

This scenario is based on several tests I've performed on KVM based virtual machines.

Boot the system with a rescue DVD (or ISO for a VM).

At the CentOS 7 boot CD prompt, choose "Troubleshooting" and "Rescue a CentOS system".  Later, choose "Continue" to allow the rescue environment to mount the machine's file systems under "/mnt/sysimage". 

At the prompt, you will be in a shell loaded by the boot CD.

sh-4.2# ...

Since we want to work directly with the broken system, we will chroot to the mounted FS.

# chroot /mnt/sysimage

Check the state of the boot directory:

# ls -la /boot

At this point if the boot partition was corrupted, you could run either parted, gdisk or fdisk to recreate the partition.  You could also run fsck to run a filesystem check.

In my case /boot was fine, but empty.


At this point we need to re-install the kernel... However, the kernel version installed is later than the one on the installation CD.  There are two different things we can do:
  1. Install the old kernel from the CD or,
  2. Start the network and install the latest kernel from yum.
NOTE: Once the kernel is reinstalled through RPM or YUM, the installation triggers dracut which re-generates the necessary initramfs files.

RE-INSTALL THE KERNEL FROM THE BOOT CD: (skip if you want to use yum and the network)

Mount the boot CD to the /run/install/repo directory:

# mount /dev/sr0 /run/install/repo

# rpm -ivh --force /run/install/repo/Packages/kernel-<...version and arch...>

RE-INSTALL THE KERNEL FROM THE NETWORK: (skip if you re-installed the kernel from the CD already)

Luckily the network configuration is sound so we can simply start the network device and use yum to reinstall the kernel:

# service network start

Run a yum clean all just in case.

# yum clean all

Reinstall the kernel.

# yum reinstall kernel


Run ls -la /boot to verify the /boot directory and you should see the new kernel and associated files listed.  Most of the /boot directory's missing files and directories will be created.  One key portion that will be missing is Grub2.

# ls -la /boot

So we now need to reinstall grub2 and to recreate the configuration.  This process is fairly simple.

Install grub2 on /dev/  -- in my case on a KVM it's /dev/vda

# grub2-install /dev/vda

If no errors were reported, you are ready to reconfigure grub (otherwise you'll need to troubleshoot why you can't write to your device.):

# grub2-mkconfig -o /etc/grub2.cfg

(While the real grub2.cfg file is actually in the /boot/ partition, /etc/grub2.cfg is a symlink and easier to reference - especially if you are using UEFI.  If you are using UEFI, the grub2.cfg filename is actually /etc/grub2-efi.cfg -> ../boot/efi/EFI/centos/grub.cfg)

 Next, since you are in a chroot shell, you need to exit before you can reboot:

# exit

# reboot

In theory,  your system should now be able to boot just fine, but the SeLinux relabeling will have been triggered and may take some time to complete.  Once it is done your system will reboot automatically one more time.

If you had multiple kernels installed, but chose to fix this system by installing the base one from the CD, you can install your version again by running:

# yum reinstall kernel-

If you don't know which kernels you had previously installed, you can get the version from the rpm query command:

# rpm -q kernel

Thats it.

Tuesday, September 12, 2017

CVE-2017-1000251 - bluetooth vulnerability

Given the scope of this vulnerability, its probably a good idea to disable bluetooth until all devices are patched.

On RH based systems:

Mask the service just in case, this will prevent another systemd unit from attempting to load it.

# systemctl mask bluetooth.service

Stop the service if it is running:

# systemctl stop bluetooth.service

Monday, September 11, 2017

ssh-chat - irc-like chat client over SSH

For a long time I've been using and maintaining an active "talk" client/server on one of my systems in order to be able to communicate and collaborate securely over SSH with whoever I needed to.

Unfortunately this is not a perfect solution for many reasons.  I've been thinking of setting up a local IRC server but there are weaknesses with that as well.

Recently I found an interesting project on github, created by a very ingenious programmer who goes by the alias of shazow.  His project, written in Go: ssh-chat.

It uses the go libraries for most of the SSH client/server code, but it adds a custom terminal which has a similar look and feel as IRC.

It's very well written and requires very little work to get it up and running.  The only thing I did was install it on a small KVM server and got it to start up automatically.

Here is the systemd file I created for it, located at:



Description=SSH-CHAT service

ExecStart=//ssh-chat/ssh-chat -i //.ssh/id_rsa


A couple of issues with this solution:

1) While it is an interesting idea, I need to keep an eye out on the golang SSH client/server libraries to make sure security vulnerabilities are kept at bay.

2) Keep in mind that like 'talk,' the conversations on the local server are not necessarily encrypted and could potentially be captured if the server is compromised.

Apart from these reservations, I really like this client and will look into it further as a potential solution.  

Wednesday, September 6, 2017

Compiling minetest on CentOS 7

HOW TO: The default gcc version of CentOS 7.3.1611 is gcc 4.8.5.  Minetest 0.4.16 requires a minimum of gcc 4.9 to compile.


Minetest is probably one of the best Open Source games of all times.  The project website is

While minetest can be compiled on many different OS'es, its easier on some than on others...  I found it easiest to compile on Fedora - all the libraries are readily available and the compilers are cutting edge.

This post doesn't go into resolving dependencies, rather it is a step-by-step document describing compiling minetest with an appropriate compiler.  The default gcc version of CentOS 7.3.1611 is gcc 4.8.5.  Minetest 0.4.16 requires a minimum of gcc 4.9 to compile:

Regarding dependencies, I think there are enough articles on the net that describe how to get what you need.  I will say this however: The third party repos that I normally use on CentOS are: Epel and RPMFusion.  Once these two are installed, getting the required dependencies is as easy as running either: yum search or yum provides */


Attempting to run cmake on the minetest project, renders:

"Insufficient gcc version, found 4.8.5.  Version 4.9 or higher is required. "

So, here is what we do in CentOS 7:

- Install the Software Collections "devtoolset-6"
- Provide CMAKE with the devtoolset compiler locations

Installing the "software collections" repositories,

# yum install centos-release-scl*

Enable the collection repository that we need:

# yum-config-manager --enable rhel-server-rhscl-7-rpms

Install SCL devtoolset-6 to get a newer version of GCC and G++.  

# yum install devtoolset-6

Go into your minetest build folder and enable the toolset's version of bash:

# scl enable devtoolset-6 bash

Unfortunately, since cmake is not included in the toolsets, we need to tell the base one where to find the right compilers, otherwise it tries to use the system's default.

The only method that I found which worked, was:

$ CXX=/opt/rh/devtoolset-6/root/usr/bin/g++ CC=/opt/rh/devtoolset-6/root/usr/bin/gcc cmake . -DRUN_IN_PLACE=TRUE ...

Now, you can exit the devtoolset-6 bash shell and compile normally.  Cmake generated all the necessary information to use the correct compilers regardless of your environment:

$ exit

$ make -j <# cpus>

Happy compiling!

P.S. Remember that when you run scl enable devtoolset-6 bash, you are in a new bash session with

Thursday, August 31, 2017

Upgraded from Fedora 25 to 26

I've just upgraded my Fedora 25 workstation to Fedora 26.  This is the same system that I had previously upgraded from 23 to24, then 24 to 25... So far, things are working out well and I have not noticed any issues directly related to the upgrade.

Kudos to the Fedora team.

Tuesday, August 29, 2017

Choosing a safe encryption algorithm for SSH on CentOS

Choosing a stronger encryption algorithm for SSH, than the default:

Regenerate a new host key using the ed25519 algorithm (ed25519 uses Curve25519 which has a high safety rating) 

ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519

# vim /etc/ssh/sshd_config

Comment all HostKey lines, except for the key using ed25519:

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

Restart the sshd service:

systemctl restart sshd

Sunday, August 27, 2017

A word about sealert

The sealert command can be run both as a CLI or GUI program.  However, when you want to run it from the CLI, it is necessary to specify the path using the -a switch.

For example, here is the result if you run sealert from a TTY:

# sealert
could not attach to desktop process

On the other hand, if you specify the file to scan:

# sealert -a /var/log/audit/audit.log

You will get the expected result.  If any failures were in the logs, they will show up with an analysis, similar to:


SELinux is preventing /opt/brother/Printers/mfcj485dw/cupswrapper/brcupsconfpt1 from execute access on the file /etc/

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that brcupsconfpt1 should be allowed execute access on the file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
allow this access for now by executing:
# ausearch -c 'brcupsconfpt1' --raw | audit2allow -M my-brcupsconfpt1
# semodule -i my-brcupsconfpt1.pp

Additional Information:
Source Context                system_u:system_r:cupsd_t:s0-s0:c0.c1023
Target Context                unconfined_u:object_r:ld_so_cache_t:s0
Target Objects                /etc/ [ file ]
Source                        brcupsconfpt1
Source Path                   /opt/brother/Printers/mfcj485dw/cupswrapper/brcups
Source RPM Packages           mfcj485dwlpr-1.0.0-0.i386
Target RPM Packages           glibc-2.17-157.el7_3.5.x86_64
Policy RPM                    selinux-policy-3.13.1-102.el7_3.16.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     SOMENAME
Platform                      Linux SOMENAME 3.10.0-514.26.2.el7.x86_64 #1 SMP
                              Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64
Alert Count                   20
First Seen                    2017-08-21 18:04:39 EDT
Last Seen                     2017-08-21 18:04:42 EDT
Local ID                      9851dcdd-6b59-4310-8e26-573219f32e7e

Raw Audit Messages
type=AVC msg=audit(1503353082.145:499): avc:  denied  { execute } for  pid=14664 comm="brmfcj485dwfilt" path="/etc/" dev="dm-0" ino=146770715 scontext=system_u:system_r:cupsd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:ld_so_cache_t:s0 tclass=file

type=SYSCALL msg=audit(1503353082.145:499): arch=i386 syscall=lgetxattr per=400000 success=no exit=EACCES a0=0 a1=22671 a2=1 a3=2 items=0 ppid=14608 pid=14664 auid=4294967295 uid=4 gid=7 euid=4 suid=4 fsuid=4 egid=7 sgid=7 fsgid=7 tty=(none) ses=4294967295 comm=brmfcj485dwfilt exe=/opt/brother/Printers/mfcj485dw/lpd/brmfcj485dwfilter subj=system_u:system_r:cupsd_t:s0-s0:c0.c1023 key=(null)

Hash: brcupsconfpt1,cupsd_t,ld_so_cache_t,file,execute