Diagnosing Kernel Panics on CentOS 7

Sometimes I have had to diagnose problems with Kernel but it is so sporadic that I always forget the steps to take information to know what the problem comes from, so this is a little guide just to remember what I did the last time.

I also give some advice to prevent what it generated in my case kernel panics.

In my case when I run “yum update” and this update installs a new kernel version there are some possibilities to generate a kernel panic the next time I boot the machine, so the first advice would be rebooting the computer when a new kernel is installed.

The last time kernel panics was due to /boot partition was filled installing the new kernel and the installation process wasn’t end completely (for example there wasn’t ), I have to control the size of this partition and take note of increasing it in a future installation.
/dev/sda1 239M 197M 25M 89% /boot

Second advice it would be run the following previously to run a system update:
# package-cleanup --oldkernels --count=2

Well once kernel panics to debug the problem it’s needed to remove this kernel parameter (if present): rhgb quiet. This is made by editing the file /etc/default/grub, the original line to edit would be:
GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet"
New line:
GRUB_CMDLINE_LINUX="crashkernel=auto"

Then if you are using Grub2 on CentOS 7 like me you can run the following command:
# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...

The path could be different if you are using an UEFI system read HowTo Grub2 to more information.

The next time you boot with this kernel more log is shown so you can find more accurately the cause of this problem, you also can search the error message on Internet if you haven’t any clue about it.

Apart from removing the mentioned kernel options you add rdshell so a shell is shown if kernel cannot boot properly.
dracut:#

A common problem when a kernel fails could that is not included in initram file miss some driver to boot properly, like LVM driver when you have root filesystem over LVM, if it is the case you must recreate the initram file:
# dracut -f -v /boot/initramfs-2.6.32-504.el6.x86_64.img 2.6.32-504.el6.x86_64

As a last shot if you like debug core files when kernel panics a core file was generated in / directory so you can use gdb to see where the problem is:
# gdb /boot/vmlinuz-3.10.0-693.11.1.el7.x86_64 core.2912
...
(gdb) bt
#0 0x00007f158da191f7 in raise () from /lib64/libc.so.6
#1 0x00007f158da1a8e8 in abort () from /lib64/libc.so.6
...
(gdb) quit


“Civil disobedience becomes a sacred duty when the state has become lawless or corrupt”
— Mahatma Gandhi

Advertisements

The Interrogation of the Good

I don’t remember exactly when I read a comment about this poem but that time I thought that it is worth to share it so here it is.
“The Interrogation of the Good” by Bertolt Brecht, which Slavov Zizek quotes in Violence to mock the neoliberalism of American progressives and professionals.

The comment I read about this poem says that “Brecht’s poem goes far beyond Hannah Arendt in blaming the average person who thinks he’s apolitical, ‘just doing his job,’ for enabling political and societal breakdowns through complacence. I can’t help from thinking that the poem is something of a joke, but there’s also an unmistakable hint at the absolute hatred that people bear against one another when things fall apart. It has to be one of the most scathing condemnations of political apathy and shallow philanthropic liberalism ever written.”

Judge by yourself.

Step forward: we hear
That you are a good man.

You cannot be bought, but the lightning
Which strikes the house, also
Cannot be bought.
You hold to what you said.
But what did you say?
You are honest, you say your opinion.
Which opinion?
You are brave.
Against whom?
You are wise.
For whom?
You do not consider your personal advantages.
Whose advantages do you consider then?
You are a good friend.
Are you also a good friend of the good people?

Hear us then: we know.
You are our enemy. This is why we shall
Now put you in front of a wall. But in consideration
of your merits and good qualities
We shall put you in front of a good wall and shoot you
With a good bullet from a good gun and bury you
With a good shovel in the good earth.


“We cannot solve a problem by using the same kind of thinking we used when created them.”
— Albert Einstein

Keyboard and controller set up on RetroPie

This entry explains some issues that I found when I installed Retropie image on my Raspberry Pi 3 Model B and how I fixed them.

KEYBOARD ON CONSOLE
To setup properly my Logitech K400r QWERTY keyboard with Spanish layout I have followed generic Debian GNU/Linux instructions however RetroPie is based on Debian Jessie and it works pretty well:
# dpkg-reconfigure keyboard-configuration
# service keyboard-setup restart

It is not enough restarting keyboard setup you can use:
# udevadm trigger --subsystem-match=input --action=change

You can find your setup in this file:
# cat /etc/default/keyboard

Just a little trick here because with the default keyboard layout I wasn’t able to find some keys just to type the previous commands so I install gpm package which enables a mouse cursor in text console and with help of tab key I could copy characters printed on screen and paste them using right mouse button once selected.
# aptitude install gpm

SCRAPING FROM COMMAND LINE
I had problems scraping new ROM games from graphical user interface so I need use a command to be able to scrap properly the last roms which I upload to Retropie and the command to fix this was:
# /opt/retropie/supplementary/scraper/scraper -scrape_all -thumb_only -workers 4
Here is the link to read more instructions.


SETUP PS3 CONTROLLER WITH BLUETOOTH CONNECTION

There are some ways to setup PS3 controller but I am going to follow the simplest one which is using RetroPie UI. First of all I setup a keyboard (it was mentioned earlier) you have to pay attention because there are some important keys that we need to remember. In my case I setup D-PAD buttons as my arrow keys, ‘Q’ key as Start button, ‘A’ as X button and ‘Z’ as back button.

Now we can use keyboard to navigate through Retropie UI, then select “Retropie Setup” select “Manage Packages” > “Manage driver packages” and selects “ps3controller” > “Install from source”.
Now it shows (installed) label in “ps3controller” so it’s time to exit from Retropie setup and enter in EmulationStation Menu using ‘Q’ key and select “Configure Input”, remember release PS3 controller from USB cable and you just map the key controller, I find useful the following picture from here
PS3 Controller keys

ABORT A GAME
Pressing a combination of buttons you can access to functions such as saving, exiting games… When I test some games and I decide that I don’t want to play more you can use HotKey + Select + Start (PS3 button in the middle of select and start button, on my PS3 console) to return to menu.
You can get more hotkeys from here


“To be independent of public opinion is the first formal condition of achieving anything great”
— G. W. F. Hegel

Commands to sysadmin duties on GNU/Linux

In this quick entry I want to summarize some commands which I have been using for a while when I have the system administrator role and I found them useful:

When you have to release space from a server you can look for bigger files:
$ du -hsx * | sort -rh | head -10
for i in G M K; do du -ah | grep [0-9]$i | sort -nr -k 1; done | head -n 11

To verify no accounts have empty passwords, in this case awk uses a condition to do the job:
# awk -F: '($2 == "") {print}' /etc/shadow

To make sure that no non-root accounts have UID Set To 0:
# awk -F: '($3 == "0") {print}' /etc/passwd

Print lines in the n-th /BEGIN/,/END/ range, not inclusive:
# awk -v n=4 '/END/{p=0}; p && c == n; /BEGIN/ && !p {p=1; c++}'

Look for multiple patterns in files (PX from patternX):
$ grep -E 'P1|P2|P3' foo.txt
$ awk '/P1|P2|P3/' foo.txt
$ awk '/P1/ || /P2/ || /P3/' foo.txt
$ sed '/P1/b; /P2/b; /P3/b; d' foo.txt

If order matters:
grep 'P1.*P2.*P3' foo.txt
If order does not matter:
$ awk '/P1/ && /P2/ && /P3/' file.txt

When /START/ is seen, start collecting lines. Each time an /END/ is seen (and /START/ was previously seen), print what we have so far, empty the buffer and start collecting lines again, in case we see another /END/ later.
Here’s an awk solution for the inclusive case:
$ awk '!ok && /START/ { ok = 1 }
ok { p = p sep $0; sep = RS }
ok && /END/ { print p; p = sep = "" }' foo.txt

The ps is very useful command and it has interesting flags one of them is “–sort” as you expect order metric passed in the second parameter, “rss” in this example:
# ps aux --sort -rss|head -5
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www-data 31771 7.2 1.2 8557572 186180 ? Sl Nov25 68:49 /usr/sbin/apache2 -k start
www-data 31230 2.1 0.8 8494032 122548 ? Sl Nov25 20:10 /usr/sbin/apache2 -k start
www-data 8725 0.8 0.6 8474156 100556 ? Sl Nov25 8:33 /usr/sbin/apache2 -k start
www-data 7222 0.4 0.5 370236 75432 ? S 08:00 0:02 php-fpm: pool p

If you remove “-” from “rss” param, the output order is reverted:
# ps aux --sort rss|head -5
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S Nov25 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Nov25 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S Nov25 0:00 [rcu_sched]

Other pretty handy flag is “-o” with ps command can be useful to calculate median memory for a specific command:
# ps --no-headers -o "rss,cmd" -C COMMAND | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"Mb") }'
And removing NR from the previous command you can get the total amount of memory:
# ps --no-headers -o "rss,cmd" -C COMMAND | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/1024,"Mb") }'

Argument passed to awk:
awk 'BEGIN{print "ARGC is " ARGC; for(i = 0; i < ARGC; i++) print "ARGV["i"] is " ARGV[i]}' foo bar

Copying remote data commands from this link:
$ scp -3 -r remote1:/src/dir remote2:/dest/dir
The same using tar:
ssh remote1 'tar -C /src/dir/ -cvzf - .' | ssh remote2 'tar -C /dest/dir/ -xzvf -'

(the -t option to SSH is to force a pseudo-tty allocation, otherwise we couldn’t be asked for the password):
$ ssh -t -R10000:remote2:22 remote1 'rsync -e "ssh -l root -p 10000" -avz /src/dir/ localhost:/dest/dir/'

The previous command is the same as:
$ ssh -R10000:remote2:10000 remote1
$ ssh -R10000:remote2:22 remote1
remote1$ rsync -e 'ssh -l root -p 10000' -avz /src/dir/ localhost:/dest/dir/

I always forget loop syntax on Bash so here a remainder for people like me:
for (( i = 0; i < 10; i++ )); do echo $i; done
While loop:
i=0
while [[ $i -lt 10 ]]; do echo $i; let i++; done

For loop using seq:
for i in $(seq 0 9); do echo $i; done
Using brace expansion:
for i in {0..19}; do echo $i; done

Backup a file using brace expansions:
cp file{,.`date +%Y%m%d`}


“Once you have tasted flight, you will forever walk the earth with your eyes turned skyward, for there you have been, and there you will always long to return.”
– Leonardo da Vinci

Cryptography: openssl scenarios

I want to summarize some examples of using openssl as tool to manage certificates. If you are interested in a good introduction about cryptography you can check this link gpgtools.

Let’s start with some basic uses:
– Create our own Certificate Authority (CA):
$ openssl genrsa -des3 -out root-ca.key 2048
$ openssl req -new -x509 -days 3650 -subj "/C=ES/ST=Zaragoza/O=Home/CN=localhost.local" -key root-ca.key -out root-ca.crt

– If you want to check a CA certificate content:
$ openssl x509 -noout -text -in root-ca.crt

You may want to change /etc/ssl/openssl.conf file to point the new CA certificate:
[ ca ]
default_ca = CA_custom
[ CA_custom ]
...
certificate = $dir/root-ca.crt # The CA certificate
private_key = $dir/private/root-ca.key # The private key

– Create a self-signed certificate using one command, remember if your are creating a server certificate Common Name line has to match with server FQDN.
# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/server-selfsigned.key -out /etc/ssl/certs/server-selfsigned.crt

– Generate private key with AES128 algorithm, this key is not signed by anybody:
$ openssl genrsa -aes128 2048 > foo.key
– Creating a Certificate Signing Request (CSR) for a private key:
$ openssl req -new -key foo.key -out foo_key.csr
– Review CSR content:
$ openssl req -in foo_key.csr -noout -text
– CA signing a CSR:
$ openssl ca -in foo_key.csr -out foo.crt

– Getting certificates from a server, this command was taken from here:
$ openssl s_client -showcerts -connect shipit.ubuntu.com:443 ("level" c ".crt")}
/---END CERTIFICATE-----/{inc=0}'

– Viewing information about downloaded certificates:
$ for i in level?.crt; do openssl x509 -noout -serial -subject -issuer -in "$i"; echo; done

– Create a strong Diffie-Hellman group, to negotiate Perfect Forward Secrecy with clients:
# openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

– Installing server certificate in Nginx /etc/nginx/snippets/selfsigned.conf:
ssl_certificate /etc/ssl/certs/server-selfsigned.crt;
ssl_certificate_key /etc/ssl/private/server-selfsigned.key;
ssl_dhparam /etc/ssl/certs/dhparam.pem;

To setup a more secure (strong encryption) settings check the following link out. Also take a look to this link HSTS
/etc/nginx/sites-available/default
...
server {
# SSL configuration
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
include snippets/self-signed.conf;

# nginx -t
# systemctl restart nginx

– SSL settings in server certificate in Apache2:
/etc/apache2/sites-available/default-ssl.conf
SSLEngine on
SSLCertificateFile /usr/local/apache/conf/ssl.crt/server-selfsigned.crt
SSLCertificateKeyFile /usr/local/apache/conf/ssl.key/server-selfsigned.key
SSLOpenSSLConfCmd DHParameters "/etc/ssl/certs/dhparam.pem"

To strong encryption settings to Apache2 check this link.

– Exporting as PKCS12 certificate from PEM format, in case you this format:
$ cat foo.key foo_key.csr > foo.pem
$ openssl pkcs12 -export -in foo.pem -out foo.p12 -name foo

– Exporting certificate in DER format:
$ openssl x509 -in foo.pem -outform DER -out foo.der

– Start simple OSCP server check link for more information:
$ openssl ocsp -index index.txt -CA root-ca.key -rsigner root-ca.key -rkey root-ca.key -port 3456
Waiting for OCSP client connections...

– Hashing: calculating Message Digests with SHA algorithm:
$ openssl dgst -sha1 foo
SHA1(foo)= 9f43d756fa00e241dd614728f5e13461bfc8dde1

– Encode text using base64:
$ openssl base64 -in foo
bGFsYWxhbAo=


“If you know the enemy and know yourself, you need not fear the result of a hundred battles.”
— Sun Tzu, (Art of War)

Fixing time out waiting for LVM2 devices booting CentOS 7

After my last system update I started to experiment troubles in the booting process, my server couldn’t to finish the process with any of kernel images installed on CentOS.
I checked booting with the rescue image which hadn’t problems detecting LVM2 devices but appeared the same time out with RAID device.
Fortunately the rescue image finished properly because /, /home and the remain partitions were correctly mounted, except the partition used by the backup software related with RAID device (previously I removed RAID device entry from /etc/fstab).
I tested changing some kernel parameters at booting time as it was suggested in some web pages, but without success.
A little bit frustrated after a lot of reboot attempts and finally I fixed the problem rebuilding all initial ramdisks using the following command:
# dracut --regenerate-all -fv --mdadmconf --fstab --add=mdraid --add-driver="raid1 raid10 raid456"

Then I entered again the RAID device entry in fstab file, and I rebooted the server and I checked that the booting process works like a charm with all kernel images, included the rescue one.

References:

Good judgement is the result of experience … Experience is the result of bad judgement.
— Fred Brooks

Installing CyanogenMod 12.1 on Nexus 7 (2012 version) (a.k.a. “grouper”)

This entry is just to remember steps that I have followed to install CyanogenMod 12.1 on my Nexus 7 (2012).
This would be the previous point to be able to make my tablet backups using BackupPC (you can check this link Nexus 7 & BackupPC).
The following information is extracted from this guide.
First of all, some packages need to be installed in the PC which is going to be used to connect with your tablet, in my case:
# aptitude install android-tools-adb android-tools-fastboot
These packages contain “adb” and “fastboot” command required to complete device flashing process.

Once tablet is connected to PC and to have enabled USB debugging option and be enabled Developer Options explained here it executes this command to boot in fastboot mode:
$ adb reboot bootloader
$ fastboot devices
XXXXXXX fastboot

XXXXXXX -> Device serial number
The next command is used to unlock the bootloader:
$ fastboot oem unlock
After this point to re-enable USB debugging it’s needed to be able to continue.

Now it’s time to install  custom recovery image, to know more about recovery images visit this page

You need to download the image to be installed, for me it was: https://dl.twrp.me/grouper/twrp-2.8.7.0-grouper.img

  • Connect the device to computer via USB (as we did in an earlier step)
  • Execute these commands:

$ adb reboot bootloader
$ fastboot devices
XXXXXXX  fastboot
$ cd Downloads && fastboot flash recovery twrp-2.8.7.0-grouper.img
sending 'recovery' (11850 KB)...
OKAY [  1.764s]
writing 'recovery'...
OKAY [  0.397s]
finished. total time: 2.160s

  • Reboot the device into recovery just to check the recovery image is correctly installated.

Now let’s continue with CyanogenMod installation from custom recovery mode, the first step would be to download the build package for your tablet. I decided to use a release version because is the stable package version (cm-12.1-20151117-SNAPSHOT-YOG7DAO1KA-grouper.zip).
I also downloaded this file open_gapps-arm-5.1-nano-20160809.zip from here.

  • Place the build package, zip file, on the root of /sdcard/ using this command:

$ cd ~/Downloads && adb push cm-12.1-20151117-SNAPSHOT-YOG7DAO1KA-grouper.zip /sdcard/
$ adb push open_gapps-arm-5.1-nano-20160809.zip

  • Boot to recovery mode (Team Win Recovery Project), for my device is needed to hold Volume Up, Volume Down and the Power button, to be able to flash CyanogenMod image.

It’s highly recommended create a backup before flashing the image zip file. After that, select Wipe and then Factory Reset, then select Install option navigate to /sdcard and select cm-12.1-20151117-SNAPSHOT-YOG7DAO1KA-grouper.zip file, follow the messages printed on screen. To install Open GApps image follow the same process.

Just you need to reboot Nexus 7 and check if CyanogenMod is running properly.

I want to thank to “The CyanogenMod Team”, its Community and “The Open GApps Team” for their effort, thanks guys!

Some acronyms related with this topic:

  1. OEM: Original Equipment Manufacturer
  2. OTA: Open Tools API
  3. USB: Universal Serial Bus

References:

Enjoy!


‘I understand that readers do not need to know these things; but I need to tell them.’
— Rousseau.

Set up RAID 1 (Mirroring) on CentOS 7

This entry explains steps which I followed to set up storage to BackupPC software.

Obviously the right choice to set up storage settings depends on your hardware and your needs. In my case I decided to use my HP ProLiant MicroServer Gen8 G1610T as my backup server and I wanted to be safe enough to avoid data lost due to hardware failures, so I decided set up 2 internal hard disks Western Digital Red as RAID 1 device.

To prevent the mistake which I made I would recommend that you read carefully the specifications from your hardware and even make some checks with dd’s…

After installing CentOS 7 on my HP server I read the following blog entry:

The Gen8 model’s 4 bays are split — Bays 1 and 2 SATA3 6Gbps, while Bays 3 and 4 are SATA2 3Gbps.

Unfortunatelly I discovered these specifications too late when OS was already installed in /dev/sda and the two disks used to RAID device were located in second and third bay, which means different speed rates so RAID device will work to the lowest transfer speed (I guess, I am not measure I/O speed in RAID device).

I haven’t much free time so I decided assume this issue (and not reinstall my server), because after all is a home server, but I learnt an important lesson, you need to know perfectly well your hardware. With all of this the steps I followed were these:

Selecting physical devices:
/dev/sdb
/dev/sdc

Let’s create partitions:
# fdisk /dev/sdb
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Welcome to fdisk (util-linux 2.23.2).
First step create partitiions:
Command (m for help): m
...
o create a new empty DOS partition table
---
Building a new DOS disklabel with disk identifier 0x9f64c2f4.
The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-3907029167, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-3907029167, default 3907029167):
Using default value 3907029167
Partition 1 of type Linux and of size 1.8 TiB is set
Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'
Command (m for help): p
Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x9f64c2f4
Device Boot Start End Blocks Id System
/dev/sdb1 2048 3907029167 1953513560 fd Linux raid autodetect

The same previous step to /dev/sdc device. Next step:
# mdadm -E /dev/sd[b-c]
/dev/sdb:
MBR Magic : aa55
Partition[0] : 3907027120 sectors at 2048 (type fd)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 3907027120 sectors at 2048 (type fd)
# mdadm -E /dev/sd[b-c]1
mdadm: No md superblock detected on /dev/sdb1.
mdadm: No md superblock detected on /dev/sdc1.

Create a RAID device:
# mdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/sd[b-c]1
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Check RAID device:
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Mon Feb 29 23:57:11 2016
Raid Level : raid1
Array Size : 1953382464 (1862.89 GiB 2000.26 GB)
Used Dev Size : 1953382464 (1862.89 GiB 2000.26 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Feb 29 23:58:17 2016
State : clean, resyncing
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Resync Status : 0% complete
Name : g8.acme:0 (local to host g8.acme)
UUID : 20d65619:0ed9ba74:36f94bc0:6fddc56e
Events : 13
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
# mdadm -E /dev/sd[b-c]1
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 20d65619:0ed9ba74:36f94bc0:6fddc56e
Name : g8.acme:0 (local to host g8.acme)
Creation Time : Mon Feb 29 23:57:11 2016
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
Array Size : 1953382464 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906764928 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=48 sectors
State : active
Device UUID : 4d9290ae:994f8d57:602be8b6:73edb241
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 29 23:59:02 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 7fcc116a - correct
Events : 22
Device Role : Active device 0
Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 20d65619:0ed9ba74:36f94bc0:6fddc56e
Name : g8.acme:0 (local to host g8.acme)
Creation Time : Mon Feb 29 23:57:11 2016
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
Array Size : 1953382464 (1862.89 GiB 2000.26 GB)
Used Dev Size : 3906764928 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=48 sectors
State : active
Device UUID : b0eb0220:52f87062:dcbf7a6a:75028466
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 29 23:59:02 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d485bd04 - correct
Events : 22
Device Role : Active device 1
Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

Reviewing the RAID configuration:
# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[1] sdb1[0]
1953382464 blocks super 1.2 [2/2] [UU]
[====>................] resync = 23.3% (456830464/1953382464) finish=193.4min speed=128915K/sec
bitmap: 12/15 pages [48KB], 65536KB chunk
unused devices:

Format the raid devices with journal file system:
# mkfs.ext4 /dev/md0
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
122093568 inodes, 488345616 blocks
24417280 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2636120064
14904 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information:

If you want to determine if a given device is a component device or a raid device you can execute the following commands:
# mdadm --query /dev/md0
/dev/md0: 1862.89GiB raid1 2 devices, 0 spares. Use mdadm --detail for more detail.
# mdadm --query /dev/sdb1
/dev/sdb1: is not an md array
/dev/sdb1: device 0 in 2 device active raid1 /dev/md0. Use mdadm --examine for more detail.

List array lines:
# mdadm --detail --scan
ARRAY /dev/md0 metadata=1.2 name=g8.acme:0 UUID=20d65619:0ed9ba74:36f94bc0:6fddc56e

At this point I decided to set up LVM2 over the RAID device, the choice was made more for curiosity than for a technical reason.

Create Physical Volume using RAID1 array:
# pvcreate /dev/md0
WARNING: ext4 signature detected on /dev/md0 at offset 1080. Wipe it? [y/n]:
WARNING: ext4 signature detected on /dev/md0 at offset 1080. Wipe it? [y/n]: y
Wiping ext4 signature on /dev/md0.
Physical volume "/dev/md0" successfully created

Check Physical volume attributes using pvs:
# pvs
PV VG Fmt Attr PSize PFree
/dev/md0 lvm2 --- 1.82t 1.82t
/dev/sda2 centos lvm2 a-- 424.00g 4.00m

Check Physical Volume information in detail using pvdisplay command:
# pvdisplay
--- Physical volume ---
PV Name /dev/sda2
VG Name centos
PV Size 424.01 GiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 108545
Free PE 1
Allocated PE 108544
PV UUID YnaiKQ-Yz9Z-UUUN-H9aa-XLRq-AT1m-7y8wqh
"/dev/md0" is a new physical volume of "1.82 TiB"
--- NEW Physical volume ---
PV Name /dev/md0
VG Name
PV Size 1.82 TiB
Allocatable NO
PE Size 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID mf9XlE-QDIs-7Xz3-qCHH-fXok-GclK-yLDfzR

Create volume group named raid1 using vgcreate command:
# vgcreate raid1 /dev/md0
Volume group "raid1" successfully created

See Volume group attributes using vgs command:
# vgs
VG #PV #LV #SN Attr VSize VFree
centos 1 3 0 wz--n- 424.00g 4.00m
raid1 1 0 0 wz--n- 1.82t 1.82t

See volume Group information in detail using vgdisplay:
# vgdisplay
--- Volume group ---
VG Name centos
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size 424.00 GiB
PE Size 4.00 MiB
Total PE 108545
Alloc PE / Size 108544 / 424.00 GiB
Free PE / Size 1 / 4.00 MiB
VG UUID ZRYdVb-NmBJ-Z6Mp-NTVo-QklF-Qy7r-OCJRr2
--- Volume group ---
VG Name raid1
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 1.82 TiB
PE Size 4.00 MiB
Total PE 476899
Alloc PE / Size 0 / 0
Free PE / Size 476899 / 1.82 TiB
VG UUID vnJGO0-g8iT-MJTo-wMVh-Zxck-ITRh-jD26H8

Logical Volume Creation using lvcreate.
# lvcreate -L 100G raid1 -n lvm0
Logical volume "lvm0" created.

View the attributes of Logical Volume:
# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home centos -wi-ao---- 200.00g
root centos -wi-ao---- 200.00g
swap centos -wi-ao---- 24.00g
lvm0 raid1 -wi-a----- 100.00g

View Logical Volume information in detail:
# lvdisplay
--- Logical volume ---
LV Path /dev/centos/root
LV Name root
VG Name centos
LV UUID RtvhTy-m8Ra-xOJ2-cxPB-ruEK-4jmC-BGA8lB
LV Write Access read/write
LV Creation host, time localhost, 2015-04-17 18:24:27 +0200
LV Status available
# open 1
LV Size 200.00 GiB
Current LE 51200
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
--- Logical volume ---
LV Path /dev/centos/home
LV Name home
VG Name centos
LV UUID p4ahvC-3Y0I-yblG-xzC0-6dJI-hDk3-PJuOt8
LV Write Access read/write
LV Creation host, time localhost, 2015-04-17 18:24:31 +0200
LV Status available
# open 1
LV Size 200.00 GiB
Current LE 51200
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2
--- Logical volume ---
LV Path /dev/centos/swap
LV Name swap
VG Name centos
LV UUID EphLec-154b-jvIY-4MAf-uAnV-4XYe-GffUKQ
LV Write Access read/write
LV Creation host, time localhost, 2015-04-17 18:24:35 +0200
LV Status available
# open 2
LV Size 24.00 GiB
Current LE 6144
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:1
--- Logical volume ---
LV Path /dev/raid1/lvm0
LV Name lvm0
VG Name raid1
LV UUID xaUN5g-f4Yc-q0jV-NGJ4-lwrR-P0Bs-ZDz8fh
LV Write Access read/write
LV Creation host, time g8.acme, 2016-03-01 01:02:14 +0100
LV Status available
# open 0
LV Size 100.00 GiB
Current LE 25600
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:4

Format lvm partition
# mkfs.ext4 /dev/raid1/lvm0
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
6553600 inodes, 26214400 blocks
1310720 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2174746624
800 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

To create mount point:
# mkdir /mnt/raid1

To mount logical volume:
# mount /dev/raid1/lvm0 /mnt/raid1
# lvmdiskscan
/dev/loop0 [ 100.00 GiB]
/dev/md0 [ 1.82 TiB] LVM physical volume
/dev/centos/root [ 200.00 GiB]
/dev/loop1 [ 2.00 GiB]
/dev/sda1 [ 250.00 MiB]
/dev/centos/swap [ 24.00 GiB]
/dev/sda2 [ 424.01 GiB] LVM physical volume
/dev/centos/home [ 200.00 GiB]
/dev/mapper/docker-253:0-11667055-pool [ 100.00 GiB]
/dev/raid1/lvm0 [ 100.00 GiB]
/dev/sdd1 [ 438.50 GiB]
/dev/sdd2 [ 4.02 GiB]
/dev/sdd3 [ 23.07 GiB]
/dev/sdd5 [ 133.32 MiB]
/dev/sdd6 [ 23.50 MiB]
4 disks
9 partitions
0 LVM physical volume whole disks
2 LVM physical volumes

If you want to know about the physical volume in detail along with the drive participated with physical volume you can get all through this file.
# less /etc/lvm/lvm.conf

Edit /etc/fstab to permanent mount:
/dev/raid1/lvm0 /mnt/raid1 ext4 defaults 0 0

If you want to make availability tests you can manually force a fail in a physical device:
# mdadm /dev/md0 --fail /dev/sdb1
# mdadm --detail /dev/md0

Interesting entries that I found investigating for this matter:


“Those who can imagine anything can create the impossible.”
— Alan Turing

How safely free space in /boot

I have to admit that I’m a newbie using CentOS, that was the point because I decided install it on my server to learn from practice. I’m get used to managing my Debian GNU/Linux box and there are things that I have to write just to remember them, that’s the reason for this entry.

Updating the system (yum update) I found the following message:

Transaction check error:
 installing package kernel-3.10.0-327.18.2.el7.x86_64 needs 30MB on the /boot filesystem

Error Summary
 -------------
 Disk Requirements:
 At least 30MB more space needed on the /boot filesystem.

The df command confirms the bad news:
# df -h
Filesystem               Size  Used Avail Use% Mounted on
...
/dev/sda1                239M  235M     0 100% /boot
...

If you have some old kernel versions the easiest way to solve space problem in boot partition in pretty obvious, remove unused old versions, so let’s to list kernel versions installed in our system:
# rpm -qa kernel |sort -V
kernel-3.10.0-229.1.2.el7.x86_64
kernel-3.10.0-229.11.1.el7.x86_64
kernel-3.10.0-229.el7.x86_64
kernel-3.10.0-327.10.1.el7.x86_64

The same information will be got with this:
# yum list installed|grep ^kernel

Now we can use package-cleanup command to remove the old ones, in this case I decided to use --count=2 the leave the latest version and other one just in case.

# package-cleanup --oldkernels --count=2
...
Removed:
kernel.x86_64 0:3.10.0-229.el7         kernel.x86_64 0:3.10.0-229.1.2.el7         kernel-devel.x86_64 0:3.10.0-229.el7         kernel-devel.x86_64 0:3.10.0-229.1.2.el7
Complete!

# df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/sda1 239M 134M 88M 61% /boot
...

As yum update wanted to install a new kernel version, we’ll have to repeat the same package-cleanup to leave just two kernel versions in our system once system update have finished properly.

“Be yourself; everyone else is already taken.”
-Oscar Wilde

Quick fix to BackupPC start on CentOS 7

A reader had commented that he had a problem when BackupPC service started and some actions needed to be done to fix the service start, this comment was related to this previous entry, if you get this message when the service is started, you could try the following fix:
$ /usr/share/BackupPC/bin/BackupPC_serverMesg status info
Can't connect to server (unix connect: No such file or directory)

First of all is needed to find the script which controls the service:
# systemctl status backuppc
● backuppc.service - SYSV: Starts and stops the BackupPC server
Loaded: loaded (/etc/rc.d/init.d/backuppc; static; vendor preset: disabled)
Active: active (running) since Sun 2016-05-29 21:26:39 CEST; 14min ago
Docs: man:systemd-sysv-generator(8)
Process: 3410 ExecStart=/etc/rc.d/init.d/backuppc start (code=exited, status=0/SUCCESS)

So let’s edit /etc/rc.d/init.d/backuppc script, and we have to add the bold line:
$ cd /etc/rc.d/init.d/
# cp backuppc backuppc.OLD.`date +%Y%m%d`
start() {...
echo -n "Starting BackupPC: "
[ ! -d /var/run/BackupPC ] && mkdir /var/run/BackupPC; chown backuppc.backuppc /var/run/BackupPC;
...

It’s time to restart the service and check its status:

# systemctl stop backuppc
# systemctl start backuppc
$ /usr/share/BackupPC/bin/BackupPC_serverMesg status info
Got reply: %Info = ("ConfigLTime" => "1464549999","poolFileCntRep" => 0,"cpoolFileCntRep" => 0,"DUDailyMaxReset" => 0,"cpoolFileCnt" => 0,"cpoolFileCntRm"...

At this point service starts correctly but, in my case, when server is restarted the service is not automatically up, let’s dig deeper to find what’s going on. If we check if the service is enabled we get this:

# systemctl is-enabled backuppc
static

But what “static” exactly means? From this page (netsuso comment) we could read the following explanation:

Static units are those which cannot be enabled/disabled, but it doesn’t mean they are always executed. They will only if another unit depends on them, or if they are manually started.
Actually, static units are simply those without an [Install] section. As enabling units means just creating a symlink to wherever [Install] mandates, those units without [Install] section cannot be enabled, as systemctl doesn’t know where to place the symlink.
Of course, you can still manually create a symlink from a static unit to (for instance) /etc/systemd/system/multi-user.target.wants/, and it will be executed as any other enabled unit. But I suppose static units are not intended to be enabled in that way, and most probably you shouldn’t need to do it wink

So, what happens if we change service descriptor and we add [Install] section:

# cp /etc/systemd/system/backuppc.service /etc/systemd/system/backuppc.service.OLD.`date +%Y%m%d`
# vim /etc/systemd/system/backuppc.service
...
[Install]
WantedBy = multi-user.target

With this modification we try to enable the static unit:
# systemctl enable backuppc
Created symlink from /etc/systemd/system/multi-user.target.wants/backuppc.service to /etc/systemd/system/backuppc.service.

It’s time to reboot our server and check if unit is automatically started. Now the service should start without any problems.

That’s all folks!

“Simplicity is the ultimate sophistication”
— Leonardo da Vinci