Notes on UNIX Pipes

• A pipe call creates two file descriptors - read and a write
• $ ls -l|more if write from ls is too much to handle by more, the write waits till read drains it.
• If there is nothing to read, the pipe read waits.
• Pipe is a kernel resource and is basically just a buffer.
• A write to a read-closed pipe or vice-versa would not work. We get a SIGPIPE.
• lseek( ) does not work on unidirectional pipes.

How shell commands piping work?

• A process has its own local file descriptor table. This table keeps track of files opned by the process and their FDs.
  Using dup() system call, we can create a new entry in the process FDT as following:

FD	filename
10	/tmp/xx

After calling dup(), we get a new entry with a new FD.

FD	Filename
10	/tmp/xx
11	/tmp/xx

Both the entries map to same slot in the global file table. It is equivalent to a file softlink. Now, we can close the fd 10 and use fd 11 to continue ops or keep both of them open.

dup() returns the lowest available FD in the process. To exploit this fact, we can close standard FDs (0,1,2) and the next call to dup() would return these FDs.

int fd[2];
int p = pipe(fd);
close(1); // frees up fd 0
dup(fd[1]);
// dup will return the lowest available FD, one for fd[1]. 

FD	File name
10	/tmp/xx
1	/tmp/xx

Now any write that was supposed to go to FD 1, would go to file /tmp/xx

Written with StackEdit.

Notes on Signals in UNIX

• Pressing a key generates an interrupt and kernel has a keyboard interrupt handler module. The KB interrupt handler would send a signal to all processes associated with the terminal.
• Once a signal handler is called, it is deleted from memory. So we call it recursively.

    void handler()
    {
        signal(SIGINT, handler);
    }

• Common Signals to know:
  • SIGSYS: Incorrect usage of a system call
  • SIGCHLD
  • SIGALRM
• SIGALRM is also used in sleep() call. Sleep call is just waiting infinitely for a signal. To create such wait, pause() function is used.

Written with StackEdit.

Notes on Linux Process Management

• All processes have a PID and a group ID. The group leader is your shell (terminal).
• So for a process to qualify as background, we change the process group usingsetgrp().
• nohup ./a,out &
• chmod 4750 file1.txt
  • 4 means SUID permission. An ordinary user can run this file with privileges of the actual owner of the file1.txt. Useful to allow an ordinary user to run commands accessible only to e.g. root.
    $chmod u+s file

Written with StackEdit.

Notes on exec( ) system call

• The new process shared old process’s file descriptor table
• A printf() used before exec might not work because its buffers were not flushed.
  • Use fflush().
• exec’ed process too gets access to environ.

Written with StackEdit.

Notes on fork()

• We can avoid orphans by Parent process calling wait( ) call.

wait(int *p)
*  p will have the return code of the child. So we can find
   if process was terminated normally or not. 

• Child process has its own copy of globals too.
• The file descriptor table of the parent is shared with child. It has FDs of all open files in the parent.
• fread is buffered read (defaults to 1024 bytes at a time). We call fflush to empty buffer. This is more efficient as write/reads are in-memory till 1024 bytes.
• read is a low level byte wise read op. There is no buffering.

The following code writes the string twice to the file. Why?

int main()
{
  char *p = "hello world";
  FILE *fp;
  fp = fopen("test", "w");
  fwrite(p, sizeof(p), 1, fp);
  fork();

• fork() also copies environment variables of the parent to the child. Type $set to access env vars.

Written with StackEdit.

dyld: Library not loaded: /usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/Python

I started getting this error after trying to install macvim as follows:

brew install macvim --override-system-vim

The error string is as following:

$ vi linkedlist.cc
dyld: Library not loaded: /usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/Python
  Referenced from: /usr/local/bin/vim
  Reason: image not found
Abort trap: 6

I checked shared libs for vim using otool.

$ otool -L /usr/local/bin/vim
/usr/local/bin/vim:
 /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
 /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
 /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
              (compatibility version 45.0.0, current version 1561.40.112)
 /usr/local/opt/lua/lib/liblua.5.3.dylib 
               (compatibility version 5.3.0, current version 5.3.4)
 /usr/local/opt/perl/lib/perl5/5.26.1/darwin-thread-multi-2level/CORE/libperl.dylib
               (compatibility version 5.26.0, current version 5.26.1)
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
 /usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/Python 
              (compatibility version 3.6.0, current version 3.6.0)
 /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
               (compatibility version 150.0.0, current version 1452.23.0)
 /usr/local/opt/ruby/lib/libruby.2.5.dylib (compatibility version 2.5.0, current version 2.5.1)
 /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
 /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices 
                (compatibility version 1.0.0, current version 822.31.0)
 /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation 
                         (compatibility version 300.0.0, current version 1452.23.0)

It appears that many shared libs versions are updated incorrectly.

To solve the problem, I tried to upgrade packages.

$ brew update
Already up-to-date.

$ brew upgrade

It started working after doing upgrades.

Written with StackEdit.

Ceph Outage with OSDs Heartbeat failure on Hammer (0.94.6)

Symptoms

• The cluster went down after 24 OSDs were added and marked in simultaneously.
• This was an erasure coded (10+5) RGW cluster on Hammer.
• All the OSDs started failing and eventually 50% of the OSDs were down.
• Manual efforts to bring them up failed and we saw heartbeat failures in OSDs log.
• All OSD were consuming ~15G RAM and OSDs were hitting Out of memory errors.

2018-07-18 08:58:12.794311 7f4aa0925700 -1 
osd.127 206901 heartbeat_check: 
no reply from osd.55 since 
back 2018-07-18 08:45:13.647493 
front 2018-07-18 08:45:13.647493 
(cutoff 2018-07-18 08:57:12.794247)

2018-07-18 08:58:12.794315 7f4aa0925700 -1 osd.127 206901
 heartbeat_check: no reply from osd.57 since back
  2018-07-18 08:45:42.452510 front 2018-07-18
   08:45:42.452510 (cutoff 2018-07-18 08:57:12.794247)

2018-07-18 08:58:12.794321 7f4aa0925700 -1 osd.127 206901
 heartbeat_check: no reply from osd.82 since back 
 2018-07-18 08:45:13.647493 front 2018-07-18 
 08:45:13.647493
  (cutoff 2018-07-18 08:57:12.794247)

• OSDs maps were out of sync

2018-07-18 08:56:52.668789 7f4886d7b700  
0 -- 10.33.49.153:6816/505502 >> 10.33.213.157:6801/2707
 pipe(0x7f4a4f39d000 sd=26 :13251
  s=1 pgs=233 cs=2 l=0 c=0x7f4a4f1b8980).connect
   claims to be 10.33.213.157:6801/1003787 not 
   10.33.213.157:6801/2707 - wrong node!   

• An OSD has ~3000 threads, most of them in sleeping state.
• Using GDB and getting a backtrace of all threads we found that most of the active threads were just Simple Messanger Pipe readers.
• We were suspecting a memory leak in Ceph code.

Band-aid Fixes

• Set norebalance, norecover, nobackfill

• Adding swap memory to OSDs

• Tuning heartbeat interval

• Tuning OSD map sync and setting noout, nodown to let OSDs sync their maps.

$ sudo ceph daemon osd.148 status
{
    "cluster_fsid": "621d76ce-a208-42d6-a15b-154fcb09xcrt",
    "osd_fsid": "09650e4c-723e-45e0-b2ef-5b6d11a6da03",
    "whoami": 148,
    "state": "booting",
    "oldest_map": 156518,
    "newest_map": 221059,
    "num_pgs": 1295
}

• Tuning OSD map cache size to 20
• Finding processes other than Ceph
  • Processes consuming network, CPU, and RAM
  • Killing them
• Starting OSDs one by one - that worked for us :-)

RCA

• The major culprit was a rogue process that was consuming massive network bandwidth on OSD nodes.
• As network bandwidth was not enough, many messenger threads were just waiting.
• The Simple Messanger threads are sync threads and would wait till they get through.
• That is one of the reasons of an OSD having ~3000 threads and consuming ~15G of memory.
• As network was saturated, OSDs heartbeat signals too were blocked and they were either committing suicide or dying of OOM.

References

• OSD Map Sync tips: https://www.mail-archive.com/ceph-users@lists.ceph.com/msg10187.html

Written with StackEdit.

s3cmd SSL connection error

Problem Statement

 File "/usr/lib/python2.7/httplib.py", line 1263, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python2.7/ssl.py", line 363, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 611, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 840, in do_handshake
    self._sslobj.do_handshake()
error: [Errno 0] Error

Environement

• Debian 9
• s3cmd 2.0.1

Solution

The problem happens due to SSL issue. To make s3cmd work, we should invoke it without SSL (–nossl) as following.

s3cmd --no-ssl get --access_key=5DTA7J1ORIQ3E7LMV9YD 
--secret_key=GIqPAez7zdHSC9r3HsMNOgJlHqHttvGi
 --host=10.xx.xx.xxx:80 --host-bucket=10.xx.xx.xx:80
 s3://TESTBUCKET/abc.tgz

References

• https://sourceforge.net/p/s3tools/mailman/message/34389949/

MySQL & Python- Error: 2006 mysql has gone away

MySQL & Python: Error: 2006 mysql has gone away

This problem occurs for multiple reasons such as DB connection problem. In our code, we hit this issue due to a subtle problem with DB cursor.

The code was as following:

with conn as cur:
    try:
        print "hello"
    except:
        print "sorry"
    finally: 
        conn.close()

The above code would throw the error 2016 mysql has gone away exception. The problem lies in with conn as cur. This statement creates a cursor on the DB and the cursor object autmatically gets destroyed.
Here, we are closing the DB connection before the automatic destruction happened.

So since connection was invalid(closed), cursor deletion hit an exception.

The solution is to close the connection after cursor object deletion.

try:
    with conn as cur:
        print "hello"
except:
    print "sorry"
finally: 
    conn.close()

Written with StackEdit.

Building Ceph on Debian Jessie

1. Checkout Ceph with --recursive
2. Comment code that installs setuptools and sudo ./install-deps.sh
3. ./ceph/do_cmake.sh.
4. Switch to root using sudo su and run the following commands:

    echo "LC_ALL=en_US.UTF-8" >> /etc/environment   
        echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen 
        echo "LANG=en_US.UTF-8" > /etc/locale.conf
        locale-gen en_US.UTF-8

6. Now switch back to your login and create ~/.bash_profile:

    $ cat ~/.bash_profile  
    export LC_ALL=en_US.UTF-8  
    export LANG=en_US.UTF-8

7. Run source ~/.bash_profile
8. cd to ./ceph/build and run make -j 4

Notes on Ceph librados Client

Cluster Connection

• A client is an application that uses librados to connect to a Ceph cluster.

• It needs a cluster object populatd with cluster info (cluster name, info from ceph.conf)

• Then the client do a rados_connect and cluster handle is populated.

• A cluster handle can bind with different pools.

​

Cluster IO context

• The I/O happens on a pool so the connection needs to bind to a pool.

• The connection to a pool gives the client an I/O context.

• The client only species an object name/xattr and librados maps it to a PG & OSD in the cluster.

• An obhect write to rados require key, value, and value size.

• librados::bufferlist is primarily used for storing object value.

​

References

• http://docs.ceph.com/docs/luminous/rados/api/librados-intro/

​

​

Written with StackEdit.

How to secure pools in a Ceph cluster

We have three flags to set per pool: nopgchange, nodelete, and nosizechange

$ for pool in $(rados lspools); do ceph osd pool set $pool nopgchange true; done

Set application on all pools
for pool in $(rados lspools); do ceph osd pool application enable $pool rgw; done

Ceph radosgw: ERROR: endpoints not configured for upstream zone, meta sync: ERROR: failed to fetch mdlog info

The s3cmd client was getting error code 500 for bucket creation.

rgw logs were showing following errors:

2018-06-06 18:57:01.808147 7f0114b8d700  0 ERROR: endpoints not configured for upstream zone
2018-06-06 18:57:01.808164 7f0114b8d700  0 meta sync: ERROR: failed to fetch mdlog info
2018-06-06 18:57:31.808298 7f0114b8d700  0 ERROR: endpoints not configured for upstream zone
2018-06-06 18:57:31.808326 7f0114b8d700  0 meta sync: ERROR: failed to fetch mdlog info

This problem can occur with an improperly configured zone.

Solution

In my case I resolved it by running the following commands:
 
$ radosgw-admin zone modify --rgw_realm=my_realm --rgw-zonegroup=in --rgw-zone=north --master --default

$ radosgw-admin period update

$ radosgw-admin period commit

Notes from http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-July/011950.html

#!/bin/sh

set -x

RADOSGW_ADMIN=radosgw-admin

echo "Exercise initialization code"
$RADOSGW_ADMIN user info --uid=foo # exercise init code (???)

echo "Get default zonegroup"
$RADOSGW_ADMIN zonegroup get --rgw-zonegroup=default | sed 's/"id":.*/"id": "default",/g' | sed 's/"master_zone.*/"master_zone": "default",/g' > default-zg.json

echo "Get default zone"
$RADOSGW_ADMIN zone get --zone-id=default > default-zone.json

echo "Creating realm"
$RADOSGW_ADMIN realm create --rgw-realm=myrealm

echo "Creating default zonegroup"
$RADOSGW_ADMIN zonegroup set --rgw-zonegroup=default < default-zg.json

echo "Creating default zone"
$RADOSGW_ADMIN zone set --rgw-zone=default < default-zone.json

echo "Setting default zonegroup to 'default'"
$RADOSGW_ADMIN zonegroup default --rgw-zonegroup=default

echo "Setting default zone to 'default'"
$RADOSGW_ADMIN zone default --rgw-zone=default

How to delete a commit in Git

$ git rebase -p --onto SHA^ SHA

The SHA is the commit ID.

Reference 

How to write idempotent tasks in Ansible

Link

What is difference between rebalance and recovery in Ceph?

We set norebalance, noecover flags on a Ceph cluster during a cluster modification such as adding/removing OSDs. A cluster OSDs modification created a new mapping of PGs.

A new mapping of PG means:

• The PG might move to a new primary
• The PG might have a new secondary
• A PG might have an all new primary and secondary

The first scenario means that PG is hosted on a new OSD so all of its data must move, by PG backfill. Since it's a new primary, the secondary OSDs might have changes too. A new seocndary means that backfill would start on secondaty OSDs too.
So, setting recover means that we don't do backfill and rather serve data from old mappings (which might have become unavailable if OSDs was down)
If we set norebalance, that means that we would not do backfill of secondary PGs and would rather live with fewer copies of PGs.

How to set up ceph-volume OSDs with ceph-ansible

LVM Overview

• https://www.digitalocean.com/community/tutorials/an-introduction-to-lvm-concepts-terminology-and-operations
• https://docs.ansible.com/ansible/2.3/lvol_module.html

For a LVM OSD, we need to create a physical volume, volume group, followed by a set of logical volumes.

Seting up OSD in ceph-ansible:

Under ceph-ansible/group_var/osds.yml:
osd_scenario: lvm

lvm_volumes:
  - data: /dev/vdb
    db: /dev/vde1
    wal: /dev/vde2

  - data: /dev/vdc
    db: /dev/vde3
    wal: /dev/vde4

  - data: /dev/vdd
    db: /dev/vde5
    wal: /dev/vde6

You would get the following volumes after running Ansible:

$ sudo lsblk
NAME                                                                                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                                                                                                   254:0    0   10G  0 disk
`-vda1                                                                                                254:1    0   10G  0 part /
vdb                                                                                                   254:16   0  3.7T  0 disk
`-ceph--6f4ea188--1115--4b01--98ac--2c0d9d5033f2-osd--block--975e8e76--6db6--4331--becb--0ec800ce6700 253:0    0  3.7T  0 lvm
vdc                                                                                                   254:32   0  3.7T  0 disk
`-ceph--26bd1be1--1f95--4389--b1b4--3a2416e38419-osd--block--ffff5f01--8925--44c9--9019--5c43e7323f14 253:1    0  3.7T  0 lvm
vdd                                                                                                   254:48   0  3.7T  0 disk
`-ceph--b2cfeafb--3c45--4f0b--8c31--fba4203877da-osd--block--d9bf9af9--63ea--48eb--9d24--f27cdb868ddd 253:2    0  3.7T  0 lvm
vde                                                                                                   254:64   0  335G  0 disk
|-vde1                                                                                                254:65   0  100G  0 part
|-vde2                                                                                                254:66   0    5G  0 part
|-vde3                                                                                                254:67   0  100G  0 part
|-vde4                                                                                                254:68   0    5G  0 part
|-vde5                                                                                                254:69   0  100G  0 part
`-vde6                                                                                                254:70   0    5G  0 part

$ sudo lvm
lvm> lvdisplay
  --- Logical volume ---
  LV Path                /dev/ceph-26bd1be1-1f95-4389-b1b4-3a2416e38419/osd-block-ffff5f01-8925-44c9-9019-5c43e7323f14
  LV Name                osd-block-ffff5f01-8925-44c9-9019-5c43e7323f14
  VG Name                ceph-26bd1be1-1f95-4389-b1b4-3a2416e38419
  LV UUID                8ejEk9-Nmxk-CJcu-qrmP-krB6-RNcV-7tJmx7
  LV Write Access        read/write
  LV Creation host, time osd-f-6-1067165, 2018-05-23 11:13:02 +0530
  LV Status              available
  # open                 4
  LV Size                3.64 TiB
  Current LE             953861
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

How to access ceph.conf variable from ceph-ansible play

There is a command 'ceph-conf' that can be used inside Ansible play to access sections in ceph.conf on a remote host.

# ceph-conf -c /etc/ceph/ceph.conf --lookup  "bluestore block wal size" -s osd
2147483648

Ref
====
http://docs.ceph.com/docs/master/man/8/ceph-conf/

How to set config for all OSDs in a Ceph cluster

To set a config on all OSDs in a cluster:
$ sudo ceph tell osd.* injectargs -- --osd_recovery_max_active=8

To set a config on all Mons in a cluster:
$ sudo ceph tell mon.* injectargs -- --{conf option}

ceph-ansible: ceph-volume with lvm (Logical Volume Manager)

The command to create a ceph-volume for Bluestore OSD on a physical disk results in physical volume and followed by volume group creation. Then finally it creats a logical volume in the volume group.

$ vgcreate --force --yes ceph-7a1d0d7f-c6f4-468c-8fc4-3e74930ac1fd /dev/vdc