Showing posts with label development. Show all posts
Showing posts with label development. Show all posts

Monday, December 10, 2018

Why use Base64 Encoding?

##What is Base64 encoding?

  • Given a stream of binary bits, it will encode 6-bits to a character from a set of 2 pow 6 (64 chracters).
  • Example “abcd”, the ASCII representation is 65666768.
  • [1000001][1000010][1000011][1000100]
  • Base64 would pics six continuous bits
  • 100000|| 110000|| 101000|| 011100||0100xx here xx would be 00 (padding)
  • gwocQ

Why use base64 encoding?

  • Transferring binary data in URLs
  • Tranferring binary data such as images as text
  • Transmit and store text that might cause delimiter collision.
    • Example is a random string followed by a delimiter (_) and a pattern and the code logic searches the delimiter to seperate the pattern.
    • The _ can appear in the generated random string too.
    • So encoding the random string in base64 would avoid such case.
    • Embed image in a XML

Tuesday, December 4, 2018

Time based Key Expiry in Redis

https://redis.io/commands/expire
It is a useful feature to expire keys based on their last access time. We can use it to develop interesting feature such as rate limits,

There are various rate limiting implementations.
https://github.com/redislabsdemo/RateLimiter/tree/master/src/com/redislabs/metering/ratelimiter

Written with StackEdit.

Monday, December 3, 2018

Sample git conf

#### Put this in your ~/.gitconfig or ~/.config/git/config
[user]
	name = Your Full Name
	email = your@email.tld
[color]
	# Enable colors in color-supporting terminals
	ui = auto
[alias]
	st = status
	ci = commit
	lg = log --graph --date=relative --pretty=tformat:'%Cred%h%Creset -%C(auto)%d%Creset %s %Cgreen(%an %ad)%Creset'
	oops = commit --amend --no-edit
	review-local = "!git lg @{push}.."
	# Or pre 2.5, as we didn't differential push and upstream in shorthands:
	# review-local = lg @{upstream}..
[core]
	# Don't paginate output by default
	pager = cat
	#
	# Out of luck: on Windows w/o msysGit? You may have Notepad++…
	# editor = 'C:/Program Files (x86)/Notepad++/notepad++.exe' -multiInst -notabbar -nosession -noPlugin
	# 
	# If you want to use Sublime Text 2's subl wrapper:
	# editor = subl -w
	# 
	# Or Atom, perhaps:
	# editor = atom --wait
	# 
	# Sublime Text 2 on Windows:
	# editor = 'c:/Program Files (x86)/Sublime Text 2/sublime_text.exe' -w
	# 
	# Sublime Text 3 on Windows:
	# editor = 'c:/Program Files/Sublime Text 3/subl.exe' -w
	#
	# Don't consider trailing space change as a cause for merge conflicts
	whitespace = -trailing-space
[diff]
	# Use better, descriptive initials (c, i, w) instead of a/b.
	mnemonicPrefix = true
	# Show renames/moves as such
	renames = true
	# When using --word-diff, assume --word-diff-regex=.
	wordRegex = .
	# Display submodule-related information (commit listings)
	submodule = log
[fetch]
	# Auto-fetch submodule changes (sadly, won't auto-update)
	recurseSubmodules = on-demand
[grep]
	# Consider most regexes to be ERE
	extendedRegexp = true
[log]
	# Use abbrev SHAs whenever possible/relevant instead of full 40 chars
	abbrevCommit = true
	# Automatically --follow when given a single path
	follow = true
[merge]
	# Display common-ancestor blocks in conflict hunks
	conflictStyle = diff3
[mergetool]
	# Clean up backup files created by merge tools on tool exit
	keepBackup = false
	# Clean up temp files created by merge tools on tool exit
	keepTemporaries = false
	# Put the temp files in a dedicated dir anyway
	writeToTemp = true
	# Auto-accept file prompts when launching merge tools
	prompt = false
[pull]
	# This is GREAT… when you know what you're doing and are careful
	# not to pull --no-rebase over a local line containing a true merge.
	# rebase = true
	# WARNING! This option, which does away with the one gotcha of
	# auto-rebasing on pulls, is only available from 1.8.5 onwards.
	rebase = preserve
[push]
	# Default push should only push the current branch to its push target, regardless of its remote name
	default = upstream
	# When pushing, also push tags whose commit-ishs are now reachable upstream
	followTags = true
[rerere]
	# If, like me, you like rerere, uncomment these
	# autoupdate = true
	# enabled = true
[status]
	# Display submodule rev change summaries in status
	submoduleSummary = true
	# Recursively traverse untracked directories to display all contents
	showUntrackedFiles = all
[color "branch"]
	# Blue on black is hard to read in git branch -vv: use cyan instead
	upstream = cyan
[tag]
	# Sort tags as version numbers whenever applicable, so 1.10.2 is AFTER 1.2.0.
	sort = version:refname
[versionsort]
	prereleaseSuffix = -pre
	prereleaseSuffix = .pre
	prereleaseSuffix = -beta
	prereleaseSuffix = .beta
	prereleaseSuffix = -rc
	prereleaseSuffix = .rc

###

Reference https://gist.github.com/tdd/470582

Written with StackEdit.

Thursday, May 24, 2018

How to set up ceph-volume OSDs with ceph-ansible

LVM Overview

  • https://www.digitalocean.com/community/tutorials/an-introduction-to-lvm-concepts-terminology-and-operations
  • https://docs.ansible.com/ansible/2.3/lvol_module.html

For a LVM OSD, we need to create a physical volume, volume group, followed by a set of logical volumes.

Seting up OSD in ceph-ansible:

Under ceph-ansible/group_var/osds.yml:
osd_scenario: lvm

lvm_volumes:
  - data: /dev/vdb
    db: /dev/vde1
    wal: /dev/vde2

  - data: /dev/vdc
    db: /dev/vde3
    wal: /dev/vde4

  - data: /dev/vdd
    db: /dev/vde5
    wal: /dev/vde6


You would get the following volumes after running Ansible:

$ sudo lsblk
NAME                                                                                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                                                                                                   254:0    0   10G  0 disk
`-vda1                                                                                                254:1    0   10G  0 part /
vdb                                                                                                   254:16   0  3.7T  0 disk
`-ceph--6f4ea188--1115--4b01--98ac--2c0d9d5033f2-osd--block--975e8e76--6db6--4331--becb--0ec800ce6700 253:0    0  3.7T  0 lvm
vdc                                                                                                   254:32   0  3.7T  0 disk
`-ceph--26bd1be1--1f95--4389--b1b4--3a2416e38419-osd--block--ffff5f01--8925--44c9--9019--5c43e7323f14 253:1    0  3.7T  0 lvm
vdd                                                                                                   254:48   0  3.7T  0 disk
`-ceph--b2cfeafb--3c45--4f0b--8c31--fba4203877da-osd--block--d9bf9af9--63ea--48eb--9d24--f27cdb868ddd 253:2    0  3.7T  0 lvm
vde                                                                                                   254:64   0  335G  0 disk
|-vde1                                                                                                254:65   0  100G  0 part
|-vde2                                                                                                254:66   0    5G  0 part
|-vde3                                                                                                254:67   0  100G  0 part
|-vde4                                                                                                254:68   0    5G  0 part
|-vde5                                                                                                254:69   0  100G  0 part
`-vde6                                                                                                254:70   0    5G  0 part


$ sudo lvm
lvm> lvdisplay
  --- Logical volume ---
  LV Path                /dev/ceph-26bd1be1-1f95-4389-b1b4-3a2416e38419/osd-block-ffff5f01-8925-44c9-9019-5c43e7323f14
  LV Name                osd-block-ffff5f01-8925-44c9-9019-5c43e7323f14
  VG Name                ceph-26bd1be1-1f95-4389-b1b4-3a2416e38419
  LV UUID                8ejEk9-Nmxk-CJcu-qrmP-krB6-RNcV-7tJmx7
  LV Write Access        read/write
  LV Creation host, time osd-f-6-1067165, 2018-05-23 11:13:02 +0530
  LV Status              available
  # open                 4
  LV Size                3.64 TiB
  Current LE             953861
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

Wednesday, May 23, 2018

How to access ceph.conf variable from ceph-ansible play

There is a command 'ceph-conf' that can be used inside Ansible play to access sections in ceph.conf on a remote host.

# ceph-conf -c /etc/ceph/ceph.conf --lookup  "bluestore block wal size" -s osd
2147483648


Ref
====
http://docs.ceph.com/docs/master/man/8/ceph-conf/