Technology


24
Oct 12

My Debian installation tips

So, after some time with Arch Linux, I’ve moved to Debian testing. Debian testing seems to strike a nice balance between being new and not breaking things (Debian stable on the other hand has really old packages). Compared to Arch, upgrades have thus far been a lot less painful. Arch, while lovely, has a tendency to break things and I was always rather annoyed with python being python3 rather than python2 like the rest of the world expected it to be. I miss the minimal base install from Arch though.

First steps

I used the amd64 (e.g. x86-64) weekly testing build rather than the stable build, because, well it gets me on testing without needing to do a stable install first and comes with more up-to-date packages.

While none of the docs make it clear, all you need is the first CD (or the netinstall build) to start the install.

Edit sudoers

su -c "visudo"

At the end of the file, add:

yourusername ALL=(ALL) NOPASSWD: ALL

Copy ssh keys over from old machine

Update and install packages

Add “contrib” and “non-free” after “main” to /etc/apt/sources.list – this is needed for stuff like the flash plugin.

If there are lines mentioning “cdrom:” as a source, delete or comment out those lines.

sudo apt-get update && sudo apt-get upgrade

sudo apt-get install git mlocate mercurial nitrogen wicd-gtk \
gnome-icon-theme htop unzip openssl chromium-browser zsh \
xterm epdfview mysql-server ruby tmux curl wget redis-server \
xcursor-themes xarchiver gzip bzip2 zip unzip unrar p7zip \
meld mplayer make gcc libc-dev build-essential openssl \
libssl-dev libx11-dev libxinerama-dev guake xfce4-notifyd \
keychain flashplugin-nonfree lxappearance tree

Install node

mkdir other
cd other
wget http://nodejs.org/dist/v0.8.12/node-v0.8.12.tar.gz
tar xvvf node-v0.8.12.tar.gz
cd node-v0.8.12
./configure && make && sudo make install

Install nwm, my window manager

git clone git://github.com/mixu/nwm.git
cd nwm
npm install

Create ~/nwm.sh and chmod +x:

#!/bin/sh
/usr/local/bin/node /home/m/nwm/nwm-user-sample.js 2> ~/nwm.err.log 1> ~/nwm.log

Create /usr/share/xsessions/nwm.desktop:

[Desktop Entry]
Encoding=UTF-8
Name=nwm
Comment=This session starts nwm
Exec=/home/m/nwm.sh
Type=Application

Set the user shell from bash to zsh

sudo chsh

Set the default xdg-open action for PDFs

In case you don’t know what xdg-open is – it’s the thing that decides what happens when you try to open a file (e.g. a downloaded file from Chrome).

I don’t want to open PDFs and PS files in Gimp, I want to open them in a document viewer

xdg-mime default evince.desktop application/pdf

xdg-mime default evince.desktop application/postscript

To query the default app: xdg-mime query default $(xdg-mime query filetype FILENAME)

Configuring appearance

Download https://wiki.ubuntu.com/Artwork/Incoming/DustTheme and unzip the theme to ~/.themes/

I find the easiest way to configure gnome2 is to use lxappearance, since it allows you to set the gnome theme and x11 cursors in a few clicks.

Guake: disable smart terminal names

gconftool-2 --set /apps/guake/general/use_vte_titles --type boolean false

Configure git

Edit ~/.gitconfig:

[color]
ui = true
[user]
email = 
name = 
[core]
editor = nano
[alias]
lg = log --graph --pretty=format:'%Cred%h%Creset-%C(yellow)%d%Creset %s %Cgreen(%cr)%C(bold blue)%Creset' --abbrev-commit --date=relative
[push]
default = current

Configure zsh

# Lines configured by zsh-newuser-install
HISTFILE=~/.histfile
HISTSIZE=1000
SAVEHIST=9999
unsetopt beep
bindkey -e
# End of lines configured by zsh-newuser-install
# The following lines were added by compinstall
zstyle :compinstall filename '/home/m/.zshrc'

autoload -Uz compinit
compinit
# End of lines added by compinstall

bindkey "\e[1~" beginning-of-line # Home
bindkey "\e[4~" end-of-line # End
bindkey "\e[5~" beginning-of-history # PageUp
bindkey "\e[6~" end-of-history # PageDown
bindkey "\e[2~" quoted-insert # Ins
bindkey "\e[3~" delete-char # Del
bindkey "\e[5C" forward-word
bindkey "\eOc" emacs-forward-word
bindkey "\e[5D" backward-word
bindkey "\eOd" emacs-backward-word
bindkey "\e\e[C" forward-word
bindkey "\e\e[D" backward-word
bindkey "\e[Z" reverse-menu-complete # Shift+Tab
# for rxvt
bindkey "\e[7~" beginning-of-line # Home
bindkey "\e[8~" end-of-line # End
# for non RH/Debian xterm
bindkey "\eOH" beginning-of-line
bindkey "\eOF" end-of-line
# for freebsd console
bindkey "\e[H" beginning-of-line
bindkey "\e[F" end-of-line
# for guake
bindkey "\eOF" end-of-line
bindkey "\eOH" beginning-of-line
bindkey "^[[1;5D" backward-word
bindkey "^[[1;5C" forward-word
bindkey "\e[3~" delete-char # Del

alias ls='ls --color=auto -F'
alias ga='git add'
alias gp='git push'
alias gl='git log'
alias gs='git status'
alias gd='git diff'
alias gdc='git diff --cached'

autoload -U colors && colors

PROMPT="%{$fg_bold[white]%}[%1~]%{$reset_color%} "

PATH=$PATH:$HOME/.rvm/bin # Add RVM to PATH for scripting
[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm"

gr() {
   local cmd="$*"
   for dir in /home/{FILL IN REPOSITORIES HERE}; do
    (
     print "\nin $dir"
     cd $dir
     eval "$cmd"
    )
   done
}

alias grs="gr git --no-pager status -sb"
alias grl="gr git --no-pager log --decorate --graph --oneline -n 3"
alias grd="gr git diff"
alias grdc="gr git diff --cached"
alias grn="gr npm ls"

eval `keychain --eval --agents ssh id_rsa`

# Stuff for git
parse_git_branch () {
    git branch 2> /dev/null | grep "*" | sed -e 's/* \(.*\)/ @\1/g'
}

# colors for prompt
BLACK=$'\033[0m'
RED=$'\033[38;5;167m'
GREEN=$'\033[38;5;71m'
BLUE=$'\033[38;5;111m'
YELLOW=$'\033[38;5;228m'
ORANGE=$'\033[38;5;173m'

function precmd() {
    export PROMPT="%{$fg_bold[white]%}[%1~%{$reset_color%}%{$YELLOW%}$(parse_git_branch)%{$fg_bold[white]%}]%{$BLACK%}%\ "
}

6
Apr 12

Git tips and tricks

This post is basically a collection of things I think I should remember about git: setup, commands, multiple repos, more obscure options etc.

Config

Two things: enable colors, and make git push push the current branch to a branch of the same name (and not all the branches, which is the default). This means when you do git push, it pushes your working branch but not other branches.

git config --global color.ui true
git config --global push.default current

Zsh/bash aliases

These are small things, but they make things so easy. I instinctively type gs after changing directory nowadays.

alias ga='git add'
alias gp='git push'
alias gl='git log'
alias gs='git status'
alias gd='git diff'
alias gdc='git diff --cached'
alias g='git'
git config --global alias.lg "log --graph 
--pretty=format:'%Cred%h%Creset
-%C(yellow)%d%Creset %s %Cgreen(%cr)
%C(bold blue)<%an>%Creset' --abbrev-commit --date=relative"

Apparently, the “git lg” above can be replicated fairly well with:

git log --decorate --graph --oneline

For more git log related discussion, see this HN thread.

Managing multiple git repositories

A fairly simple way of doing this is to add a function that you can use to run the same command in multiple directories. This one is for zsh:

gr() {
  local cmd="$*"
  for dir in /home/m/repos/{one,two,three}; do
   (
    print "nin $dir"
    cd $dir
    eval "$cmd"
   )
   done
}

alias grs="gr git --no-pager status -sb"
alias grl="gr git --no-pager log --decorate --graph --oneline -n 3"
alias grd="gr git diff"
alias grdc="gr git diff --cached"
alias grn="gr npm ls"

In the example below, I 1) switch all working repos to master, 2) fetch the recent remote changes, 3) check the status of every repository, and 4) after going into those couple of dirs and stashing stuff (not shown in detail), pull all repos to update them

gr git checkout master
gr git fetch
grs
in /home/m/repos/one
## master
in /home/m/repos/two
## master...origin/master [behind 8]
in /home/m/repos/three
## master...origin/master [behind 4]
 M Makefile
gr git pull

Add all changes or all changes to existing files

$ git add -A

Adds all files and file deletes, so you don’t need to enter them manually.

$ git add -u

Same, but doesn’t add files that didn’t exist before.

Log-fu

$ git log -p

.. to show the patches alongside the log

$ git fetch && \
git log --author="`git config --get user.name`"\
--pretty=format:'%h %an %Cred%ar %Cgreen%s' refactor..master
Shows the commits that you’ve made between “refactor” and “master”, e.g. for emailing a changelog with your commits:
6591466 Mikito Takada 7 months ago Port over more code
1f464fb Mikito Takada 7 months ago Refactor window
41a3ec9 Mikito Takada 9 months ago Handle switching from window to root window
$ git log staging -1 -p -- my/subdir

Show the latest commit that affects the my/subdir path on the branch “staging” with the patch.

$ git log --stat

Show the changed files only.

$ git log --author=foo

Only commits by author foo.

$ git log --stat -- subdirectory

Show the commits affecting only “./subdirectory”, with a summary of the files (–stat).

For example:

$ git log --stat -- .

Shows the summary of commits affecting the current directory.

$ git log --grep='.*foo.*'

Grep for a specific log message.

$ git log --since="1 week ago" --until="2 weeks ago"

Filter commits by date.

$ git shortlog -sn

Shows a “top contributors list” by commits.

Diff-fu

Differences between the staged files and the last commit:

$ git diff --cached

I use that a lot just before committing the changes to check what my commit changed.

So git diff is for the files that are unstaged, while git diff –cached is for staged files.

What’s different between my branch and some other branch, e.g. master?

$ git diff my_branch..master

What changed in the last three commits?

$ git diff HEAD^3

Ignore whitespace?

$ git diff HEAD^ --word-diff

What’s different between my branch and some other branch in subdirectory (or file)?

$ git diff my_branch..master subdirectory

Remember, git diff outputs an applicable patch:

$ git diff > my.patch
$ git apply < my.patch

Grep-fu

$ git grep -e foo --or -e bar --not -e baz

Grep for foo or bar, but exclude the expression baz.

$ git grep --untracked foo

Include untracked files in grep.

$ git grep --cached foo

Include cached (staged) files in grep.

$ git grep --no-index foo /var/bar

Use “git grep” instead of regular grep, grep for foo in /var/bar. Compared to regular grep, git grep is nicer due to the colors and paging defaults.

Creating a feature branch and merging it back

Checkout the branch you want to base your work on:

$ git checkout master

Then create a branch and check it out:

$ git checkout -b local_name

Push the branch to the remote

$ git push -u origin local_name

Later on, merge it:

$ git checkout master
$ git merge local_name

Look at the unmerged branches:

$ git branch --no-merged

Look at branches that contain a particular commit (e.g. when cherrypicking):

$ git branch --contains 1234abcd

Stashing changes

$ git stash
$ git stash list
$ git stash apply

Blaming

$ git blame path/to/file

See who changed what.

$ git blame abce1234^ path/to/file

See the blame, starting one commit before abce1234. E.g. when you need to trace back a particular line of code in history.

Working with remotes

Changing from Github to Bitbucket

$ git remote -v # look at remotes
$ git remove rm origin # delete
$ git remote add origin git://addr/to/repo # add new remote
$ git push -u origin master # push to new remote

Adding a upstream repo to pull in new changes from a forked project

$ git remote add upstream git://addr/to/repo
$ git checkout master && git pull upstream master

Keeping the number of merges lower on small commits

For small commits made on top of a frequently changing branch like master, you might want to rebase your local changes on top of the current remote before you merge a feature branch (more in depth):

$ git pull --rebase
$ git push

Resetting the current branch

When you want to reset to the current HEAD.

$ git reset --hard HEAD

When a branch tracking a remote has become outdated (e.g. you are on staging now but your commits have diverged):

$ git reset --hard origin/staging

Cherry-picking a commit

$ git cherry-pick sha1_of_commit

Reverting (unapplying) a bad commit:

$ git revert sha1_of_commit

Total number of remote branches

First, remove local branches that do not exist on origin (–dry-run if you want to preview):

$ git remote prune origin
$ git branch -r | wc -l
 123

Get the latest commit of all remote branches

Get the latest commit of all remote branches, and summary, ordered by age

Use this Node script:

https://gist.github.com/91ba46dee32b221b3a84

Delete a local or remote branch

Delete a local branch:

git branch -d some_branch_name

Really delete a local branch, even if git complains about it in the previous command:

git branch -D some_branch_name

Delete a remote branch, e.g. in Github:

 git push origin :some_branch_name

The reason for that syntax is that you can do `git push origin local_branch_name:remote_branch_name` so what that line above is doing is essentially pushing NULL to the remote branch, thereby killing it off.


17
Feb 12

Mechanical keyboards rock!

I’m a big believer in having the best tools possible for the job. I’ve gone through at least 3 Microsoft Ergonomic Keyboard 4000′s, and used to think those were the best keyboards ever (as did Jeff Atwood).

Well, there is whole world of keyboards that are even better, namely: buckling spring keyboards (e.g. the IBM Model M), Cherry MX switch keyboards and Topre switch keyboards.

I got this one for my place – Cherry MX brown switches, which are less noisy (though definitely they could be even more silent):

The major difference between a mechanical keyboard and a regular one is in the feel of the keys. Mechanical keyboards have metallic springs, while cheaper keyboards just have some flexible plastic.  After a short while typing on the MS Ergonomic keyboard started feeling like typing on a mushy piece of plastic compared to the mechanical keyboard. I do kind of miss the more ergonomic split layout, but typing is just a lot nicer overall on a mechanical keyboard.

I ended up buying two keyboards, both with Cherry MX switches (~110 USD from here). I didn’t get a buckling spring keyboard, because they are more noisy and there aren’t any space-saving tenkeyless models available. The third alternative – keyboards with Topre switches – are more expensive (~300+ USD) and have been described as less tactile. I’ll probably get a Topre keyboard eventually anyway, just so I can try it out.

Here is the new keyboard I use at work, which has Cherry MX blue switches (a bit more noisy than my other keyboard, but definitely one I prefer):

Best purchase in a long time!


22
Nov 11

Performance benchmarking Socket.io 0.8.7, 0.7.11 and 0.6.17 and Node's native TCP

I’ve been working with Socket.io quite a bit recently. It’s a great library. However, after upgrading to 0.8.x, I ran into problems with increased CPU usage. Since performance is very important for high traffic pubsub implementations, I decided to investigate this further – and try to quantify the performance impact of upgrading to a newer version of Socket.io.

I wrote a benchmarking suite (siobench). The benchmark is rather simple. Clients connect one at a time, and a new client is only allowed to connect when the previous one is connected. When the server has used up 5000 milliseconds of CPU time, the benchmark is stopped. Every second, every connected client sends a single message which is echoed back by the server (more details).

This workload is geared towards a situation where Socket.io is used to notify people of things as part of a larger application: e.g. most of the load is assumed to be idling connections rather than real-time messaging like in, say, a multiplayer game.

The “end of test” condition is 5000 ms of CPU time, because this seemed to be a easy way to give all implementations the same amount of time. CPU usage % is not accurate, since it is dependent on how much CPU time the process gets over a particular amount of wallclock time. In the graphs the CPU usage % calculated over a 100ms interval, while usertime and systime are the actual numbers reported at that particular time.

Summary

Node (0.4.12) using tcp ~ 8000 connections on a single core
socket.io 0.6.17 using websockets ~ 2300 connections on a single core
socket.io 0.7.11 using websockets ~ 1800 connections on a single core
socket.io 0.8.6 using websockets ~ 1900 connections on a single core

Remember, this is just one server on one core, with 5000 ms of CPU time on that core. The rest of the cores are used to generate sufficient load. The full graphs are at the end of the post.

Note that the absolute numbers are mostly unimportant – I ran this on the following 15″ Macbook Pro running Arch with the 3.1.04 Linux kernel in Virtualbox with 4096 Mb of RAM, a SSD and four cores (Intel(R) Core(TM) i7-2635QM CPU @ 2.00GHz GenuineIntel GNU/Linux). You can get numbers that are more representative of your system by getting siobench and running it:

Usage: node siobench.js [env]
A tool for benchmarking your Socket.io server.

Available environments:
	0.6.17
	0.6.17_poll
	0.7.11
	0.8.7
	0.8.7_poll
	tcp

You can also write your own benchmarks under ./bench, by writing a new server.js (example #1, #2) and a new client.js (example #1, #2). Each benchmark has it’s own set of npm dependencies installed, so that one can run benchmarks against many versions of socket.io.

Some notes on performance

The relative performance is more interesting.

First, the node TCP speed represents the highest achievable performance on this benchmark, since it only uses the built-in TCP implementation. Compared to this, Socket.io is has about 1/3 of the performance (~ 2300 vs ~8000 connections) when using WebSockets.

Second, it appears that 0.8.7 is about 20% slower than 0.6.17 on this benchmark. If I remember correctly, Socket.io 0.7 switched to a new protocol, and there are clearly some performance improvements over 0.7.11 in 0.8.7 (+100 connections in this bench); it’s just that the overall performance is still worse in this benchmark than in the old 0.6.17 branch.

Working towards higher-performance

As this is just a simple benchmark, I don’t really have solutions – only some suggestions.

1) A CI build that includes benchmarks and community contributed test cases

First, I’d love to see a CI build for Socket.io that would include performance benchmarks and community contributed test cases.

However, currently setting up a CI build for Socket.io is difficult because the bundled test suite only works on OSX. It would be a lot easier to contribute if the tests worked on other platforms.

I am hoping that as Engine.io gets going, the test suite will be fixed so that it can be run on other platforms. Otherwise, contributing improvements will be tricky/impossible since there is no way to tell whether the code works.

2) More realistic performance test scenarios

The current test scenario is rather limited in that it mostly tests performance in terms of establishing connections (without terminating them). I’d love to hear more realistic scenario suggestions, particularly from people who have run into memory usage issues.

siobench is only a starting point: it’s way better than just looking at htop and wondering whether performance was better in the last version or not. There are still specific questions that should be formulated as replicable tests.

3) A polling transport that works on Node.js

I did write tests for the xhr-polling transport for Socket.io as well. These showed much worse performance, around:

  • ~ 550 connections on Socket.io 0.6.17 (vs ~2300 using WS)
  • ~ 450 connections on Socket.io 0.8.7 (vs ~ 1900 using WS)

However, the xhr-polling is severely broken in that it stops connecting after 4-5 connections on Node v0.4.12. So I had to force each load generating client to only make four connections and then spawn a new load generating process to work around the problem. I wouldn’t vouch for the accuracy of the test with xhr-polling until the xhr-polling transport is fixed on Node when using socket.io-client (it’s been broken for the last three releases, though).

4) Comparative benchmarks

Hopefully, this will help with performance testing new releases of Socket.io and other Comet libraries. Since the plan is that Engine.io will allow people to work with a lower level than Socket.io, there might be new performance oriented versions, and it would be useful to see benchmarks for those. Re: the other Node.js pubsub frameworks: I can’t benchmark Faye, because it does not provide the right API out of the box, and Juggernaut uses Socket.io internally.

I’m going to use siobench it for internal testing to ensure that the pubsub implementation I am working on (built over Socket.io) will not have performance regressions.

The full graphs are below. Please leave comments and suggestions for improvements – I am hoping that the developer community around Socket.io can help in improving the performance going forward, kind of like what Mozilla did with “arewefastyet.com“.

Socket.io 0.6.17 – Websockets – CPU usage and time

 

Socket.io 0.6.17 – Websockets – resident set size


 

Socket.io 0.7.11 – Websockets – CPU usage and time

Socket.io 0.7.11 – Websockets – resident set size

Socket.io 0.8.7 – Websockets – CPU usage and time

Socket.io 0.8.7 – Websockets – resident set size

Node 0.4.12 – TCP – CPU usage and time

Node 0.4.12 – TCP – resident set size

 

 


10
Nov 11

My Arch Linux setup

This is mostly just a reminder for myself – but I always learn new things when I read how other people set up their system. Leave a comment if you have a tip – that’s how I learned about wicd-gtk :) . Oh, and install my window manager (tiling, written in C++ and node.js, configurable using Javascript).

First steps

  • do the basic arch setup first (or VMware, or Virtualbox)

Update the system (and install/setup sudo)

dhcpcd eth0 #if you did not add "interface=eth0" in rc.conf during setup
pacman -Syu

Fixes:

http://www.archlinux.org/news/initscripts-update-manual-intervention-required/

rm /etc/profile.d/locale.sh

http://www.archlinux.org/news/filesystem-upgrade-manual-intervention-required/

pacman -S filesystem --force
pacman -S sudo
vim /etc/sudoers # add yourself to sudoers
sudo vim /etc/pacman.conf # set SigLevel = Never TrustAll
sudo shutdown-r now

Install X11:

pacman -S xorg-server xorg-xinit xorg-utils xorg-server-utils xterm

If virtualized in VirtualBox, make copy-paste work first:

pacman -S virtualbox-archlinux-additions

Details

Create a new user

pacman -S zsh
useradd -m -g users -G audio,lp,optical,storage,video,games,power,scanner -s /bin/zsh USERNAME
su USERNAME
passwd

Add x11

pacman -S xorg-server xorg-xinit xorg-utils xorg-server-utils

Copy ssh keys over from old machine
Useful packages

pacman -S base-devel sudo python2 git libev mlocate mercurial nitrogen 
sakura wicd-gtk pcmanfm gnome-icon-theme htop unzip 
openssl chromium flashplugin bash-completion xterm 
epdfview mysql ruby tilda tmux wget redis xcursor-vanilla-dmz 
xarchiver gzip bzip2 zip unzip unrar p7zip 
meld ttf-ubuntu-font-family mpg123 alsa-utils redis mysql ruby libxslt
  • base-devel and python2 for compiling node
  • libev for nwm
  • mlocate for locate command
  • nitrogen is better than feh for multiple screens
  • sakura is a nice terminal
  • wicd-gtk is a simple wifi network gui
  • pcmanfm gnome-icon-theme are for pcmanfm, a Nautilus alternative

Remember to visudo and remove the password requirement from wheel. And add dbus and wicd to /etc/rc.conf just like pacman tells you to.
Configuring X11

Add ~/.Xresources:

Xcursor.theme: vanilla-dmz
Xcursor.size:  16       !  32, 48 or 64 may also be good values

Configuring git

git config --global color.ui true

Configuring sakura

I want to use ctrl + Page_Up / Page_Down to switch tabs, so edit ~/.config/sakura/sakura.conf:

switch_tab_accelerator=4 # since GDK_CONTROL_MASK is 1 << 2, e.g. 4.
prev_tab_key=Page_Down
next_tab_key=Page_Up

Some basic niceties: whatprovides, service and chkconfig

pacman -S pkgtools
  • pkgtools provides the pkgfile tool. It works like yum whatprovides (e.g. allows you to search for a particular command or dependency in all the pacman packages)
  • “sudo pkgfile -u” to update the db
  • “pgkfile zipinfo” to search for zipinfo

Arch don’t have a service and chkconfig, but we can make the new things curve a bit less steep by adding some functions to .bashrc:

function service() {
  sudo /etc/rc.d/$1 $2
}

alias chkconfig='cat /etc/rc.conf | grep DAEMONS && echo "cat + grep /etc/rc.conf"'

This makes service an alias for /etc/rc.d/ and prints out the enabled services from /etc/rc.conf. While we’re editing .bashrc, might as well add:

PS1="[W]$ " # my preferred bash prompt (e.g. only the current dirname).
ulimit -s 16400 # higher stack limit
# ssh-agent
SSH_ENV="$HOME/.ssh/environment"
function start_agent {
     echo "Initialising new SSH agent..."
     /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
     echo succeeded
     chmod 600 "${SSH_ENV}"
     . "${SSH_ENV}" > /dev/null
     /usr/bin/ssh-add;
}
# Source SSH settings, if applicable
if [ -f "${SSH_ENV}" ]; then
     . "${SSH_ENV}" > /dev/null
     #ps ${SSH_AGENT_PID} doesn't work under cywgin
     ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
         start_agent;
     }
else
     start_agent;
fi

Installing Node and NPM

You can just do:

pacman -S nodejs

If you’re OK with that version, which seems to track the Node releases pretty well.

Arch uses python3 as python. You need to change python to python2 (thanks Rob Searles!)

# node.js fix for arch (use python2) 
mkdir /tmp/bin
ln -s /usr/bin/python2 /tmp/bin/python
export PATH=/tmp/bin:$PATH

You can then do a regular node install:

git clone git://github.com/joyent/node.git

git checkout v0.4.12

./configure

make

sudo make install

Remember to install npm as well:

curl http://npmjs.org/install.sh | sudo sh

Installing my window manager and personal config

git clone git://github.com/mixu/nwm.git
cd nwm
node-waf clean || true && node-waf configure build
sudo npm link # add a global npm symlink to this repository - so nwm-user can find it (man npm link)
cd ..
git clone git://github.com/mixu/nwm-user.git
cd nwm-user
npm link nwm # now make a symlink to the nwm installation

Add it to ~/.xinitrc (change paths!!):

exec /usr/local/bin/node ~/mnt/nwm-user/nwm-user.js 2>~/nwm.err.log 1>~/nwm.log

And while we’re at it, lets add some other stuff:

VBoxClient-all &
export PATH=/tmp/bin:$PATH # for node-waf, too lazy to work on a better solution
xset +fp /usr/share/fonts/local
xset fp rehash

Run “startx” to start X11 with nwm.

Installing my mp3 player

First, we need to configure alsa (included by default):

pacman -S mpg123 alsa-utils

Run:

alsamixer

and turn on Master and PCM channels (by pressing m) as they are muted by default.

sudo alsactl store

Then continue:

git clone git://github.com/mixu/nplay.git

Run nplay with

node nplay.js

TODO: fix directory in source code and change backend from mpg321 to mpg123.

Switching the keyboard language in X11

I sometimes need to write emails in Finnish, so here is how to switch the layout:

setxkbmap -layout fi # revert back setxkbmap -layout us 

Install yaourt

Install dependencies

yaourt libpng12 gtk2-theme-dust

 

Install Sublime Text 2

Sublime Text needs libpng12, which you have to install from AUR:

wget http://aur.archlinux.org/packages/li/libpng12/libpng12.tar.gz
tar -xzvf libpng12.tar.gz
cd libpng12
makepkg
pacman -U ./libpng12-1.2.46-2-x86_64.pkg.tar.xz

Then download and run Sublime Text 2.

Also, you might want to ger http://aur.archlinux.org/packages/gt/gtk2-theme-dust/gtk2-theme-dust.tar.gz.

Configuring Sublime Text 2

locate Packages # returns ~/.config/sublime-text-2/Packages
cd ~/.config/sublime-text-2/Packages
git clone https://github.com/buymeasoda/soda-theme/ "Theme - Soda"
cd User
wget http://blog.mixu.net/files/2010/05/my_themes.zip # Install my themes
unzip my_themes
rm my_themes.zip

Base File settings

{
  // FONTS and COLORS
  "color_scheme": "Packages/User/Mixu Espresso.tmTheme",
  "font_size": 11,
  "tab_size": 2,
  // WHITESPACE
    // Set to true to insert spaces when tab is pressed
    "translate_tabs_to_spaces": true,
    "trim_automatic_white_space": true,
    "trim_trailing_white_space_on_save": true,
    // Set to false to disable detection of tabs vs. spaces on load
    "detect_indentation": false,
  "shift_tab_unindent": true,
  // Set to false to disable highlighting any line with a caret
  "highlight_line": true,
  // Set to "none" to turn off drawing white space, "selection" to draw only the
  // white space within the selection, and "all" to draw all white space
  "draw_white_space": "selection",
  // Set to true to ensure the last line of the file ends in a newline
  // character when saving
  "ensure_newline_at_eof_on_save": true,
  "fold_buttons": false
}

Global Settings

{
  "theme": "Soda Light.sublime-theme"
}

Other tweaks

I put these in my usual “startup” command, nwm-setup.sh and run it manually:

xrandr --output VBOX0 --auto --left-of VBOX1 # Virtualbox displays
chromium & # start chromium
export PS1="[W]$ "
xsetroot -cursor_name left_ptr # Set pointer
nitrogen --restore & # Restore desktop background using nitrogen

Other dependencies

pacman -S redis mysql ruby libxslt

Setting up ree:

cd
bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)
echo '[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" # Load RVM function' >> ~/.bashrc
source .bashrc
rvm install ree-1.8.7-2011.03

Installing REE will fail. You need torun the installer from /home/m/.rvm/src/ree-1.8.7-2011.03/installer manually:

./installer -no-tcmalloc

Then continue on:

rvm ree-1.8.7-2011.03 --default

13
Aug 11

Nginx, Websockets, SSL and Socket.IO deployment

I’ve spent some time recently figuring out the options for deploying Websockets with SSL and load balancing – and more specifically, Socket.IO – while allowing for dual stacks (e.g. Node.js and another dev platform). Since there seems to be very little concrete guidance on this topic, here are my notes – I’d love to hear from you on your implementation  (leave a comment or write about and link back)…

The goal here is to:

  1. Expose Socket.io and your main application from a single port — avoiding cross-domain communication
  2. Support HTTPS for both connections — enabling secure messaging
  3. Support the Websockets and Flashsockets transports from Socket.io — for performance
  4. Perform load balancing for both the backends somewhere — for performance

Socket.io’s various transports

Socket.io supports multiple different transports:

  • WebSockets — which are essentially long lived HTTP 1.1 requests, which after a handshake upgrade to the Websockets protocol
  • Flash sockets — which are plain TCP sockets with optional SSL support (but Flash seems to use some older SSL encryption method)
  • various kinds of polling — which work over long lived HTTP 1.0 requests

Starting point: Nginx and Websockets

Nginx is generally the first recommendation for Node.js deployments. It’s a high-performance server and even includes support for proxying requests via the HttpProxyModule.

However, — and this should be made much obvious to people starting with Socket.io — the problem is that while Nginx can talk HTTP/1.1 to the client (browser), it talks HTTP/1.0 to the server. Nginx’s default HttpProxyModule does not support HTTP/1.1, which is needed for Websockets.

Websockets 76 requires support for HTTP/1.1 as the handshake mechanism is not compatible with HTTP/1.0. What this means is that if Nginx is used to reverse proxy a Websockets server (like Socket.io), then the WS connections will fail. So no Websockets for you if you’re behind Nginx.

There is a workaround, but I don’t see the benefit: use a TCP proxy (there is a custom module for this by Weibin Yao, see here ). However, you cannot run another service on the same port (e.g. your main app and Socket.io on port 80) as the TCP proxy does not support routing based on the URL (e.g. /socket.io/ to Socket.io and the rest to the main app), only simple load balancing.

So the benefit gained from doing this is quite marginal: sure, you can use Nginx for load balancing, but you will still be working with alternative ports for your main app and Socket.io.

Alternatives to Nginx

Since you can’t use Nginx and support Websockets,  you’ll need to deal with two separate problems:

  1. How to terminate SSL connections and
  2. How to route HTTP traffic to the right backend based on the URL / load balance

If you want to run two services on the same port, then you will have to terminate SSL connections before doing anything else. There are several alternatives for SSL termination:

  • Stunnel. Supports multiple SSL certificates per process, does simple SSL termination to another port.
  • Stud. Only supports one SSL certificate per invocation, does simple SSL termination to another port.
  • Pound. An SSL-termination-capable reverse proxy and load balancer.
  • Node’s https. Can be made to do anything, but you’ll have to write it yourself.

If you choose Stunnel or Stud, then you need a load balancer as well if you plan on having more than one Node instance in the backend.

HAProxy is not generally compatible with Websockets, but Socket.IO contains code which works around this issue and allows you to use HAProxy. This means that the alternatives are:

  • Stunnel for SSL termination + HAProxy for routing/load balancing
  • Stud for SSL termination + HAProxy for routing/load balancing
  • Pound (SSL and routing/load balancing)

I haven’t looked into Pound more – mainly as I could not find info on it’s TCP reverse proxying capabilities (see the section on Flash sockets below), but it seems to work for these guys.

Setting up Stunnel

The Stunnel part is quite simple:

cert = /path/to/certfile.pem
; Service-level configuration
[https]
accept  = 443
connect = 8443

If you only have one Node instance, you can skip setting up HAProxy, since you don’t need load balancing.

Setting up HAProxy

Would you like Flash Sockets with that?

Note that we need TCP mode in order to support Flash sockets, which do not speak HTTP.

Flash sockets are just plain and simple TCP sockets, which will start by sending the following payload: ‘<policy-file-request/>’. They expect to receive a Flash cross domain policy as a response.

Since Flash sockets don’t use HTTP, we need a load balancer which is capable of detecting the protocol of the request, and of forwarding non-HTTP requests to Socket.io.

HAProxy can do that, as it has two different modes of operation:

  • HTTP mode – which allows you to specify the backend based on the URI
  • TCP mode – which can be used to load balance non-HTTP transports.

Main frontend

We accept connections on two ports: 80 (HTTP) and 8443 (Stunnel-terminated HTTPS connections).

By default, everything goes to the backend app at port 3000. Some HTTP paths are selectively routed to socket.io

TCP mode is needed so that Flash socket connections can be passed through, and all non HTTP connections are sent to the TCP mode socket.io backend.

# Main frontend
frontend app
  bind 0.0.0.0:80
  bind 0.0.0.0:8443
  # Mode is TCP
  mode tcp
  # allow for many connections, with long timeout
  maxconn 200000
  timeout client 86400000
 
  # default to webapp backend
  default_backend webapp
 
  # two URLs need to go to the node pubsub backend
  acl is_socket_io path_beg /node
  acl is_socket_io path_beg /socket.io
     use_backend socket_io if is_socket_io
 
   tcp-request inspect-delay 500ms
   tcp-request content accept if HTTP
   use_backend sio_tcp if !HTTP

Port 843: Flash policy

Flash policy should be made available on 843.

# Flash policy frontend
frontend flashpolicy 0.0.0.0:843
   mode tcp
   default_backend sio_tcp

Default backend

This is just for your main application.

backend webapp
   mode http
   option httplog
   option httpclose
   server nginx1s localhost:3000 check

Socket.io backend

Here, we have a bunch of settings in order to allow Websockets connections through HAProxy.

backend socket_io
  mode http
  option                  httplog
  # long timeout
  timeout server 86400000
  # check frequently to allow restarting
  # the node backend
  timeout check 1s
  # add X-Forwarded-For
   option forwardfor
  # Do not use httpclose (= client and server
  # connections get closed), since it will close
  # Websockets connections
  no   option httpclose
  # Use "option http-server-close" to preserve
  # client persistent connections while handling
  # every incoming request individually, dispatching
  # them one after another to servers, in HTTP close mode
  option http-server-close
  option forceclose
  # just one node server at :8000
  server node1 localhost:8000 maxconn 2000 check

Socket.io backend in TCP mode

This is the same server as above, but accessed in TCP mode.

backend sio_tcp
  mode tcp
  server node2 localhost:8000 maxconn 2000 check

Conclusion

The configs above allow you to serve Websockets, Flash and polling from a single port.

However, I am dissatisfied by the complexity of this configuration. In particular, Flash sockets’ TCP requirements are rather painful since they require protocol detection in order to work from a single port.

The alternative is of course to run Socket.io on a different port than your main app. This would mean that you configure HAProxy to just do TCP mode load balancing at that port, with SSL termination in front of HAProxy.

If you do that, you might want to configure a fallback from Nginx at port 80 to Socket.io for those clients who are behind draconian corporate firewalls which disallow ports other than 80 and 443. The fallback will only support long polling and I don’t think Socket.io itself supports automatically switching ports during transport negotiation, but you can detect a failure in Socket.io and re-initialize manually with a different port and polling-only transport.

Do you have a better way? How do you deploy Socket.io? Let me know in the comments below.

 


6
Aug 11

Collaborative git reference

Here is a basic reference for collaborative git commands:

Checkout a remote branch

# list branches first
git branch -a
# * master
# remotes/origin/branch_name
git checkout -b local_name remotes/origin/branch_name

Merging

# switch to master
git checkout master
# merge with experimental branch
git merge experimental

Create a tag

# list tags
git tag
# add a tag
git tag tagname
# push to remote
git push origin master --tags

 

 


11
Jun 11

dwm tips on Fedora

I’ve been testing out Fedora 15′s Gnome 3 and Ubuntu’s Unity, and didn’t like either of them. They both take up too much precious screen space just to show a fancy UI, and requiring hardware acceleration is a pain for low end netbooks and virtual machines.

So I decided to move to an alternative window manager. DWM (dynamic window manager, http://dwm.suckless.org/) is an extremely lightweight tiling window manager written in C which saves screen space and works pretty well as long as you don’t need to connect to wireless networks.

I’ve been pretty happy with it. The main drawback is that connecting to wireless networks is a pain in the ass as there are no proper GUI tools to do this. Check out these tips to get started with dwm

0. Installing dwm on Fedora, keyboard shortcuts

To install DWM, run yum install dwm. You can then choose to use dwm or Gnome or Kde in the login screen.

The default keyboard shortcuts are listed at man dwm or at http://man.suckless.org/dwm/1/dwm.

1. Customizing dwm

Customizing dwm can be done by making changes to config.h and recompiling the window manager.

Fedora has a really nice package called dwm-user, which automates this process! Here is the package description:

dwm-start is a helper script for running and reconfiguring dwm if neccessarry. It’s the preferred way of starting dwm in Fedora.
Running  dwm-start starts Fedora build by default. If you wish to customize your configuration, put the dwm config header file to $HOME/.dwm/config.h and adjust it according to your needs. Every time the user configuration file has changed, dwm-start will rebuild the user dwm binary prior to its execution.

All you need to do is:

sudo yum install dwm-user
mkdir ~/.dwm
cp /usr/src/dwm-user-5.8.2-6.fc14/config.def.h ~/.dwm/config.h

E.g. install via yum, then make a ~/.dwm folder, then copy the config.h file and edit it. When you restart, you can choose dwm-user as your window manager which uses you custom version of dwm. For example, I remapped Meta (Cmd/Windows key) + h and meta + l to meta + pg up / pg down and meta + shift + q to meta + shift + end since I’m currently running Fedora on an OSX host.

You will probably make changes to the keyboard shortcuts. To find the keymap:

sudo updatedb
locate keysymdef.h

Keysymdef.h lists the names of the keys in X11.

2. Tip: Guake is just as awesome on dwm

By default, dwm launches xterm. I prefer to use guake, since that allows me to get the tabbed terminal window on any workspace when I need them. Just launch guake& to run it in the background.

UPDATE: I moved to F15 (deciding simply to ignore Gnome 3) and noticed that guake has problems starting. To fix those:

sudo yum install xfce4-notifyd

Basically you need a notify daemon to allow Guake to print that pretty message “Guake is running”, and xfce4-notifyd provides an alternative notifications daemon.

3. Launch netbeans and other Java programs with font smoothing and GTK look and feel

You need to specify a couple of extra switches to get the GTK look and feel in Java programs, for example:

/home/username/netbeans-7.0/bin/netbeans -J-Dswing.aatext=true -J-Dawt.useSystemAAFontSettings=on –laf com.sun.java.swing.plaf.gtk.GTKLookAndFeel

4. Launch nautilus without the desktop

nautilus --no-desktop

5. Use dwm with a dual screen setup

If dwm starts with mirroring output to your secondary screen, then you need to run xrandr to get the names your screens. E.g. VBOX0 and VBOX1.

Then configure the screen layout:

xrandr --output VBOX1 --auto --right-of VBOX0

dwm will now let you have your own workspaces for each screen.

6. Change your desktop background

Use feh to change your desktop background:

feh --bg-tile /path/to/background/image

7. xterm config

For a usable xterm, create the following ~/.Xresources and run

xrdb -merge .Xresources
xterm*faceName:           monospace:pixelsize=14
xterm*saveLines:          9999
xterm*scrollBar:          false
xterm*background:  #000000
xterm*foreground:  #dfdfdf
xterm*color0:      #000000
xterm*color1:      #9e1828
xterm*color2:      #aece92
xterm*color3:      #968a38
xterm*color4:      #414171
xterm*color5:      #963c59
xterm*color6:      #418179
xterm*color7:      #bebebe
xterm*color8:      #666666
xterm*color9:      #cf6171
xterm*color10:     #c5f779
xterm*color11:     #fff796
xterm*color12:     #4186be
xterm*color13:     #cf9ebe
xterm*color14:     #71bebe
xterm*color15:     #ffffff

8. Add a clock using xsetroot

You can do something like this in a bash script to show the time in dwm on the top right corner.

while true; do
   xsetroot -name "$( date +"%F %R" )"
   sleep 1m    # Update time every minute
done

9. Connect to wifi

This is rather painful. The instructions here were collected from the mailing list, and I did get them to work, but I’m too lazy to write a full tutorial on this right now.

Basically, you need to scan, then do different things depending on whether the wifi uses WEP or WPA for authentication.

Start by running:

iwlist scan

9.1 WEP wifi

> > #wep connect to a wep wifi
> > #! /bin/sh
> >
> > key="`grep $1 /home/pmarin/wep | cut -d' ' -f2`"
> > sudo ifconfig wlan0 up
> > sudo iwconfig wlan0 essid $1
> > sudo iwconfig wlan0 key s:$key
> > sudo dhclient wlan0
> > #end
> > The wep is a plain file with to columms
> >
> > essid  key

9.2 WPA wifi

> >
> > #wpa connect to a wpa wifi
> > #! /bin/sh
> >
> > sudo ifconfig wlan0 up
> > sudo iwconfig wlan0 essid $1
> > sudo wpa_supplicant -iwlan0 -c/home/pmarin/wpa -B
> > sudo dhclient wlan0
> > #end
> >
> > the wpa file is similar than /etc/wpa_supplicant.conf

To create the wpa file:

wpa_passphrase your_ssid_of_network your_network_password
Create the file:
ctrl_interface=/var/run/wpa_supplicant
#ap_scan=2

network={
       ssid="your_ssid"
       scan_ssid=1
       proto=WPA RSN
       key_mgmt=WPA-PSK
       pairwise=CCMP TKIP
       group=CCMP TKIP
       psk=your_psk_from_wpa_passphrase
}
sudo wpa_supplicant -Bw -Dwext -i eth0 -c/etc/wpa_supplicant.conf

9.3 Wifi troubleshooting:

1) CHECK THAT YOU DON’T have the NetworkManager service or wpa_supplicant running already!!!

You can run wpa_supplicant with -dd flag for a detailed debug output.1) If you don’t manage to connect to the AccessPoint, try to uncomment line 2 in /etc/wpa_supplicant.conf.

2) If that doesn’t help, try change its value to 0 or 1.

3) If you get troubles while authenticating, try removing “RSN” and/or”CCMP” strings from /etc/wpa_supplicant.conf.

Sources for Wifi stuff:

http://ubuntuforums.org/showthread.php?t=263136

http://www.mail-archive.com/dwm@suckless.org/msg06800.html


24
Feb 11

Quick tip: Fix Flash audio stutter on Fedora 14 (64bit)

On my FC14 machine, I had a problem with Flash (64bit) audio playback: the sound on sites other than Youtube would stutter terribly. It appears that this a systematic problem; but luckily there is a fix!

Check out Ahmed Abdo’s post on Flash audio stutter for the details. Works perfectly for me!

Details: https://bugzilla.redhat.com/show_bug.cgi?id=638477

The bug is triggered by a change in glibc. Who proposed the fix? Linus Torvalds. So I guess the following isn’t quite true?

I love the pragmatism from his part:

So in the kernel we have a pretty strict “no regressions” rule, and that if people depend on interfaces we exported having side effects that weren’t intentional, we try to fix things so that they still work unless there is a major reason not to.

So I’m disappointed glibc just closes this as NOTABUG. There’s no real reason to do the copy backwards that I can see, so doing it that way is just stupid.

But whatever. You can do a LD_PRELOAD trick to get a sane memcpy(), and it does indeed fix the sound for me.
[...]
The fact that the glibc people don’t do that, and that this hasn’t been elevated despite clearly being a big usability problem (normal users SHOULD NOT HAVE TO google bugzillas and play with LD_PRELOAD to have a working system), is just sad.

Although overall, as an end user it the conversation around this bug and its persistence makes me sad. I know it’s selfish not to care about the technical superiority of a solution or about who is to blame here – but I’d just like to have my smooth Flash playback…


18
Feb 11

Cascading file loading in Node.js

One of my favorite features of Kohana 3 is it’s cascading filesystem – so I decided to implement it for Node.js. A cascading filesystem is an elegant solution to a common problem: how to provide a mechanism for loading modules and reusing code?

The following image from Kohana 3′s docs shows an example:

Benefits

The key benefits are:

  1. Consistency. All your application files, including views, controllers, models and other data such as translation messages are loaded using one, easy-to-understand mechanism.
  2. Easy reuse. Without a cascading file system, you’ll have to copy and move files around if you want to use someone else’s libraries or modules. With a cascading file system, you just place the module in your application, and enable cascading for that directory.
  3. Transparent extensibility. What if you want to override one part of a module (say, a view) but don’t want to modify your copy of the module (e.g. so that you can update without manually merging changes). A cascading filesystem allows you to selectively replace files in 3rd party code simply by providing your own version of the file.

The code

Load order and file name resolution

The load order for my implementation is:

  1. Application path –  files under ./application/ are always checked first.
  2. Module paths – set modules(['./modules/my-module']) to enable module loading. Files from modules are loaded from in the order they are added.
  3. System path – files under ./system/ are loaded if no alternative exists.

Assumptions about file and class names

Files are assumed to be lowercase. Underscores in class names are replaced by slashes (so Controller_User becomes ./application/classes/ controller/user.js).

Performance impact

Requests are cached, so that additional calls to find_file() do not cause additional stat() calls in the filesystem. This is insignificant anyway, since Node.js servers are persistent so the cascading search is only done once per server instance for each file (not once per request).

Loading 3rd party code

The loaded files do not need to be “compatible” in any way other than layout in the file system. For example, while Cfs.factory(‘some_other_lib’) loads the file from ./application/ classes/some/other/lib.js, that file does not actually need to contain a class named some_other_lib; just that it returns something via module.exports.

Methods

The methods are:

  • Cfs.modules(['./modules/path-to-module']) – set the modules directories to search.
  • Cfs.find_file(dir, file, ext) – Search each path under dir (e.g. ‘classes’, ‘views’) for file (filename) with the extension (ext, default is “.js”).
  • Cfs.factory(class_name) – Return a new instance of the given class after loading the corresponding file from the cascading file system. Note that classes should be in the classes subdirectory.
  • Cfs.load(class_name) – Return whatever require(file-which-contains-the-class) returns. Useful for extending classes, see below for an example

Example usage:

var Cfs = require('./cfs.js');
 
// test class loading:
// e.g. check ./application/classes/test.js
// ./modules/modulename/classes/test.js
// ./system/classes/test.js
var t = Cfs.factory('test');
t.run();
 
// test view loading
// e.g ./application/views/user/index.html
// ./modules/modulename/views/user/index.html
// ./system/views/user/index.html
fs.readFile(Cfs.find_file('views', 'user/index', '.html'), function (err, data) {
  if (err) throw err;
  sys.puts(data);
});

To set modules:

// set only once, before calling any other functions!
Cfs.modules([
         "./modules/testmodule/",
         "./modules/testmodule2/",
         ]);

Extending classes:

// test extending class (see code in /application/classes/controller/extend.js
// to see how extension is achieved)
// e.g. ./application/classes/controller/extend.js
// ./modules/modulename/classes/controller/extend.js
// ./system/classes/controller/extend.js
var t3 = Cfs.factory('Controller_Extend');
t3.run();
t3.run_parent();

Note that if you put cfs.js in ~/node_modules/cfs.js, you don’t need to specify the path to it… see Modules in node.js docs.

// in extend.js:
var Controller_Extend = function () {
}
// extend the class
var util = require('util'), Cfs = require('../../../../cfs.js');
util.inherits(Controller_Extend, Cfs.load('Controller_Base'));
 
Controller_Extend.prototype.run = function() {
   console.log("Controller_Extend from testmodule2.");
};
Controller_Extend.prototype.run_parent = function() {
   // run the parent function
   Controller_Extend.super_.prototype.run();
};
 
module.exports = Controller_Extend;