1. Edit /etc/rc.conf. Add thinkpad_acpi to MODULES=() in the hardware section.

2. Create a new file enabling fan control as /etc/modprobe.d/thinkfan.conf:

options thinkpad_acpi fan_control=1

3. Reload the thinkpad_acpi module:

sudo modprobe -r thinkpad_acpi && sudo modprobe thinkpad_acpi

To view the current fan speed:

[~]$ cat /proc/acpi/ibm/fan
status: enabled
speed: 2887
level: 4
commands: level ( is 0-7, auto, disengaged, full-speed)
commands: enable, disable
commands: watchdog ( is 0 (off), 1-120 (seconds))

To control fan speed:

# echo level 0 | sudo tee /proc/acpi/ibm/fan (fan off)
# echo level 2 | sudo tee /proc/acpi/ibm/fan (low speed)
# echo level 4 | sudo tee /proc/acpi/ibm/fan (medium speed)
# echo level 7 | sudo tee /proc/acpi/ibm/fan (maximum speed)
# echo level auto | sudo tee /proc/acpi/ibm/fan (automatic - default)

For easy control from the command line:

function fan() {
  sensors
  echo level $@ | sudo tee /proc/acpi/ibm/fan
}

Now you can run fan 2 to set the fan level to 2 and so on.

To view fan speed, install lm_sensors following the instructions on the Arch wiki. Example output:

[~]$ sensors
acpitz-virtual-0
Adapter: Virtual device
temp1:        +37.0°C  (crit = +99.0°C)

thinkpad-isa-0000
Adapter: ISA adapter
fan1:        2904 RPM

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +39.0°C  (high = +86.0°C, crit = +100.0°C)
Core 0:         +37.0°C  (high = +86.0°C, crit = +100.0°C)
Core 1:         +37.0°C  (high = +86.0°C, crit = +100.0°C)
Core 2:         +35.0°C  (high = +86.0°C, crit = +100.0°C)
Core 3:         +39.0°C  (high = +86.0°C, crit = +100.0°C)

First steps

• do the basic arch setup first (or VMware, or Virtualbox)

Update the system (and install/setup sudo)

dhcpcd eth0 #if you did not add "interface=eth0" in rc.conf during setup
pacman -Syu

Fixes:

http://www.archlinux.org/news/initscripts-update-manual-intervention-required/

rm /etc/profile.d/locale.sh

http://www.archlinux.org/news/filesystem-upgrade-manual-intervention-required/

pacman -S filesystem --force

pacman -S sudo
vim /etc/sudoers # add yourself to sudoers
sudo vim /etc/pacman.conf # set SigLevel = Never TrustAll
sudo shutdown-r now

Install X11:

pacman -S xorg-server xorg-xinit xorg-utils xorg-server-utils xterm

If virtualized in VirtualBox, make copy-paste work first:

pacman -S virtualbox-archlinux-additions

Details

Create a new user

pacman -S zsh
useradd -m -g users -G audio,lp,optical,storage,video,games,power,scanner -s /bin/zsh USERNAME
su USERNAME
passwd

Add x11

pacman -S xorg-server xorg-xinit xorg-utils xorg-server-utils

Copy ssh keys over from old machine
Useful packages

pacman -S base-devel sudo python2 git libev mlocate mercurial nitrogen \
sakura wicd-gtk pcmanfm gnome-icon-theme htop unzip \
openssl chromium flashplugin bash-completion xterm \
epdfview mysql ruby tilda tmux wget redis xcursor-vanilla-dmz \
xarchiver gzip bzip2 zip unzip unrar p7zip \
meld ttf-ubuntu-font-family mpg123 alsa-utils redis mysql ruby libxslt

• base-devel and python2 for compiling node
• libev for nwm
• mlocate for locate command
• nitrogen is better than feh for multiple screens
• sakura is a nice terminal
• wicd-gtk is a simple wifi network gui
• pcmanfm gnome-icon-theme are for pcmanfm, a Nautilus alternative

Remember to visudo and remove the password requirement from wheel. And add dbus and wicd to /etc/rc.conf just like pacman tells you to.
Configuring X11

Add ~/.Xresources:

Xcursor.theme: vanilla-dmz
Xcursor.size:  16       !  32, 48 or 64 may also be good values

Configuring git

git config --global color.ui true

Configuring sakura

I want to use ctrl + Page_Up / Page_Down to switch tabs, so edit ~/.config/sakura/sakura.conf:

switch_tab_accelerator=4 # since GDK_CONTROL_MASK is 1 << 2, e.g. 4.
prev_tab_key=Page_Down
next_tab_key=Page_Up

Some basic niceties: whatprovides, service and chkconfig

pacman -S pkgtools

• pkgtools provides the pkgfile tool. It works like yum whatprovides (e.g. allows you to search for a particular command or dependency in all the pacman packages)
• “sudo pkgfile -u” to update the db
• “pgkfile zipinfo” to search for zipinfo

Arch don’t have a service and chkconfig, but we can make the new things curve a bit less steep by adding some functions to .bashrc:

function service() {
  sudo /etc/rc.d/$1 $2
}

alias chkconfig='cat /etc/rc.conf | grep DAEMONS && echo "cat + grep /etc/rc.conf"'

This makes service an alias for /etc/rc.d/ and prints out the enabled services from /etc/rc.conf. While we’re editing .bashrc, might as well add:

PS1="[\W]\$ " # my preferred bash prompt (e.g. only the current dirname).
ulimit -s 16400 # higher stack limit
# ssh-agent
SSH_ENV="$HOME/.ssh/environment"
function start_agent {
     echo "Initialising new SSH agent..."
     /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
     echo succeeded
     chmod 600 "${SSH_ENV}"
     . "${SSH_ENV}" > /dev/null
     /usr/bin/ssh-add;
}
# Source SSH settings, if applicable
if [ -f "${SSH_ENV}" ]; then
     . "${SSH_ENV}" > /dev/null
     #ps ${SSH_AGENT_PID} doesn't work under cywgin
     ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
         start_agent;
     }
else
     start_agent;
fi

Installing Node and NPM

You can just do:

pacman -S nodejs

If you’re OK with that version, which seems to track the Node releases pretty well.

Arch uses python3 as python. You need to change python to python2 (thanks Rob Searles!)

# node.js fix for arch (use python2) 
mkdir /tmp/bin
ln -s /usr/bin/python2 /tmp/bin/python
export PATH=/tmp/bin:$PATH

You can then do a regular node install:

git clone git://github.com/joyent/node.git

git checkout v0.4.12

./configure

make

sudo make install

Remember to install npm as well:

curl http://npmjs.org/install.sh | sudo sh

Installing my window manager and personal config

git clone git://github.com/mixu/nwm.git
cd nwm
node-waf clean || true && node-waf configure build
sudo npm link # add a global npm symlink to this repository - so nwm-user can find it (man npm link)
cd ..
git clone git://github.com/mixu/nwm-user.git
cd nwm-user
npm link nwm # now make a symlink to the nwm installation

Add it to ~/.xinitrc (change paths!!):

exec /usr/local/bin/node ~/mnt/nwm-user/nwm-user.js 2>~/nwm.err.log 1>~/nwm.log

And while we’re at it, lets add some other stuff:

VBoxClient-all &
export PATH=/tmp/bin:$PATH # for node-waf, too lazy to work on a better solution
xset +fp /usr/share/fonts/local
xset fp rehash

Run “startx” to start X11 with nwm.

Installing my mp3 player

First, we need to configure alsa (included by default):

pacman -S mpg123 alsa-utils

Run:

alsamixer

and turn on Master and PCM channels (by pressing m) as they are muted by default.

sudo alsactl store

Then continue:

git clone git://github.com/mixu/nplay.git

Run nplay with

node nplay.js

TODO: fix directory in source code and change backend from mpg321 to mpg123.

Switching the keyboard language in X11

I sometimes need to write emails in Finnish, so here is how to switch the layout:

setxkbmap -layout fi # revert back setxkbmap -layout us 

Install yaourt

Install dependencies

yaourt libpng12 gtk2-theme-dust

Install Sublime Text 2

Sublime Text needs libpng12, which you have to install from AUR:

wget http://aur.archlinux.org/packages/li/libpng12/libpng12.tar.gz
tar -xzvf libpng12.tar.gz
cd libpng12
makepkg
pacman -U ./libpng12-1.2.46-2-x86_64.pkg.tar.xz

Then download and run Sublime Text 2.

Also, you might want to ger http://aur.archlinux.org/packages/gt/gtk2-theme-dust/gtk2-theme-dust.tar.gz.

Configuring Sublime Text 2

locate Packages # returns ~/.config/sublime-text-2/Packages
cd ~/.config/sublime-text-2/Packages
git clone https://github.com/buymeasoda/soda-theme/ "Theme - Soda"
cd User
wget http://blog.mixu.net/files/2010/05/my_themes.zip # Install my themes
unzip my_themes
rm my_themes.zip

Base File settings

{
  // FONTS and COLORS
  "color_scheme": "Packages/User/Mixu Espresso.tmTheme",
  "font_size": 11,
  "tab_size": 2,
  // WHITESPACE
    // Set to true to insert spaces when tab is pressed
    "translate_tabs_to_spaces": true,
    "trim_automatic_white_space": true,
    "trim_trailing_white_space_on_save": true,
    // Set to false to disable detection of tabs vs. spaces on load
    "detect_indentation": false,
  "shift_tab_unindent": true,
  // Set to false to disable highlighting any line with a caret
  "highlight_line": true,
  // Set to "none" to turn off drawing white space, "selection" to draw only the
  // white space within the selection, and "all" to draw all white space
  "draw_white_space": "selection",
  // Set to true to ensure the last line of the file ends in a newline
  // character when saving
  "ensure_newline_at_eof_on_save": true,
  "fold_buttons": false
}

Global Settings

{
  "theme": "Soda Light.sublime-theme"
}

Other tweaks

I put these in my usual “startup” command, nwm-setup.sh and run it manually:

xrandr --output VBOX0 --auto --left-of VBOX1 # Virtualbox displays
chromium & # start chromium
export PS1="[\W]\$ "
xsetroot -cursor_name left_ptr # Set pointer
nitrogen --restore & # Restore desktop background using nitrogen

Other dependencies

pacman -S redis mysql ruby libxslt

Setting up ree:

cd
bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)
echo '[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" # Load RVM function' >> ~/.bashrc
source .bashrc
rvm install ree-1.8.7-2011.03

Installing REE will fail. You need torun the installer from /home/m/.rvm/src/ree-1.8.7-2011.03/installer manually:

./installer -no-tcmalloc

Then continue on:

rvm ree-1.8.7-2011.03 --default

Basically, you can’t use the registration tool unless you really want to replace your registration dialog with Facebook’s own version. You can’t redirect to it conditionally, e.g. if the user has Facebook and clicks the login button, since the login will already trigger the Facebook authorization process.

So unless you want two logins (Facebook login link and regular login form) and two registration links (regular registration page and Facebook register page), you can’t use the registration tool. My ideal implementation would just have a single link for everything related to Facebook, but present the full registration form if the user clicks it and has not yet registered with my site… no luck.

Hence, I will do this the traditional way:

• Use the Javascript SDK to render the Facebook Login link and popup.
• Ask for additional permissions on the Login.
• When the authorization returns, process the user login / registration.
• If the user wants to register using Facebook, then they will still need to fill in additional fields in a separate field after authorizing the information we can read from Facebook.

Preparations

Get an app_id and a secret from Facebook. Register your website/app here to get an app_id and secret.

Add the Facebook namespace. Your HTML tag should contain the additional xmlns:fb -attribute so that the Facebook custom tags work with IE:

Source code

<html xmlns="http://www.w3.org/1999/xhtml" xmlns:fb="http://www.facebook.com/2008/fbml">

I did not know you could add XML namespaces to your documents and actually have custom tags which work in all current browsers, but since Facebook does it, it must have been tested well.

Rendering the Facebook login link and popup

Source code

<div id="fb-root"></div>
<script>
    window.fbAsyncInit = function() {
        FB.init({
            appId   : '<?php echo $facebook->getAppId(); ?>',
            session : <?php echo json_encode($session); ?>, // don't refetch the session when PHP already has it
            status  : true, // check login status
            cookie  : true, // enable cookies to allow the server to access the session
            xfbml   : true // parse XFBML
    });
    // whenever the user logs in, we tell our login service
    FB.Event.subscribe('auth.login', function() {
       window.location = "<?php echo URL::site('/user/fb_login') ?>";
    });
  };

  (function() {
    var e = document.createElement('script');
    e.src = document.location.protocol + '//connect.facebook.net/en_US/all.js';
    e.async = true;
    document.getElementById('fb-root').appendChild(e);
  }());
</script>

Asking for additional permissions

You have to do this by adding the perms attribute to the fb:login-button tag. The links in the docs are broken, but here is the page documenting the login button and the extended permissions you can ask for are listed here.

We’ll just ask for the email address:

Source code

<fb:login-button perms="email" size="large"></fb:login-button>

Processing the resulting login or registration

As you can see above, we subscribe to the auth.login event using Javascript and specify that on completion, the user will be redirected to /user/fb_login/.

On that page, we need to:

1. Check whether a user account exists with the returned Facebook user id.
2. If no user is found with the FB uid, retrieve the user’s email via the Graph API and check whether a user exists with that email. If so, we will merge the user.
3. If no user is found with the FB email, create a new account.
4. If creating a new user fails, then accept defeat and show the regular registration page.

A complete implementation of this available from Bitbucket – soon! I’m releasing it as part of my Useradmin module for Kohana 3.

Logging out

We need to handle logging out sensibly. I prefer that the user explicitly clicks a login button if they do not currently have a session with our application. Automatic relogin is confusing, particularly when logging out usually redirects to the login page.

So we need to handle the case where the person is connected (logged in to Facebook and authorized with us) by displaying a custom login button.

You can find the Facebook logo from the branding site. Of course, a Google search for facebook logo won’t find it, so click here to get the image that you are allowed to use for this purpose.

Source code

window.fbAsyncInit = function() {
        FB.init({
            appId   : '<?php echo Kohana::config('facebook')->app_id; ?>',
            status  : true, // check login status
            cookie  : true, // enable cookies to allow the server to access the session
            xfbml   : true // parse XFBML
    });
    // whenever the user logs in, we tell our login service
    FB.Event.subscribe('auth.login', function() {
       window.location = "<?php echo URL::site('/user/fb_login') ?>";
    });
    // if the user is already logged in, redirect them to the login action
    // they cannot reach the login page if they are already logged in
    // since login() redirects to profile if the user is logged in
   FB.getLoginStatus(function(response) {
     if (response.status == 'connected') {
        document.getElementById('fb-login-li').innerHTML = '<a href="<?php echo URL::site('/user/fb_login') ?>"><img src="/img/fb-login.png"></a>';
     } else {
        document.getElementById('fb-login-li').innerHTML = '<fb:login-button perms="email" size="large"><?php echo __('Login / Register with Facebook')?></fb:login-button>';
        FB.XFBML.parse(document.getElementById('fb-login-li'));
     }
   });
  };

Appendix: all fb:login-button attributes

The Facebook documentation does not list the fb:login-button attributes, which is infuriating. I found a forum post (here), which documents the optional attributes of fb:login-button:

• condition; string Indicates whether the button is visible or hidden.
• size; string Specifies the size of the button. Specify icon to display a favicon only, or small, medium, large, or xlarge. (Default value is medium.)
• autologoutlink; bool If true and the user is already connected and has a session, then the button image changes to indicate the user can log out. Clicking the button logs the user out of Facebook and all connected sessions. (Default value is false.)
• background; string Specifies the button image to use that is anti-aliased to match the background of your site — whether it’s pure white, light, or dark. Specify white, dark, or light. (Default value is light.) Note: You don’t specify this attribute if you are using v=”2″.
• length; string Specifies which text label to use on a button with size specified as small, medium, large, or xlarge. Specify short for the text label Connect only or long for the text label Connect with Facebook. If you are rendering the login button text by including it within the fb:login-button tags, you don’t specify a length at all. (Default value is short.)
• onlogin;     string JavaScript code to execute when the user gains a Facebook session (that is, after logging into Facebook and authorizing the site).
• v;     string Specify “2″ to use the latest Facebook Connect login buttons, (examples available in the Facebook Connect wizard). Don’t use the attribute if you need to use the original Facebook Connect login buttons.

• Nginx as a fast HTTP server with reverse proxy to node.js
• php-fpm for running PHP scripts. The php-fpm (PHP FastCGI sapi) is built into the PHP core but only since 5.3.3, so we need a recent version of PHP.
• node.js for handling comet and high-concurrency/persistent connections.
• Monit is used to restart node.js in case of errors.

For all the packages other than node.js (obviously, since it’s new/actively changing), you can get the most recent version without to patching/compiling anything anymore.

Start by updating your packages and uninstalling httpd if you had it installed:

yum update

yum remove httpd

Add repositories for nginx and PHP-fpm

Just add the EPEL and Remi repos:

Source code

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-5.rpm

Install nginx and PHP-fpm

Source code

yum --enablerepo=remi install php-fpm nginx

Source code

yum --enablerepo=remi install mysql mysql-server php-mysql php-common php-gd php-mbstring php-mcrypt php-xml php-gd php-bcmath

PHPunit and PHPdocumentor:

Source code

yum --enablerepo=remi install php-channel-phpunit php-pear-PhpDocumentor php-phpunit-PHPUnit

Create a test application

mkdir /var/www/

In /var/www/index.php:

Source code

<?php
  phpinfo();

Configure nginx

Source code

server {
   listen       80;
   server_name  _;
   access_log  logs/host.access.log  main;
   root   /var/www;
   index  index.php index.html index.htm;
   location ~ \.php$ {
      # Security: must set cgi.fixpathinfo to 0 in php.ini!
     fastcgi_split_path_info ^(.+\.php)(/.+)$;
      fastcgi_pass 127.0.0.1:9000;
      fastcgi_index index.php;
      fastcgi_param SCRIPT_FILENAME         $document_root$fastcgi_script_name;
      fastcgi_param PATH_INFO $fastcgi_path_info;
      include /etc/nginx/fastcgi_params;
   }
}

Default fastcgi_params

I start the php location by including /etc/nginx/fastcgi_params. This is because I want to use those as defaults, then override them individually afterwards. It is also recommended to have one root directive per server.

PHP path

I  use fastcgi_split_path_info which became available in 0.7.31 to pass PATH_INFO.

Security settings

IMPORTANT: Change cgi.fix_pathinfo to 0 in php.ini to prevent a security issue which arises with the default PHP configuration when PHP incorrectly tries to guess which file you want for URLS specifying nonexistent files. Setting cgi.fix_pathinfo=0 causes PHP to only try the literal path given. Alternatively check that the file exists: if (!-f $request_filename) { return 404; }

See also: Nginx pitfalls.

Start/restart and test

service php-fpm start

service nginx restart

Test by going to your server root, it should show you your phpinfo().

Prepare to install node.js

Excellent post: http://wavded.tumblr.com/post/475957278/hosting-nodejs-apps-on-centos-5

Add a user for node.js:

Source code

groupadd -r node
useradd -r --shell /bin/bash --comment 'User for running node.js' -g node  --home /var/lib/node node

Install node.js

Source code

yum install gcc-c++ openssl-devel
wget --no-check-certificate https://github.com/ry/node/tarball/v0.3.3
tar -xzvf ry-node-v0.3.3-0-g57544ba.tar.gz
cd ry-node-v0.3.3-0-g57544bac1
./configure
make
make install
mkdir /var/node

Create a test application

Source code

var sys = require("sys"),
   http = require("http");
http.createServer(function (request, response) {
  response.writeHead(200, {"Content-Type": "text/plain"});
  response.end("Hello World\n");
}).listen(8000);
sys.puts("Server running at 127.0.0.1:8000");

Configure nginx for node.js (subdirectory approach)

Source code

location /node {
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header Host $http_host;
   proxy_set_header X-NginX-Proxy true;
   proxy_pass http://127.0.0.1:8000/;
   proxy_redirect off;
}

Install monit

yum install monit

Create monit script

/etc/monit.d/hello_world content:

Source code

check host hello_world with address 127.0.0.1
    start program = "/usr/local/bin/node /var/node/hello_world/example.js" as uid node and gid node
    stop program  = "/usr/bin/pkill -f 'node /var/node/hello_world/example.js'"
    if failed port 8000 protocol HTTP
        request /
        with timeout 10 seconds
        then restart

Start/restart and test

service monit start

service nginx restart

Test by going to /node on your server.

Also, run ps -Af to verify that node is running with the uid node.

The reboot test

Finally, test everything by restarting your server. A friendly reminder:

chkconfig nginx on

chkconfig monit on

chkconfig php-fpm on

chkconfig mysqld on

chkconfig –list

Getting started: some terminology

If you are like me and haven’t implemented Facebook integration before, the terminology and various API’s can be confusing. And the Facebook developer documentation is a combination of too-much-information and not-enough-explanation.

First, the Facebook documentation talks about different kinds of applications, which are:

• Facebook canvas applications: Applications-within-Facebook, or what users think of as “Facebook apps”.
  • FBML / FBJS apps (built using FB-specific markup)
  • Iframe canvas apps (built using Javascript and the FB JS API)
• Facebook desktop applications (rare; basically anything that cannot run within a browser, like a desktop client for Facebook)
• Facebook web applications (websites with Facebook-integration such as single sign-on and custom Facebook elements within them such as CNN and Digg)

And there are multiple developer tools that Facebook offers:

• Facebook Connect – no such thing exists anymore (e.g. http://mashable.com/2010/04/21/facebook-kills-facebook-connect/). The new API is called the Graph API, and talking about Facebook Connect is inaccurate..
• APIs (always used, either on the server side or on the client’s web browser via Javascript)
  • Graph API: The new version of Facebook’s API. The Graph API is much more than just authentication: it is the mechanism which powers Facebook applications and allows you to read and write date to Facebook.
  • Old REST API: The early version of Facebook’s Graph API; Facebook does not recommend you use it because they are in the process of deprecating it.
  • FQL (Facebook Query Language): A SQL-like language for querying the Graph API. Supposedly makes using the API easier for apps.
• Markup
  • FBML (Facebook Markup Language): A HTML-like language for building Facebook applications. FBML is rendered on the Facebook server. Facebook does not recommend that you use it; instead suggesting that you use the Graph API via their Javascript SDK with XFBML.
  • XFBML: FBML-within-(X)HTML pages rendered using Facebook’s Javascript SDK.
• SDKs
  • Language-specific libraries for querying the API. Other than the Javascript SDK, it’s all server-side and based on HTTP requests.
  • Note that the Javascript SDK allows you to render XFBML embedded on a (X)HTML page! This was confusing because I had done something with FBML before, and didn’t see where exactly the transformation from fb:login-button to HTML was occurring…
• Other stuff from Facebook:
  • Open Graph Protocol: A convention of meta and other tags which allows you to add metadata to Like button clicks, integrating them with Facebook (e.g. a like on a profile).
  • Social plugins: Ready-made Facebook widgets you can embed using an iframe.

How does Facebook authentication work?

Conceptually, it works like this (picture of user represents page shown to user):

There most important authentication methods are:

1. Javascript single sign-on: The Javascript single sign-on is simplest, because it provides a function (FB.login) which handles everything from the details of OAuth 2.0 requests to rendering the actual login button. When using the Javascript API, the user id and access token are stored in a cookie (e.g. fbs_app_id) which you can access both server-side and via Javascript in the client.
2. web application authentication (using server side SDK’s): When you do web application authentication, you need redirect the user to Facebook in your app. When the user allows access for your app, Facebook redirects them back and sends the access token and user ID as part of the GET request to your authentication-complete redirection page. Then you can access the user’s information using the returned token.

Next part

In part 2 of my Facebook login tutorial I address database design considerations and look at how Facebook login can be integrated with your existing user management.

In part 3 of my Facebook login tutorial I go into implementation specifics using the Javascript SDK.

References

• http://developers.facebook.com/docs/
• http://developers.facebook.com/docs/guides/performance (good illustration on the difference between FBML and IFrame applications)
• http://stackoverflow.com/questions/1175047/facebook-app-vs-facebook-connect-site

Why?

• Ease of updating. Mercurial keeps track of the changes and only sends the necessary changes -  you don’t need to worry about transferring files.
• Make it possible to roll back changes on the deployed site. You can use hg to roll back from a bad update if necessary.
• Once set up, it’s beautiful. “hg deploy”. How can you not like that?

How is this different from the other guide you wrote about setting up private repo hosting?

• While the differences aren’t that big, this setup is better for deployment rather than code distribution via repositories:
  • Minimal dependencies. This approach only uses the hg-ssh script from the Mercurial core contrib.
  • Manual configuration. You can set different directories for each repository which allows you to work with your existing webroot setup. However, you will need to manually add new repositories, since hg-ssh does not support adding new repositories remotely.

If you want a private version of Bitbucket (without any additional features, of course), e.g.  to be able to remotely init/clone new repositories, check out my other tutorial about setting up private repo hosting.

1. Make sure that the .hg directories are never served to the public

To prevent .hg directories from being accessible to other people, you can take a number of (optional, but recommended) precautions:

Move the web root of your project to a different folder

You can probably alter your project  repository so that you have an explicit webroot folder in which you only have the files that should be accessible to the public (bootstrap + js/css resources). This makes it less likely that you accidentally serve your .hg directory. You move the files within hg do this using hg move:

Source code

mkdir webroot
hg move . webroot

Or you can use hg addremove –similarity=100 after moving the files manually. It will detect identical files as moved.

After this, you may need to update your apache config to serve from the webroot (e.g. adjust the DocumentRoot directive for the site / virtual host).

Set up Apache not to serve .hg directories

You can also prevent Apache from serving directories with .hg in them by adding the following to your httpd.conf:

Source code

<DirectoryMatch \.hg>
   Order allow,deny
   Deny from all
</DirectoryMatch>

Restart Apache if you change httpd.conf.

2. Setup the server

Create the user

Source code

# make a system user (-r) with a home directory (-m)
useradd -r -m hg
# lock the user account
usermod -L hg

Setup ssh

Source code

#install mercurial if not previously installed
yum install mercurial
su hg
cd /home/hg/
wget http://www.selenic.com/repo/hg-stable/raw-file/tip/contrib/hg-ssh
chmod u+x ~hg/hg-ssh
mkdir /home/hg/.ssh/
nano /home/hg/.ssh/authorized_keys
# remember to chmod the ssh config
chmod -R 0700 /home/hg/.ssh/

Copy the public key: ssh-rsa …(key data)== to /home/hg/.ssh/authorized_keys, one line for each authorized key.

Add the following in front of each authorized key in /home/hg/.ssh/authorized_keys:

Source code
   
command="~hg/hg-ssh /path/to/repository",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-rsa ...==

If necessary, enable public key auth in /etc/ssh/sshd_config and add hg to the AllowUsers directive. Restart sshd if you change sshd_config.

3. Init the repo

New repo on the remote server

Just run hg init in the new repo folder, then hg clone it on your computer from the remote server:

hg clone -v --debug ssh://hg@server:port/path/to/repo

The -v and –debug make the clone show more information about what is going on.

From existing sources

Unfortunately hg-ssh does not support the hg init/hg clone command remotely. If you have an existing repo, you have to copy it first to the repository directory on the server. Since there are many files to move, I recommend gzipping the whole directory before moving it, then unzipping on the server. Do a hg log and hg status to see that everything transferred correctly.

Adding more repos

To add more repositories, simply add another repository path (separated by a space) immediately after the first repo path in .ssh/authorized_keys, and either init a new repo or copy an existing repo.

4. Push updates

After you have the same repository on both the server and locally, you can start pushing  stuff to the server:

Source code
   
hg push ssh://hg@server:port/path/relative/to/home/dir

If your repo path is not relative to the home dir, you need an extra slash in front of the push:

Source code
   
hg push ssh://hg@server:port//path/relative/to/base/dir

If you run into problems, try connecting via ssh – see my previous post for some tips on this.

5. Automate hg update

You should automate hg update so that each push causes the repository to be updated automatically to the pushed version. You can automate hg update on the remote repo by adding the following to the remote .hg/hgrc:

Source code

[hooks]
changegroup = hg update >&2

Note that output has to be redirected to stderr (or /dev/null), because stdout is used for the data stream -which is why there is the >&2 at the end of the command.

If you get errors in updating automatically, check the permissions. In particular, make sure that .hg/ is owned by hg and that the content is readable by apache.

6. Set up default push or create an alias

If you only push to one location, then you can set up the default locations for pull and push in the local  .hg/hgrc:

[paths]
default = ssh://hg@servername/reponame
default-push = ssh//:hg@servername/reponame

This allows you to simply use “hg push” to deploy.

If you need more than one push location, create an alias in the local  .hg/hgrc:

[alias]
deploy = push -v --debug ssh://hg@server:port/path/to/repo

This allows you use “hg deploy” as an alias for deploying.

What about configuration files?

One simple solution is to use “hg forget” to forget them once you have deployed the configuration files. This means that hg will not track the configuration files, but will not delete them either.

GDocs toggle

Just grab the link with your mouse, and drag it to your bookmarks bar. Click on it when you have a Google Docs page open, and it will hide / show the extra stuff on top of the page.

Note: You may  have to resize your window once so that the text area expands, press F11 to do this quickly.

I don’t understand why this isn’t included as a default feature – it used to be supported in Google Docs as a menu option.

The main technologies are reverse DNS checking, SPF, SenderID, Domainkeys and DKIM. I will discuss all of these here and provide my tips on setting up SPF, SenderID and DKIM. Please keep in mind that I am not an expert on email servers – but I hope this helps someone! It took me about half a day to figure this all out, so it is probably worth doing to improve your email delivery rates.

This blog entry from Dave Zohrom provides a nice discussion of sending email to a general audience (part 2 of a 3-post series).

Introduction

SPF or Sender Policy Framework

SPF is easy to setup. It uses DNS TXT records to allow the email providers to check which servers are authorized to send email for a particular domain. The SPF project has done a great job at making this simple to setup. Just go to their homepage and use the setup wizard to generate the appropriate TXT records (the text field underneath “Deploying SPF”). Then change the values in your DNS records.

One thing to note is that if you have multiple servers and send email from a server with a different hostname (e.g. “mail.example.com”), you need to setup a record for that server as well. See the common mistakes page on the SPF site. Also, if you have hostnames that are not supposed to send mail, you ought to indicate this as well.

Another thing is that if you are using Google Apps for Your Domain to send email, then you will need to have to add “include:aspmx.googlemail.com” to also allow mail from Google to validate. See this answer for more.

SenderID from Microsoft

SenderID is a variant of SPF, which for most practical cases is the same as SPF. Just setup SPF and this should produce a validation pass for SenderID as well. The semantics of the validation are a bit different,  but this does not seem to be a major practical problem. See testing tips below to make sure that this is also true in your case.

DomainKeys

DomainKeys is an older version of DKIM (DomainKeys Identified Mail) developed by Yahoo. Despite having very similar names, these ARE NOT the same!

Both DomainKeys and DKIM store public key information in DNS records and sign the message headers of every email sent. The recipient can then verify the signature.

DomainKeys was deprecated in 2007, but some email providers may still be using it. However, these are a shrinking minority and Yahoo does support the newer DKIM. Because of this I did not add DomainKeys support but opted only to use DKIM.

You can add it to sendmail or Postfix using the dk-milter project code, but the unofficial RPM release is not maintained anymore, which means you will need to install it from source (available via SourceForge).

DKIM, or DomainKeys Identified Mail

DKIM on the other hand seems to be gaining momentum. It is used by Gmail, Yahoo and AOL and many others, and also works by publishing public keys via DNS TXT records and by signing the emails at the email server.

To setup DKIM, you need an additional filter which takes the completed email and adds the DKIM signature to the email prior to sending it out.

Postfix and sendmail support “milters“, which is apparently short for “mail filter”. There is a DKIM-milter package available for Centos at the EPEL repositories (see Centos page for 3rd party repos).

Setting up DKIM-milter with sendmail

Updated: Linked tutorial is DEAD. Here are the steps:

Step 1. Generate a private key

openssl genrsa -out default.private 1024

A “default.private” key file will be generated. It will be moved to a specific location later.

Step 2. Generate a public key for this private key

openssl rsa -in default.private -pubout -out default.public -outform PEM

A file with filename “default.public” will be generated with content like

—–BEGIN PUBLIC KEY—–

…

—–END PUBLIC KEY—-

It will be used to create a DNS TXT record. See next step.

Step 3. Create a DNS record of type TXT

Modify DNS records and add a record of type TXT:

TXT record name

default._domainkey

TXT record value

v=DKIM1; g=*; k=rsa; p=<content of default.public>

Note that the prefix “—–BEGIN PUBLIC KEY—–” and suffix “—–END PUBLIC KEY—-” should not be put in the TXT record value.

This DNS record will be retrieved by mail receivers who want to verify emails with DKIM signatures. The record name “default._domainkey” tells verifier that the “selector” of this signature is “default”, therefore if you are changing selector name to something else, make sure you change all of them consistently.

Step 4. Install dkim-milter in Fedora

Run the following as root to install the dkim-milter pacakge.

yum install dkim-milter

Step 5. Enable dkim-milter to run on start-up

Make sure dkim-milter service will run on start-up by running this command:

chkconfig –level 3 dkim-milter on

Note that your server may use a different “runlevel”. You can check “/etc/inittab” to see which run level you are on.

Step 6. Move private key to appropriate location

As root, copy the private key to the location specified by the “keylist” (refer to next step) and make sure it is readable by dkim-milter:

mkdir /etc/dkim-milter/

mv default.private /etc/dkim-milter/default

chown dkim-milter.dkim-milter /etc/dkim-milter/default

Make sure the filename of private key file matches the “selector” name specified in the DNS record.

Step 7. Add an entry to the keylist for dkim-milter to read

Add the following line to /etc/mail/dkim-milter/keys/keylist. Replace <domain.com> with your domain name.

*:<domain.com>:/etc/dkim-milter/default

Step 8. Configure dkim-milter

Open configuration file /etc/mail/dkim-milter/dkim-filter.conf and use the following configuration:

Canonicalization simple
Domain example.com
KeyFile /some/path/to/whatever-your-keyfile-was
Selector name-of-the-selector
SignatureAlgorithm rsa-sha256
Socket inet:8891@localhost
Syslog Yes
Userid dkim-milter

NOTE: you will be configuring dkim-milter to use the loopback interface instead of a socket file. I was unable to get dkim-milter to work via the socket file with sendmail. If you get it working, let me know.

You may also want to setup the following:

SubDomains Yes
SyslogSuccess Yes
X-Header Yes

The X-Header and Syslog options are useful for debugging. See the config file, each option should be documented there.

Step 9. Change the default init.d script to use the loopback interface

The default init script uses a socket, this needs to be changed. Open /etc/init.d/dkim-milter and change/comment the line:

SOCKET=local:/var/run/${name}/${name}.sock

to:

SOCKET=inet:8891@localhost

Step 10. Configure sendmail to use dkim-milter

First a few reminders about sendmail configuration, remember that:

1. Sendmail comments DO NOT USE # as the comment, instead “dnl”  (delete through newline) at beginning of the line is used to comment lines out.
2. Sendmail configuration is built from the *.mc script files using the M4 macro processor.
3. You need to install the sendmail-cf package for dependencies and install the m4 macro processor separately.
4. In the configuration, the opening quote is a grave accent ` and the closing quote is a straight quote ‘.

To configure sendmail, open the “/etc/mail/submit.mc” file (which contains the settings for message sending; in older sendmail versions this config was in sendmail.mc).

10.1 Edit submit.mc by adding the following entry to it:

INPUT_MAIL_FILTER(`dk-filter', `S=inet:8891@localhost')dnl

(for example just before “FEATURE(`msp’, `[127.0.0.1]‘)dnl”). Make sure that there are no define(`confINPUT_MAIL_FILTERS’, `…’)dnl lines after this; if there are, you will need to add dkim-milter manually to the INPUT_MAIL_FILTERS list.

10.2 Build and install a new submit.cf:

m4 /etc/mail/submit.mc > submit.cf

Tip: use m4 -d /etc/mail/submit.mc to debug first.

10.3 Restart sendmail

service sendmail restart

Testing tips

Some possible errors:

1. Errors getting the sendmail configuration generated: Check whether the dependencies (m4 and sendmail-cf) are installed and paths are correct.
2. Errors starting sendmail: Make sure you replaced the correct files (submit.mc to submit.cf and sendmail.mc to sendmail.cf) and if you modified sendmail.mc in addition to submit.mc, make sure you have regenerated both.
3. Errors starting dkim-milter: check the permissions on the key file
4. Sendmail seems to ignore the dkim-milter, no X-dkim-milter header is in the mail even after you enabled the X-Header option in dkim-filter.conf:
   1. Check /var/log/maillog.
   2. Sendmail seems to default to ignoring the milter if it cannot connect to it. This was the problem I ran into when using sockets; after switching to the loopback interface everything started working.
   3. Also, check whether your mail is sent from a recipient for whom mail is supposed to be signed! If you haven’t setup your hostname, then this may lead to email not being signed (eg. hostname -F hostname plus /etc/hosts plus /etc/sysconfig/network).  See, for example this tutorial for configuring sendmail.
   4. Also, check if you need to configure masquerading for Sendmail, see http://www.sendmail.org/m4/masquerading_relaying.html

Sending email from the console using only sendmail

Create a file with the following content:

To: "Recipient name" <john.doe@example.com>
From: "Sender name" <admin@example.com>
Reply-To: admin@example.com
Subject: Hello world
This is the content of the message, end it with a line containing only a period as sendmail expects this.
.

Then cat the file and pipe to sendmail -t:

cat message.txt |sendmail -t

Checking all of the technologies mentioned above

Port25.com offers a free service which check SPF, SenderID, DomainKeys and DKIM: http://port25.com/domainkeys/

Quote:

• If you wish to receive the results at the address in the “mail_from,” the sample message should be sent to check-auth@verifier.port25.com.
• If you wish to receive the results at the address in the “from” header, the sample message should be sent to check-auth2@verifier.port25.com.

A reply email will be sent back to you with an analysis of the message’s authentication status. The report will perform the following checks: SPF, SenderID, DomainKeys, DKIM and SpamAssassin.

Additional resources:

How to Setup DKIM for Postfix in Fedora using dkim-milter

How to Prevent Web Server Emails from being Marked as SPAM

DomainKey Implementor’s Tools and Library for email servers & clients

Sendmail DKIM

Setting up DKIM, SPF, Domainkeys DNS, Regular DNS on CentOS 5.3 at Pacificrack.com

How to manually install DKIM-Filter with Sendmail

Over the years I’ve read quite a pile of books particularly related to programming, general business, entrepreneurship and software engineering. Now that I am reading up on new topics such as organizational development, lean methods and statistical research methods I am having real trouble finding books I would consider reading just based on the table of contents and Amazon reviews. This is probably a good thing, since the most efficient way to get information is to only read good books.

Here are a couple of guidelines for picking good non-fiction books:

1. Number of reviews. If there is only a small number of reviews, there is a risk that those reviews are written by the author or his/her friends. A book with more reviews is more credible, as is a book which has had more editions.
2. Distribution of reviews. If the reviews are split between 5-star and 1-star reviews, the 5-star reviews are most likely fake. This is an absolutely disgusting practice, but unfortunately common.
3. Credentials of the authors. If the authors are practitioners and not scientists, make sure they do not work for a consultancy. I have unfortunately 1-2 books that I did not check adequately and ended up getting a useless brochure book.
4. The table of contents.
   1. What is the starting point of the discussion? This defines the prerequisite knowledge that you are expected to have. Reading a book which is for beginners is incredible boring, while reading an advanced book as a beginner is frustrating.Evaluate whether you have the prerequisite knowledge and could read the first few chapters without getting bored to tears. Remember that most popular business books are written for an audience that reads just a few books per year – this pretty much makes them too repetitive if you read more than that.
   2. What is the emphasis of the book? Is it oriented towards: a) practice or b) checklists, c) proper theory or d) undergraduate students textbook? (See below for more)
   3. How many pages are dedicated towards each topic? Don’t expect anything good from a book that has a 200 pages and cover 200 topics: that’s just one tiny page per topic.Good books don’t cover everything with the same small amount of pages, because everything is not equally important in real life.
5. The actual content of the reviews. Don’t bother with 5- or 4-star reviews, they will just tell you the book is good. Read the 1-star and 3-star reviews.See if the 1-star reviews seem to be justified or are just a result of the reviewer being an idiot (e.g. “Haven’t received book from Amazon” or “This book is too difficult” -reviews).3-star reviews are usually the best, since the author did like some aspects but did not like others. Pay attention to reviews that say that a book is too basic or simple. If this is said more than once, you probably don’t want to read that book – American writing in particular is already very verbose and too many books are written where a five-page article would do, hence reviews which state that the book is too simple should be taken very seriously.
6. Number of and affiliation of authors. If all the authors are from the same institution or organization, the book is probably of lower quality. A huge warning sign is a “collection” book that has multiple chapters on different topics by the same authors, all from the same decade. This indicates that the authors have not bothered to contact the people who are at the top of their respective fields; articles which are all from the same decade are likely to be just a random set of articles which were written for the book rather than the result of good editorial work.
7. Order of topics. Scientific collection books usually order the chapters so that the better and more widely applicable material is at the front of the book, and the more specialized and less interesting material is at the end. It is safe to assume that if the chapter that seems decent for your purpose is at the end of the book, you are better off finding a more specialized book.

Four classes of books:

I would put books in four broad categories, two of which are useful and two are useless:

1. Practical books
2. Academic books
3. Ego-booster books
4. Motivational books

Practical books: experience-based vs checklist books

Practical books are in my opinion divisible into two categories: advice from practitioners and advice from people who like to write checklists. That is, you have books that are based on the experiences of a practitioner, and then you have books that are written by people who have not actually done that much but wanted to write a book and ended up writing a checklist – a list of things that might be of use without really emphasizing what is important.

Try to get a feel of the book from the preview (if you can’t, don’t buy it!), search for a couple of phrases that interest you. Does the author know what they are doing, or are they just collecting information from other people? Sometimes a “checklist book” is not bad – when you just need the basics and can’t be bothered to collect all the information. However, usually the experience-based book is better.

Academic books: theory vs textbook

When it comes to academic books, the two main types are proper theory books and textbooks. The more I have seen and read textbooks, the less I like them. Generally speaking they try to cover everything and end up saying nothing.

If you care about the topic, don’t get a textbook. Instead, get a proper collection of influential articles if it exists (ex. Oxford Handbook of X). You will perhaps not be fully prepared for everything you read, but you will learn something useful. Unlike after reading a textbook, where you just have fuzzy awareness of the fact that “all kinds of things happen” in a particular field.

Ego-booster books: academic and practical

There is also another category of books which are basically books written for the author’s ego. Some books are not written to be read, they are written either to impress others or to act as marketing tools.

The academic substitute for having a nice car sometimes seems to be writing an obtuse treatise which is downright hostile towards the reader, while consultancies in particular are guilty of writing books in which the answer to every question is “hire our consultancy”. The academic ego book is harder to spot, so pay attention when previewing the book.

Motivational books: feeling good doesn’t make it so

Finally, there is a category of fiction books which pretend to be non-fiction books. The motivational books may be a fun read at times, and are usually the most popular books in their category.

However, they are fiction for your entertainment. Feeling good is nice, but that does not change the fact that there are people who are knowledgeable, talented and well-connected doing whatever the book is about, and that feeling good does not make you knowledgeable, talented or well-connected.

0. Preliminary setup: make sure each computer gets a constant IP address

Before setting up the DNS server, you need to ensure that each computer or virtual machine gets a fixed IP address. Otherwise the IP address of the computer may change on each reboot.

This should be done on your router, which is usually located either at 192.168.0.1 or 192.168.1.1 (check your current local IP address by running the Command Prompt and “ipconfig”, which shows your current IP address). Check your router manual on instructions how to log in, most routers have a HTTP-based configuration system which can be accessed when connected to the router.

In my case (I have a Asus RT-N11), the address was 192.168.1.1. Make sure that the computers you want to setup IPs for are connected, then find the “Status” or “DHCP Leases” listing on the router web interface. This listing will contain the MAC addresses for each host. Here is mine:

Host Name       MAC Address       IP Address      Lease
--------------------------------------------------------------
HOST1        00-11-22-33-F0-AC 192.168.1.5     60016 secs.
HOST2        00-11-33-44-F2-2C 192.168.1.6     85074 secs.

A MAC address is a unique identifier given to each network adapter. It allows you to setup fixed IPs.

Find the router functionality which allows you to “Assign IP Addresses Manually”. This should enable you to specify a MAC address and a corresponding fixed IP address. Do this for each of the network adapters you wish to have a fixed IP address. On the Asus RT-N11 this was under “IP Config” -> “DHCP Server”.

Add a MAC address <-> IP address pair for each computer.

1. Get MaraDNS

MaraDNS is a free, lightweight and relatively easy-to-configure DNS server for Windows and Linux. Download it from here and unzip it to some folder.

2. Configure MaraDNS

Open “secret.txt” and change the value to something else (random characters).

The MaraDNS configuration is in the “mararc” file in the same directory. DNS servers have two sets of functionality. They can function as a “Authoritative name server” or a “Recursive/caching name server”.

Authoritative name servers specify IP addresses for domain names. Recursive name servers store information from authoritative name servers and pass on queries in a recursive manner.

We will be configuring both authoritative and recursive functionality in MaraDNS.

2.1 Authoritative configuration

We will configure the server to provide authoritative names of the LAN domain names. Pick any domain, I chose “local.com” (note though that you will not be able to access the actual “local.com” website if you pick an existing domain name).

Add configuration lines to “mararc” like these:

csv2 = {}
csv2["local.com."] = "db.lan.txt"

Where local.com is the domain name you picked, and db.lan.txt is the name of the second configuration file which we will be creating next (change it if you want to name the second configuration file).

Create a new file named “db.lan.txt” in the same directory as MaraDNS.

For each of the computers you want to resolve to a name, add a line to “db.lan.txt”. For example, for two machines, one “dev.local.com” and the other “blog.local.com”, add the following lines:

dev.%       192.168.1.4 ~
blog.%        192.168.1.6 ~

Done!

2.2 Recursive configuration

We will setup MaraDNS to ask your default name servers for all other domains so that you can resolve all other domain names to their correct IP addresses.

Find out your ISP’s DNS server addresses. These are likely to be listed either on the Router status page, or by checking the details on your network adapter.

Now add your ISP’s DNS servers as upstream servers in “mararc”:

upstream_servers = {}
upstream_servers["."] = "xxx.xxx.xxx.xxx, yyy.yyy.yyy.yyy"

Where xxx.xxx.xxx.xxx and yyy.yyy.yyy.yyy are your ISP’s DNS servers.

Done!

3. Run MaraDNS and test it using askmara.exe

Double-click “runmara.bat” , and leave the server running.

Open a command prompt, navigate to the MaraDNS directory and try running:

askmara.exe Agoogle.com.

and

askmara.exe Ablog.local.com.

You should get replies like this:

# Querying the server with the IP 127.0.0.1
# Question: Agoogle.com.
google.com. +300 a 74.125.67.100
google.com. +300 a 74.125.53.100
google.com. +300 a 74.125.45.100
# NS replies:
# AR replies:

and:

# Querying the server with the IP 127.0.0.1
# Question: Ablog.local.com.
blog.local.com. +86400 a 192.168.1.6
# NS replies:
#local.com. +86400 ns synth-ip-7f000001.local.com.
# AR replies:
#synth-ip-7f000001.local.com. +86400 a 127.0.0.1

If you get problems with the first query, you messed up the recursive DNS settings (are your ISP DNS server addresses correct?), and if you get an error with the second query, you messed up the authoritative settings.

4. Change MaraDNS to reply to queries from your LAN

Shutdown the MaraDNS window, and change the first two lines of “mararc” to something like:

ipv4_bind_addresses = "192.168.1.2
recursive_acl = "192.168.1.0/24"

Where 192.168.1.2 is the IP address of the computer on which the server will be running and the “192.168.1″ part of recursive_acl is the same as on your network (might be 192.168.0.0/24).

Start MaraDNS again, and leave it running.

5. Setup your router to hand out your new DNS server

Open your router’s web interface and find the DHCP server settings. There should be an option to set up a DNS server. Write the IP address of the computer on which the DNS server will be running.

For each of your computers, disconnect the network (e.g. by disabling and enabling it in Windows, or by using “ifconfig eth0 down”/”ifconfig eth0 up” on Linux).

That’s it, you should now be able to refer to your LAN computers by their domain names.