things which you probably knew how to do in Linux but which i didn't

this collection of little howtos was built up as i spent time figuring out how to do little things in Linux. most of these were first performed in Fedora, since that was my first distribution, but later i began dabbling in Debian and Ubuntu. some of these are specific to one distribution or another, but most should apply to any distribution.

mistakes? questions? know a better way? i can be contacted at bart at bnagel.NOT (net, not not).

the primary audience for this document is myself and for that reason i don’t provide any guarantee that any of this will work – i’ll only say that it worked for me at one point – and i haven’t gone into a great amount of detail. i thought perhaps this would be useful to others, so here it is.

transcode WMA and such to something better

i generally go for Vorbis quality 6 when the source is lossless, but if going from a WMA it probably isn’t worth it. go for the default, which is 3, probably.

extract, then encode
  1. mplayer -ao pcm:file=tempfile source
  2. oggenc -o destination -q quality --artist artist --album album --title title --date year --tracknum tracknum tempfile
extract and encode at the same time with a named pipe
mkfifo fifo; (oggenc -o destination -q quality --artist artist --album album --title title --date year --tracknum tracknum fifo; rm fifo) & mplayer -ao pcm:file=fifo source

copy stuff over the network

scp file user@remotehost:destinationpath

either or both files can be remote, destinationpath can be a directory or full path including filename.

bandwidth can be limited with the l switch (Kbit s−1), for instance scp -l 90000 justripped/*.mkv perigee:/srv/media/video/film/ to leave a bit of room on a 100Mbit network

use the p switch to preserve modification times, access times and modes

that means the port switch has to be P port rather than lowercase p.

start an x app remotely, on the remote machine

ssh in – the user who owns the x session must run this (not even root)

DISPLAY=:0 command &

probably want to combine this with nohup so it doesn't die when you log out.

start an x app remotely but have it display on the local machine

ssh in with the X flag, then just run the program: ssh perigee -X

ssh without a password

  1. make sure you have a public/private RSA key pair (get one with ssh-keygen -t rsa, no passphrase is fine)
  2. add id_rsa.pub which is now in ~/.ssh as a new line to ~/.ssh/authorized_keys on the remote machine (which should be chmodded 600): cat id_rsa.pub >> ~/.ssh/authorized_keys

disable samba unix extensions

put unix extensions = no in /etc/samba/smb.conf in the [global] section

search within files

on particular files

grep searchstring file list

recursively

this searches in all files in this directory and subdirectories

grep -r searchstring .

use the l switch to print the names of the files with matches rather than the matches themselves

the i switch for case-insensitive

the v switch for an inverse search (return lines/files which don’t match)

egrep instead of grep to use extended regexes

list all installed packages from a certain repository (Fedora)

rpm -qa --qf "%{VENDOR} -- %{NAME}\n" | grep -i repo

set the command run when power button is pressed

Fedora

edit the file /etc/acpi/events/power.conf – mine looks like this:

event=button/power.*
action=/usr/bin/bwarnshutdown

Ubuntu
see that /etc/acpi/events/powerbtn points to /etc/acpi/events/powerbtn.sh, so edit that

send a message to all users’ consoles

use the wall command. note that consoles can be set not to receive these (local ones are usually set like that for instance)

fix the presence of both official ati and fglrx drivers (Fedora Core 6)

  1. remove kmod-fglrx
  2. updatedb
  3. locate fglrx | xargs rm -rf
  4. rm -rf /usr/share/ati /usr/src/ati
  5. backup xorg.conf
  6. install kmod-fglrx again
  7. restart x
  8. replace xorg.conf

boot in runlevel 3 instead of 5 (Fedora)

change the 5 to 3 in /etc/inittab

flash a dvd drive to be region free

  1. go to tdb.rpc1.org and get the right file
  2. get binflash from binflash.cdfreaks.com
  3. as root: ./necflash -scan
  4. get the exsting firmware: ./necflash -dump destination /dev/drive
  5. flash the new firmware: ./necflash -flash firmware.bin /dev/drive

find out what plugins firefox has installed

go to about:plugins

run a process which won't die when you log out

either start it with nohup:

nohup command &

or, to make it not die when already running use disown after getting it running in the background:

disown %jid

list directories only

ls -d */ or ll | grep ^d

to go recursive, you can use tree -d to show a tree, or stick the -R flag on the one with grep, or use a find . -type d

split a wave file

wavbreaker (an x app)

sync two folders

only newer files are copied. anything not in the source folder is deleted from the destination. be careful with characters which aren't allowed on the target filesystem. my script:

#!/bin/bash
rsync --delete --modify-window=1 -av ~/media/music/ /media/IAUDIO/music/ > /var/log/syncx5.log >>&1 | grep -i argument

fat is retarded and makes folders made up of just uppercase lowercase. therefore it'll delete and re-copy the contents of such folders over each time. never mind, eh?

alternatively, use unison, which is based on rsync and can sync in both directions.

install british english support for KDE (Fedora)

yum install kde-i18n-British

stop standard input

control-d

run a command at a specific time

at time

then give commands and stop standard input. no seconds. instead of time, can also give a datetime. understands noon, teatime (4pm), midnight, tomorrow and some other bits.

can also add a relative time, for instance now + 20 minutes (now/tomorrow) (minutes/hours/days/weeks).

check the manpage

output is mailed to the user with sendmail. the DISPLAY environment variable isn't preserved, so if an X app should be loaded, prefix the command with DISPLAY=:0 or similar.

have colours on the bash prompt

add this stuff to ~/.bashrc (can’t remember where i got it)

and then add things like $RED to $PS1 or $PS2, the prompt variables

split output to redirect to a file and to screen

either use tee, as in ls | tee file or use tail -f as in ls > file & tail -f file. the latter might not print the first bit to standard output if it starts quickly but it'll all be in the file. nevertheless, it's probably better since viewing output can be switched on and off by stopping or starting tail.

better still, run tee while in a screen. then the screen can be detached and reattached later and all previous output will still be visible. screen commands: control-a and then

c
start a new screen
d
detach screen
n/p
next/previous screen
esc
enter copy mode, letting you scroll up and down. esc to abort

log out to end any particular screen. screen -r to reattach a screen if it’s been detached.

install windows fonts (Fedora)

follow instrux at corefonts (sourceforge). the RPM isn’t in a repo so you build it yourself and have to rebuild to update it.

log into mysql without a password

create the file ~/.my.cnf with contents like:

[mysql]
user=user
password=pass
[mysqladmin]
user=user
password=pass

have local mail send out to an external email address (eg Gmail)

  1. install postfix
  2. edit /etc/aliases to point mail to local users on to the external address
  3. have a look at /etc/postfix/main.cf, in particular the stuff about the local domain

note that since it’s not sending from a proper domain it way well end up in the spam box.

reencode a folder full of audio files (FLAC, wav, whatever) to Vorbis

oggenc -q quality --discard-comments --artist artistname --album albumname --date year *.flac

i usually go for quality level 6. it might be overkill, but never mind. the discard-comments bit gets rid of existing tags (eg id3, which are sometimes on flac files). then run easytag . to tag the track numbers and titles and rename the files or picard it up.

strip id3 tags from mp3 files

on Fedora:

on Ubuntu, install id3v2, then

watch what a program does

you can find out which files a program is opening, for instance

strace command 2> outputfile

set up an ftp server

  1. install vsftpd
  2. edit /etc/vsftpd/vsftpd.conf
  3. switch the service on

get a remote Windows desktop from uni

this works now – before i had the wrong port number for RDP. mushroom is the host to connect to once through the tunnel, bpp should probably be 8, 16 or 24 and geometry should look like 1024x768.

ssh -f -L 3389:mushroom:3389 bjn104@linuxproj.ecs.soton.ac.uk sleep 5; rdesktop -a bpp -g geometry localhost

make programs (in wine) think IE is installed

from an faq linked in #winehq on freenode: go to the useful registry keys list on winehq and add the internet explorer ones using regedit. then run wineprefixcreate.

mount an iso CD image

mount -t iso9660 isopath mountpoint -o loop

merge AVI files

avimerge -o outfile -i infile infile

send a custom http header (or any other network traffic)

use nc:

echo -ne "GET /path/including/inital/slash.html HTTP/1.1\r\nHost: host\r\n\r\n" | nc host port

eg echo -ne "GET / HTTP/1.1\r\nHost: www.bnagel.net\r\n\r\n" | nc www.bnagel.net 80

upload offline email to Google mail, preserving dates

  1. set up a POP server on your computer. i used dovecot, which was very easy to set up.
  2. store the mail which is to be transferred in mbox format (or anything else dovecot supports) where dovecot expects that user’s mail to be.
  3. can verify dovecot’s working locally, and then remotely (remember to forward port 110), with netcat
  4. point google mail’s mail downloader to your computer and watch the mail appear

scale a JPEG from the commandline

use convert.

convert infile -resize widthxheight -quality 85 outfile

also look at mogrify, which i’ve used in the past. i don’t recall what the difference is. it could be that one resamples while the other does not. it could be that convert lets you specify an output file rather than replacing the input.

append images from the commandline

this code takes two images and puts them together side by side with no gaps:

convert desk_left.png desk_right.png +append wholedesk.png

-append would do it vertically

install web server with PHP (and MySQL) support (Fedora Core 6)

  1. install lighttpd and lighttpd-fastcgi
  2. i already had php and mysql, so can’t say for sure, but probably need php-cli, php-mysql and mysql-server
  3. edit /etc/lighttpd/lighttpd.conf:
  4. update permissions for the sessions directory (in my case, it was owned by the apache group): chown :lighttpd -R /var/lib/php/session
  5. disable httpd (apache) for all runlevels if applicable
  6. enable lighttpd for all runlevels: chkconfig lighttpd on
  7. start it up: service lighttpd restart

batch rename

i was writing little bash scripts every time i wanted to do this, but there's a much better way.

rename "s/regexfrom/regexto/" files

use the n switch to test before making actual changes.

according to the manual, you can also pipe filenames to it, eg from find. haven’t tested this though.

note: i did this on Ubuntu and Debian. i’ve just tried on Fedora and there seems to be a completely different version there, which doesn’t take a s/from/to/ thing.

delete all backup (*~) files beneath the present directory

find . -type f -name "*~" -exec rm {} \;

search and replace text in all files found with find

find . -type f -name whatever -exec sed -i 's/regexfrom/regexto/' {} \;

careful. check the sed command on one file first.

for proper regexes, add the r switch to sed.

set up a tcp tunnel

this sets up a tunnel to uni.

ssh -D 23456 bjn104@uglogin.ecs.soton.ac.uk

then set up whatever program to go through a proxy server (localhost) on that arbitrary port (23456).

set up a reverse tcp tunnel

awesome! this means if you want to log into a computer but don’t want to mess around with its network’s firewall, but you can run a command on that machine (tell someone to do it over the phone, or do it physically) you can still get to it.

this (run on otherbox) sets up a tunnel from perigee to otherbox:

ssh -nNT -R 2222:localhost:22 username@perigee.tremby.net

then on perigee username can run ssh -p 2222 username@localhost

rot13

it’s not in Ubuntu’s repo, so i found this command on linuxquestions.org:

echo string | tr 'a-zA-Z' 'n-za-mN-ZA-M'

or put it in a bash script:

#!/bin/bash
echo $@ | tr 'a-zA-Z' 'n-za-mN-ZA-M'

and then run rot13 "fbzr grkg"

mount a remote filesystem through ssh

sshfs username@host:path mountpoint

and then to unmount, fusermount -u mountpoint

make an ISO from a CD

  1. put the CD in
  2. ensure it isn’t mounted
  3. dd if=/dev/cdrom of=file.iso

switch off screens from commandline

xset dpms force off – it comes back on with input like it usually would.

to force back on from the commandline, xset dpms force on

losslessly rotate JPEGs

rotate 90 degrees clockwise (use 270 for ccw):

jpegtran -rotate 90 -outfile outfile infile

use FTP on other operating systems

Windows
  1. Open the Run dialog with windows+r
  2. Type explorer ftp://username@server and click ok
Mac OS 10+
  1. Open Finder
  2. Bring down the go menu and click connect to server
  3. Fill in the server address at the bottom and connect

backup a DVD

work in progress

mashed together from a howto, a post on the Ubuntu forums and my own experience.

  1. install some apps: mplayer mencoder vorbis-tools ogmtools mkvtoolnix mkvtoolnix-gui gpac x264 transcode subtitleripper
  2. rip DVD to disk: STREAM=streamnum; DVDDEVICE=devicepath; VOBFILE=vobfile; mplayer dvd://$STREAM -dvd-device $DVDDEVICE -dumpstream -dumpfile $VOBFILE
    streamnum is often 1, but not always, and the dvd-device switch may not be needed (was on Yappy’s machine, and it was /dev/hda)
  3. get chapter titles: STREAM=streamnum; DVDDEVICE=devicepath; dvdxchap -t $STREAM $DVDDEVICE > chapters.txt and then edit them if necessary

  4. check which audio and subtitle ids are which with STREAM=streamnum; DVDDEVICE=devicepath; mplayer dvd://$STREAM -dvd-device $DVDDEVICE -v -vo null -ao null -frames 0 2>/dev/null </dev/null | grep "aid\|sid\|Found subtitle"

    alternatively, use lsdvd, with the a switch for audio, s for subtitles or x for all, which includes title lengths. to convert its hex stream ids into decimal for AID (or SID+32), do echo "ibase=16; capitalhex" | bc

    if they’re ambiguous (28 Days Later has two English subtitle tracks and two English audio tracks – one of each is commentary), test with mplayer: mplayer vobfile -sid sid and mplayer vobfile -aid aid

  5. extract each subtitle required

    1. VOBFILE=vobfile; SID=sid; SUBFILE=subtitlefilename; tccat -i $VOBFILE | tcextract -x ps1 -t vob -a 0x`echo "obase=16; 32+$SID" | bc` > $SUBFILE
    2. extract the images: SUBFILE=subtitlefile; mkdir ${SUBFILE}.images; cd ${SUBFILE}.images; subtitle2pgm -o ${SUBFILE}.i -c 255,255,255,0 < ../$SUBFILE (experiment to see which as 0 gives the centre part of the text – there seems to be no standard)
    3. perform OCR: SUBFILE=subtitlefilename; pgm2txt ${SUBFILE}.i
    4. collect the separate files into one SRT file: SUBFILE=subtitlefilename; srttool -s -w < ${SUBFILE}.i.srtx > ../${SUBFILE}.srt
    5. edit the output – it helps to do a spellcheck first, a few global search and replaces and then have a quick read through
  6. get audio tracks

    extract existing AC3 audio

    AC3 is lossily compressed, and Vorbis isn’t good at multichannel

    VOBFILE=vobfile; AID=aid; AC3FILE=ac3file; mplayer $VOBFILE -aid $AID -dumpaudio -dumpfile $AC3FILE -vc dummy -vo null

    there is suspiciously little output, but it’s going.

    transcode to Vorbis

    a good idea for commentary tracks, which don’t need to be fantastic quality

    1. get rescale factor: VOBFILE=vobfile; AID=aid; (tcscan -x pcm -i fifo | grep "suggested volume rescale") & mplayer $VOBFILE -aid $AID -ao pcm:nowaveheader:file=fifo -vo null -vc dummy -benchmark >/dev/null 2>&1
    2. extract the audio with calculated boost and encode to Vorbis (quality 0 is plenty for commentary, haven’t experimented yet for main audio, but 4 probably tastes great. 3’s fine for TV shows): VOBFILE=vobfile; AID=aid; VORBISQ=quality; VORBISOUT=oggfile; SCALEFACTOR=scalefactor; oggenc -q $VORBISQ -o $VORBISOUT fifo & mplayer $VOBFILE -aid $AID -af volume=`echo "20*l($SCALEFACTOR)/l(10)" | bc -l` -ao pcm:file=fifo -vo null -vc dummy -benchmark >/dev/null 2>&1

    quite disgusting, but those two steps in one command: VOBFILE=vobfile; AID=aid; VORBISQ=quality; VORBISOUT=oggfile; mkfifo fifo; (RESCALE=`tcscan -x pcm -i fifo | grep "suggested volume rescale" | sed -r "s/.*rescale=(.*)/\1/"`; echo "boosting ${RESCALE}dB"; oggenc -q $VORBISQ -o $VORBISOUT fifo & mplayer $VOBFILE -aid $AID -af volume=`echo "20*l($RESCALE)/l(10)" | bc -l` -ao pcm:file=fifo -vo null -vc dummy -benchmark >/dev/null 2>&1 </dev/null) & mplayer $VOBFILE -aid $AID -ao pcm:nowaveheader:file=fifo -vo null -vc dummy -benchmark >/dev/null 2>&1 </dev/null

    the </dev/null bit towards the end lets mplayer wait patiently in the background – otherwise it and other interactive programs block until they are in the foreground. no good if it’s waiting for the other end of a pipe to open.

  7. figure out crop settings: run mplayer -vf cropdetect -sb 50000000 vobfile, navigate until all areas have been covered, then quit. see the last crop area output line. test it: mplayer vobfile -vf crop=cropsettings
  8. check if it’s interlaced (teeth wherever there’s movement), telecine (steady pattern of progressive and interlaced frames), progressive (each frame is complete) or some horrible mixture (check intro sequence, scene transitions and so on).

    interlaced
    experiment with deinterlacers – i found yadif to look good but you get some frames of green at the start. lavcdeint seems ok. kerndeint seems better, and needs the scale filter to be active (though it doesn’t actually have to do any scaling).
    telecine
    try the pullup and filmdint filters to see which is best. each of these needs the softskip filter to go after it. inverse telecine slows the framerate to 24000/1001.
    progressive
    rejoice, no extra filters are needed
    some horrible mixture
    it’ll have to be a compromise (bloody NTSC)
  9. encode video. since i don’t care about the file size (i’m not into archiving to CDs and so there’s no point in having a 700M target size) i’m using x264’s constant quality mode, CRF. this also means only one pass is necessary, ensures quality and apparently gives results equal to two-pass average bitrate mode.

    VOBFILE=vobfile; FRAMERATE=framerate; CROPSETTINGS=cropsettings; CRF=crfvalue; OUTFILE=outfile; mencoder $VOBFILE -nosound -ofps $FRAMERATE -vf crop=$CROPSETTINGS -ovc x264 -x264encopts b_pyramid:weight_b:bframes=3:threads=auto:me=umh:trellis=1:mixed-refs:bime:b-rdo:subq=7:frameref=5:8x8dct:crf=$CRF -of rawvideo -o $OUTFILE

    crfvalue is a scale from 0 (lossless) through 18 (pretty much transparent) through 24 (very decent) through 30 (pretty lossy) to 35 (crap) and beyond. 26 seems great.

    for animation change frameref and bframes both to 16.

    framerate is best using the fractional value rather than the decimal, for instance 24000/1001 instead of 23.976.

  10. multiplex it: load mmg and add all the files, chapters etc. add DVD cover titled “DVD cover” (not standard, but anything with “cover” is noticed by at least one shell extension). don’t forget to specify the framerate for the x264 file.

have PHP interpret a script

this is probably quite obvious, but i hadn’t tried it until now. simply use the hashbang, so something like this:

#!/usr/bin/php
<?php
echo "here i am\n";
?>

run commands at login or logout

kde
put scripts or symlinks in ~/.kde/Autostart or ~/.kde/shutdown but beware that shutdown ones won’t happen unless a KDE shutdown/reboot button is pressed (ie not counting reboot etc)
gnome
for startup, add stuff in the preferences -> sessions dialog. haven’t tried to do stuff at logout yet

if something must always run at boot/shutdown, make an init script for it. that will run as root, but you can use su to run as any given user.

output the nth line of a file

head -n file | tail -1

-n is short for -n n.

allow passwordless shutdown for admin users (Ubuntu)

add the following to the end of /etc/sudoers:

Cmnd_Alias SYSTEMPOWER = /sbin/poweroff, /sbin/reboot, /sbin/shutdown
%admin ALL = NOPASSWD: SYSTEMPOWER

have Bluecurve cursors on Ubuntu

i have missed these lovely things (the friendly blue smudge in the middle, particularly) since switching to Ubuntu.

someone has packaged them on kde-look (but they’re fine for any WM)

split FLAC+cue into many FLAC files

  1. install shntool and cuetools
  2. cuebreakpoints cuefile | shnsplit -o flac flacfile

set up wake on lan at boot

network cards seem to forget that they're meant to wake on lan next time they boot, so they need reminding with a command like ethtool -s eth0 wol g

stick a script which runs that in /etc/init.d and then install the rc scripts with update-rc.d wakeonlanconfig start 20 2 3 4 5 . stop 80 0 1 6 .

convert MDF CD image (Alcohol 120%) to ISO

  1. install mdf2iso (in Ubunutu’s repos)
  2. mdf2iso file.mdf

pan something hard left or hard right in mplayer

mplayer whatever -af pan=2:1:0:1:0

the 2 is the number of output channels (probably always 2), then the next arguments are proportion of the each input channel to each output channel:

iL→oL:iL→oR:iR→oL:iR→oR

the above command will pan it hard left.

change the button order in Metacity’s window decorations

  1. run gconf-editor
  2. navigate to /apps/metacity/general
  3. edit “button_layout”

change a MAC address

sudo ifconfig device hw class address

class is most likely ether. network interface can’t be in use when this is run – disable networking first (harmless error otherwise).

this is not permanent – it’ll have to be repeated after a reboot/suspend.

copy something to the clipboard or paste from the clipboard, but on the commandline

use xclip. its default action is to copy from stdin or a file if a filename is given. pass -o to paste to stdout.

by default, the “x selection” register is used, which is set whenever you select something and can be pasted with a middle click. this is not the same as the clipboard (control-c/v, control/shift-insert) – to use that pass -selection clipboard.

turn caps lock into another control key

use the X keyboard option ctrl:nocaps, for instance setxkbmap gb dvorak compose:ralt ctrl:nocaps

turn right control into another mod4 key

xmodmap -e "remove control = Control_R"
xmodmap -e "add mod4 = Control_R"

the two expressions can instead go in a file, for instance .xmodmaprc, and loaded with xmodmap .xmodmaprc from elsewhere, like .xinitrc

Get random data from the command line

Pick the first 64 hex characters out of random data:

cat /dev/urandom | tr -dc '0-9a-f' | head -c 64

Pastebin some stdin

My sprunge script:

#!/bin/bash
url=$(curl -F 'sprunge=<-' -H "Expect: " http://sprunge.us 2>/dev/null)
if [ $? -ne 0 ]; then
	echo "request failed"
	exit 1
fi
if [ $# -gt 0 ]; then
	url="$url?$1"
fi
echo $url
if [ $DISPLAY ]; then
	which xclip >/dev/null 2>/dev/null
	if [ $? -eq 0 ]; then
		echo -n $url | xclip
	fi
fi

The -H "Expect: " gets around a problem with sending the request through the Squid proxy. Not sure if that's a bug in Squid or what.