Assumptions:
- webserver root directory is /var/web
- production node is called prod
- development node is called dev
- WordPress database is called wpdb

Procedure to copy production WordPress instance to the development node:
1. Copy webserver www root dir via a tarball
tar czf prod-20110909.tgz /var/web

2. Dump the WordPress database to a MySQL dmp file:
mysqldump -u$mysqluser -p$mysqlpass wpdb | \
 gzip -c > prod-20110909.dmp.gz

3. Copy these two backup files to the dev node:
scp prod-20110909* user@dev:.

On the development node:
4. Unpack the webserver tarball:
mv /var/web /var/web.previous
cd /
tar xzvf prod-20110909.tgz

5. Drop the WordPress database and restore the new version:
mysql> drop database wpdb;
mysql> create database wpdp;
$ gunzip prod-20110909.dmp.gz
$ mysql -u$mysqluser -p wpdb < prod-20110909.dmp

6. Update the WordPress 'siteurl' and 'home' options to point to the development node:
update wp_options set option_value='http://dev.pomeroy.us' where option_name='siteurl';
update wp_options set option_value='http://dev.pomeroy.us' where option_name='home';

Should be all done!

Technology has evolved since the last MythTV PVR I built, as chronicled here.  Here’s the latest techniques and tech that I’ve used to (start) build(ing) my current PVR. I’ll update this article as I go, as there’s been some bumps along the way, so completion of the project has been slower than I anticipated.

Requirements for my new PVR include:

• Linux operating system for cost and flexibility reasons
• Quiet! Fan-less operation if at all possible, external power supply ok
• Small form factor, black case to fit in with my current home theater gear
• Video capture with MPEG-2 hardware acceleration to help keep the CPU needed as small as possible, in an expansion card format for the most compact physical footprint .. additionally there must be at least two independent tuners
• Analog tuners, but would be good if they were capable of digital for when I eventually move to digital/HD
• IR receiver and transmitter capability for easy remote control and ability of the PVR to use my current set-top box as a source (gives me all the cable company movies and channels that are not available via the basic cable connection
• Ability to schedule at least 10 shows and retain 5 episodes of each show .. also ability to schedule based on show name alone
• Ability to perform post-recording processing, such as removing commercials or changing formats
• Should be able to use a pre-packaged distribution for most if not all of the functions .. I know it’s a home-brew, but I’m tired of messing with individual packages, firmware, and custom codes to make it work. Using a distribution package makes it easier to maintain through updates.
• Want to purchase the parts from the same supplier if possible (ended up using newegg.ca)

Since I already run MythTV, it was an obvious starting point and given I don’t have an affinity to a specific Linux distribution, I looked at Mythbuntu and Mythdora since I’m familiar with and already run both Ubuntu and Fedora distributions.

After downloading the Mythbuntu 10.10 ISO disk image, I discovered I didn’t have my USB DVD drive, so I wanted to create a bootable USB flash disk.  I followed the excellent instructions at https://help.ubuntu.com/community/Installation/FromUSBStick and successfully burned a bootable Mythbuntu disk on a 2GB USB flash disk via a Ubuntu VM running on my MacBook Pro.

The Hardware

The hardware that I chose to use included:

• An Antec ISK-300-65 case, good for fan-less operation
• ASUS AT5IONT-I mainboard dual core Atom D525 CPU
• Hauppauge WinPVR-2250 dual tuner PVR card with MPEG-2 hardware acceleration (PCI-express)
• 4GB DDR3 SO-DIMM memory (2x 2GB)
• 2x 750GB 2.5″ SATA HDDs
• My existing Microsoft MCE USB IR receiver/blaster and remote

I evaluated the very cool and potentially high performance hybrid HDD/SSD disks, but there were too many experiences users expressed that were sub-optimal, most stating the technology is too new. Having a terabyte 2.5″ disk with 4GB of SSD would be sweet, but for now I’m just sticking with 750GB 7200RPM 2.5″ SATA disks. Since I changed my mind and I’m not going to put a DVD drive into the case, I chose to put another HDD in and mirror them up (since there are two SATA adapters on the mainboard and space in the case for two HDD).

The ASUS mainboard is designed for fan-less operations, and coupled with the Antek case as one massive heat sink, it is incredibly quiet. Video outputs are all handled by the mainboard versus the video capture card and include DVI, HDMI and component video outputs. On initial power on, I was somewhat underwhelmed, since although the power on button turned on the blue power light on the mainboard, then spun up the disk and fan, no joy on the mainboard BIOS POST. After some Googling, I found the Asus board uses the very finicky Intel memory controller that is used with the Atom CPU. I purchased a pair of KVR1066D3S7/1G (Kingston 1GB 204-Pin DDR3 SO-DIMM DDR3 1066 (PC3 8500) Laptop Memory) to boot the AT5IONT-I far enough to get the BIOS updated. See the forum thread here for other people’s experiences. Version 312 of the ASUS BIOS did not support the 2GB DIMMs so I was a bit annoyed that I had to purchase 1GB DIMMs (Kingston KVR1066D3S7/1G) in order to get into the BIOS.  I downloaded the 316 BIOS ROM image from the ASUS website and put it onto a FAT formatted USB memory stick, thinking I’d have to go through the pain of booting some form of Windows or DOS to run some lame BIOS updater utility. I was pleasantly surprised to find a BIOS update utility built into the BIOS! All I had to do is plug in the USB stick and select the option to update the BIOS. It worked! Not only the most painless BIOS update I’ve ever done, now the 2GB memory DIMMs work (anyone want to buy my 1GB DIMMs for the cost of shipping?). On to the installation of Mythbuntu.

I originally wanted to have a slim DVD drive to play DVDs but then realized that I don’t even have any movies on DVD any more.  All the oldie goldies that I had, I now have copies in iTunes. Since the mainboard only supports two SATA interfaces, I chose to reserve one for a future redundant HDD (as it turns out I just ordered the extra disk when I purchased the 1GB DIMMs).

The Hauppauge card is a dual-tuner analog/digital that has an IR receiver and blaster – so it can change channels on a cable set top box. The 2250 also has dual tuners so that the conflicts that I often encountered with a single tuner can be avoided. 

OS Install

I tried a couple of All-In-One distributions (Mythdora and Mythbuntu) and even a couple of versions of each.  Seemed like I ran into issues with both distros in different areas. Mythbuntu 10.10 wouldn’t save the Video Sources. Mythdora had a better setup interface than Mythbuntu 10.10, but would not setup a default route for some reason – all the subsequent updates and package installs would obviously fail.  Sigh. Doing a base install of Fedora 14 then installing from ATrpm repositories would go better for the OS install (including full mdadm mirroring of the two SATA drives), but compiling the Hauppauge HVR 2250 analog driver from Steve Toth’s excellent support site would fail with usb_ function call mismatch errors. Apparently the usb_ memory function definitions have changed in recent 2.6 kernels. Arrrg!

Fortunately I set this aside for a while and in the mean time, Mythbuntu came out with release 11.04 … would it work??

So now it works for analog .. exactly what I wanted. Ironically I don’t need the digital tuners for a while yet.

Here’s how:

I downloaded Mythbuntu 11.04 64 bit ISO and created bootable USB flashdrive via
http://www.pendrivelinux.com/universal-usb-installer-easy-as-1-2-3 on my HP notebook (Windows 7). Booted off the USB and selected the Install option. Ultimately I wanted to partition the drives and use mdadm software RAID 1 with LVM2 on top for partition and filesystem management options. No matter how I tried, the Mythbuntu 11.04 installer just would not let me do an install in that configuration. So I did a vanilla install, configured things the way I wanted THEN did a transition to LVM2 mirroring setup.

1. Use USB stick to boot Mythbuntu 11.04 and perform MythTV install
Use the following partition table on /dev/sda with all primary partitions and ignore /dev/sdb for now – note you’ll need to use /srv for the MythTV storage fs as those are the mount point options available in the install image. Don’t worry, we’ll change it later to /storage and only /dev/sda1 (/boot) will remain after we’re done the conversion to LVM mirroring.
/dev/sda1 /boot 150MB
/dev/sda2 / 8GB
/dev/sda3 swap 8GB
/dev/sda4 /srv 630GB

Once the install is done, change the /srv filesystem to /storage and make it owned by user mythtv then create the storage directories that MythTV will use for LiveTV and Recordings.
umount /srv
vi /etc/fstab (change /srv to /storage)
mkdir /storage
mount /storage
mkdir /storage/livetv
mkdir /storage/recordings
mkdir /storage/db_backups
chown mythtv:mythtv /storage/*

2. Compile V4L code
Install tools needed to get and build the code
apt-get install git patch patchutils libproc-processtable-perl gcc make

Get the code base and build it – based on the forum post “How to Obtain, Build and Install V4L-DVB Device Drivers”
git clone git://linuxtv.org/media_build.git
cd media_build
./build
sudo make install


3. Download HVR 2250 firmware and install in /lib/firmware
wget http://www.steventoth.net/linux/hvr22xx/firmwares/4019072/NXP7164-2010-03-10.1.fw
sudo cp NXP7164-2010-03-10.1.fw /lib/firmware

I found that the HVR 2250 card wasn’t completely recognized even with the firmware in place, as seen in dmesg output .. and no /dev/video* or /dev/dvb/… devices were created. Googling found a few forum posts that discuss the issue .. the solution: create /etc/modprobe.d/saa7164.conf and force a card selection.

Jun 23 22:17:54 pvr kernel: [10.642158] saa7164 driver loaded
Jun 23 22:17:54 pvr kernel: [10.642321] saa7164 0000:07:00.0: PCI INT A -> GSI 19 (level, low) ...
Jun 23 22:17:54 pvr kernel: [10.643371] saa7164[0]: Your board isn't known (yet) to the driver.
Jun 23 22:17:54 pvr kernel: [10.643376] saa7164[0]: Try to pick one of the existing card configs via
Jun 23 22:17:54 pvr kernel: [10.643380] saa7164[0]: card=<n> insmod option.  Updating to the latest
Jun 23 22:17:54 pvr kernel: [10.643384] saa7164[0]: version might help as well.
Jun 23 22:17:54 pvr kernel: [10.643395] saa7164[0]: Here are valid choices for the card=<n> insmod option:
Jun 23 22:17:54 pvr kernel: [10.643403] saa7164[0]:    card=0 -> Unknown
Jun 23 22:17:54 pvr kernel: [10.643410] saa7164[0]:    card=1 -> Generic Rev2
Jun 23 22:17:54 pvr kernel: [10.643417] saa7164[0]:    card=2 -> Generic Rev3
Jun 23 22:17:54 pvr kernel: [10.643424] saa7164[0]:    card=3 -> Hauppauge WinTV-HVR2250

To set a card number option, create a modprobe directive file /etc/modprobe.d/saa7164.conf
options saa7164 card=3

Now reboot and watch the dmesg output to ensure the firmware is loaded properly
Aug 18 19:31:29 pvr1 kernel: [24.480644] saa7164 driver loaded
Aug 18 19:31:29 pvr1 kernel: [24.480891] saa7164 0000:07:00.0: PCI INT A -> GSI 19 (level, low) ...
Aug 18 19:31:29 pvr1 kernel: [24.490973] CORE saa7164[0]: subsystem: 0070:8891, board: Hauppauge WinTV-HVR2250 [card=3,insmod option]
Aug 18 19:31:29 pvr1 kernel: [24.490992] saa7164[0]/0: found at 0000:07:00.0, rev: 129, irq: 19, latency: 0, mmio: 0xfb800000
Aug 18 19:31:29 pvr1 kernel: [24.700362] saa7164_downloadfirmware() Waiting for firmware upload (NXP7164-2010-03-10.1.fw)
Aug 18 19:31:29 pvr1 kernel: [27.153217] saa7164_downloadfirmware() firmware read 4019072 bytes.
Aug 18 19:31:29 pvr1 kernel: [27.153227] saa7164_downloadfirmware() firmware loaded.
Aug 18 19:31:29 pvr1 kernel: [27.153257] saa7164_downloadfirmware() SecBootLoader.FileSize = 4019072
Aug 18 19:31:29 pvr1 kernel: [27.153269] saa7164_downloadfirmware() FirmwareSize = 0x1fd6
Aug 18 19:31:29 pvr1 kernel: [27.153276] saa7164_downloadfirmware() BSLSize = 0x0
Aug 18 19:31:29 pvr1 kernel: [27.153282] saa7164_downloadfirmware() Reserved = 0x0
Aug 18 19:31:29 pvr1 kernel: [27.153289] saa7164_downloadfirmware() Version = 0x1661c00
Aug 18 19:31:29 pvr1 kernel: [27.304006] Modules linked in: nvidia(P+) snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel(+) ir_lirc_codec lirc_dev ir_mce_kbd_decoder ir_sony_decoder snd_hda_codec ir_jvc_decoder snd_hwdep snd_pcm snd_seq_midi rc_rc6_mce psmouse ir_rc6_decoder ir_rc5_decoder mceusb serio_raw snd_rawmidi ir_nec_decoder joydev snd_seq_midi_event rc_core snd_seq saa7164(+) snd_timer snd_seq_device snd xhci_hcd asus_atk0110 dvb_core v4l2_common videodev media v4l2_compat_ioctl32 tveeprom soundcore snd_page_alloc lp parport usbhid hid ahci r8169 libahci
Aug 18 19:31:35 pvr1 kernel: [34.380040] saa7164_downloadimage() Image downloaded, booting...
Aug 18 19:31:35 pvr1 kernel: [34.490037] saa7164_downloadimage() Image booted successfully.
Aug 18 19:31:36 pvr1 kernel: [36.830033] saa7164_downloadimage() Image downloaded, booting...
Aug 18 19:31:38 pvr1 kernel: [38.270702] saa7164_downloadimage() Image booted successfully.
Aug 18 19:31:38 pvr1 kernel: [38.996628] DVB: registering new adapter (saa7164)
Aug 18 19:31:38 pvr1 kernel: [38.999488] saa7164[0]: registered device video0 [mpeg]
Aug 18 19:31:39 pvr1 kernel: [39.231624] saa7164[0]: registered device video1 [mpeg]
Aug 18 19:31:39 pvr1 kernel: [39.443888] saa7164[0]: registered device vbi0 [vbi]
Aug 18 19:31:39 pvr1 kernel: [39.444038] saa7164[0]: registered device vbi1 [vbi]


4. Run MythTV Backend setup, use IVTV MPEG driver and /dev/video0, /dev/video1

This configuration will result in the NTSC (analog) tuners to function with MythTV by initializing the tuner as an “IVTV MPEG-2 Encoder” defined for /dev/video0 and /dev/video1.

<I’ll post an update here for each of the backend setup steps, but for now follow the setup steps in order starting with General>

Under Storage, change the default directories for Recordings, LiveTV and Database Backups to the directories you created in step 1.

5. Setup mail transport agent so we can send status email
apt-get install postfix bsd-mailx

Since I locate the PVR behind a firewall on a residential Internet connection, I choose “Internet connected with Smart (Relay) host”
shawmail.cg.shawcable.net

6. Fix up some of the annoying outstanding problems

Arrow Buttons Repeat

For some reason, the Windows Media Center remote control IR codes are working but arrow button presses cause double movements. Huh? Ok, Google solves it:

MCE remote menu entries skipping twice for every button push
http://www.mythtv.org/wiki/MCE_Remote#Arrow_Buttons_Repeat

When navigating the menus in MythTV, it may appear that each time you hit the up or down arrow, that the button his hit twice but if you are using irw, the button only appears to be pressed once. This is likely caused by another kernel module that is attempting to treat the MCE remote as a keyboard. As a test try unloading a few modules
modprobe -r ir_rc6_decoder
modprobe -r rc_rc6_mce
modprobe -r ir_rc5_decoder

Since this solved the problem, the post author suggests the following command on system boot (in /etc/rc.local)
echo lirc > /sys/class/rc/rc0/protocols

Audio Pauses and Stutters

Turns out the default 4096 buffer size is too small for this system, so I increase it to 16384 (trial and error).

Put these lines in /etc/rc.local:

# prevent Arrow Button repeat
echo lirc > /sys/class/rc/rc0/protocols
# increase of audio buffer - from 4096 default to 16384
echo 16384 | tee /proc/asound/card0/pcm1p/sub0/prealloc

7. Setup the Disk Mirroring

I’m not going to use mdadm (dang it!) because mdadm (3.1.4) breaks initramfs (forum posts here). People upgrading Ubuntu to the “natty” release experienced this behavior where initramfs could not mount /root. For now I’ll use LVM2 to do the mirroring of

/
swap
/storage

but /boot will still be a standalone ext4 partition and filesystem on /dev/sda1. For recovery if /dev/sda dies, I’ll partition /dev/sdb the same and keep /dev/sdb1 in sync with /dev/sda1 – as well as have grub install a boot loader on both /dev/sda and /dev/sdb.  With the Asus AT5IONT-I mainboard, you can designate which SATA disk is the “Primary” and which is the “Secondary”. Worst case, if /dev/sda dies the Secondary drive can be manually mapped to the Primary disk (/dev/sda). Whew. A lot of extra work because mdadm is broken!

7.1. Install the lvm2 package

apt-get install lvm2

7.2. Partition the second drive with the desired end state

fdisk /dev/sdb

primary partition 1, 150M, set active, partition type flags “83″ (normal Linux fs)
primary partition 2, rest of the disk, partition type flags “fd” (Linux logical volume)

7.3. Setup the first LVM partition for pvr on /dev/sdb

# initialize the LVM volume
pvcreate /dev/sdb2
pvdisplay /dev/sdb2
# create the volume group
vgcreate rootvg /dev/sdb2
# create the logical volumes (with extents from one physical disk)
lvcreate -L 8G -n lv_root rootvg
lvcreate -L 8G -n lv_swap rootvg
lvcreate -L 650G -n lv_storage rootvg

This gives us the volumes we’ll use in our final configuration. Go ahead a get the filesystem contents copied across to the new LVM volumes.

# create the filesystems
mkfs.ext4 /dev/rootvg/lv_root
mkfs.ext4 /dev/rootvg/lv_storage
mkswap /dev/rootvg/lv_swap
# copy the old fs to new fs
mkdir /mnt/root
mount /dev/rootvg/lv_root /mnt/root
cd /
find . -xdev -print | cpio -pmd /mnt/root
umount /mnt/root
mkdir /mnt/storage
mount /dev/rootvg/lv_storage /mnt/storage
cd /storage
find . -xdev -print | cpio -pmd /mnt/storage
umount /mnt/storage

7.4. Update boot configuration to use LVM root volume

Copy the /dev/sda1 /boot filesystem to /dev/sdb1 for a backup in case something goes horribly wrong, you’ll at least have a starting point to recover.

To update GRUB to use the LVM device, add GRUB_DEVICE=/dev/mapper/rootvg-lv_root
to /etc/default/grub and disable the UUID volume label tracking by uncommenting the GRUB_DISABLE_LINUX_UUID=true line.

Update the /boot/grub/grub.cfg by running
cp /boot/grub/grub.cfg /boot/grub/grub.cfg.orig
update-grub -o /boot/grub/grub.cfg
You should see update-grub detect the original boot env on /dev/sda1 (/boot) with a root of /dev/sda2 (where we configured /) and you should also see it detect the new root environment on /dev/mapper/rootvg-lv_root.

Then install the grub boot environment on /dev/sda and optionally /dev/sdb. Note /dev/sdb will not have a boot block or env loaded yet, so no worries about having to save what might be there.
grub-install /dev/sda
grub-install /dev/sdb

Update the /etc/fstab to swing the filesystems over to the LVM volumes – update the device specs from their UUID labels to /dev/mapper/rootvg-lv_root, /dev/mapper/rootvg-lv_swap and /dev/mapper/rootvg-lv_storage
cp /etc/fstab /etc/fstab.orig
vi /etc/fstab

It should look something like
# / was on /dev/sda2 during installation
#UUID=fc0fa1e9-e2b6-4d11-9a51-d3c432bb3137 / ext4 errors=remount-ro 0 1
/dev/mapper/rootvg-lv_root / ext4 errors=remount-ro 0 1
# leave /boot alone
/dev/sda1 /boot ext4 defaults 0 2
# /storage was on /dev/sda4 during installation
#UUID=131514d2-3911-45df-8d6f-b9a19f2379bb /storage ext4 defaults 0 2
/dev/mapper/rootvg-lv_storage /storage ext4 errors=remount-ro 2
# swap was on /dev/sda3 during installation
#UUID=eedc9a3b-d957-4904-988e-32b117def5ac none swap sw 0 0
/dev/mapper/rootvg-lv_swap none swap sw 0 0

This is the nail-biting time, now reboot. When GRUB comes up and shows you the boot environments, select the normal boot Ubuntu Linux with the root on /dev/mapper/rootvg-lv_root.

7.5. Extend the LVM to use /dev/sda
Now we’re running on the LVM volumes on /dev/sdb, we want to reclaim the plain 0×83 Linux filesystem partitions off /dev/sda and add them to the rootvg volume group then extend each logical volume so it has a mirror on /dev/sda.

# delete /dev/sda2, /dev/sda3, /dev/sda4 partitions
# add /dev/sda2 as the remaining disk, toggle the partition
# type flags to 0xfd (Linux LVM)
fdisk /dev/sda

Now clear the first few blocks of /dev/sda2 since it will still have a Linux ext4 filesystem signature on it and we don’t want to confuse LVM.
dd if=/dev/zero of=/dev/sda2 count=100

Extend the volume group to include /dev/sda2 and add a mirror onto each logical volume.

# initialize the /dev/sda2 partition for LVM
pvcreate /dev/sda2
pvdisplay /dev/sda2
# extend the volume group to include /dev/sda2
vgextend rootvg /dev/sda2
vgdisplay -v
# now extend each logical volume to /dev/sda2
lvconvert -m1 --mirrorlog core /dev/rootvg/lv_root /dev/sda2
lvconvert -m1 --mirrorlog core /dev/rootvg/lv_swap /dev/sda2
lvconvert -m1 --mirrorlog core /dev/rootvg/lv_storage /dev/sda2

Let this run for a while, the system will be very busy syncing (re-silvering) the physical extents on /dev/sdb2 to /dev/sda2.

Side note: If something messes up and you need to remove the /dev/sda2 or /dev/sdb2 volume, or if you need to tear down the lvm2 setup (such as to remove or play around with RAID volumes), use

lvremove /dev/mapper/rootvg-lv_root
vgremove rootvg
pvremove /dev/sdb2

so you don’t have problems with residual signatures when you try to initialize the LVM volume group an physical devices again.

7.6. Update /boot on /dev/sdb

mount /dev/sdb1 /mnt/boot
cd /boot
find . -print | cpio -pvmd /mnt/boot
umount /mnt/boot

Now you should be able to boot off either /dev/sda or /dev/sdb.

8. Remaining issues
Despite turning off the screen saver, the HDMI to TV output dims occasionally and I’m pretty sure it’s not my Sony Bravia that’s doing it.

Audio is still a pain in the butt – I’m using the analog audio out on the Asus mainboard into a pair of Audio Engine speakers to get audio, since I think the HDMI driver needs to be changed to support audio over HDMI. I’m using the Open Source video driver instead of the nVidia and I think that’s the culprit. No time to test it right now though.

Update:
Tried out a LVM volume extend for the /storage filesystem:

df -h /storage
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-lv_storage
669G 70G 565G 11% /storage

vgdisplay
— Volume group —
VG Name rootvg
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 11
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 2
Act PV 2
VG Size 1.36 TiB
PE Size 4.00 MiB
Total PE 357628
Alloc PE / Size 355728 / 1.36 TiB
Free PE / Size 1900 / 7.42 GiB
VG UUID JwB28k-Eeg6-HNq0-Ghdn-r4db-mNqd-fZWYfG

Since vgdisplay shows we have free space (7GB), issue the lvextend command:
lvextend -L +1G /dev/rootvg/lv_storage
Extending 2 mirror images.
Extending logical volume lv_storage to 679.78 GiB
Logical volume lv_storage successfully resized

After extending the logical volume, we can extend the filesystem. We will extend the fs while it’s mounted, since the current versions of resize2fs allow online extension or shrinkage.

resize2fs /dev/mapper/rootvg-lv_storage
resize2fs 1.41.14 (22-Dec-2010)
Filesystem at /dev/mapper/rootvg-lv_storage is mounted on /storage; on-line resizing required
old desc_blocks = 43, new_desc_blocks = 43
Performing an on-line resize of /dev/mapper/rootvg-lv_storage to 178200576 (4k) blocks.
The filesystem on /dev/mapper/rootvg-lv_storage is now 178200576 blocks long.

Yay! LVM sure makes fs and volume manipulation easy.

df -h /storage
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-lv_storage
670G 70G 566G 11% /storage

Drop tables with wildcard:

There are multiple ways to specify MySQL credentials, this is not the best, but simply an example of how to drop tables using a wildcard pattern. In this case, command line history such as .bash_history will store your MySQL username and password plaintext, and an extended process listing will also reveal both username and password. When run from the command line like this, the SQL commands and the credentials are not stored in the MySQL history file (.mysql_history).  On closed (private) systems, the risk is low, especially if you clean up after these maintenance activities by purging the command histories.

mysql -u user -p password database -e "show tables" | grep "table_pattern_to_drop_" | awk '{print "drop table " $1 ";"}' | mysql -u user -p password database

A simple MySQL update can match the WordPress blog to a new site URL:

mysql> select option_value from wp_options where option_name = 'siteurl';

+--------------------------------+
| option_value                   |
+--------------------------------+
| http://www.example.com |
+--------------------------------+
1 row in set (0.00 sec)

mysql> select option_value from wp_options where option_name = 'home';

+--------------------------------+
| option_value                   |
+--------------------------------+
| http://www.example.com |
+--------------------------------+
1 row in set (0.00 sec)

mysql> update wp_options set option_value='http://server.newsite.com' where option_name='siteurl';

Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> update wp_options set option_value='http://server.newsite.com' where option_name='home';

Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
 

Websites like sectools.org maintain lists of various security tools and point to numerous open source web application fuzzing and testing tools, including BurpSuite, Nikto, WebScarab, Whisker and Wikto. Although each of the open source tools I use have various strengths, w3af is IMHO the first reasonable challenger to commercial web application testing tools like IBM’s AppScan.

Can we please get rid of bad input validation errors now??

For a commercial IT security professional that wants to help an internal web application development team improve the security of their applications, tools like IBM’s AppScan and Acunetix WVS can save valuable time by generating reports that include not only the vulnerable URI but also include vulnerability background information (CVSS, OWASP, WASC), the specific HTTP request/response strings and suggested code fixes. This is particularly valuable to a security architect or operations role that is pressed for time (an army of one anyone?).

The w3af support from Rapid7 will enable this excellent tool to mature more quickly and improves the capability for any web development team, regardless of funding, to improve their security. Can we please get rid of bad input validation errors now?? My recent thesis illustrated the downright depressing numbers of SQL injection flaws that continue to exist. With tools like w3af, there is no excuse left for web developers to press applications into production with these injection flaws that are trivial to avoid. At the very least a survey of the NIST National Vulnerability Database does show the number of SQL injection flaws starting to drop. Unfortunately they still substantially outnumber traditional memory corruption flaws such as buffer overflows.

Explosion of SQL buffer errors

As you can see, the story up to 2008 was pretty grim for web applications – SQL injection flaws increased by over 1,500% in the same time buffer overflow errors increased by just over 500%.

Although it looks like there has been a reversal of the shocking explosion of SQL injection flaws, the sheer volume of these web application flaws is astonishing .. especially since injection flaws have been around for about 10 years. Not exactly a problem that has recently snuck up on us.

Web developers that still turn out applications that contain SQL or command injection errors and most cross site request forgery errors are simply guilty of gross negligence.

Despite the web development industry knowing these errors exist and good developers designing and coding to avoid these issues, there is still a need to build sufficient forensics around externally facing (publicly accessible) applications to enable reconstruction of attacks. In my next post, I outline a summary of my thesis “Effective SQL injection attack reconstruction using network recording”.

Connect to the database via your favorite GUI (phpMyAdmin, Navicat) or command line with either the WordPress role account or any other MySQL user account with select and update privileges on the WordPress database:

update wp_x_users set user_pass = MD5('123abc890') where user_login = 'administrator';

This will update the password for user ‘administrator’ to ’123abc890′.  Once this has completed, either flush the wp_x_users table or exit the tool used to access the database to cause the updates to be committed.  Sign into WordPress with the new password and optionally change the password via the user interface.

Sample query to see if you’re on any Spamhaus block list:  http://www.spamhaus.org/query/bl?ip=10.11.12.13

It seems it’s time to relinquish the care and feeding of my own Postfix mail system and turn to a hosted solution.  This means I need to migrate about 5GB of IMAP store to another site (again).  Last time I did a wholesale migration, I used imapsync to make the transition painless. In the code example below, an SSL connection to the IMAPS server at imap-server.sourcedomain.com is made with username@sourcedomain.com and the password stored in the plaintext file secret1. An SSL connection is made to the target system (which happens to be the server on which the imapsync tool is running, but could just as easily be another IMAPS server somewhere on a network accessible to the host where imapsync is running). The –delete and –expunge1 arguments will clean the successfully moved messages from IMAP store #1 .. so be sure you have your messages on the target successfully! Imapsync can be run iteratively to ensure you have got all the messages from your source.


/usr/bin/imapsync \
--host1 imap-server.sourcedomain.com \
--ssl1 \
--authmech1 LOGIN \
--user1 username@sourcedomain.com --passfile1 secret1 \
--host2 127.0.0.1 --user2 username@targetdomain.com --passfile2 secret2 \
--ssl2 \
--delete --expunge1 \
--buffersize=128

And one can use the

--dry

option to just test the process but not actually move any of the messages.

So that’s it – I’m about half way though migrating my current IMAP stores over to a hosted mail solution, so that I don’t need to keep up with the increasing level of care and feeding that running your own mail service requires.  Before I get too many darts about that .. I first started running my own personal MTA in 1995, adding spam and av filtering over time, and adding substantial redundancy (servers, sites, storage) so I could rely on it and fix things that broke as I had time rather than right when they broke (which was always at a bad time).  My new hosted solution takes over from two VPS servers running Postfix, Spamassassin, ClamAV, Greylisting with the IMAP store replicated across data centers in different states (15 minute rsyncs).  So soon, the (hopefully) last Allen Pomeroy owned and operated MTA can be turned off, while I get to work on fun stuff, rather than figuring out why my email is bouncing. 

Problem: VMware Fusion 3.0 doesn’t give a way to edit the virtual network settings via the GUI.
Solution: To change the subnet used by the NAT or HostOnly networks, go root in Mac OS X and edit

/Library/Application Support/VMware Fusion/networking

and set the following lines to the subnets desired:

answer VNET_1_HOSTONLY_SUBNET 192.168.35.0
answer VNET_8_HOSTONLY_SUBNET 10.10.1.0

To add additional custom isolated host only VLANs, also edit the networking file and add additional VNET definitions. There can apparently only be 8 VLANs with VLAN 1 and 8 already pre-defined.

answer VNET_2_DHCP no
answer VNET_2_HOSTONLY_NETMASK 255.255.255.0
answer VNET_2_HOSTONLY_SUBNET 10.10.21.0
answer VNET_2_VIRTUAL_ADAPTER yes
answer VNET_3_DHCP no
answer VNET_3_HOSTONLY_NETMASK 255.255.255.0
answer VNET_3_HOSTONLY_SUBNET 10.10.22.0
answer VNET_3_VIRTUAL_ADAPTER yes
answer VNET_4_DHCP no
answer VNET_4_HOSTONLY_NETMASK 255.255.255.0
answer VNET_4_HOSTONLY_SUBNET 10.10.23.0
answer VNET_4_VIRTUAL_ADAPTER yes

Now create your vm with as many network interfaces as you have separate VLANs (vnet) then edit the node.vmx vm configuration file and change the interfacename.connectionType to custom, and define the VLAN (vnet) that interface will attach to:

#ethernet0.connectionType = "nat"
ethernet0.connectionType = "custom"
ethernet0.vnet = "vmnet3"

Also realize that VMware will take the .1 host address on each vmnet – so you cannot assign .1 to any of your VMs.

Problem: Ubuntu 9.10 persistent network configuration (stores the MAC address of network adapters), so if you copy a machine, by default Ubuntu will setup a new logical adapter (eth1) since the MAC address has changed (when you answer I Copied It in VMware).
Solution: Tell VMware you copied the machine, so it will chose a unique MAC address. Boot Ubuntu into single user mode (another article on that to follow) then edit the MAC address associated with eth0.

sudo vi /etc/udev/rules.d/70-persistent-net.rules

find the stanza of the network interface in question (NAME=”eth0″) and set the following ATTR tag to the new MAC address:

ATTR{address}=="new-mac-address-here"


I wanted to upgrade the disks in my Linux PVR to a 1TB pair and thus had to migrate from one existing disk (/dev/sda) to the new (/dev/sdb):

1. Add new physical disk to system

2. Partition disk to have a linux LVM partition – use flag 0x8e

# fdisk /dev/sdb

3. Add to LVM

# pvcreate /dev/sdb2

4. Add physical LVM volume to a LVM volume group (VolGroup00)

# vgextend /dev/VolGroup00 /dev/sdb2

2. Move all lvm volumes off old lvm disk

# vgdisplay -v (look for old physical volume name)

# pvmove /dev/olddisk      # will move all physical extents from olddisk to any available pv in the vg

3. Remove old disk from vg

# vgreduce /dev/olddisk

4. Remove old disk from LVM

# pvremove /dev/olddisk

RAID Notes
Debian RAID setup on my PVR:
/dev/md0  /boot
/dev/hda1
/dev/hdb1
/dev/md1  /
/dev/hda2
/dev/hdb2
/dev/md2  swap
/dev/hda3
/dev/hdb3
/dev/md3  /data
/dev/hda4
/dev/hdb4

Show detail of RAID set:
# mdadm –detail /dev/md0

Detach mirror member:
- first mark member as bad (unless is really is bad, in which case it’ll already be marked faulty):
# mdadm –set-faulty /dev/md0 /dev/hdb1
- now remove it from the RAID1 set
# mdadm –remove  /dev/md0 /dev/hdb1

To reattach member (after partitioning, or if it’s the same disk):
# mdadm   /dev/md0  –add  /dev/hdb1
- to watch the progress on the resync, look at /proc/mdstat
# cat /proc/mdstat

I think now (2010/01/24) the faulty syntax is:

mdadm /dev/md0 –fail /dev/sdb1

then

mdadm /dev/md0 –remove /dev/sdb1

Crypto Filesystem Notes

Linux (2.6) crypto filesystems are supported via a loopback device. Various ciphers can be specified.  This example, default AES cipher is used and the disk partition is /dev/sdb1 – which is just setup as a normal Linux (0×83) partition.

1. Load the crypto filesystem module

modprobe cryptoloop


2. Start the crypto device (I’ll insert initialization instructions here later)

Note – you don’t need losetup, if the parameters are specified in fstab and mount does the startup. When losetup runs, it will prompt for the passphrase used to encrypt the partition. Once the crypto driver has the correct key to allow on the fly encryption/decryption, then processes that use the partition see cleartext (such as mount).

losetup -e aes /dev/loop0 /dev/sdb1 || exit 1
mount /bu