Tuesday, 24 July 2012

Creating a loopback file-system image with partitions

With building the OpenELEC images for Raspberry Pi - I ran into an odd problem with creating system images for people to download.
The first image I made was DD'd from an 8GB SD card that I had already created and tested. I formatted the SD card, tested in the Pi and then took an image with DD. Then, rewrote it to the card and tested again. All good. Then a few people commented that the image was slightly too large to fit on other 8GB cards which is when I checked and my card it was indeed a few sectors large than some others. So how do you create images which are useful for more people without going out and buying an army of various sized SD cards? The answer is creating loopback devices which are basically file-systems in a single file (which is what you create when you DD a working image from an SD card, for example).

1. Decide what size you would like your image to be. For this example and the OpenELEC image I posted previously, I used 1GB.
2. Create your empty container file using DD. You can replace the expression after seek with 1024000000 if you prefer - this is the size of your image in bytes.

 dd if=/dev/zero of=~/1gb_file_image.img bs=1024 count=0 seek=$[1000*1000]  

3. Mount this device as a loopback device

 sudo losetup /dev/loop0 ~/1gb_file_image.img  

4. This will give you a loopback block device under /dev/loop0. You can now create your partitions using fdisk / parted. I've used the examples from the OpenELEC wiki here.

 sudo parted -s /dev/loop0 mklabel msdos
 sudo parted -s /dev/loop0 unit cyl mkpart primary fat32 -- 0 16  
 sudo parted -s /dev/loop0 set 1 boot on  
 sudo parted -s /dev/loop0 unit cyl mkpart primary ext2 -- 16 -2  
 sudo mkfs.vfat -n System /dev/loop0p1  
 sudo mkfs.ext4 -L Storage /dev/loop0p2  

5. Now you have a filesystem in a file with two partitions. Things can get a little tricky here when you need to mount them individually. You need to umount the current loopback device so you can remount it with some sector offsets and size limits. If you search and go by the scraps of mailing lists that mention this, it can be a little daunting but it's not as bad as it seems. Lets look at an fdisk output of our new filesystem in a file.

 sudo fdisk -l ~/1gb_file_image  
 Disk 1gb_file_image.img: 1024 MB, 1024000000 bytes  
 255 heads, 63 sectors/track, 124 cylinders, total 2000000 sectors  
 Units = sectors of 1 * 512 = 512 bytes  
 Sector size (logical/physical): 512 bytes / 512 bytes  
 I/O size (minimum/optimal): 512 bytes / 512 bytes  
 Disk identifier: 0xd00251a6  
 Device              Boot Start    End     Blocks Id  System  
 1gb_file_image.img1 *    2048     264191  131072 c   W95 FAT32 (LBA)  
 1gb_file_image.img2      264192   1999999 867904 83  Linux  

What we need to note here are the start and end sectors as we will use these to calculate the offset and sizelimit parameters to pass to losetup when mounting the partitions in the file. Taking the first partition as an example (shown as an actual file with a different number at the end, that's just how fdisk rolls I guess). We have a starting sector of 2048 and and end sector of 264191. We need to multiply each of these by 512 to get the total amount of bytes instead of sectors (bytes per sector is in the output - line 4 above). So, our starting offset becomes 1048576, and the size being  (end - start, multiplied by 512 again) 134217216.
We'll add these to our next command.

6. Mount the loopback image file again, specifying the offset and sizelimit for the first partition.

 sudo losetup /dev/loop0 ~/1gb_image_file.img -o 1048576 --sizelimit 134217216  

7. Now, you can mount this loopback device as a block device with the standard mount command

 sudo mount /dev/loop0 /mnt/my_mount_point  

This should give you the first FAT partition (if you used the OpenELEC example anyway) mounted under /mnt/my_mount_point (make sure it exists first!). Then you can copy files into your loopback device to populate your image. To mount the second partition, follow the same procedure as in steps 5-7 but use a new loopback device (/dev/loop1 is a good starting point) and calculate the new offsets and sizelimits based on the start and end sectors of the second partition. When you're finished, make sure to unmount the loop devices, then detach the loopback devices with

 sudo losetup -d /dev/loop0   
 sudo losetup -d /dev/loop1  

It can be a finicky procedure, but once you've got it nailed it's pretty handy. Let me know if any parts don't work as advertised (especially seeing as it took several attempts to get mine right too!).

OpenELEC Build R11639 available

It's been a while since I compiled the latest OpenELEC GIT commit and created an image. So long ago apparently I forgot how to use anything Linux based (long story short - don't do 'apt-get purge ruby* on ubuntu, it removes grub :-/ ).
Built from the GIT source as of 23/07/2012. Binaries and 1GB SD image available below.

OpenELEC Binaries for manual setup

OpenELEC 1GB SD image for DD / Win32imager

Enjoy - any problems let me know in the comments (unless you're going to leave a comment like 'it not work' - if you're going to do that at least leave a description of the problem please).


Wednesday, 11 July 2012

Puppet - 'host name was not match with the server'

...or another way to put it 'you're going to lose several hours of your life to this stupid error message that you'll never ever get back'.

Puppet is a great tool for automated system builds. But as with anything that uses SSL, it can be an utter pig if you start doing things that stray from the standard install guide.

Our scenario looked fairly simple; a handful of puppet agents finding their master by looking up 'puppet.[domain]' - but with a small twist of puppet being a DNS CNAME to another host. My belief was that the agent would be clever enough to follow the CNAME reference and accept the correctly issued certificate from the alternative name. No. Not the case...

The agent (unless configured otherwise) will only lookup 'puppet' and 'puppet.[domain]'. So no matter how you configure your puppetmaster with certs for different DNS names, unless you've added those DNS names to the agent /etc/puppet/puppet.conf file under the 'server=' setting - it won't work. Ever.
There are articles about setting dns_alternative_names when you generate the puppetmaster certificates which is great - but nothing tells you that won't be any use until you configure the client to accept that new hostname.
The page here http://docs.puppetlabs.com/guides/troubleshooting.html points you in the right direction - but is a bit misleading when it doesn't mention that you have to edit the 'server' property on the AGENT too.
The simplest way to solve this that I could find was to configure your puppetmaster to use the setting in /etc/puppet/puppet.conf of 'certname=puppet'. OK, internally the cert name will be different to the actual hostname but it will please the puppet agents and you won't have to edit all of your agent settings in the long run. You can still have the host and DNS name as whatever you like because puppet doesn't ever change the hostname it's requesting after following DNS.
I hope other people getting caught up on this manage to find this and save at least a few minutes of angry typing / throwing heavy things across the room.