Category Archives: Mac

A European's experience in [NOT] buying a MacBook Pro in US

I’m visiting NYC this week. I’ve come to US thinking hard whether or not I want to buy a MacBook Pro. I decided that yes, I do. Now I changed my mind. Why?

I wanted to do a couple of things. Use employee discount, pay with my Irish card, and I absolutely required a European-style keyboard layout. This last thing is the ultimate reason why I decided to skip it.

Apple is excellent if you are an average customer. Walk into one of the stores (there’s one 5 minutes of walk from me), pick up what you want, walk out happy. I could have done that. I can still do that. I don’t want to.

First, I want my discount. It saves me ~~a non-insignificant~~ an interesting amount of money. I doubt I can apply the discount from within the store. [UPDATE: Americans love to apply sales tax AFTER quoting you the price. So my total savings are a bit less than expected. Still, they exist. And my estimates include Irish VAT, because I’m nice like that. Smugglers would really save a lot.]

Second, I want a European-style keyboard (British at the very least). I have a US layout wireless keyboard, and if I switch to Croatian layout in software, I cannot type < and > on it.

Turns out that Apple does not stock laptops with different layouts in their stores. Turns out that I need to wait 1-3 business days for them to build the custom laptop (!!!) and then another 1-3 business days for them to expedite-ship it. Oh you want standard shipping? 3-5 days. You dared to desire to pick it up at the store (which should actually be easier for them)? Well, feelin’ bad for you buddy — it’s as if you picked standard shipping.

Maybe you wanted to pay with your Irish card? No go — they want your billing address phone number to be a US number, and your billing address zip code to be a US zip code and a US address.

Especially if you include the fact I’d need to deal with Irish customs people as well, that’s enough divine signs telling me I should skip this purchase. Maybe I change my mind, but I doubt it.

What kind of bullshit is not stocking some British-layout MBPs somewhere in NYC? What kind of bullshit is “it takes 1-3 days to replace a keyboard”? With minimum training, it could be done in-store, even if you couldn’t stock 2 variants in each store. They don’t sell often enough? Don’t keep 50 of them, but do keep 2 of them.

And I was so looking forward to a Mac with a decent GPU.

Running a physically installed Ubuntu in VirtualBox on Mac

DISCLAIMER

Everything in this post is dangerous to your data, even if you are an expert. Think thrice before doing anything, back up everything, triple-check everything I suggest you do for hidden assumptions that may be related to my machine.

Some steps may have inadvertently been omitted. I did not note everything I did, and I don’t feel like reproducing each and every of my steps.

I know this disclaimer will probably not dissuade you from trying this, but this includes particularly dangerous stuff and I don’t want anyone misunderstanding: I am NOT advising you to do this. I am NOT advising you to follow my advice. I am only documenting this process for my own private future use.

Mac specifics

Mac runs EFI instead of BIOS. This is quite similar to today’s UEFI machines, but does present its problems.

The partition table format used is GPT. Again, this is something normal and expected on today’s new UEFI machines, but something people may be unfamiliar with.

Somewhat specific to Mac, if one installs Windows, then the resulting partition table format includes both MBR and GPT. This is a result of the fact that 32-bit Windows don’t boot from GPT and don’t like EFI, and even more so, a result of the support for Windows on Macs being produced before -Longhorn- Vista was released. So the lowest common denominator is, in this case, 32-bit Windows XP; so, “BootCamp” produces a GPT+MBR hybrid partition table. (Experts seem to dislike hybrid MBRs.)

Under (U)EFI, the equivalent of BIOS does very little. It has support for FAT (usually FAT32), for loading additional drivers from the FAT partition, and for running .EFI binaries (mostly boot loaders, although in some cases these may be utility programs or shells).

On PCs, we will be mostly interested in 32-bit and 64-bit Intel .EFI binaries (i386 and x86_64 binaries). There are even universal binaries. Running an appropriate .EFI binary is how we boot an operating system. Since 3.3, Linux kernels seem to include an EFI stub; this means you can take a “vmlinux” binary and feed it directly to the EFI subsystem of an (U)EFI machine, without use of ELILO, GRUB2, or some other boot loader.

To expose the binary to the EFI system of the machine, you need to mount the FAT-formatted partition of type “EFI System Partition”, change to the \EFI directory on this partition, create a “vendor directory” (that’s the name of your OS or your OS vendor), and put “boot.efi” binary in that directory. Some EFI systems may expose other binaries as well, but this is what one is supposed to do.

Alternatively, systems may include a “Compatibility Support Module”. Macs do. This is a BIOS emulation layer and allows booting “legacy” operating systems. Microsoft says this must be turned off by default. Note that it is not a requirement; there is no need to include CSM. (VirtualBox does not.)

Please read around to familiarize yourself with EFI and GPT, and how they differ from BIOS and MBR. Trying to describe them here would be either a lot of work or would result in even more incorrect article than it’s already bound to be.

Easy parts and troublesome parts

VirtualBox supports physical partitions in the VMDK format. You need to use the command line to create the VMDK disk image, but it’s easily doable. It’s essentially a one-liner under normal, tested conditions.

Even when the conditions are not as normal, I’ve successfully ran BootCamp’d Windows XP previously. (Search my blog if you’re interested in hacks I had to do.)

That was easy: Windows XP deals with MBR, and MBR is trivial.

The problematic part with supporting Ubuntu is that its boot loader, GRUB2, actually appears to be too smart for its own good. Having installed the 32-bit version of Ubuntu 12.04, it has (by design) not deployed an EFI boot loader. Instead of deploying grub-efi, it has deployed grub-pc. This is great; the rEFIt boot loader that I use when booting the physical machine has picked up on this and allowed me to pick the Linux system. It does, unfortunately, depend on turning off the “bootable” flag in MBR with fdisk. This appears to be a limitation (or a feature?) of the Apple-provided CSM.

That’s all great. But when I tried to load this system with VirtualBox using default options, I got nothing. Zip. Nada. A black screen and that’s it.

Why? I have no idea. But I presume that GRUB2 picked up on GPT partition table and then got very confused by this on a non-EFI system. Why? Again, no idea. Switching to EFI got the setup to work — but only after installing grub-efi and deploying it to the appropriate place.

Creating virtual disk

You want to have a virtual disk that includes all relevant GPT information.

As I was originally playing with my previous MBR-based method, and the EFI+GPT method came from that, so does my script include some dding of GPT data: the first 40 sectors of the disk, and the last 40 sectors of the disk.

IMPORTANT: Always check device names (GPT partition IDs) with sudo diskutil list /dev/disk0. Always check MBR partition IDs with sudo frisk /dev/disk0. My setup includes Windows 8 and is very weird.

IMPORTANT: This script is NOT intended to be run as-is! Read it to learn what’s going on, triple check every single number, ensure you understand every single line, customize it for your machine and only then think twice before running anything below. Playing with partitions, with disk devices etc is dangerous. Ensure you have backups of everything that’s even remotely significant.

I personally have a Time Machine backup of my important data under OS X, every important project is stored on online code hosting, and everything else under physically installed Ubuntu and Windows 8 is not important. What’s your situation? Can you afford to lose data?

NOTE NOTE NOTE: As of OS X 10.9 Mavericks, Detection of disk size in blocks is broken in the script below. I haven’t updated it; when I needed to run it, I read it from a manually-ran fdisk and put it in appropriate place. (n.b. this could possibly be doable from native code: ioctl(fd, DKIOCGETPHYSICALBLOCKSIZE, &block_size); see http://stackoverflow.com/a/15947809/39974)

# ALWAYS check devices with:
# sudo diskutil list /dev/disk0

# ALWAYS check MBR partition IDs with:
# sudo fdisk /dev/disk0

EFISYSTEMPARTITIONOSXDEVICEID=1
RECOVERYPARTITIONOSXDEVICEID=3
LINUXPARTITIONOSXDEVICEID=4
LINUXPARTITIONMBRDEVICEID=3

sudo chmod 777 /dev/disk0s$EFISYSTEMPARTITIONSOXDEVICEID
sudo chmod 777 /dev/disk0s$RECOVERYPARTITIONSOXDEVICEID
sudo chmod 777 /dev/disk0s$LINUXPARTITIONSOXDEVICEID

sudo VBoxManage internalcommands createrawvmdk -filename PhysicalDisk.vmdk -rawdisk /dev/disk0 -partitions $EFISYSTEMPARTITIONOSXDEVICEID,$RECOVERYPARTITIONOSXDEVICEID,$LINUXPARTITIONOSXDEVICEID


# (primary mbr=1, primary gpt header=1, primary gpt table = 32, extra = 6) * 512
PRIMARY=40
sudo dd bs=512 count=$PRIMARY if=/dev/disk0 of=PhysicalDisk-pt.vmdk

# secondary gpt table=32 + secondary gpt table = 1
# see http://7bits.nl/blog/2012/01/02/mac-gpt-partition-table-recovery
SECONDARY=33
DISKSIZE=`diskutil info disk0 | grep "Total Size: .*exactly .* 512-Byte-Blocks"|sed 's/.*Total Size:.*exactly \(.*\) 512-Byte-Blocks)/\1/
'`
OFFSET=`calc $DISKSIZE-$SECONDARY`
OFFSET=`echo $OFFSET|sed 's/^ *//'` # ltrim

sudo dd bs=512 count=$SECONDARY iseek=$OFFSET oseek=$PRIMARY if=/dev/disk0 of=PhysicalDisk-pt.vmdk conv=notrunc

First, note the chmods. These will be required after each reboot. OS X only allows root to access the disks (for very important security reasons); VirtualBox does not run with root privileges. I don’t do this lightly; be VERY mindful that this is actually creating a local security hole, allowing user processes to read and even write to the disk.

Next, if you take a look at PhysicalDisk.vmdk, it’s a text file. You can see how various virtual disk sectors are mapped to various physical disk sectors, to “zero”, or to PhysicalDisk-pt.vmdk. (Please do check that you can find a section that matches this; if not, something went wrong in VBoxManage, and you should delete both .vmdk files.)

dd might not be necessary; but I ensure that whatever’s in GPT is not “accidentally” changed by VirtualBox’s VBoxManage.

Could one map even these 40 initial and 33 trailing sectors to the actual physical disk? Sure. But, why risk anything?

Installing rEFIt

Download rEFIt from its homepage. The install instructions say all you need to do with latest version, 0.14, is open the installer package and hit “Next” repeatedly.

Installing grub-efi

I decided to reuse EFI System Partition. I could have just as easily used the system partition; Apple ships a HFS+ driver, so the EFI subsystem can boot directly from the system partition.

The thing is, Ubuntu can’t write to the HFS partition, so it’s slightly easier to reuse the EFI System Partition.

What am I risking? Well, Apple might wipe this partition clean in an OS update. I hope they won’t.

IMPORTANT: The following can mess up GRUB. I can still boot using the “BIOS” GRUB2, but your mileage may vary.

What follows is inspired by Rod Smith’s EFI-Booting Ubuntu on a Mac.

Boot physical Ubuntu.
sudo apt-get install grub-efi – This removed grub-pc on my machine, although I still seem to have the ability to boot using BIOS. (Anything else would be… troublesome.)
sudo mkdir /boot/efi – This is the place where we’ll mount the EFI System Partition.
sudo mount /dev/sda1 /boot/efi
sudo mkdir -p /boot/efi/EFI/Ubuntu – Apple doesn’t ship an \EFI folder. We’ll create it, along with the “vendor” directory for Ubuntu.
sudo grub-install /dev/sda1 – This should install grub-efi to \EFI\Ubuntu.
ls -al /boot/efi/EFI/Ubuntu – You should see two files from Ubuntu here.

It’s important to understand: 32-bit Ubuntu installs 32-bit GRUB2. This will not be bootable on a 64-bit capable Mac. This is solely useful for VirtualBox.

So, ensure that you can still use the BIOS GRUB2, or have an alternative boot method, or else you’re now converting your physical installation into a VirtualBox-only installation!

Creating virtual machine

I don’t have a script for this one. Go back to OS X, go to VirtualBox GUI and create an Ubuntu-type virtual machine. Don’t pick the 64-bit version; this changes the type of EFI that the virtual machine will use!

Pick the previously created PhysicalDisk.vmdk while creating the machine.

Now edit the settings. Right click on machine name, pick “Settings”, and change the machine to be an EFI machine on the System tab. So: right click [machine name]->Settings->System->Motherboard->Enable EFI (special OSes only).

Don’t boot yet! Did you chmod the disk devices? Remember, you rebooted. Please sudo chmod 777 all partition devices in /dev (and be mindful that this is a security hole you’re creating, which you might somehow avoid with UNIX user groups, but meh).

After this point, do not recreate the PhysicalDisk.vmdk without keeping in mind that this file includes disk image IDs in several places. VirtualBox keeps track of the disk images, and will NOT be happy if the ID changes.

So, done now? Great. Boot.

You’ll be shown the EFI shell. Hoorah!

Now, let’s change to the EFI System Partition‘s filesystem and boot GRUB2.

fs0:
cd EFI\Ubuntu
boot.efi

This should show you your physical machine’s GRUB menu and the booting should move on. Observe the disk light on the bottom of VirtualBox’s window; if it stops flickering for longer than 15 seconds, and Ubuntu does not boot, you can presume you have some sort of an issue.

Note that virtual machine does have different hardware than your physical machine; for example, NVIDIA graphics driver does not work for me. I get the console, but not X11. It would be trivial to fix (replace the selected driver with vesa or something similar in Xorg.conf) but I don’t care: I need to SSH into the machine and tunnel X11 to XQuartz on OS X. I don’t need Unity: I need the ability to work on my code and display the X windows.

So, this works for me. Huzzah!

Small updates

fstab

Add this to /etc/fstab (based on Rod Smith’s post, too):

/dev/sda1       /boot/efi       vfat    ro,fmask=133    0       0

Alternatively, change that ro to rw to get the partition to mount read-write; this may be important for grub updates.

grub-efi-amd64

Ubuntu 12.04 also ships with grub-efi-amd64.

sudo mount /boot/efi -o remount,rw # if not already mounted read-write
sudo apt-get install grub-efi-amd64

Don’t forget to change machine type to “Ubuntu (64-bit)” to update the EFI type.

Note, grub-efi-amd64 conflicts with grub-efi-ia32 and grub-efi, so you’ll end up losing the 32-bit version of the boot loader. This may or may not conflict with ability to boot from BIOS/CSM – I didn’t test this yet.

Core Data: Migrating ignores manual mapping model (or fails migration) despite mapping model's existence

Mini tip on clearing iCloud bookmarks and starting over

1 Reply

I’ve got a backup of my several thousand bookmarks (mostly junk, but who’s got time to dig through it?), but I’ve played a bit with importing and exporting bookmarks from all over the place so the syncing with iCloud never ends. Too many changes!

So, how did I just finish dealing with that?

What follows is error prone, and if followed, may cause data loss. I’m not documenting this for your benefit, but for my own. Consider this a letter to myself in the future. Please don’t follow this unless you yourself know exactly what you’re doing and have three backups of your important data. Everything that has to do with syncing is error prone out of the box, even without a user messing with it.

As previously discussed, iCloud’s bookmark sync actually involves WebDAV and XBEL. It also involves a file containing all bookmarks: ~/Library/Safari/Bookmarks.plist.

So after ruthlessly murdering the SafariDAVClient process (and ensuring it doesn’t wake up until I want it to: watch -n1 killall SafariDAVClient), I opened the aforementioned file. There were over 30000 pending changes. That’s unacceptable.

So I went and deleted all bookmarks using Safari, for convenience. The plist editor I used didn’t allow selection of multiple items, so deleting bookmarks would take a while. I left alone the Reading List, because I didn’t have a proper backup of that. I took note of what the remote folder name of the Reading List was.

Then I mounted the root WebDAV mount point, and went and deleted every folder and bookmark except for the Reading List — thus bringing iCloud into sync with the local dataset.

Then I watched the progress as the changes were applied, and imported the bookmarks several times using a certain tool I use that also reduplicates bookmarks and keeps a copy of bookmarks, etc. I watched the progress by opening Bookmarks.plist and looking at how many changes were in the ./Sync/Changes array.

Bulk uploading of Google Apps accounts via CSV using Apple Numbers

2 Replies

I’m using Apple Numbers on a Mac with Croatian locale. This means CSV files not only don’t have a BOM (byte order mark) which Google Apps requires, but its CSV files are not actually comma-separated, but semicolon-separated. (Croats use decimal comma instead of decimal point, so any CSV files generated by spreadsheets can’t use commas. Weird, yep, my dear US reader.)

So… follow Google’s instructions in building the CSV file, and then process it with the following commands in Vim.

:set bomb
:%s/;/,/g
:wq

You’re welcome.

Installing IMAP extension for PHP on Mountain Lion

12 Replies

Based on Dao Hoang Son’s tutorial (itself based on Dan Spencer’s Lion tutorial), as well as a few corrections of my own, I came up with the following script. It fetches and installs University of Washington’s IMAP library, then the PCRE – Perl Compatible Regular Expressions library, and finally the PHP IMAP extension itself.

Please study the script yourself before running it, or run it line by line. It performs no error checking, meaning it might totally wreck your machine — and I’m not responsible if it does. Keep a Time Machine backup around and unplugged from the computer, in case it does.

The script wasn’t yet tested in its complete form; it’s an approximate transcript of commands I ran in shell. But since I’ll keep it around in case I ever need to re-run it, I’m fairly certain it works.

UPDATE December 24th 2012 – Merry Christmas! I’ve applied corrections based on Mehmet’s comment. He also comments you need to install autoconf before running the script (so phpize can work). I already had it installed.

Filesizes for stuff I downloaded: 1.990.304 – imap-2007f.tar.gz, 1.539.766 – pcre-8.20.tar.gz, 12.926.535 – PHP-5.3.15.tar.gz

#!/bin/bash
BUILDDIR=/tmp/phpimapmountainlion

mkdir "$BUILDDIR"
echo " "
echo "= FETCHING AND INSTALLING IMAP"
echo " "
cd "$BUILDDIR"
wget -c ftp://ftp.cac.washington.edu/imap/imap-2007f.tar.gz
rm -rf imap-2007f
tar xvvfz imap-2007f.tar.gz
cd imap-2007f
make osx EXTRACFLAGS="-arch i386 -arch x86_64 -g -Os -pipe -no-cpp-precomp"
sudo mkdir -p /usr/local/imap-2007f/include
sudo cp c-client/*.h /usr/local/imap-2007f/include
sudo mkdir -p /usr/local/imap-2007f/lib
sudo cp c-client/c-client.a /usr/local/imap-2007f/lib/libc-client.a

echo " "
echo "= FETCHING AND INSTALLING PCRE"
echo " "
cd "$BUILDDIR"
wget "http://sourceforge.net/projects/pcre/files/pcre/8.20/pcre-8.20.tar.gz"
rm -rf pcre-8.20
tar xvvfz pcre-8.20.tar.gz
cd pcre-8.20
./configure --prefix=/usr/local
make
sudo make install

echo " "
echo "= FETCHING AND INSTALLING PHP-IMAP"
echo " "
cd "$BUILDDIR"
PHPVERSION=`php --version|head -n1|cut -f 2 -d ' '`
##git clone -b "PHP-$PHPVERSION" https://github.com/php/php-src.git php-src
wget --no-check-certificate -c https://github.com/php/php-src/tarball/PHP-5.3.15 -O PHP-5.3.15.tar.gz
tar xvvfz PHP-5.3.15.tar.gz
cd `ls |grep php-php-src-|head -n1`
cd ext/imap
phpize
./configure --with-imap=/usr/local/imap-2007f --with-kerberos --with-imap-ssl
make
sudo cp modules/imap.so /usr/lib/php/extensions/no-debug-non-zts-20090626/

ivucica blog

a programmer's thoughts