Running a physically installed Ubuntu in VirtualBox on Mac

Leave a reply

What follows includes my thoughts and notes more than definitive knowledge. I did manage to run Ubuntu in VirtualBox, but the process how I got there may include incorrect conclusions. Here goes.

  • VirtualBox 4.2.16 under OS X
  • OS X 10.8.4
  • Ubuntu 12.04 32-bit
  • Windows 8 32-bit
  • MacBook Unibody late 2009

(Yes, I presume you have the above setup; if you have something that’s 64-bit, unfortunately, I can’t help you there.)

DISCLAIMER

Everything in this post is dangerous to your data, even if you are an expert. Think thrice before doing anything, back up everything, triple-check everything I suggest you do for hidden assumptions that may be related to my machine.

Some steps may have inadvertently been omitted. I did not note everything I did, and I don’t feel like reproducing each and every of my steps.

I know this disclaimer will probably not dissuade you from trying this, but this includes particularly dangerous stuff and I don’t want anyone misunderstanding: I am NOT advising you to do this. I am NOT advising you to follow my advice. I am only documenting this process for my own private future use.

Mac specifics

Mac runs EFI instead of BIOS. This is quite similar to today’s UEFI machines, but does present its problems.

The partition table format used is GPT. Again, this is something normal and expected on today’s new UEFI machines, but something people may be unfamiliar with.

Somewhat specific to Mac, if one installs Windows, then the resulting partition table format includes both MBR and GPT. This is a result of the fact that 32-bit Windows don’t boot from GPT and don’t like EFI, and even more so, a result of the support for Windows on Macs being produced before -Longhorn- Vista was released. So the lowest common denominator is, in this case, 32-bit Windows XP; so, “BootCamp” produces a GPT+MBR hybrid partition table. (Experts seem to dislike hybrid MBRs.)

Under (U)EFI, the equivalent of BIOS does very little. It has support for FAT (usually FAT32), for loading additional drivers from the FAT partition, and for running .EFI binaries (mostly boot loaders, although in some cases these may be utility programs or shells).

On PCs, we will be mostly interested in 32-bit and 64-bit Intel .EFI binaries (i386 and x86_64 binaries). There are even universal binaries. Running an appropriate .EFI binary is how we boot an operating system. Since 3.3, Linux kernels seem to include an EFI stub; this means you can take a “vmlinux” binary and feed it directly to the EFI subsystem of an (U)EFI machine, without use of ELILO, GRUB2, or some other boot loader.

To expose the binary to the EFI system of the machine, you need to mount the FAT-formatted partition of type “EFI System Partition”, change to the \EFI directory on this partition, create a “vendor directory” (that’s the name of your OS or your OS vendor), and put “boot.efi” binary in that directory. Some EFI systems may expose other binaries as well, but this is what one is supposed to do.

Alternatively, systems may include a “Compatibility Support Module”. Macs do. This is a BIOS emulation layer and allows booting “legacy” operating systems. Microsoft says this must be turned off by default. Note that it is not a requirement; there is no need to include CSM. (VirtualBox does not.)

Please read around to familiarize yourself with EFI and GPT, and how they differ from BIOS and MBR. Trying to describe them here would be either a lot of work or would result in even more incorrect article than it’s already bound to be.

Easy parts and troublesome parts

VirtualBox supports physical partitions in the VMDK format. You need to use the command line to create the VMDK disk image, but it’s easily doable. It’s essentially a one-liner under normal, tested conditions.

Even when the conditions are not as normal, I’ve successfully ran BootCamp’d Windows XP previously. (Search my blog if you’re interested in hacks I had to do.)

That was easy: Windows XP deals with MBR, and MBR is trivial.

The problematic part with supporting Ubuntu is that its boot loader, GRUB2, actually appears to be too smart for its own good. Having installed the 32-bit version of Ubuntu 12.04, it has (by design) not deployed an EFI boot loader. Instead of deploying grub-efi, it has deployed grub-pc. This is great; the rEFIt boot loader that I use when booting the physical machine has picked up on this and allowed me to pick the Linux system. It does, unfortunately, depend on turning off the “bootable” flag in MBR with fdisk. This appears to be a limitation (or a feature?) of the Apple-provided CSM.

That’s all great. But when I tried to load this system with VirtualBox using default options, I got nothing. Zip. Nada. A black screen and that’s it.

Why? I have no idea. But I presume that GRUB2 picked up on GPT partition table and then got very confused by this on a non-EFI system. Why? Again, no idea. Switching to EFI got the setup to work — but only after installing grub-efi and deploying it to the appropriate place.

Creating virtual disk

You want to have a virtual disk that includes all relevant GPT information.

As I was originally playing with my previous MBR-based method, and the EFI+GPT method came from that, so does my script include some dding of GPT data: the first 40 sectors of the disk, and the last 40 sectors of the disk.

IMPORTANT: Always check device names (GPT partition IDs) with sudo diskutil list /dev/disk0. Always check MBR partition IDs with sudo frisk /dev/disk0. My setup includes Windows 8 and is very weird.

IMPORTANT: This script is NOT intended to be run as-is! Read it to learn what’s going on, triple check every single number, ensure you understand every single line, customize it for your machine and only then think twice before running anything below. Playing with partitions, with disk devices etc is dangerous. Ensure you have backups of everything that’s even remotely significant.

I personally have a Time Machine backup of my important data under OS X, every important project is stored on online code hosting, and everything else under physically installed Ubuntu and Windows 8 is not important. What’s your situation? Can you afford to lose data?

NOTE NOTE NOTE: As of OS X 10.9 Mavericks, Detection of disk size in blocks is broken in the script below. I haven’t updated it; when I needed to run it, I read it from a manually-ran fdisk and put it in appropriate place. (n.b. this could possibly be doable from native code: ioctl(fd, DKIOCGETPHYSICALBLOCKSIZE, &block_size); see http://stackoverflow.com/a/15947809/39974)

# ALWAYS check devices with:
# sudo diskutil list /dev/disk0

# ALWAYS check MBR partition IDs with:
# sudo fdisk /dev/disk0

EFISYSTEMPARTITIONOSXDEVICEID=1
RECOVERYPARTITIONOSXDEVICEID=3
LINUXPARTITIONOSXDEVICEID=4
LINUXPARTITIONMBRDEVICEID=3

sudo chmod 777 /dev/disk0s$EFISYSTEMPARTITIONSOXDEVICEID
sudo chmod 777 /dev/disk0s$RECOVERYPARTITIONSOXDEVICEID
sudo chmod 777 /dev/disk0s$LINUXPARTITIONSOXDEVICEID

sudo VBoxManage internalcommands createrawvmdk -filename PhysicalDisk.vmdk -rawdisk /dev/disk0 -partitions $EFISYSTEMPARTITIONOSXDEVICEID,$RECOVERYPARTITIONOSXDEVICEID,$LINUXPARTITIONOSXDEVICEID


# (primary mbr=1, primary gpt header=1, primary gpt table = 32, extra = 6) * 512
PRIMARY=40
sudo dd bs=512 count=$PRIMARY if=/dev/disk0 of=PhysicalDisk-pt.vmdk

# secondary gpt table=32 + secondary gpt table = 1
# see http://7bits.nl/blog/2012/01/02/mac-gpt-partition-table-recovery
SECONDARY=33
DISKSIZE=`diskutil info disk0 | grep "Total Size: .*exactly .* 512-Byte-Blocks"|sed 's/.*Total Size:.*exactly \(.*\) 512-Byte-Blocks)/\1/
'`
OFFSET=`calc $DISKSIZE-$SECONDARY`
OFFSET=`echo $OFFSET|sed 's/^ *//'` # ltrim

sudo dd bs=512 count=$SECONDARY iseek=$OFFSET oseek=$PRIMARY if=/dev/disk0 of=PhysicalDisk-pt.vmdk conv=notrunc

First, note the chmods. These will be required after each reboot. OS X only allows root to access the disks (for very important security reasons); VirtualBox does not run with root privileges. I don’t do this lightly; be VERY mindful that this is actually creating a local security hole, allowing user processes to read and even write to the disk.

Next, if you take a look at PhysicalDisk.vmdk, it’s a text file. You can see how various virtual disk sectors are mapped to various physical disk sectors, to “zero”, or to PhysicalDisk-pt.vmdk. (Please do check that you can find a section that matches this; if not, something went wrong in VBoxManage, and you should delete both .vmdk files.)

dd might not be necessary; but I ensure that whatever’s in GPT is not “accidentally” changed by VirtualBox’s VBoxManage.

Could one map even these 40 initial and 33 trailing sectors to the actual physical disk? Sure. But, why risk anything?

Installing rEFIt

Download rEFIt from its homepage. The install instructions say all you need to do with latest version, 0.14, is open the installer package and hit “Next” repeatedly.

Installing grub-efi

I decided to reuse EFI System Partition. I could have just as easily used the system partition; Apple ships a HFS+ driver, so the EFI subsystem can boot directly from the system partition.

The thing is, Ubuntu can’t write to the HFS partition, so it’s slightly easier to reuse the EFI System Partition.

What am I risking? Well, Apple might wipe this partition clean in an OS update. I hope they won’t.

IMPORTANT: The following can mess up GRUB. I can still boot using the “BIOS” GRUB2, but your mileage may vary.

What follows is inspired by Rod Smith’s EFI-Booting Ubuntu on a Mac.

  1. Boot physical Ubuntu.
  2. sudo apt-get install grub-efi – This removed grub-pc on my machine, although I still seem to have the ability to boot using BIOS. (Anything else would be… troublesome.)
  3. sudo mkdir /boot/efi – This is the place where we’ll mount the EFI System Partition.
  4. sudo mount /dev/sda1 /boot/efi
  5. sudo mkdir -p /boot/efi/EFI/Ubuntu – Apple doesn’t ship an \EFI folder. We’ll create it, along with the “vendor” directory for Ubuntu.
  6. sudo grub-install /dev/sda1 – This should install grub-efi to \EFI\Ubuntu.
  7. ls -al /boot/efi/EFI/Ubuntu – You should see two files from Ubuntu here.

It’s important to understand: 32-bit Ubuntu installs 32-bit GRUB2. This will not be bootable on a 64-bit capable Mac. This is solely useful for VirtualBox.

So, ensure that you can still use the BIOS GRUB2, or have an alternative boot method, or else you’re now converting your physical installation into a VirtualBox-only installation!

Creating virtual machine

I don’t have a script for this one. Go back to OS X, go to VirtualBox GUI and create an Ubuntu-type virtual machine. Don’t pick the 64-bit version; this changes the type of EFI that the virtual machine will use!

Pick the previously created PhysicalDisk.vmdk while creating the machine.

Now edit the settings. Right click on machine name, pick “Settings”, and change the machine to be an EFI machine on the System tab. So: right click [machine name]->Settings->System->Motherboard->Enable EFI (special OSes only).

Don’t boot yet! Did you chmod the disk devices? Remember, you rebooted. Please sudo chmod 777 all partition devices in /dev (and be mindful that this is a security hole you’re creating, which you might somehow avoid with UNIX user groups, but meh).

After this point, do not recreate the PhysicalDisk.vmdk without keeping in mind that this file includes disk image IDs in several places. VirtualBox keeps track of the disk images, and will NOT be happy if the ID changes.

So, done now? Great. Boot.

You’ll be shown the EFI shell. Hoorah!

Now, let’s change to the EFI System Partition‘s filesystem and boot GRUB2.

fs0:
cd EFI\Ubuntu
boot.efi

This should show you your physical machine’s GRUB menu and the booting should move on. Observe the disk light on the bottom of VirtualBox’s window; if it stops flickering for longer than 15 seconds, and Ubuntu does not boot, you can presume you have some sort of an issue.

Note that virtual machine does have different hardware than your physical machine; for example, NVIDIA graphics driver does not work for me. I get the console, but not X11. It would be trivial to fix (replace the selected driver with vesa or something similar in Xorg.conf) but I don’t care: I need to SSH into the machine and tunnel X11 to XQuartz on OS X. I don’t need Unity: I need the ability to work on my code and display the X windows.

So, this works for me. Huzzah!

Small updates

fstab

Add this to /etc/fstab (based on Rod Smith’s post, too):

/dev/sda1       /boot/efi       vfat    ro,fmask=133    0       0

Alternatively, change that ro to rw to get the partition to mount read-write; this may be important for grub updates.

grub-efi-amd64

Ubuntu 12.04 also ships with grub-efi-amd64.

sudo mount /boot/efi -o remount,rw # if not already mounted read-write
sudo apt-get install grub-efi-amd64

Don’t forget to change machine type to “Ubuntu (64-bit)” to update the EFI type.

Note, grub-efi-amd64 conflicts with grub-efi-ia32 and grub-efi, so you’ll end up losing the 32-bit version of the boot loader. This may or may not conflict with ability to boot from BIOS/CSM – I didn’t test this yet.

GSoC 2013: Doublebuffered painting!

1 Reply

Thanks to a fix that Eric applied to Opal’s handling of painting CGBitmapContext->CGImage on top of another CGContext, the backend now actually paints something sensible on a doublebuffered window. Hooray!

Well, actually, it only paints something sensible on a menu window. And as soon as the backend has to paint an update, for some reason the updated stuff is painted (mostly) upside down. I’ve tested with the Gtk theme, and the in-window menus also seem to be updated upside down.

I also seem to be unable to get menu items to be wider despite various hacks when it comes to fonts. It seems that OpalFontInfo‘s -widthOfString: is not used at all, and a series of calls to -advancementForGlyph: is instead performed, with no effect on the width of the menu whatsoever.

I’ll explore further what’s wrong with both of these issues tomorrow; preparing for a job interview today will take the rest of this day.

Note: I did not commit my yesterday’s work described above, as it mostly consists of hacks that have no useful effect onscreen. Eric’s changes to Opal, combined with a change of #if 0 to #if 1 in gnustep-back’s OpalSurface‘s init method do produce some of the visible changes described above.

GSoC 2013: Drawing some actual content

Leave a reply

Looking at the existing [...GState DPS...] methods and comparing them to the Core Graphics functions, it looked like implementing the backend will be as much of a cakewalk as I originally presumed.

Turns out it is not meant to be.

(Today’s text is a report as much as it is a way for me to contemplate next steps. It’s a noisy stream of thoughts. Sorry about that.)

All is drawn, but nothing is seen (a.k.a. doublebuffering)

The Cairo backend contains code intended to render into a backing store and then draw the content on screen when requested to do so by X. (Presumably, also to blit the content when it updates.) Theoretically, it should be trivial to add this functionality: add a new CGBitmapContext, redirect all drawing into this new context, and then blit this content on screen when requested.

When creating the surface that targets the screen, one gets a gswindow_device_t* that describes the X11 window. To tell the rest of the gnustep-back that this surface can handle its doublebuffering (although it really can’t), we set this gswindow_device_t‘s gdriver to self, and we add flags GDriverHandlesExpose and GDriverHandlesBacking to its gdriverProtocol bitmask. This is done only if the surface is not NSBackingStoreNonRetained.

(From the above description one would think by default, backing stores would be already handled elsewhere in gnustep-back. Unfortunately, although it seems like gnustep-back might be trying to handle the case where the surface does not handle X11’s Expose event nor does it handle having the backing store, it does not do so, at least not properly.)

So, I’ve tried to do it. All drawing is redirected to a CGBitmapContext, and then in -handleExpose I’m trying to blit the content on the X11-backed Opal surface. It seems that Opal currently has some issues with the following code (i.e. I get everything drawn in pale yellow):

- (void) handleExposeRect: (NSRect)rect
{
  CGRect cgRect = CGRectMake(rect.origin.x, rect.origin.y, 
                      rect.size.width, rect.size.height);

  CGImageRef backingImage = CGBitmapContextCreateImage(_backingCGContext);        
  CGContextDrawImage(_x11CGContext, cgRect, backingImage);  
  CGImageRelease(backingImage);
}

Debug output from Opal suggest it’s going through all the expected dance moves: creating that image, converting it into destination context’s colorspace, et al. I’ll poke Eric to see if he has a better idea of what’s going wrong.

An interesting behavior is that if I simply don’t redirect drawing to the bitmap context, then I get some content on screen. Despite not having an actual backing store. Maybe GDriverHandlesBacking has side effects? I’ll have to dig in more.

(Oh, and gotShmCompletion methods seemed like they might be related. No change if I implement them.)

Everything is flipped

When I first got some content on the screen, it turned out that GNUstep’s zero-zero seems to be based on top-left corner of the drawable — opposite from what Opal expects.

So as a first attempt I went in and implemented matrix functions. While things don’t get drawn in the appropriate place yet, it seems like I’m on the right track. The thing that worries me is that I’m not sure what’s the orientation of DPS matrix as opposed to the orientation of the CG matrix. Also, I’m now overriding DPS methods and not calling super, yet there’s a bunch of code in GSGState‘s implementation that makes me think this is not such a good idea. DPS functions seem to be maintaining matrix state; hopefully they don’t ever use it directly. If they don’t, then it might be safe to simply override the getter for the matrix.

To be able to reset the Opal matrix to the identity (or, near-identity, considering it’s also flipping the coordinate system), I’ve also added a method to Opal that allows this precise operation. This is an extension over what Apple provides: Apple provides absolutely no way to set the CTM (presumably, this stands for current transform matrix) directly, which makes only a little bit of sense. I don’t see a good reason this functionality is not provided to developers. (I can think of one that’s just a tiny little bit good.)

Menus have a single item

Now all I get in a menu window is a single menu item. Drawn in place of the menu window’s title.

Meaning, drawn on a completely incorrect place. I’m not really sure if this is a menu item drawn in an incorrect place, or if this is caused by the fact that -compositeGState:..., which seems to permit blitting one GState‘s backing surface’s content on top of another GState‘s content, is not implemented.

If this is so, this may prove very troublesome to implement without adding further hacks into Opal. Core Graphics seems to provide no real way to copy a portion of a context on top of another context, apart from using CGBitmapContext.

Fonts. Oooh, boy.

Cairo backend does a lot of stuff when it comes fonts. A lot, lot of stuff. I wonder why most of that is not exposed to the rest of gnustep-back, so I’ll probably be moving that out of the Cairo backend and into an additional module within gnustep-back called fonts, or something like that. Or perhaps even into the existing gsc module.

Just taking a quick look, art backend seems to be supporting just fonts with the .nfont extension (explaining Helvetica.nfont), but I may be wrong. Taking a quick look at xlib backend, and it seems to be doing something similar to the cairo backend.

It’s going to be a pain to implement fonts, especially to do it properly. I’ll have to think about how to deal with refactoring this — cairo backend is popular and accidentally breaking it won’t be looked at kindly by the community. Still, it’ll be worth it.

Unless it turns out I should have dealt with Opal. In which case Cairo backend’s implementation will stay Cairo backend’s implementation, and Opal backend will leverage Opal for its text needs.

Yes — just as the title of this section implies, I’m not sure what I’m talking about.

Other upcoming steps

Among other things, I’ll look into getting image painting to work. GNUstep does a lot of things with images, so seeing the images might help me figure out what’s going on more easily.

GSoC 2013: Red rectangle!

Leave a reply

It turns out I’m slower with GSoC progress than anticipated. GNUstep meeting in Cambridge, which I’ll discuss in upcoming posts (and do so as soon possible, so I don’t forget all the wonderful sights and experiences, and all the wonderful people I met), was informative, but it turns out not quite supportive of my GSoC productivity. I spent a week being nervous before my first international trip, and I spent the last week (after returning) being impressed with how wonderful Germany and England are.

So, what progress did I make?

Preparations

First, before the trip, I prepared the backend. I figured out the simple things, such as where do I have to modify configure scripts in order to add support for the new backend, as well as which classes are expected by gnustep-gui. I identified the minimum set of methods that need to exist in the relevant classes (mostly OpalGState) and I added stubs.

Opal exposing X11 drawables as contexts

In Cambridge, thanks to help from Opal’s author Eric Wasylishen, we started exposing the creation of CGContext specific to X11 drawables. The function already existed in Opal; we just refactored the code a bit. (Rest of the hacking time in Cambridge was spent on other tasks.)

Drawing onscreen

Now that I finally cleared my head of amazement at how beautiful Cambridge is and how great hosts we had, I decided that it’s really long overdue that we get some content drawn on-screen. It took me a while to figure out where to draw in order to actually get the content drawn on-screen. It turns out the safest place to insert my dummy draw method is -[OpalGState DPSfill] — probably because it’s intended as the place to draw stuff, as opposed to various initialization methods such as -[OpalGState DPSinitgraphics].

The dummy draw method draws a red rectangle 1024×768, so I’m very happy to report that the test application’s NSWindow is, as of today, fully red.

Thoughts on next steps

Overall, I hope that things will roll much, much faster from here onward. There are bound to be many bugs along the way — and many assumptions by existing apps and themes. It’ll slow implementation down, but I hope that along the way I’ll figure out a way to clean up the current messy situation with gnustep-back‘s numerous classes such as GState, Context, et al.

The scariest part is that the Cairo backend looks like it handles many complex behaviors and different use situations. Let’s see how many bugs the Opal backend will have in the end and how much of Cairo backend’s behavior I’ll have to replicate to fix them. 🙂

GSoC13: Kicking off

Leave a reply

Last year, I did not document my work on the Core Animation implementation for GNUstep. This year I’m participating again, and as I study Opal and gnustep-back, and work on the new backend for GNUstep, I’ll try to document what I learn continuously and publicly.

You can check out the project page on Google Melange, as well as the somewhat lengthier and more detailed proposal page on Google Melange, which contains the rationale for the project and the general idea of how I’ll proceed.

Now, let’s figure out how to get only the GSoC2013 category to show up in an WordPress RSS feed… 😉

Starting the default launcher using adb

2 Replies

If you’re stuck with a custom launcher (e.g. OUYA’s) and want to get back to the stock launcher, first connect with adb:

adb connect 192.168.56.101
adb shell

Then, in the shell, punch in the following

am start -c android.intent.category.HOME -a android.intent.action.MAIN

This will dispatch an intent filtered by category android.intent.category.HOME and action android.intent.action.MAIN. You should also be able to dispatch an explicit intent, passing the full package and class name:

am start -n com.android.launcher/com.android.launcher.Launcher

Sadly, this didn’t work on AndroVM’s 4.1.x; perhaps the launcher’s name is different. Not that I care particularly 😉