VirtualBox Grub2 Rescue ISO

Happily booting for over a year, one day the upgrade of Ubuntu came and I bit the bug. From 10.10 to 11.04 LTS. Smooth as glass for Ubuntu, but it shattered into pieces with Grub2.

The first symptom was booting into this:
error: symbol not found: 'grub_env_export' grub rescue>
No Grub boot menu, so no selection between dual-boot, not even a default OS. This is corrected by booting a Ubuntu Live CD and running this:
# sudo mount /dev/sdXx /mnt # sudo mount -o bind /dev /mnt/dev # sudo mount -o bind /sys /mnt/sys # sudo mount -o bind /proc /mnt/proc # sudo chroot /mnt
Once at the prompt, reinstall grub (sda is the first disk. It may ask if you want to update /dev/sdaX, where X is the Ubuntu partition, I did this.):
# grub-install /dev/sda
Now update grub:
# update-grub
then Ctrl-D twice to exit, and reboot.

Now I can boot Windows or the squeaky clean Ubuntu 11.04. So forging into VirtualBox to run Ubuntu raw disk under Windows 7, there is another massive failure.
... nothing but blackness ...
There was no Grub menu. Euek.
Not to despair, I created a Grub2 ISO.

To create a bootable iso image from Ubuntu:
# grub-mkrescue --output=rescue.iso /boot/grub
This can be used in a Virtualbox by assigning it as the primary CD disk, then starting the instance. It contains all the grub files from /boot/grub, including the latest menu.

Attach a bootable Grub2.iso to VirtualBox VM

To use this Grub ISO, copy the rescue.iso file from Ubuntu to Windows under the VirtualBox VM directory containing the Ubuntu VM, and add it as the first CD disk to the VirtualBox manager’s Storage IDE Controller definition of Ubuntu.

Now when you start the VM and see a black box with a grub> prompt, enter:
grub> configfile (hdX,Y)/boot/grub/grub.cfg
where X=the hard drive number, 0=first, 1=second…
and Y=the partition of the boot filesystem on the drive
Ex:
configfile (hd0,msdos6)/boot/grub/grub.cfg
Tip:
use ls to find the right disk
Reference: https://help.ubuntu.com/community/Grub2

If the menu does not display, you can try these things:

In file /etc/default/grub, un-comment
GRUB_HIDDEN_TIMEOUT=0 GRUB_HIDDEN_TIMEOUT_QUIET=false GRUB_GXFMODE=640x480

Finally having things back to normal, I booting into Ubuntu from VirtualBox, installed an update from Netbeans 7.0 to 7.1. Luckly I did no more, shut down for the night and came back the next day. Wowza! Netbeans 7.1 is missing! Sure enough, if I created a file, rebooted within VirtualBox, my new file is gone…. Flashback… If I boot the same raw disk outside of VirtualBox, create a new file, the new files are retained after rebooting.

You can’t rely on your memory anymore.

As a wild guess I created a snapshot in VirtualBox with the VM stopped. Did my test again, and voila! Like magic my new file remains intact. Thanks to the post here:
https://forums.virtualbox.org/viewtopic.php?f=7&p=155510
for the hint.

VirtualBox

Virtualbox is a way to run another operating system within an already running operating system. Originally written by Sun Microsystems, and now owned by Oracle when they bought Sun in 2010, it is one of a dozen Virtual Machine software offerings.

One way I use it is to run Windows and Linux on the same PC at the same time with near-native speed. I call it ‘simultaneous boot’. Here is how I did it.

Start with a multi-core GB memory machine. I suggest at least 4 64-bit cores that support VM in the BIOS (i5-Intel) and 8 GB of RAM. Be sure you have a large disk, such as Seagate’s 1TB $59.00 Barracuda special. You will NEED 64-bit processor and motherboard support for more than 3GB of memory.

Partition and install Windows (XP, 7, etc…) 64-bit on a hard drive first. Windows assumes it owns everything so it is best to let it think that way.

Next partition and install Linux (Ubuntu, Fedora, etc…) 64-bit on the same drive, being careful to skip the existing windows partitions. It will install a boot loader (GRUB, etc…) that allows you to select which operating system you want to run.

http://pagead2.googlesyndication.com/pagead/show_ads.js

Test it out and boot several times between Windows and Linux.

Enable the VM extensions on the motherboard’s BIOS. This will allow one VM to have exclusive rights to whatever number of CPUs you have assigned it.

Download VirtualBox 64-bit from Oracle’s web site, as it has the VRDP, and USB extensions, but mostly because it can handle raw virtual disks. This is where the magic comes in. Normal VMs create files under the host operating system that the guest thinks are real disk drive partitions. In raw VM disks the filesystem for the guest are native to the guest, thereby making it as fast as greased lighting.

If your first drive on the machine is what you want, use ‘PhysicalDrive0’. For me, it’s the second one, so I use ‘PhysicalDrive1’. Be sure to Right Click cmd.exe in the Windows menu and select ‘Run as Administrator’ to enable access to raw disk.

Here is the disk we want to use, Partitions:
1 = Windows 7 ‘System Space’ (type 07)
2 = Windows ‘C:’ (type 07)
5 = Ubuntu ‘swap’ (type 82)
6 = Ubuntu ‘/’ filesystem (type 83
–> they show up as SATA drive 1 <–

cd C:Program FilesOracleVirtualBox
VBoxManage.exe internalcommands listpartitions -rawdisk \.PhysicalDrive1
Number  Type   StartCHS       EndCHS      Size (MiB)  Start (Sect)
1       0x07  0   /32 /33  12  /223/19           100         2048
2       0x07  12  /223/20  1023/254/63        499899       206848
5       0x82  1023/254/63  1023/254/63          7720   1024002048
6       0x83  1023/254/63  1023/254/63        446147   1039814656

We need to register the Ubuntu partitions with VB using ‘internalcommands createrawvmdk…’. I want partitions 5 and 6 from the list above.

VBoxManage.exe internalcommands createrawvmdk -filename D:VirtualBoxubuntu.vmdk -rawdisk \.PhysicalDrive1 -partitions 5,6

Now create a new VM in VirtualBox, and choose the existing disk you created above (D:VirtualBoxubuntu.vmdk). Also choose ‘Use Host Disk Cache’ for the SATA controller, and PXII System extensions. Allocate two (half) the CPUs you have, and half (4GB) of memory.

When you boot Ubuntu in VirtualBox it will display the Grub menu:
===> Choose the Linux partition ONLY!!! booting back into a running Windows system will corrupt the disk!!! <====

When I booted into Linux from VirtualBox the window manager did not start, leaving me looking at a black screen with the Login: prompt. To fix this I booted the machine into Linux and removed the proprietary Nvidia video driver, restarted the machine again and selected a lower screen resolution from the default video driver. Now boot back to Windows, run VirtualBox and select Linux from the GRUB menu. The GUI window manager video worked for me!

Now is a good time to install the VirtualBox Host Add-ons from the Device menu. This gets you better video, mouse, USB and file sharing.

References:
http://chaits.accosted.net/?p=54
http://www.neowin.net/forum/topic/784138-howto-boot-existing-ubuntu-partition-using-virtualbox-inside-windows/
http://www.virtualbox.org/manual/ch09.html#rawdisk

Remote Desktop Windows Protocol

To access the box remotely set the VM Display tab ‘Remote Access’ check box on, and change the port to something other than 3389, such as 5000. Leave the Authentication to NULL and time out 5000. Make sure the 3D and 2D video acceleration are turned off.

Start the VM, but leave it at the Login window. Start a Windows Remote Desktop program (RDP, not VNC) and enter the IP address of the Host PC, NOT the VM’s IP, with the trailing port number. For example:
1.2.3.4:5000

If all goes well you will see some warning about non-authentication connection, then click OK and your Linux Login GUI should display. You are in! One funny aspect of this is that using RDP on the local machine has a better display than VirtualBox, due to lowered screen resolution and it has better control over the full-screen view. I need to experiment more with the display settings.

References:
Mac OS X
Windows XP
Windows 7
Linux

Remote Desktop VNC Protocol

To set up VNC to Ubuntu running under Windows 7 Virtualbox you can use the NAT Advanced port forwarding settings and Ubuntu’s tightvncserver (sudo apt-get install tightvncserver). On Windows you can get a VNC client from TightVNC.

Under Virtualbox setting for Ubuntu, select Network
Change NAT adapter (the default) port forward:

host port: 5902 – guest port: 5902

Start Ubuntu, run:
tightvncserver :2
Test- Windows 7-> Connect to ‘localhost:2’
Test- Other host -> Connect to ‘<ip address of Windows 7 PC>:2’

References:
Mac OS X
Windows and Linux

The speed of Windows and VM-Ubuntu are exceptionally fast, as if they are both running natively. I have half the CPU and memory allocated to the VM. The other nice advantage is if VirtualBox quits working, I can still dual-boot into both operating systems.

You can also export your entire VM setup and data, then import it somewhere else. Oracle has several examples of that here where you can try out some of their complicated-to-set-up tools without all the muss-and-fuss. I’m not sure how it would export the raw disks.

Hope this helps.

Music Server

Sockso is a Java based web server to play your mp3 music collection from a central computer to any computer. Based on open source, it is extremely easy to set up and run. You simply point it to a filesystem directory, let it scan, and when finished point your browser to http://localhost:4444.

A few things about security you should be aware of, lest the DCMA police disconnect your ISP and send you downtown. Secure your music site with passwords, and require pre-registration.

When you unzip the install, go to the bin directory and start it up:
If you are on windows -> windows.bat
If you are on linux -> linux.sh

A nice GUI will appear with tabs across the top.

Go to the Users tab, and click the check-boxes: Require login, and Disable registering. Then press the button: Create User, and make yourself an account. This will require you to pre-create an account for anyone wanting to play your music collection.

Now go to Collection and press the Add Folder button. Navigate to your collection of mp3s. When you have all your folders, press the Scan Now button and let it rip for a while.

An hsql database will be created in your home folder; ~/.sockso

-rw-r--r-- 1 me me 2572141 2011-05-01 09:48 database.script
-rw-r--r-- 1 me me     419 2011-05-01 09:48 database.properties
-rw-r--r-- 1 me me   13468 2011-05-01 10:08 database.log
-rw-r--r-- 1 me me      16 2011-05-01 10:15 database.lck

This is a memory-only database that is saved in ascii file database.script. This is important because you want to back your mp3 database up if you create playlists and users, otherwise they will be lost.

Another important item with sockso is how it gets the music mp3 file details from ID tags embedded in the .mp3 file itself. If you buy songs through Amazon they come pre-loaded with all the info you need, but if you copy your CDs, they may not. You can download an mp3 editor and add ID-tags yourself. A good open source and cross platform one is EasyTAG.

Apache sites can host Sockso in a virtual host, like this:

<VirtualHost *:80>
        ServerAdmin webmaster@example.com
        DocumentRoot /data/domains/example.com/www/public/
        ProxyRequests Off
        ProxyPass / http://www.example.com:4444/
        ProxyPassReverse /  http://www.example.com:4444/

        ServerName music.example.com
        ServerAlias music.example.com

        ErrorLog ${APACHE_LOG_DIR}/error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn

        CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>

Which allows a URL such as => http://music.example.com
Just be aware that you need to keep the sockso GUI running to server the web pages and songs.

Songs can be played in several formats, such as flash, javascript, external winAmp pls playlists, etc…

In summary, it is nice to have access to your music wherever you go and to be able to share it with close friends. Organizing playlists can be a fun way to discover your lost songs, if you have 1000s of files, and also having your friends start their own music server.

Pleasant sounds….

Test and Debug

Design by tests. When you have the goal in mind you built a test case. Every functional piece should be able to be tested in isolation.

For instance, validating a login, listing an article, and updating a row in the database all should be testable separate from the whole application.

Accomplishing this goal is helped by using common utilities such as:
– command line parser
– logging
– web browser debugger

Main interfaces will be in most every class. HTML files can be opened on the file system. Firebug in Firefox. Log4j and Java Command Line Option (JCLO) libraries are crucial elements of a successful test driven development strategy. Even JQuery has a log function.

For example, we can create a Log class to encapsulate log4j that will allow us to run without log4j, into stdout and stderr. All our classes can call Log.write(…) with various levels of detail, error, warn, info, debug, etc…

Like this:

package com.doncohoon.util;

import java.util.Date;

import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import org.apache.log4j.Priority;

import com.doncohoon.db.Access;

public class Log {
	static boolean v = false;
	public final int all_int    = Level.ALL_INT;
	public final int trace_int  = Level.TRACE_INT;
	public final int debug_int  = Level.DEBUG_INT;
	public final int error_int  = Level.ERROR_INT;
	public final int fatal_int  = Level.FATAL_INT;
	public final int info_int   = Level.INFO_INT;
	public final int off_int    = Level.OFF_INT;
	public final int stdout_int = 1; // emergency standard log
	public final int stderr_int = 2; // emergency error log
	private Logger logger = null;
	
	public Log(Logger L) {
	   this.logger = L;	
	}
	
	public Log(Logger L, boolean debug) {
		if (debug) {
			v = true;
		}
		this.logger = L;		
	}
		
	public void setVerbose (Logger logger) {
		logger.setLevel(Level.DEBUG);
		v = true;
	}
	
	public void all(String msg, Exception e) {
		write (all_int, msg);
		logger.info("Exception: ", e);
	}
	public void write(int level, String msg, Exception e) {
		write (level, msg);
		logger.fatal("Exception: ", e);
	}
	public void write(int level, String msg) {

		// Switch
		switch (level) {
		case Level.ALL_INT:
			logger.info(msg);
			break;
		case Level.TRACE_INT:
			logger.trace(msg);
			break;			
		case Level.DEBUG_INT:
			if (logger.isDebugEnabled())
			    logger.debug(msg);
			else if (v)
				write (stdout_int, msg);
			break;
		case Level.ERROR_INT:
			logger.error(msg);
			break;
		case Level.FATAL_INT:
			logger.fatal(msg);
			break;
		case Level.INFO_INT:
			logger.info(msg);
			break;
		case Level.OFF_INT:
			logger.setLevel(Level.OFF);  
			break;
		case Level.WARN_INT:
			logger.warn(msg);
			break;
		case stdout_int:
			Date now1 = new Date();
			System.out.println(now1.toString() + "-App:" + msg);
			break;
		case stderr_int:
			Date now2 = new Date();
			System.err.println(now2.toString() + "-App:" + msg);
			break;
		default:
			Date now = new Date();
			System.err.println(now.toString() + "-App:Invalid level."
					+ level);
			break;
		} // switch

	} // write
}

The other technique is to be able to call each class from the command line and exercise its methods. For this we will need a command line argument parser for the main(…) method. JCLO works quite well.

Like this, we add a Args class to each class. For class NanoHTTPD we add the following:

 /*
  * Command line arguments
  */
class NanoHTTPDArgs {
	private String confdir = "config/";
	private String log4j_xml = "LocalServer_log4j.xml";
	private String db = "DB.xml";
	private String dbdir = null;
	//
	private boolean v = false;
	private String[] additional;
}

and add a parse method:

	/**
	 * Parse command line inputs
	 */	
	private static void parseArguments(String[] args) {
		try {		
			JCLO jclo = new JCLO(new NanoHTTPDArgs(), args);
	
			// Allow debug to override log4j on the command line
			v = jclo.getBoolean("v");
			
			// Allow override APP directory, w/trailing slash
			confdir = jclo.getString("confdir");
			
			// Allow override of log4j xml configuration file
			log4j_xml = jclo.getString("log4j_xml");
			
			// Allow override DB directory, w/trailing slash
			dbdir = jclo.getString("dbdir");
	
			// Allow different database connections
			// i.e.: Database_derby_client.xml, or Database_oracle_client
			db = jclo.getString("db");
	
		} catch (IllegalArgumentException e) {
			String[] arg2 = { "Debug" };
			log(stderr,"---> App: Unknown parameter <---");
		     for (int i = 0; i < args.length; i = i + 1) { // Test and Loop
		    	 log(stderr,args[i]);  
		       }		    	
			JCLO jclo2 = new JCLO(new NanoHTTPDArgs(), arg2);
			log(stderr,jclo2.usage());
		}		
	} // parseArguments

then finally parse the main input parameters, like this:

	public static void main( String[] args )
	{
		/* parse the arguments */
        parseArguments(args);
.  .  .		
       // call all the normal methods here

      // then print some results to System.out.println(...);

      // and exit with a zero, one, two, etc... so the unit test driver can determine success
       }

If the class is called from command line, main is called, does some processing and returns results via stdout, or if the class is instantiated via another class, the same processing may be done and returned results sent back via some object result set. See? NanoHTTPD.java

Model View Control

Control ties the web server (view) to our database (model). In this implementation views are further defined using HTML files, Java classes will glue database stored procedure models containing SQL.

Example of Model View Control Thought Process

HTML5 files will use JavaScript to call APIs which determine the next step, or control. Models are entirely accessed in stored procedures within the database, via call statements, keeping CReate Update and Delete (CRUD) SQL separate from the decisions.

Security controls are also in the controller while delegating either an LDAP or DB model repository to store usernames and passwords.

These defined roles will allow changes to display formats; laptop vs mobile, or database; Derby vs Oracle, to evolve.

Don Cohoon

Thoughts into words