Apache: Full Long File Names in Index

There are cases that the default index file in Apache server trimmed your long file name. Instead you'd like to show them in full length. To do this, you can edit the configuration file of your Apache server (httpd.conf). Add "NameWidth=*" to IndexOptions, so that it looks like:

IndexOptions FancyIndexing NameWidth=*

Reference
http://httpd.apache.org/docs/1.3/mod/mod_autoindex.html#indexoptions

Migrate from Goolge Pages to Appengine

Migration without losing any data.

When it comes to Google Pages, will you opt-out or do nothing and wait for your pages to be migrated to sites smoothly? We can wait, but at a cost. Google Sites do not provide equal or more features than Pages do. No custom Javascript; no uploaded HTML; limited layouts, themes. It dose not seem possible to migrate without losing any data. Even the appearance will change. So, where else can we migrate to?

Our choice is Appengine.

Pros:

• Powerful.
• From the same vendor.

Cons:

• Technical skills required

Step by step migration:

1. Download your pages as a zip from Google Pages
2. Create a new appspot handle
3. Download appengine SDK from http://code.google.com/appengine/
4. Create a new project
5. Configure the project
6. Unzip pages into static content directory
7. Test on localhost
8. Deploy to appspot
9. Test with http://<your-handle>.appspot.com/
10. (Optional) Bind your domain

The long waited automatical migration from Pages to Sites is still not done! Is Google waiting for everyone to opt-out? The magic is a rumor.

Option 1 - Do nothing, and your pages will automatically be moved to Google Sites

• We'll set up the new site and move your pages for you.
• Visits to your googlepages.com URL will redirect to your new site.
• Note that Google Sites does not support custom JavaScript or CSS in its pages.

Option 2 - Opt out of the move, and take your pages to a new location

• Download your site.
• Once you've moved your site to its new location, opt out of migration by setting up a redirect below.

Let's wait and see.

Distinguish the two words in reCAPTCHA

reCAPTCHA is a great idea, but people can `easily` distinguish the known word from the unknown one.

Moebius Gears

Here's a picture of Moebius Gears.

Look cool? If you are on a Linux box, you might already have a lively demo installed. For Ubuntu users, it is located at /usr/lib/xscreensaver/moebiusgears.

Press Alt+F2 and type /usr/lib/xscreensaver/moebiusgears and run!

The xscreensaver source code could be downloaded from here.

The Risk of OpenID

OpenID seems a promising standard for user authentication. If service providers support OpenID, users can login without creating another pair of account and password. Ideally, a user need only remember one pair of account and password. However, such convenience comes at a cost. The only pair of account and password or the OpenID provider becomes a Single Point Of Failure. If you ever forgot your password or the Identity Provider withdrew your account because you didn't login in the past three months, you will have no way to login to any of the services. Here, we assume that the service providers only accept authentication from OpenID providers. This also applies to other forms of third party authentications.

For users, care should be taken on choosing OpenID providers. Users should keep their OpenID account active and secure the password. For service providers, they should allow other ways of authentication besides OpenID. Users should not lose the service if they lose their OpenID.

OpenID is not that open, not even as open as telephone numbers which could be transfered among telephone service providers.

Dimensional Modeling vs Interval Tree

The concept Dimensional Modeling is like the Interval Tree (Segment Tree). Both speed up queries by storing information at different levels. The difference is that in Dimensional Modeling, levels are defined by users, while in Interval Tree, levels are usually defined by a complete binary tree. Interval Tree usually solves problems on one dimension, but it could also be applied to 2D problems. One dimension is not a limit.

Different outputs in Tcl - Stdout vs Return Value

When both set a {hi} and puts "hi" echo a "hi" in an interactive tcl shell, what is the difference between them?

Clarification on "output"
1. standard output
puts command will output to stdout by default.

% puts "hello"

hello

2. return values
set hi {hello} has a return value hello and will echo back to user

% set hi {hello}

hello

To make things clear, look at this example.

% set ret [puts "hello"]

hello

% puts $ret

(blank)

So that the return value from puts is blank.

Compound statements
Let move on to compound statements.

% set a "hello"; set b "hello2"

hello2

This gives a single line of output from `set b`, and a and b are both set.

% puts $a

hello

% puts $b

hello2

Here's two lines of output from stdout.

% puts "hello"; puts "hello2"

hello

hello2

Importance
Why is this important? What if we would like to store a calculated value into variable a.

% set a [expr 3+3]

6

% puts $a

6

This is what we expected. However, if we wrote another version of expr that output the result into stdout.

% set a [set b [expr 3+3]; puts $b]

6

% puts $a

(blank)
We get nothing.

Generally speaking, it is not a good idea to print to screen from many functions. Because this would make the testing difficult, if not impossible. Usually, assert will compare the return result to expected result, while the output to standard output are ignored. Despite all that, we can still try to test function which output to stdout. The idea is to trap the stdout. This assert for stdout would look like this pseudo code:

procedure assertStdout (expected, command) {

  create out for this instance

  execute command > out

  flush out

  return expected == out

}

One trick is that if multiple assertStdout procedures are running, 'out's must be distinct in order to keep thread-safe.

The echo back of tcl might be of use, but it could confuse people with the output from within procedures. However, it is mandatory to distinguish them for a programmer. Other scripting languages like shell does not have such echo back, but that doesn't mean it is OK to output from many functions for them. Especially, for compicated software that requires a thorough test, standard out or output to file everywhere would definitely bring down the testing.

Snapshot in UI Testing

If you have experience in JUnit or other testing tools, you probably know the importance of setup and teardown methods. These two methods help build a testing environment. However, when it comes to UI testing, it is not the case. UI testing means taking a long time, boring, and random exceptions. What's more, every time an error occurs, which might be caused either by a typo or by the tested codes, the only solution is to start over again from the very beginning. This is boring!

If we could take a snapshot at certain points (which are called save points), we could easily get started from those points. It should be time efficient. It certainly save a lot of time otherwise wasted in setting up the environment again. What's more, you can benefit from reproducing some random errors. If a snapshot were taken just before it occurs, the error would probably show up again, which also helps reduce the number of bugs.

It is not too hard to implement such a system. There are various ways though. It can be implemented as a functionality of the testing tools, or it could be bounded with the help of underlying systems. One good candidate is the virtual machine. Most VMMs, including open source ones, have a similar function. Taking snapshot takes seconds to minutes, depending on the implementations of VMM and the size of the virtual machine. We could take multiple snapshots, and rollback to a certain one, and even make branches. Although making use of VMM might seem easier, it is harder, but necessary, to integrate the snapshot into the testing tools. Taking snapshot should be as simple as setting breakpoints in debug. 

Expecting analytical tools for individuals

Analytical tools for websites are everywhere. However, analytical tools for individuals are rare. Google web history might be one, but has much less functions compared to its analytics.

One may wonder why would we need an analytical tool for individuals. The reason is that tracing one's activities is interesting (as long as it's not privacy related). As it should be helpful to a website, an analytical tool for individual would benefit the user and even provide suggestions on time management. The analytical tool for individual has to be

1. automatic; users won't log each into a document manually.
2. complete; a partial result is of mere use.
3. precise; an imprecise log will lead to fault result.

Of these characters, automatic is the most important. A user is usually not willing to log his/her activities manually. Users are not able to log each activity precisely. However logging is a required step in time management, without which analyzing would be impossible.

It would be difficult to implement such an analytical tool, especially for fine-grained user activities. There are different operating systems, browsers, etc. Besides, there are hundreds of thousands of activities outside computer world.

os.system() and its return value

When os.system('exit 1') returns 256, you might wonder what is going on with python? Why isn't it simply 1 (one)? Let's take a look at this command.

A system() command might be frequently used when you were interacting with the system or other scripts.

Try help(os.system) will only show:
system(command) -> exit_status

Execute the command (a string) in a subshell.

And it talks nothing about the exit status.

This function appears in the tutorial as an example and in library reference:

system(command)

Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to posix.environ, sys.stdin, etc. are not reflected in the environment of the executed command.

On Unix, the return value is the exit status of the process encoded in the format specified for wait(). Note that POSIX does not specify the meaning of the return value of the C system() function, so the return value of the Python function is system-dependent.

On Windows, the return value is that returned by the system shell after running command, given by the Windows environment variable COMSPEC: on command.com systems (Windows 95, 98 and ME) this is always 0; on cmd.exe systems (Windows NT, 2000 and XP) this is the exit status of the command run; on systems using a non-native shell, consult your shell documentation.

Availability: Macintosh, Unix, Windows.

wait(command)

Wait for completion of a child process, and return a tuple containing its pid and exit status indication: a 16-bit number, whose low byte is the signal number that killed the process, and whose high byte is the exit status (if the signal number is zero); the high bit of the low byte is set if a core file was produced. Availability: Macintosh, Unix.

So this explains why os.system('exit 1') returns 256 instead of 1. Actually, the signal number killed the process is zero. To get the exit status, just shift the value 8-bit to the right.

Here is a link talking about the return code of os.system(). http://mail.python.org/pipermail/python-list/2001-March/073147.html

Virtualization and Video Card

Virtualization helps utilize the computer power. (It would consume more power, than a single server, but much less compared to a group of single servers). Virtualization is a cost-effective solution.

Although there are great enhancement of virtualiztion technology, there is not so much in the field of video virtualization. So far, no released virtual machine could provide a modern video card. What they do provide is an emulated slow video card with likely 8mb ram or even worse. It has limited the usability of virtual machine. For example, a game user certainly could not play his/her most favorable 3D game on such a VM. An 8mb ram might not be sufficient to render a quite large LCD. Further more, many CAD software requires a compatible video card.

There are other possibilities. When CPU and main memory are fast enough, the GPU could be replaced. Thus, things will be simpler. Only an average video is necessary.

It would be great for people requiring the flexibility of *nix who can't live without games which only have windows versions. In an ideal world, users would be able to

• switch between VMs as easy as a simple click;
• run any application at native speed;

When Parted ruins it, Testdisk rescues it

Notice that this article contains many useful yet dangerous tools. You should always NOT use them unless you have to and you know what your are doing.

Yesterday, I was going to make a new Linux LVM partition on the hard disk with a Debian system and a Windows installed. I tried parted (GNU Parted) and everything went well until the misleading command

mklabel LABEL-TYPE create a new disklabel (partition table)

which I thought of labeling the partition, but which actually erase your partition table. This command also gives a warning like

Warning: Partition(s) on /dev/sda are being used.
Ignore/Cancel?

and if you ignored it, you would regret.

So the partition table was wiped out. However, the good news was that only the partition table was lost. Parted provided a rescue command which is useless. Let's just skip the boring part of trying 'rescue 0 20GB' a hundred times. After several search queries, I came to gpart (not GParted). It turned out to be of use. With

gpart /dev/sda

I found an NTFS partition and a Linux swap partition. The gpart did not recognize ext3 and lvm2. Then I found the last release dated back to 2001, and it certainly won't support these new formats.

Whenever you are to give up, the savior comes.

Testdisk came in and became handy. Comparatively, it is user-friendly. Testdisk analyzed the disk and gave out a list of partitions it found. I modified some attributes (boot flag etc) and write the partition table back.

It should marked the end of the partition table recovery, but things were a little more complicated than expected.

The partition was not ordered correctly. It is not a problem of testdisk though. Testdisk ordered the partitions by cylinder number, but the partition table on the disk used to be in random order. The reason for that is when I was installing debian and made partitions using partman, the ordered was determined by the operation order. I had to change the partition order back manually. I chose fdisk this time. It never modify the partition table before the w (write) command.

1. First, list the partitions using fdisk and record them somewhere.
   

   fdisk -l /dev/sda         (list by cylinders)
   fdisk -ul /dev/sda         (list by sectors, it will be of use)
   fdisk -u /dev/sda         (using a sector mode)
   

2. Second, delete those partitions not in the expected order with command d
   
   
3. Third, add primary partitions using the expected number. (command n).
   Add an extended partition that will contain all logical partitions.
   Add logical partitions from number 5. Note that you can't choose the partition number for logical partitions.
   
   
4. [Very Important] Fourth, verify that the sectors you entered are EXACTLY as the listed ones.
   Set the boot flag on the boot partition.
   Set partition types (LVM, NTFS, FAT etc.).
   Make sure that everything is correct and you know what you are doing.
   
   
5. Finally, write back and quit with w. [Dangerous]

I verified that the partition table is correct with testdisk.

And re-install grub on MBR. With

grub-install /dev/sda

Now the most exciting part. Reboot...

Everything went and the system was not affected.

Links:

• TestDisk
• Partition-Rescue HOWTO

iocharset, deprecated option but might be of use

Thanks to various encoding systems which makes the world diverse.

First of all, this is extracted from manual from mount on Debian Linux

Mount options for ntfs
  iocharset=name
         Character set to use when returning file  names.   Unlike  VFAT,
         NTFS  suppresses  names  that  contain unconvertible characters.
         Deprecated.

  nls=name
         New name for the option earlier called iocharset.

  utf8   Use UTF-8 for converting file names.

Typically, you don't need the iocharset option when you are using an UTF-8 locale. But when it comes that you have to transfer some ugly-named files to other systems that don't have an UTF-8 support (or when you can't turn it on, like on Windows), the iocharset option turns out useful. What I was doing recently is to transfer files from an old PC to a new laptop. The files are in an NTFS partition, but for some reason, the driver is used as a secondary HDD on a running Debian Linux, which means I can not unplug it or reboot to windows. The new laptop is running windows. To solve this problem is easy. Mount the NTFS partition and transfer files using scp. (Installation of scp clients on the laptop is necessary though). As my windows is using codepage 936 or GBK as default, mount the partition as follows

mount -t ntfs -o iocharset=cp936,umask=022 /dev/hdd1 /mnt/ntfspart

and that is all. And the command (pscp is a scp client from putty)

pscp -ls someone@someaddr:/some/dir

returned well formed results, which will makes me happy.

Note that you might mess up the file names on your NTFS partition when writing data using an incorrect codepage. Please make sure you know what you are doing before striking the enter/return key.