MindRetrieve Blog

From Simple to Complex - A Reflection on Framework Design

2006-05-31T13:31:00.000-07:00

For a while I have used the Jakarta Struts framework in web development. I was quite overwhelmed by the number of configuration files and entries I have to wade through to get anything done. All these configurations are put in for a reason. The offer users a degree for flexibility. These are knobs you can tune to set the system up. With these knobs you no longer need to open the box and rewire things when you want to fine tunes things later.

Alas this makes a steep learning curve. It takes a lot of work to get just some simple thing done. Also when things go wrong, it is very hard to track it down. Unhappy programmers are the result.

It just occur to me that there is a pattern in this kind of issues in framework design. A lot of simple things adds up. When looked at a system scope, it results in something very complicated. The options of struts all serve a purpose and is not hard to understand by itself. But because you need to use all of them in your application, the overall pictures is complicated.

I will use foreign language learning as an example. You learned and memorized many grammars rules: singular v.s. plural, subject v.s. object and so on. You went through the drilling exercises quiet well. Then the challenge came when you need to read or write a full sentence. Your tongue tied, mind gone blank. Why? Because a real life sentence is free formatted. Out of the many grammar rules you've learned perhaps two or three is relevant in the sentence in front of you. The hard part is you don't know which two or three should apply. Although you understand each rule individually, together they become a challenge.

Some lessons I have learned as a user and designer of framework are:

1. Look at things at system level, not just individual feature level.

2. Use common pattern that user have already learned. Clever one-off trick may be an advantage on individual level, but it complicates the system as a whole.

3. Design a framework can do simple work without much training. User can learn gradually when they need to do something complex.

4. Spend time to learn the detail of the system. Like the foreign language analog, sometimes there is no shortcut but to learn really hard. If you are a user who stuck with a complex system you have to use this applies to you. For framework designer, better stuck with point 3 than assume your users' dedication.

5. Finally, provide a tool to manage complexity. If an action have to involve 2 beans, 3 templates, and 5 validation rules, make a tool to help user to track them down. Surprising much of my time in software development is spent simply in tracking these things down manually.

Rudy on Rail's convention over configuration philosophy is a good example. It uses a simple rule consistently (naming notation as the main logic). It can get simple thing to work easily and complex thing possible.

MindRetrieve got Ajax

2006-05-29T23:13:00.000-07:00

Thanks for the Prototype library, MindRetrieve has started to add some Ajax UI, like an auto-completer for entering tags makes it so much easier to from pick existing tags.

I have spent much time struggling with Javascript and the poorly documented DOM before. Relying on trial and error and tribal knowledge to program DHTML is so painful. I'm glad to find these new generation Ajax libraries has brought a big improvement to this situation. They have done things the DOM standard has failed to do, namely to provide a consistent interface to programming dynamic elements across browsers.

Prototype's implementation has a lot of neat things. The $() method as a shorthand to document.getElementById() is just brilliant.

Took a RSS break

2006-05-23T09:10:00.000-07:00

I have took a 2 weeks break in reading the technology blogs and industrial news. Now each of the 20 feeds I've subscribed has at least 50 unread items. Not knowing the 20 new products that has since launched, I feel that I am out of touch with the industry.

It is interesting how much a RSS feed can change a person. As long as I am keeping up the news I feel like I am an active player. But the world moves so fast that if you standstill for a little while you will have a lot of catch up to do.

berlios outage

2006-04-27T08:26:00.000-07:00

berlios, the hosting service of this project, is suffering from another outage. This time I'm counting 16 hours since yestarday afternoon. Over the course I've seen many small outages that last minutes or a few hours. I should get very serious about finding alternative provider, especially now that sourceforge offer subversion finally.

For now, I hope berlios will get back to life soon.

Fixing TortoiseCVS's shattering glass sound

2006-04-10T16:08:00.000-07:00

TortoiseCVS plays an irritating sound when it encounters an error. It is downright embarrassing when you are sitting in a cafe and from your laptop comes this
shattering glass noise.

What's annoying is I couldn't find a way to change it or turn it off. I tried the usual places and searched the web. Still I couldn't get the noise to leave me alone. Finally I found the right place to do it. Go to Windows - Control Panel - Sounds and Audio Devices - Sounds Tab. Change the TortoiseCVS Error sound to something more gentle.

Upgrade notification with Atom

2006-04-04T21:38:00.000-07:00

One of the biggest advantage of building a web application over a desktop one is the ease of upgrade and bug fixes. For MindRetrieve I've been looking for a mechanism to notify user for upgrade. At the same time I have to keep the mechanism non-intrusive and respect the user's privacy.

After looking at different mechanisms like CGI and Javascript on browser, it just dawn on me than upgrade notification has a lot in common with news syndication. All I have to do is to publish a news feed and then put a mini-newsreader into the application. Of course the user can always opt out or even choose to fetch it using their own newsreader.

It is a good day to find a satisfactory solution to a nuisance issue.

xfolk - Microformats

2006-03-27T23:41:00.000-08:00

I was looking for a bookmark sharing standard. Sure enough there is a microformat xFolk proposed.

xFolk is a simple and open format for publishing collections of bookmarks. It better enables services for improving user experience and sharing data in web-based bookmarking software. xFolk may be embedded in (X)HTML, Atom, RSS, and arbitrary XML. It is one of several open microformat standards.

Microformat is simple conventions for embedding semantics in HTML to enable decentralized development. It remixing of websites possible, as shown in the wonderful Live clipboard demo.

MindRetrieve postcard

2006-03-24T08:22:00.000-08:00

This is a postcard I've made as a promotion material. I find it a more creative compare to handing out name card. You can print it on photo paper at home. Or you can send it to a photo lab to print a big stack. Either case it probably turns out quicker and cheaper than name cards. The color is great. The 4x6 inch space afford a great deal of creativity.

The amateurish graphics design is mine. You think I got a lot of nerve to actually hand this out to people. At least I can claim some novelty :)

Announce: interactive javascript console

2006-03-22T07:57:00.000-08:00

I'm rather productive in cranking out little tools lately. Here is another one.

js_console is an javascript console you can insert into you web page. It allows you to test javascript statements interactively in the context of your web page. Try a demo and download from

http://tungwaiyip.info/software/js_console.html

Google Desktop Search cannot find files moved

2006-03-08T13:00:00.000-08:00

The story about Google Desktop Search cannot find files moved stir up some nosie in the blogosphere. Admittedly it is a hard problem. And it would require good platform support to do right. It is the same dilemma I have faced when implementing the tag the files feature. I choose to leave this problem alone for now (while leaving enough nexus so that it can be fixed later). This allow me to put out a simple features in days. If it is proven popular I can alway go back and spend time to resolve it.

Announce: HTMLTestRunner - generates HTML test report for unittest

2006-02-13T09:12:00.000-08:00

I'd like to annouce the release of Python library HTMLTestRunner, a by-product of the MindRetrieve project.

------------------------------------------------------------------------
HTMLTestRunner is an extension to the Python standard library's unittest
module. It generates easy to use HTML test reports. See a sample report at
http://tungwaiyip.info/software/sample_test_report.html.

Check more information and download from
http://tungwaiyip.info/software/#htmltestrunner

------------------------------------------------------------------------

Actually this was posted on comp.lang.python newsgroup a while ago. Thank you Cameron Laird for picking it up in Dr. Dobb's Python-URL!

Great gathering last night

2006-01-20T12:29:00.000-08:00

Thanks to Scott of Ookles for organizing the great gathering in San Francisco last night. It was a great pleasure to meet so many prominent people who are pushing the Internet front.

Geeks are officially cool again!

Also I got to thank Scott for being the inspiration behind the new tag the file feature!

Announce release of version 0.8.0

2006-01-19T12:01:00.000-08:00

I'm gload to announce the first major release in almost a year. New features in version 0.8.0 include:

Web library - tag base bookmarking system
Tag based categorization
Tag files in local disk (Windows XP)

Download from via the MindRetrieve website.

Added IE support

2006-01-18T09:22:00.000-08:00

After months of developing on Firefox and Opera only, I have finally ported the DHTML code to IE. Well almost except some small bugs. IE6 is just hideous. I should really have wait to support IE7 only.

Efficient character escapes decoding

2006-01-13T18:32:00.000-08:00

I ran into this technical issue using Python's 'unicode_escape' codecs on unicode string. Put a question in the comp.lang.python newsgroup and thought this is probably too technical for anyone to care. But the Python community never fails me. Thank you for Steven Bethard to come up with a great suggestion that works for me. I have written up the solution in a Python cookbook recipe.

Hope it will be useful for other people too.

Synchronization with Simple Sharing Extensions

2006-01-12T12:15:00.000-08:00

I have come across this Simple Sharing Extensions for RSS and OPML (SSE) specification from Microsoft. It is a minimum extensions necessary to enable loosely-cooperating apps to use RSS as the basis for item sharing – that is, the bi-directional, asynchronous replication of new and changed items amongst two or more cross-subscribed feeds.

This seems to be a great fit for synchronizing the weblib among different repositories. The essence of SSE is a set of items, each has a globally unique id, a timestamp and an increasing version number. This has inspired me to make another revamp to the Weblib file specification to add all those elements. The only remaining issue for me is the globally unique id. I need to figure out a simple way to generate them across distributed systems, (I feel UUID too heavy weight). The implementation is available in SVN, although I haven't updated the specification yet.

Synchronization is a though issue I haven't have it all thought through. But it probably can't go wrong to go with Ray Ozzie who has created Lotus Notes.

Quick key and the Petname system

2006-01-11T08:16:00.000-08:00

I have come across an interesting article An Introduction to Petname Systems. It echoes with a quick key feature I am building, which associates a short phrase with frequently used URL. For example phrase 'amex' can be used as the key to the 'American Express' web site. The main goal of the article is in security (against phishing). While for MindRetrieve the goal is to make a slick user interface. Nevertheless it helps me to understand what I am doing, an implicit system for user to assign petname to items!

Text v.s. multimedia?

2006-01-03T21:02:00.000-08:00

The heat is on once again in the advent of internet technologies. The word is that we will move beyond text based communication to audios and videos, from text blogging to audio podcasting to video blogging, from SMS evolves MMS, from search for text to search for audios and videos, etc. The humble text is only the first step in technology development. Eventually technologies would be advanced enough to enable the full glory of multimedia.

I feel rather lukewarm for this. I often past up the news video for text article as I grow impatient with video's pace. With text I can scan back and fro much more easily. User generated content? That reminds me of those home videos, where the camera never stop panning left and right. And when the subject talks, he is actually off-screen ;)

Multimedia was the buzz world of last generation. With the enabling technology CD-ROM become popular, people envision our PC would be filled with sight and sound. While we now see lot more graphics on our PC than the early days, enough to say the focus is still on text, whether we are using email or the web. Anyway none of the multimedia companies has become Google.

I think people consider who video and audio are more advance than text are only looking at it from computer engineering's perspective. This really does not do justice to text. From a different perspective I would say audio and video more primitive because they are base on our biological sensory perception. Whether as text is truly the great innovation than revolutionized communication.

Incremental Development

2005-12-30T08:04:00.000-08:00

I was looking at the subversion checkin log at

http://svn.berlios.de/wsvn/mindretrieve/trunk/?op=log&rev=0&sc=1&isdir=1

I notice I have made many many small checkins. Often several times a day and each checkin include a group of several files (taking into consideration I am not working on this full time). This style is quite different from my work on other projects when I do a lot less checkin but usually in a larger chunk.

Perhaps this say something about the productivity? Perhaps I was acting thoughtless because I'm the only developer right now. But just now I have come to another characterization - this is incremental development!

Each time I made small changes, add some new feature or a methods, refactor code, fix a bug, add a test. I made the code changes, test it, and then I check in. The code base is functional most of time. Seldom did I make big changes that break the code base for several days or more.

Is incremental development the best development process? I'll leave it to other discussion. But from a developer's perspective, having a functional system most of time and being able to test and verify any code change easily is wonderful. Everytime I do a checkin I have the satisfaction that something is done. Coming from an environment where changing only one line of code would lead to tedious work of building a test environment and a painful testing process, this is just pure joy.

Keep Your Article in One Single Web Page

2005-12-28T17:08:00.000-08:00

There is a style widely used in web publishing. It says people don't like to read long article, one should keep modem user in consideration that a long web page would result in slow download and so. If you have a long article to publish, break it down into several short sections and let user read it page by page.

I have arrived in a contrarian view. I think breaking a long article down into several pages is a hassle to the users. It is best to publish the entire article in one single page. The issue is in order to finish the article I need to click next page several times, every time there is a delay and it interrupts the momentum. Usually the delay is short, like one or two seconds. But it is a noticeable delay. Scrolling is seamless in comparison. For slow sites the delay is much worst. Some sites routinely take 10 seconds or more. That would feel like an episode ended in a cliffhanger and we have to wait for the next episode.

Is several short page a better layout than a long one? I actually prefer to have a long one. The scroll bar give a good indication of how far in the article I have progressed. The scroll wheel is very handy in navigation. I can also use the browser to search for a word that appears anywhere in the article. All these is better than arbitrary break an article into several parts and then leave only a narrow window to the user. The speed issue is a non-issue. Any browser should be able to render incrementally so that you can start reading as soon as any text arrives. Even my cellphone do this flawlessly.

In some case the multiple pages format backfire badly. I'm glad that Yahoo has a mobile version made for cellphone at http://mobile.yahoo.com/. I thought it would be great for reading email. Turns out the issue with cellphone data network is not just low bandwidth, every time when I click a link there would be a long delay before I can get any response. As Yahoo mobile break every screen into 15 lines or so, even the simplest email cost me multiple clicks to read though. The long delay between clicks make it plain unusable. I end up went back to the regular web interface. Even I have to scroll through tons of irrelevant stuff to get to the email body I still prefer it to the mobile version.

Of course my comment should not be generalized too far. I have tried to load an entire technical manual as a single web page (in my harddrive). The browser have noticeably delay to process a web page of this size. But I still keep the manual in this format for the ease of searching.

ISO 8601, the metric of date format

2005-12-16T08:53:00.000-08:00

Different countries have different convention of writing date. Some write in m/d/y order and some write in d/m/y order and so on. One thing to do for localization is to show date in the right format...

I say ____ the convention. Write it in ISO 8601 format. That is in the YYYY-MM-DD format. No more confusion of whether the day or month come first. And when you sort it, it comes out in chronological order. ISO 8601 is the metric of date format.

Your Site Has Vulnerability

2005-12-10T12:05:00.000-08:00

I was testing my web application for security problem. Failure to escape user input is a very common class of security problem. So I created an input string like this:

'"></script><h1><font size=7 color=red>GOTCHA<iframe src=http://mindretrieve.blogspot.com/2005/12/your-site-has-vulnerability.html width=500 height=300>
Cut and paste it into any input fields and then click submit. If you see something you don't expect, that site probably has a problem.

Hurry! Test out your site before hackers do!

17000 lines of code

2005-12-08T08:01:00.000-08:00

It has been a while since I did a tally on the code size. The new statistics shows there are about 17000 lines of code in about 100 modules. That is for a non-trivial application with everything deliberately designed to be as simple as possible.

A closer look inside, among the them is 5000 lines of unit test code. It on track with the the rule of thumb of 1 to 1 ratio of production code to test code.

Two of the largest module has around 700 lines of code. The median is around 100 lines. This sounds minuscule to most other software code. But with Python, 700 lines can be a really sophisticated module. In many cases I would break a module down into smaller components before they even reach 700 lines.

Google v.s. bookmark?

2005-12-07T18:02:00.000-08:00

While we are working hard on MindRetrieve to improve bookmarking, some people argue that the entire idea of bookmarking is outdated. Instead of bookmark, we just google. Indeed Google gives such excellent experience that people expect the right answer instantaneously. More easy than you would looking things up from the bookmark menu.

While this is true for some obvious sites, perhaps for 'dell' or 'walmart', it ain't necessary useful for everything you'd ever interested. How many times have you flip through pages of marginally useful search result until you finally found the one that gives what you need? (assuming, of course, you an advance searcher who actually go beyond the first page). You will not want to repeat this search. And if you do, you won't necessary found the same item. How about a page that is not a direct search result but is one click, or two clicks away? Let's say from Google you found a passionated travelogue about an Italy trip. From there you found more links about some hotel recommendation that you really want. Google is a good starting point. But you have also make additional effort to arrive in what you needed.

When you search and evaluate the result, you are actually creating something of value. If you put together the link of the travelogue, a guide on how to use the Italy train system and the two cute hotel you have seen, you will in effect created some kind of travel guide, even if you have not authored any of them. Personal web is a way for you to capture the creation.

Weblib file specification

2005-12-01T08:34:00.000-08:00

The work on MindRetrieve has reached another mile stone as I have published the weblib file specification and that it supports updating now. Previously I was using a hack to rewrite the entire file whenever I update a tiny little piece of data. That actually served me well for several months during development and personal use. Rewriting a 200k file everytime never seems to add any drag. In anycase the newer format seems well designed and is more ready for future scalability.

I've included a snapshot of the spec here or you can find the latest version from the
source code.


MindRetrieve Weblib Data File Specification Version 0.5

MindRetrieve weblib data is an UTF-8 encoded text file (no other
encoding is supported as this time). The overall format is a block of
headers followed by a blank line and then the body similar to email and
HTTP messages. Each line of the body part represents a webpage or tag
item. Update to the weblib is appended as change records to the end of
the file. The entire weblib can be represented by a single file.

file            = headers BR body
headers         = *(header BR)
header          = field-name ":" [ field-value ]
field-name      = token
field-value     = DSV encoded value
body            = column-header BR *((data-line | comment-line | *SP) BR)
column-header   = column-name *( "|" column-name)
column-name     = token
comment-line    = "#" any string
data-line       = [change-prefix] (data-record | header)
change-prefix   = '[' YYYY-MM-DD SP HH:MM:SS ']' SP ['r' | 'u' | 'h'] '!'
data-record     = ["@"] id *( "|" field-value)
BR              = CR | LF | CR LF
SP              = space characters


Note

* token is defined according to RFC 2616 Section 2.2.

* DSV encoded value is an unicode string with the characters "\", "|",
  CR and LF encoded as "\\", "\|", "\r" and "\n" respectively.

* There are two kind of data records, a webpage has a numeric id, while
  a tag has a numeric id prefixed by "@".

//* A record with the same id can appears multiple times in the data file.
//  The last record overwritten preceding records.

* A data-record preceded by a change-prefix denote update to the file.

* A record prefixed by "[ISO8601 time] r!" is a remove record. The item
  with the corresponding id is to be removed.

* A record prefixed by "[ISO8601 time] u!" is an update record. The item
  with the corresponding id is to be replaced.

* A record with by "[ISO8601 time] h!" is an header update record. The
  header value is to be updated. There is no remove header record. A
  header value can be set to empty string however.

* The last line should always ended with BR. If the last line is not
  terminated with BR it is considered a corrupted record and must be
  discarded. Moreover never append change record to a corrupted record
  because the line break would be misplaced.

* The encoding header is defined for future extension only. Only UTF-8
  encoding is supported right now.

How do I find out this web page?

2005-11-23T07:42:00.000-08:00

One of the puzzle I try to solve with this project is to find the trail that leads me to certain web page. Often time when I am researching on the web I find some interesting page but do not remember how I get there. Maybe it is a fruitful search result with certain search terms and I want to explore more. Or it maybe it is point out by some knowledgeable blogger and I want to know the him better and what more he has to offer.

One idea is to have a "always present back button". MindRetrieve keep enough information that you can find the source page of any page most of time. I probably have to experiment with various UI before I know how effective it can be.

I am not alone in having this problem. Marc Hedlund calls for a tool to find the tab history in his blog. This New York times article look at a similar issue in a larger context. Modern people are heavily multitasked. But once we are interrupted it often takes a lot of work to reestablish the context when we return to the original task.

HTML 3-state checkbox

2005-11-15T08:06:00.000-08:00

I have been experimenting with a 3-state checkbox GUI. When we edit the tags of multiple items, there would be some tags that is only used in some items. Just and on and off setting is not enough. We also need a "grey" state to mean leave the tag untouched rather than turning everything on or off.

Since HTML form does not support 3-state checkbox, I have experimented with various simulations using Javascript and CSS. While the widget work resaonble well, I find the "grey" state rather unintuitive. It is obvious what on or off will do. You have to stop to think what grey's action would be.

So I have decided against using 3-state checkbox after all and stick to simple widgets. I think the user in this case would have a strong intent to make properties common among all items. If they leave the checkbox untouched, the tag setting will be unchanged. Otherwise it will be on or off for all.

I have posted one version of the working HTML code here in case if there is any interest. It is tested in Firefox and Opera. Need some simple change to make it work in IE.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3c.org/TR/html4/loose.dtd">
<html>
<head>
<style>
  #tagControl {
    line-height:200%;
  }

  #tagControl span {
      padding-left: 0.3em;
      padding-right: 0.5em;
      padding-top: 0.3ex;
      padding-bottom: 0.3ex;
      border: thin outset #777;
      font-weight: bold;
      color: #000;
  }

  #tagControl .tagSelect {
  }

  #tagControl .tagUnchange {
      color: #777;
  }

  #tagControl .tagDelete {
      color: #777;
      text-decoration: line-through;
  }

  #tagControl .tagSelectPressed {
      color: #fff;
      background-color: #777;
      border: thin inset #707;
  }

  #tagControl .tagUnchangePressed {
      color: #fff;
      background-color: #777;
      border: thin inset #707;
  }

  #tagControl .tagDeletePressed {
      color: #fff;
      background-color: #777;
      border: thin inset #707;
  }
</style>

<script>

function toggleTagSelect(e) {
    p = this.parentNode
    var cb = this;
    if (p.className == 'tagSelect') {
        p.className = 'tagDelete';
        cb.checked = false;
    }
    else if (p.className == 'tagDelete') {
        p.className = 'tagUnchange';
        cb.checked = true;
    }
    else if (p.className == 'tagUnchange') {
        p.className = 'tagSelect';
        cb.checked = true;
    }
}

function onload(e) {
    // attach event handlers
    var tc = document.getElementById('tagControl');
    var elems = tc.getElementsByTagName('input');
    for (var i=0; i < elems.length; i++) {
        var elem = elems[i];
        elem.addEventListener('click', toggleTagSelect, false);
    }
}

window.addEventListener('load', onload, false);
</script>
</head>

<body>
<table>
<tr>
    <td>Tags</td>
    <td>
    <div id='tagControl'>
    <span  class='tagSelect' node='rep:tag'>
        <input type='checkbox' />
        <a node='rep:tag'>
        Gas
        </a>
    </span>
     
    <span  class='tagUnchange' node='rep:tag'>
        <input type='checkbox' />
        <a node='rep:tag'>
        Food
        </a>
    </span>
     
    <span  class='tagDelete' node='rep:tag'>
        <input type='checkbox' />
        <a node='rep:tag'>
        Lodging
        </a>
    </span>
     
    </div>
    </td>
</tr>
</table>

</body>
</html>

Progress on categorization

2005-11-04T13:20:00.000-08:00

I have make a lot of progress designing the categorization for the tags. Once things are ready I plan to put an online version for people to get a feel.

This time I want to show the visualization of the category tree. The most common way to show it is the explorer style tree view. With tree view you can represent categories any level deep. User can navigate by expanding and collapsing the branches. I find the expanding and collapsing fairly cumbersome. So I put a twist to that idea by expanding all categories by default. I use different font size and color to represent different level of categories. It works quite well that all items are presented on a single webpage and you can do an incremental search (with Opera or Firefox) without going to the web server again.

Right now it can show reasonable well up to three levels. While I don't have categorization any deeper I would like to find solutions for deeper categorizations. Either a specific way to represent fourth or fifth level items or a scheme that can work at arbitrary level deep.

spywareaid says MindRetrieve.exe is Legitimate

2005-10-10T21:10:00.000-07:00

MindRetrieve.exe is a legitimate application according to spywareaid.

Thanks for some of you, who are among the hundreds of people who have downloaded MindRetrieve, for updating spywareaid about this.

Actually not only is it not a spyware, MindRetrieve is committed to respect user's privacy. The way we ensure this is by being open source. Anyone can audit the software to see what it is really doing, and should there be anything objectionable, to be able to disabled them or even to fork a new branch.

A life time archive?

2005-10-06T15:50:00.000-07:00

One thing that MindRetrieve does differently is there is no limit on how many webpages it archives. That may surprise some people intially as a disk hungry application. But after one year I collected about 120MB of archive. Project this number for 10 years would result in mere 1.2GB. You can buy a flashdrive today and have enough capacity for the next 10 years. There is rising awareness that technology has advanced to a stage that we can record everything we ever read or done, documents, emails, digital video, phone calls, etc, and keep an archive of you entire lifetime. MindRetrieve is designed with this idea in mind.

That said I find MindRetrieve current works best finding webpages you have just recently viewed, perhaps on the same day. Like things that are a few clicks away from a google result but you forgot how you get there. What about thing you've seen 6 months ago? Initially I think MindRetrieve would work like google of your mind. You enter a keyword and the desired result shows up on top. Actual experience is a lot less effective. One issue is you've simply used the wrong keyword. Memory fades. You might try to search for the article about the Vietnamese restaurant you went six months ago in vain. Turns out you didn't go to a Vietnamese restaurant six months ago. It was really an Indian restaurant you are looking for.

Beyond that there is a deep user interface problem for us to solve. Having a TiVo of you life is one thing. Being able to rewind over hundreds of thousands of hours of recording and find the exact one moment is a different thing. Full text search is helpful. But that alone is not the solution.

So I am still adding to my 125MB of web archive. And I believe sooner or later we'll find some interesting ways to explore the memory land.

Blog moved to blogger.com!

2005-10-06T08:18:00.000-07:00

Mindretrieve blog has moved to a new location at http://mindretrieve.blogspot.com/!

I have been lazy to repurpose berlios' discussion forum as the MindRetrieve blog. This works out to some extend. But its UI is a bit clumsy and there is no RSS support.

I finally give up and setup an account with blogger instead. I have manually imported the past posting there. From now on please visit the new location!

P.S. The spammer there is fierce! Even before I finish importing past postings they are already adding spam there! Got to update the posting policy.

HTTrack Website Copier - Offline Browser

2005-09-02T08:08:00.000-07:00

Think you have a novel idea in software? Chances are there are many other people working on the same thing. I just found one that have significant overlapping. Description from its web site:

----------------------------------------------
HTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility.

It allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure. Simply open a page of the "mirrored" website in your browser, and you can browse the site from link to link, as if you were viewing it online. HTTrack can also update an existing mirrored site, and resume interrupted downloads. HTTrack is fully configurable, and has an integrated help system.

WinHTTrack is the Windows 9x/NT/2000/XP release of HTTrack, and WebHTTrack the Linux/Unix/BSD release. See the download page.

http://www.httrack.com/

When will MindRetrieve move out of alpha?

2005-08-15T08:07:00.000-07:00

If Google introduce the notion of putting out software in perpetual beta state, MindRetrieve maybe push this further for being in perpetual alpha state. This wasn't a statement that the quality of software is subpar or buggy. Since MindRetrieve sit between me and the world wide web every day, I would be the first one to be upset if my web experience suffer in any way. It is in alpha because I see this as a research project. The idea of personal web and ways to organize it is fairly novel. I enjoy toying it in this project.

That say, once the web library feature is complete I feel it would reach a certain mile stone. By then MindRetrieve should be mature enough to be in beta.

Wikipedia:Categorization

2005-08-05T08:07:00.000-07:00

Here is an interesting guideline from wikipedia on categorization.

http://en.wikipedia.org/wiki/Wikipedia:Categorization

Notice categories do not form a tree but a more general directed acyclic graph.

Status update

2005-08-05T08:06:00.000-07:00

Development has been slow since I came back from PyCon. During that time I have moved to a new day job. While I am very glad to shave a large part off my monstrous two-hour one way commute, the unintented consequence is I also lost a lot of development time I had on the commuter train. You never how productive a long train commute can be.

In anycase now I have craved out a new time to move things forward, often during marginal hours at coffee shops. A lot of good things in managing personal web library are coming out already.

Also looking at SVN I notice I have spent weeks hacking perhaps no more than a thousand line of code over and over again. Despite the code size some very interesting functionarity has slowly come into shape. It must some unimpressive if you measure productivity by number of lines of code written. But I find this a very interesting experience as I have high regard of simplicity.

using del.icio.us

2005-04-13T08:05:00.000-07:00

The idea of del.icio.us is rather simple. But its utility is sometimes very subtle. For example the URL below shows the people who have bookmarked MindRetrieve:

http://del.icio.us/url/532b38dcd4526f40faef56f8c94896e9

21 people has bookmarked this project. As a popularity contest this is a respectable number. More interestingly is the common tags box on the right. This show what people think this project is about.

------------------
common tags
13 search
12 python
7 web
4 lucene
3 software
3 tool
2 firefox
2 desktop
2 useful
2 pylucene
2 apps
2 browser
------------------

13 people tagged this as search. Another 12 tagged Python, the programming language used in this project. 4 people mention lucene, the open source search engine that powers MindRetrieve.

Python is not exactly what I think should characterize MindRetrieve but merely an implementation detail. Since I have made an demo in PyCon, the Python community has help to spread the word about this project. del.icio.us' user build classification system just reflected this zeitgeist.

Congratulations to Joshua Schachter of del.icio.us. Recently he announced that he has received funding to start working on the project full time. Money aside, he has show how an individual innovator can make significant achievement.

http://lists.del.icio.us/pipermail/discuss/2005-April/002801.html

Help make the mac version happens!

2005-03-28T07:58:00.000-08:00

My mac mini has finally arrived :-) I have gotten myself oriented. Fooled around and eventually found out how to launch the terminal. Then got some taste of downloading mac applications. Now I am ready to port MindRetrieve to Mac!

I am still totally new to Mac application development. Would you like to contribute development effort to make this happen? Or point me to literatures I should know about? In particular I like to know how an application is supposed to be packaged (it is written in Python). Also how to setup a daemon application on Mac (auto startup when the system starts, stop when the system shutdown).

Email me at tungwaiyip at users.berlios.de. Thanks a bunch!

Lightning talks in PyCon

2005-03-28T07:54:00.000-08:00

I have made one of the first public demo of MindRerieve at PyCon [http://www.pycon.org/dc2005/]. This informal 5 minutes demo got some friendly reception.

PyCon was a great gathering for Python developers. Read more about my experience at my blog [http://tungwaiyip.info/blog/2005/03/25/].

browser statistics

2005-03-04T07:51:00.000-08:00

Below is some statistics of what people use to visit the MindRetrieve home page from a snapshot of server log. Everyone seems to be using updated version of browser. Firefox is used by more than half of the visitors. Note that at present MindRetrieve has only released to a very small circle. Visitors are likely to be Java programmers are those who know something about Lucene.

----------------------------------------------------------
Breakdown by browser

Mozilla 72 (all seems to be recent Gecko or Firefox)
Win 60
Linux 11
mac 1

Mozilla 4.0 5

IE 38
5.0 2
5.2 mac 1
6.0 35

Opera 8

Safari 7

Konqueror 2

OmniWeb 1

133

----------------------------------------------------------
Breakdown by OS

Win 105 (all win NT except a few win 98)
Linux 13
Mac 10
Unknown 5

133
----------------------------------------------------------

organizing bookmarks thoughts

2005-03-03T07:49:00.000-08:00

First we deride the folder based hierarchical taxonomy as too strict and unrealistic data organization. Then we have the liberal tags based deli.cio.us style organizing.

Thinking more, we have another familiar way of organization, that is the hyperlinked web. The web may be a superior mean than the above two methods.

But then it is easy to get lost just wandering in the web. There is where the hierarchical map comes in. It gives you a starting point and known paths to the destinations....

How to save a web pages?

2005-02-23T08:08:00.000-08:00

Design A
--------
Save the web page and all dependent files as one unit

Pros
- Easy to implement. Easy deletion.
- Good performance. (No extra lookup, all data in one place)

Cons
- Duplicated .jpg and .gif, etc are saved for each web page.

Design B
--------
Build a URL (or MD5) to resources database.

Pros
- Web pages referencing same resources only have 1 copy saved. Experiments show that in compressed form, the original document takes only 1/5 to 1/10 of storage. The rest are potentially duplicated. (Note dependent resources like .jpg are often not well compressed).

Cons
- Delete requires reference counting before removing shared resources.
- Opening a saved web page requires multiple lookups.
- Saving in storage only happens incrementally. I.e. the first web pages saved take same amount of storage in either design. Saving only happends when adding subsequent web pages from the same site (referencing similar resources)

Design C
--------
Save resources from the same DNS domain in one place. Items under one domain are more managable.

Compromise? Best of both world or complexity of both?

Caveat: one web page can have resources coming from several domains.

Why don't we all have 10,000 bookmarks?

2005-02-23T08:02:00.000-08:00

Is that because there is not so much worthy materials from the web? At least not for me. If there are tools to support it I am confident that I would a bookmark of 10,000 items within 10 years.

One obvious issue is that web browser's bookmark UI is not scalable. The primary UI is a pull down menu with items organized under hierarchical folders. Access become dramatically more difficult when a menu is longer than a screenful or when items are organized in a second level folder or below. I painstakingly organized my bookmark list to keep frequently used items accessible and avoid having some went out of the radar screen. It was never fully successful. It also mean I bookmark a lot less than I want to.

A search based access is an improvement. Organize by a flat keyword list (tags a la deli.cio.us) also improve upon the strict hierarchical taxonomy.

When you really have 10,000 bookmarks, another issue comes up. People don't seems to have the ability to pay attention to such a large amount of items. I purge bookmark regularly after I think I got a good grip on the information and that it cannot provide me extra value. This is usually a mistake. I seldom want to purge something completely from my consciousness. What I was trying to do is to keep the bookmark list tidy to prepare for new information. The tools force me to erase the memory completely.

An alternative is keep recent items handy while pushing old items into an archive. Gmail introduced such organization technique. It recommends you to use 'archive' instead of 'delete' so to get an item out of sight but still accessible via search in the long term archive. This process is very promising but unconventional. How much success does Google has?

Practical File System Design

2005-02-21T07:49:00.000-08:00

Part of the design is reminiscience of a file system. It store's web pages instead of files. And there are attributes like URL and title that is indexed and searchable. I just found Dominic Giampaolo's book "Practical File System Design with the Be File System" available online (http://www.nobius.org/~dbg/practical-file-system-design.pdf). This seems to be a good study on this subject.

A Comparison of Hyperstructures

2005-02-19T07:49:00.000-08:00

A Comparison of Hyperstructures: Zzstructures, mSpaces, and Polyarchies

McGuffin, M. J. and schraefel, m. c. (2004) A Comparison of Hyperstructures: Zzstructures, mSpaces, and Polyarchies. In Proceedings of ACM Conference on Hypertext and Hypermedia, 2004 (in press), pages pp. 153-162, Santa Cruz, California, USA.
--------------------------------------------------------

Just found this paper at http://eprints.ecs.soton.ac.uk/9230/. The descriptions of Hyperstructures has enlighten me on a lot of unstructure thought on how to organize & structure bookmarks. The deli.cio.us way to eschew hierarchical structure in favor of associated keyword is a simple and effective approach. However the paper says

"A disadvantage of using this approach alone is that users cannot exploit their spatial memory and learn where things are located, or remember paths of links to find them"

It also helps me to understand what semantic web is all about. Great work.

article gone? MindRetrieve save the day

2005-02-02T07:48:00.000-08:00

I have bookmarked an article "INNOVATION, INFORMATION TECHNOLOGY. AND THE CULTURE OF FREEDOM" by Manuel Castells for later reading at

http://www.softwarelivre.org/news/3636

When I go back today somehow the original article is pulled and replaced with another Spanish piece. The site is in Portuguese and there is no indication of any language button. I begin to question my memory if there was ever an English article there.

A search in MindRetrieve quickly dispel my doubt. In the cache is the English article I've seen.

Anyway it is largely things about open source I already know. Nothing particularly insightful there.

TortoiseSVN + Total Commander

2005-02-02T07:47:00.000-08:00

I've just setup the TortoiseSVN client with the fresh release of Total Commander 6.5. The result is a wonderful integration of SVN client with my favourite file manager. A little icon next to the file indicates whether the file is modified or not. Diff, commit and revert commands are readily available in the context menu. Even through the SVN repository is half way around the globe and I am working off DSL, the speed still blow away the ClearCase that I use in corporate environment. Also SVN works great offline and diff works without the network!

The only thing I haven't figure out is how to store the session for the svn+ssh protocol. Right now it asks for password for every remote operation, sometimes even twice.

thought on search the search?

2005-01-27T07:46:00.000-08:00

2005-Jan-27 02:59

The basic thing to do is to separate search engine result page from other pages. If users is searching for a topic chances are they have look for it in search engine before. It is distracting to see a lot of google result pages among the result.

More than this is to tackle some frustrations a search often encounter:

* beating around the bush, keep search the same item and revisiting same result. Can it help to point out dead end earlier?

* How do I get here? You're looking at a promising result. If it can backtrack to the search result page (or some other directory page) it may lead to more useful lead.

Would a software helps in these cases? How to present an intuitive and non-intrusive UI? That's something to experiment with in the next few months.