Chandler Notes. Ever since I joined Lotus and played with Lotus Agenda I have been interested in what Mitch Kapor is up to. It's really painful to see Chandler being developed out in the open as I want to use it today! I'm really hoping that it will be as powerful if not more powerful than Agenda on it's first release. Then anticipation is killing me!
I'm doing some work now with a set top box but before I can use it I need to prepare a machine which has a TV "video in" port to view it on. Luckily I have a machine at hand. I reinstalled XP from scratch on it as I prefer the English version over the Japanese. If you use the English version you can add Japanese reading and writing capabilities to it easily but if you install the Japanese version unless you are multinational company with clout you can't even purchase the English language pack to add English menus to it. sigh.
After installing I thought I would install the latest Service Packs and upgrades before I use it for anything critical. There are critical 34 upgrades that I need which makes me wonder why Microsoft dosen't just install over the network once they have established that you are connected to the net. I feel like I have reinstalled the everything several times now after applying all these patches.
Today we are debugging a problem with our search code. Everything works fine (thanks Victor) except Zenkaku Romaji. It appears that the search is creating case insenstive SQL by using Java's toLower() on the query string and then using LOWER on the data column in SQL to complete the search query. This has one broken assumption that Java's toLower() and, in the current case in question, Postgres's LOWER produce the same result. They don't! The answer it seems is to use ILIKE (Japanese Link).
This is easier said than done! Also I'm concerned about whether other databases support ILIKE or not.
Later: my trusty (Japanese) coworker after thinking about it mentions that if we use the database's LOWER to lowercase both the query and data column then we will solve the problem. Doh!
My Ant build was failing yesterday with the code: RESULT=137 sortly after something was Killed. After a certain amount of frantic Googling with no results I decided to look at the processes which are running on the server. There were about 20 Javas there and one defunct one so I decided to kill them all off. That seemed to do the tricked. I think Ant was killing the compile tasj that was taking too long as it was getting blocked some how by the extraneous Java processes. Had me worried there for a second. It's interesting though that I immediately assumed a third party had changed something when the daily build broke. My new mantra is: when Linux goes bad on you, check the ps and check lsof before you attempt anything wackier.
On whim I was searching to see if anyone had made a truetype font which contains KDDI Au mobile phone's emoji picture characters. I get some mails forwarded to my PC and if they contain Au emoji picture characters they all look like dots!
Tom of Nooper suggested Enfour but they only seem to have an iMode emoji font.
While I was searching I came across this nice iMode introduction from IBM
Yesterday we had a question from a customer regarding the behaviour of one our Servlets. In the end it seemed that it was loosing the session information when they went back in the browser. I can understand why, as they actually cross domains doing in going back. (Don't ask why their store is spread over two domains). When customers cross to the second domain they are not sending a cookie, the servlet puts new session information in the urls and the session is getting restarted.
From the support point of view it also seems that the problem is only related to IE 6. (In fact it is only related to non cookie users). In the end it turns out that by default IE 6 rejects cookies from sites that don't have XML P3P privacy information on their server. When I installed IE 6 I turned this feature off straight away so I never saw the problem. Turning IE6 back to it's default settings makes the session problem manifest itself.
Fair enough. There are few ways around this. One is to ask everyone to set the non default cookie setting. Since most of the users use the default settings we can't do this as some people are not going to get it and this will cause problems for the store's support people. Secondly we can submit and install the privacy information on the server. This is a good thing but a pain as the store has to decide it's privacy policy which is going to take some time. Do we just say, "Sorry you need to install P3P policies on your server" or do we do this work under our maintenance agreement. After all, since the problem occurs with our software it must be our fault. Also remember that this store is in Japan and the electronic privacy concerns may not be the same as the US. People are aware of privacy issues but the solution may be different.
Microsoft, it was a nice idea to trying get everyone to install privacy information on their servers but to force people to do it by breaking their applications is not on. Anyway, it's a pain. Thanks again Microsoft.
I was looking yesterday about how Basic User authentication works on a servlet. It always intriguing to find out how things work under a hood and before I looked I didn't know how this worked. To the beginner it's not really obvious who is popping up the password box when you hit a password protected folder. It's hardly changed since early Netscape.
I'm still looking for how you can customize the page that gets shown when you input the wrong name or password.
One of our severs was reaching 95% usage capacity but df and du were giving wildly different figures for disk usage. JimT gave me the clue that du wouldn't see open files. When I checked with lsof there were a lot of files in the Apache logs folder that were "deleted" but were still open. Restarting Apache brought the figures back to normal.
Hmm read this Unix Administration Course lecture notes to know all those things that your System Administrator is doing without telling...
In the past we have sometimes had problems with bad characters when converting from Unicode to Shift-JIS in Java. It seems that there are a few characters that don't make the round trip correctly. The most troubling character is Shift JIS character "817C" which maps to Unicode "ff0d". This is a dash that is commonly used in Japanese addresses but gets converted to a question mark somewhere in the process. In the past we have replaced this character with a similar dash before converting to shift JIS.
This problem also seems to be described here Conversion Mapping Inconsistency for SJIS in an Oracle technical note.
I'm writing this in the hope that someone who understands the problem more can explain where the problem lies.
Later:
The problem is that unicode "ff0d" does not round trip when you use the encoding "SJIS" but does round trip when the Microsoft version of Shift JIS "MS932".
Even Later:
Here's the deal. Unicode contains some characters that are only present to support MS932. The dash that I mentioned is really easy to type using the IME on Windows so is often found in Japanese addresses.
When Java converts this code to the "SJIS" form of Shift JIS it converts it to the question mark signifying that it is unknown. It could have converted it to a similar character like the double byte minus sign but that would have lead to the character round tripping back to exactly the same character Unicode when it was read in again. Unfortunately converting it to a question mark means that the character is useless outside of MS932 or Unicode and has to be stripped out or converted if passed to another system which doesn't use those encodings.
Another option would have been to throw an "UncovertableCharacter" exception but I guess this data trashing happens to often to make that practical.
My co-worker Sunaga-san showed me a cool trick with native2ascii today. He ran it from the command line with no arguments and then used the IME to type in some Japanese. It output the unicode equivalent. This is what the tool is for, but I had never thought of using it interactively before. We needed to check what the unicode character was for a particular Japanese Kanji and he had an instant answer.
Today I had to add a new certifcate to Mojo's trusted certificate jar so that Mojo could connect via https to a as yet previously untried remote server. "Sugar!" I thought, "I don't remember the password for the trusted certificate jar!" but luckly I had made a memo of it in the code at the appropriate point. I didn't however remember how to add a certificate to the jar but with a couple of minutes tinkering and reading of the help file I got it to work. I have now put an example of that command in the source file for next time. I'm still left with the problem that that certificate will suddenly expire sometime next year and we have to remember to fix it before then....
Here's a great example of a University project: A site which explains how to write CGIs in Python.