What's it all about, Alfie?

I got a visit from a developer who is using a shared Java library of mine - BKit, and two components of it, specifically: the BKTable and the expression parser - BKJEP, itself based on the JEP parser while it was Open Source and free. They have since closed the source on JEP and upgraded it's capabilities, but for what we need it is more than adequate, and I really don't want to get into a licensing situation with a group that started Open Source and then closed it down. They sound like they're on the brink of closing shop.

Anyway, the developer wanted to return a null value for the computed column. Originally, the idea of the computed column was to give the ability to the user of the BKTable a way to create a simple computed column without having to do it themselves. They could say "Give me a column called 'Area' that is 'width * height'" - where the columns called 'width' and 'height' were already defined in the table. When either 'width' or 'height' of any row in the table changed, the resulting 'Area' column was also updated. Nice, but not Rocket Science.

It got more complicated when they wanted to have Excel-like functions. Things like dateFormat() to turn a Date object into a String for comparison... isNull() to test if the value is null... and then if_then() to give it a simple conditional. And it's this last one that's opened the particular can of worms I'm dealing with right now.

When data is set in the expression, it's impossible to give it a true null value. The core JEP parser simply doesn't allow it. So we encoded the null to be a special String (__null__ if it matters, and it doesn't) so that it could be 'set' and 'get' from the expression. We also created the isNull() function so that you could test for it. So far so good. But what if the output of the if_then() is to be a null? Ah... there we have an issue.

There was no way to create a properly encoded null value within a JEP expression. This developer was the first to ask for it. There's a lot of good that could be done with this variable defined in all expressions... but there's a lot of danger as well.

Say the user snuck an expression past us that generated a true null value. Then used that value in the test: x == null... well... the value of 'x' would not have been encoded to the proper string constant and so it'd fail. So we have the problem that there are cases where something that should work - won't work. The easiest way to think about this is to just be careful and use the variable 'null' only when you have to. And test... test... test.

But with that, I think this adds more than it harms, so I added it in.

UPDATE: wow... that was a lot harder than I had originally thought. The first cut I had seemed to work, but it really didn't. I had assumed that within JEP the constants and the variables were kept apart as the first are immutable and the second aren't. But in fact, they are the same storage class. So... when I scanned through the variables in the BKTable's method, I still caught them and they didn't appear as column headers and so the expression was tossed out.

I had to change the way the BKTable's method worked. When I got a variable from the expression I had to check to see if it had a defined value. If it did, then it was a constant as only constants had defined values at this point in the processing of the expression. This skipping was essential to get the expressions not only passed in, but also properly evaluated. I ran all the tests on the table and they all came up OK. Whew! That was a lot more complex than I had originally thought.

I was looking at my pencil today and noticed that I had been using the same pencil - or copies of that original pencil, for more than 22 years. It's the Pentel Kerry pencil, and it's about as great a writing instrument as I've ever experienced.

I love the removable cap that protects the tip of the pencil so that it doesn't get bent, but can easily fit into a pocket where it might otherwise damage the garment - or get damaged by rough handling. The way in which the pencil can operate either without the cap on the back - or with it, shows incredible attention to the engineering of the instrument.

But the biggest thing is the balance. The weight is perfect, and while you can write without the cap on the back, the balance is really far better when you have it on the back. Then, it's a joy to use. Absolute joy. If you've ever tried a mechanical pencil and found them wanting in one regard or another, please, do yourself a favor and try this guy out. It's something I doubt you'll regret.

Oh... and mine are all in black. I have three on my desk right now. They are incredible.

IMAP vs POP3 - classic debate these days. IMAP has 'push' email, POP3 doesn't. Pop3 is old, IMAP it new. GMail is using IMAP now... MobileMe is using IMAP... Everyone is using IMAP - or so they might have us all believe. So I looked into it this morning and decided that POP3 is really quite good enough for me.

First, I had to see if the email servers I hit supported IMAP. First, Google's GMail - Yup. Next, HostMonster - Yup. Lastly, Comcast - Nope. Shucks... we'll, 2 out of 3 isn't bad. I could still move those two to IMAP. And then I started looking at the security of IMAP - just to be safe.

See... I use POP3 and SMTP over SSL to all my mail servers. I simply do not want my mail snatched as I send or receive it. Sure, from server to server it's all in plaintext, but that's going to take a lot more effort on the part of the "snooper" to intercept a major ISP's email. Sure, they can do it, but it's harder. For the last several years, I've been using POP3/SMTP over SSL and have been very happy with it. Every mail server I've needed supports it... and it's just plain secure.

IMAP is plaintext. Period. I'm not a big fan of plaintext passwords and the logins to GMail and Comcast are the logins to my accounts there. So if they snoop it once, they are golden for those accounts. I don't like that.

So I'm going to stick with POP3/SMTP over SSL. It would have been nice, but it's just not something I'm willing to compromise on at this time. Security of my email comes first.

My favorite IM client - Adium, released version 1.3 this morning and I had to get it. The release notes indicate a ton of improvements, but there's enough bug fixes to make it worth the time to download the version.

They have made it a big deal to revamp the entire contact manager system in the app. Now you have much more control over your buddy list as well as being able to set different activity alerts (sounds, events, etc.) for each buddy's different actions. While I don't do that now, I can certainly see why that would be a very nice thing to have.

There's also now Facebook chat as well as new MSN chatting libraries to make that smoother than before. It's a heck of an upgrade, and if you haven't tried it, you really should. Highly recommended.

I haven't had a lot to do today on current projects as I'm waiting for others to get things lined up or deployed for me to do my stuff. So I've spent most of the day transferring the contents of my HTML journal at work to the WordPress weblog I've got there. I accomplished about 4 months of postings - quite a lot of work, actually. At this rate, I should be able to get all the entries into WordPress in another week.

But even making this nice progress is tiring. I'm really tired. Sitting still... hammering in changes. Trying to stay focused so as not to make mistakes... it's just plain tiring. But it's still good to see the "To Do" list shrink. My goal is still to get it all into WordPress - even if it takes longer than a week to finish.

Today is the first day of school for my kids. They are all excited - some more nervous than others. I can remember loving school and being really excited to go back in the Fall. It's a great time to be a kid. Sure, my new High Schooler is a little intimidated - as all changers are... but after a week he's going to love it! Have fun kids!

UPDATE: everyone had a great time. Joseph didn't get lost in the halls at the new High School... Marie loved being in 7^th grade and knowing the school like the back of her hand... and Angelina loved being the big 5^th grader and Traffic Guard. It was a banner first day!

It's not been a particularly great day... nothing really outstanding has happened at all. I'm keeping busy moving things from my old journal to the WordPress blog... I'm getting faster at it. It's pretty interesting when you make competitions with yourself to make the time pass more quickly. It's doubtful that anyone is going to say "Hey, this is great!" when I'm done, but it keeps me from sitting doing nothing.

Some days, it's just a matter of slugging through the day-to-day mud of a job. It's not glamorous, it's not even really interesting. It's just the job. Today is one of those days.

I'm taking off a little early because I'd like to have a nice long weekend. Nothing planned... just no need to be here, as evidenced by the fact that there's nothing really to do.

Today I was back-filling a lot of journal entries from my old HTML-based work journal to my WordPress work weblog and I noticed that there was a serious problem with the table representation. There was a hugs gap at the top of this one table, and on another the gap was smaller. I could not, for the life of me figure out what the problem was until I glanced at the page source one time and looked at the data in the table.

I'd been focusing on the CSS and classes and such, but when I looked at the table's data, I saw that there were TD tags and not td. This may seem very innocent, but I've learned from experience that WordPress does not like the upper-case tags, and will not even recognize them. What was happening was that WordPress was putting line breaks ('br' tags) at the end of each row, and somehow that was getting turned into the large gap at the top of the table.

When I went back and made all the TD tags td, the gaps at the top of the tables were exactly the same. Then I went into the CSS for the theme I was using and made classes for the header row, and even and odd rows so that I didn't have to remember the coloring of each row. This is going to make the tables all look the same and that's a lot nicer.

Also, I learned an interesting point. In a margin: statement, you can use negative values to "encroach" on someone's space. I used this to offset what I thought was an excessive buffer at the top of the table. Works like a charm now.

Today I've spent far more time than I'd like to have on a problem that has seemed to plague me for months. There's a perl client app to my market data server and it is run to get end-of-day prices for a variety of instruments that we don't get (easily) any other way. Seems reasonable.

Oh, how I wish it were...

I've been hearing that it's working, and then it's not. I look into the data, and it's fine. Then it's not. I don't want to take over this application because it's something that the guys should be able to deal with, but it has been getting closer and closer to that point because I keep thinking it's clear, and then there's an issue.

Just yesterday, I had an issue with a wild bug in the market data server. I deployed it and all looked OK. Then today it's broken again. I'm not sure exactly why this is happening, but I wasn't going to take any unnecessary chances: get the latest driver libraries, and make it from scratch. Maybe there's a header file/library mismatch? Who knows, but I was getting tired of this.

I also got their perl application - including the data, and stripped it down to the point that it's getting the data, testing it, logging failures, and then skipping the updating of the database. I've run this and it's returned good data for everything - save the ones where the ticker is bad, or there's not sufficient data in the database to generate a good ticker.

At this point, I'm convinced that the market data server is OK, and the perl interface into it is also OK. The application should run without problem, but if it picks up another mysterious problem, I'm going to be looking at the machine it runs on to see if there's any problems. It's got to be something, and the code just isn't it.

Digging into other people's code and not re-writing it is one of the less fun things I have to do from time to time. It's like a tacit approval of the horrible way they have put together this application. It was (and is) a mess, and should be recoded to be a lot more streamlined and efficient, but that's not my call. Thankfully, for now, it's working. We'll see what tomorrow brings.

This afternoon I found a wild bug in my MarketData server related to the symbology mapping and F/X rate conversion code for my ticker plant. I know this may seem to be a little 'specialized', but finding it was one of those 'Eureka!' moments that comes with a flash of insight.

This particular data provider for the market data server was the ticker plant I'd written to interface to the Reuters RMDS system. The symbology in our Shop for international tickers has an '=' separating the base ticker from the primary country of trading. So Vodaphone in Great Britain is VOD=GB while Google in the US is just GOOG - the '=US' is implied.

These need to be converted into RIC-like aliases for the ticker plant. I say RIC-like because there are synthetic prices in the system created by basket prices and/or mathematical operations done on 'natural' tickers. Say we had an F/X rate in GBP/USD, but we wanted it in USD/GBP - we'd make a synthetic price that's essentially '1/x' where x is the original F/X rate.

Well... it was this '=' that was giving me the problems. When a user asked for a piece of data for a symbol, they could ask for 'native' or dollarized (USD) figures. This means that I needed to have the F/X rate for all symbols at hand for returning the dollarized figures. But if the user happened to give me an alias as opposed to a symbol, I would happily process that and return the values. But there's the rub... if I were given the alias - that happened to include an '=', I'd try to get the F/X rate and that would be impossible.

An example is JPY=.RTL the USD/JPY F/X rate. If I were asked for that, I'd look at is as if it were a non-US symbol with the country code of '=.RTL' when in fact, it was an alias for an F/X rate and as such had no business being converted to dollars. I would try to get this mythical F/X rate (for this F/X rate) and it would be zero - causing the data to be zero.

What I needed was to be more aware of the symbols coming in and to be able to detect which were tickers and which were aliases and not try to get the F/X rate on the aliases. Once I did that, I was golden and everything worked fine. But it was interesting that I was looking at the data and the '=' sign hit me like a ton of bricks. I'm sure glad it did. The fix took all of 5 mins and everything is working fine. Whew!