Archive for August, 2009

Fighting to Get Out of a Slump

Friday, August 14th, 2009

It's been a hard week, and I'm finding it more and more difficult to focus and get motivated to get work done. It's a slump, I know it. It happens every so often when I'm not really having a fun project to work on and the hours are getting long and there's no real outlet in the evenings and weekends. It's a slump.

I know that time will take care of this, it always has in the past. The only real question is How long? Clearly, I'm hoping for "not too bloody long", but there's no way to tell. It could be something as simple as an update to some program that's got something I really have needed for a while... or maybe it's Snow Leopard... or maybe it'll be a few days off at home.

Something will start the ball rolling and then it'll build on itself just like the slump did, and in the end, I'll be back on top where the sun is shining. Just wish it'd hurry up and get here.

Hubble Ultra Deep Field in 3D – What I Love Most About Scientists

Friday, August 14th, 2009

Courtesy of BoingBoing this morning is this fantastic video about what scientists found when they pointed the Hubble Space telescope into the blackest of blacks in space. They were probably told by many that there's nothing there - after all, shouldn't there be at least a few places in space where "nothing" is there? But they didn't listen.

Instead, they believed it was a great experiment - something worthy of human time and effort to see the results - if only for the knowledge itself.

To realize what's out there - that blackness of space is really just faintness of source, and not absence of source, well... it's just what I love most about science and scientists. It is, after all, just knowledge. No one is going to have a better day today because they know this. It's not going to put food on the tables of millions, or change the GDP of the poorest countries in the world. But that's not to say that it's not something that should be done.

Liza's tagline on emails is simple: The Human Race is a relay., and while it took me no time to understand what she was saying, it took me a while to really understand the significance to her, and now, to me.

We aren't all there is. We aren't even really all that important - except to ourselves and those around us. But we are, as a species, the intellectual custodians of this existence. It is up to us to learn, document, and preserve things we see and learn about this life of ours for future generations that might find importance in some little bit of knowledge we maintain.

I got into science, as a kid, from this point of view. Learning and experimenting for the knowledge. Just knowing it was enough. Later, realities sunk in and I had to pay a mortgage, but in the beginning, the purest form of my interest in knowledge was just that - to read, to understand, and to coalesce.

Bless these men and women that to this for all of us.

It Seems Very Easy to Misuse Google Collections

Thursday, August 13th, 2009

I like reusable code as much as the next guy. I really do. For what I've seen of the Google Collections Java code, I like that too, though to be honest, I haven't seen all that much. But that's not to say that it's not possible - even easy, to create a mess with the Collections. In fact, it can make code very difficult to read. Take as an example, the code I ran into today:

  public Set<Trade> getTradesForReport(PositionService aPositionService) {
    // get all the trades from the service
    Set<Trade>  retval = aPositionService.getAllTrades();
    // now filter them on the trading parameter
    for (Iterator<Trade> iter = retval.iterator(); iter.hasNext(); ) {
      Trade   t = iter.next();
      if (!allowedUsers.contains(t.getUser())) {
        iter.remove();
      }
    }
    return retval;
  }

Java's definition of the initial Set<Trade> does not address the mutability of the Set. In fact, Java has no native immutability, and the way that Google achieved this is to make an Iterator that overrides the remove() method and throws an Exception.

That's dangerous. Not illegal for the language, but certainly dangerous.

If you look at NeXTSTEP/OPENSTEP/Cocoa on the Mac, the 'base class' is by definition immutable. You have to create a mutable version deliberately. This makes it clear what you are dealing with. The 'default' (base) behavior is to assume that you can't mess with it. Makes perfect sense.

But Java is the opposite. By default, all the objects, collections, etc. are mutable, and they handle immutability by removing the iterator() method, and using a different Enumerator. Look at the ConcurrentHashMap. No way to remove an item during a 'scan' of the data because the Enumerator doesn't have a remove() method. Period.

While I admire Google's work, it's not clear what they are doing. Say, in the example above, there were several position services, and some of them decided to return Google's ImmutableSet. Now I'm sunk. The compiler won't see that I can't do this, because as far as it's concerned, I can. I'd have to trap for the exception and reverse the logic - create a blank one and add those I wanted to it. But if I do that, I might as well use that logic for all cases:

  public Set<Trade> getTradesForReport(PositionService aPositionService) {
    // get all the trades from the service
    Set<Trade>  src = aPositionService.getAllTrades();
    // ...and make a place to put the good ones
    Set<Trade>  retval = new HashSet();
    // now filter them on the trading parameter
    for (Trade t : src) {
      if (allowedUsers.contains(t.getUser())) {
        retval.add(t);
      }
    }
    return retval;
  }

While this works in all cases, it doubles the references used, and that's not a good thing when you get into a tight memory application and garbage collection is already an issue.

What they should have done isn't clear. Face it, they wanted this to fit into the standard Collections in Java. But they are all mutable by default. No... the problem here lies with the coder that uses these. You need to be more explicit on the return types and explicitly say they are ImmutableSets. Then, the user/maintainer of the code can see what was intended.

Alas, such was not the case for me today. I had a production problem because some of the code returned a standard HashSet to the method signature and some returned a Google ImmutableSet. Unfortunately, this has been in production for several weeks, but it's just now getting hit. Lovely bug to catch and fix in a hurry.

Another Security Update for Mac OS X 10.5.8 on Software Updates

Thursday, August 13th, 2009

Leopard.jpg

This morning Apple released another Security Update - this time for a problem in the DNS component, if I read the notes correctly. Given that we had 10.5.8 released about a week ago, and Safari 4.0.3 yesterday, this is a bit of a surprise, so it must be a pretty bad problem, or very widespread - to warrant a new release.

But I'm not about to be unprotected on my Mac, so upgrade it is.

Security Update to WordPress 2.8.4 at HostMonster

Thursday, August 13th, 2009

wordpress.gif

Well, this morning I read about an exploit for WordPress 2.8.3 that allows a user to change the admin's password, and effectively lock them out of their own account. Pretty nasty stuff.

Thankfully, SimpleScripts had WordPress 2.8.4 ready to go and I was able to quickly update my installs to the version with the fix. Whew!

A Hallmark of a Good System – Clean Mid-Day Restarts

Wednesday, August 12th, 2009

Well... when I left last evening, I left my system in a less-than-perfect state: if you restarted the web app in the middle of the day, it's possible that you might not get correct firm totals for P/L, etc. Why? Because it was based on a table trigger, and if you didn't get new rows, they weren't being counted. That's not going to work when some data sources send data only during fixed times of the day. Crud.

And while it's true that I don't restart production mid-day very often, it's still not the sign of a really good app to restart poorly in the middle of the day. The problem was, properly restarting mid-day was a real problem.

First, the firm totals needed to be stored on each alert - thankfully, they already were. Why? Because the alert could be an 11-point moving average filter or a 31-point moving average. In these cases, the firm totals will be different because of the different smoothing employed on the raw data. So it's got to be "local" to the alert.

Second, the firm totals needed to be updated for all incoming data. Initially, I had first checked to see if the portfolio was one of the ones I was interested in, but that was a mistake. I needed to update the firm totals and then filter on the appropriateness of the portfolio. That was a simple fix, so no big deal.

Finally, it's how the data needed to be fed into the alerts in order to get them primed for action on the restart. If there are n alerts, then we need to push n copies of each portfolio. Doing this at the alert level means there's a lot of hits to the database, and while that's logically reasonable, it's not a good plan as the number of alerts grows. So we need a different plan.

This morning I came up with a plan that seems to be working quite well.

In the alert controller code, just after creating all the alerts, I'm going to look at all the portfolios that have sent in data so far today. I'll then look at all the alerts and ask them how many data points they need, add one to that, and know that this is the number of "recent rows" in the data table I need for each portfolio.

I'll then run through all the portfolios, get the maximum number of rows I'll need from the table, and then feed those into each of the alerts, one at a time. Of course, I'll limit the data I feed any one alert to be that which is needs, but there will be at least one that will need all the data I've obtained, and that's not bad. Then, I'll be able to use this to calculate the firm totals for each alert.

What's interesting is that this process is really quite fast. One database connection, and to the H2 in-memory database at that. Most of the pulls there are less than 150 msec. in duration (yeah, I timed them in the code), and then pushing the data to the alerts is really fast as it's all just a bunch of data structures.

In the end, I have firm totals that survive a mid-day restart quite nicely. I'm more than a little pleased that all this work took no more than three hours. I was expecting quite a bit more. Nice surprise.

Safari 4.0.3 is on Software Update

Wednesday, August 12th, 2009

Safari.jpg

Well, this morning it was time to reboot my laptop after updating Safari in Software Updates. The release notes talk about the typical round of security fixes and stability enhancements, but this time they also include fixes for logging onto iWork.com and a few non-maintenance things. Sounds reasonable, even though I haven't used iWork.com, they're making progress, and that's good enough for me.

Within the month, I'll probably be getting Snow Leopard, and that will bring a lot of little changes - not the least of which is a much faster system. Looking forward to that.

Apple is Coming to Naperville!

Tuesday, August 11th, 2009

Apple-logo.jpg

I can't believe it, but Naperville is getting an Apple Store! Opening August 15th, 10:00 am, I'll have an Apple Store on the way home - or any time I want to hit downtown Naperville.

This is the most amazing news. I just can't believe I'm lucky enough to have one so close to my home. I know Liza won't be thrilled, as it'll mean more trips to the store, and that will mean more purchasing of Apple stuff, but that's the cost of progress.

What a lucky break!

Finally Started Using My Gitosis Server

Tuesday, August 11th, 2009

gitLogo.gif

After I had upgraded to Git 1.6.4, I decided to take a little bit of time and push up one of my projects to the gitosis server I have set up at home. It really was as simple as the gitosis docs say: you set up a project in the gitosis.conf file, commit that and push it back to the server, and then push your project to the server. From then on, you'll be able to sync with that server and you're done.

Simple.

OK, maybe not simple, and maybe not exactly easy, but with some notes, it's very reasonable to do and takes only a few minutes. First, get a clone of the gitosis-admin project:

  git clone git@git.themanfromspud.com:gitosis-admin.git

and add in the new project, and possibly new team to the gitosis.conf file witin the newly cloned project:

  [gitosis]

  [group gitosis-admin]
  writeable = gitosis-admin
  members = drbob@sherman

  [group myteam]
  members = drbob@sherman steve todd
  writeable = Spill MServer

where I've added the [group myteam] section with the users and projects indicated. Then I can simply put this all back up to the server with:

  git commit -a -m "Added new Team for Projects with Steve and Todd"
  git push

and then we're ready to push up the projects.

Simply create the project:

  git init

put all the files in there, commit them, do all that normal stuff, and then push it to the server with:

  git remote add origin git@git.themanfromspud.com:MServer.git
  git push origin master:refs/heads/master

and it's done!

At this point, you can pull and push changes to and from the server and you'll not have to hassle with this ever again. It's all linked in. Very nice.

SubEthaEdit 3.5 is Out

Tuesday, August 11th, 2009

subethaedit.jpg

One of the editors that I want to use, but just isn't there - yet, is SubEthaEdit. It's collaborative features and vision of being lightweight and fast are things I want to use, but there have always been a few things that BBEdit or Vim does better and I can't seem to get behind using SubEthaEdit over either one of those.

Well, this morning I noticed that SubEthaEdit 3.5 was out, and I certainly upgraded to see if any of the things I needed were included in the update. Sadly, the big feature improvement is code-folding, and while that's really neat, I've never really used it - regardless of the editor. I've had it in many editors, and I just prefer to see all the code as opposed to seeing just the highlights and then "pop open" a section to work on it. I can see the value, I just never got used to it.

Old School, I guess.

So it's nice to see the update, but it's still not there - quite yet.