Archive for the ‘Coding’ Category

Miracles Don’t Warrant Attention Any Longer

Monday, August 17th, 2009

cubeLifeView.gif

Well... maybe it's the Slump talking, but today I was asked to put in a feature quickly and in less than two hours I had a new view of the data available to my clients just like they'd asked for. Two hours. One would call that pretty nice - maybe even pretty wild. But today it didn't even warrant a "Nice job!" email.

I'm sure they expect this from me now, and that's a double-edge sword. One one hand, it's great because I can then expect impressive bonuses, but on the other hand, it's expected now, and if I just do a "good" job, it's seen as me 'slipping'.

Everyone likes to hear that what they are doing is appreciated. I'm no different. I don't require it, but there are times when I hammer out a new feature on a 'rush' schedule and it'd be nice to hear something. Oh well... I do it for myself. I do it because I'm who I am.

It's just a little tough to take this hit in the middle of a slump. But I'll get over it. I always do.

[8/18] UPDATE: I did get a 'good job' email today, so I guess it was noticed. It helps.

Base 1.3.2 is Out

Monday, August 17th, 2009

Base.jpg

I haven't been keeping really good tabs on Base recently, but this morning I was thinking about an interesting take on SQLite3 in Mac OS X development voiced by a pretty good developer a while back. Basically, he sees SQLite3 on the Mac as an easy way to skip using fopen() ever again, and to that I can see his point: if you're using files to save data, be it parameters, or application data, you can use a file and deal with encoding and decoding, or you can simply use a database that's a single file and have it organized any way you like.

It's a neat concept. Using a database system as an alternative to the standard file formats. When that happens, it's easy to dig into the file and see what data you've written - as opposed to making a custom file reader/displayer. Slick.

Anyway, Base is a nice GUI tool for the Mac that would be that GUI viewer tool for SQLite3 databases. Today's update to v1.3.2 means it's moving along nicely which is a real treat.

MacVim Snapshot 49 is Out

Monday, August 17th, 2009

MacVim.jpg

The guys working on MacVim have been busy once again, and released Snapshot 49 (actually 48 and then 49) to refactor the keyboard code and make it a little more international-friendly as well as allowing key bindings on more key combinations. Interestingly, this also ends up help us ASCII folks, and that a nice perk.

Still one of the bet editors on the planet.

It Seems Very Easy to Misuse Google Collections

Thursday, August 13th, 2009

I like reusable code as much as the next guy. I really do. For what I've seen of the Google Collections Java code, I like that too, though to be honest, I haven't seen all that much. But that's not to say that it's not possible - even easy, to create a mess with the Collections. In fact, it can make code very difficult to read. Take as an example, the code I ran into today:

  public Set<Trade> getTradesForReport(PositionService aPositionService) {
    // get all the trades from the service
    Set<Trade>  retval = aPositionService.getAllTrades();
    // now filter them on the trading parameter
    for (Iterator<Trade> iter = retval.iterator(); iter.hasNext(); ) {
      Trade   t = iter.next();
      if (!allowedUsers.contains(t.getUser())) {
        iter.remove();
      }
    }
    return retval;
  }

Java's definition of the initial Set<Trade> does not address the mutability of the Set. In fact, Java has no native immutability, and the way that Google achieved this is to make an Iterator that overrides the remove() method and throws an Exception.

That's dangerous. Not illegal for the language, but certainly dangerous.

If you look at NeXTSTEP/OPENSTEP/Cocoa on the Mac, the 'base class' is by definition immutable. You have to create a mutable version deliberately. This makes it clear what you are dealing with. The 'default' (base) behavior is to assume that you can't mess with it. Makes perfect sense.

But Java is the opposite. By default, all the objects, collections, etc. are mutable, and they handle immutability by removing the iterator() method, and using a different Enumerator. Look at the ConcurrentHashMap. No way to remove an item during a 'scan' of the data because the Enumerator doesn't have a remove() method. Period.

While I admire Google's work, it's not clear what they are doing. Say, in the example above, there were several position services, and some of them decided to return Google's ImmutableSet. Now I'm sunk. The compiler won't see that I can't do this, because as far as it's concerned, I can. I'd have to trap for the exception and reverse the logic - create a blank one and add those I wanted to it. But if I do that, I might as well use that logic for all cases:

  public Set<Trade> getTradesForReport(PositionService aPositionService) {
    // get all the trades from the service
    Set<Trade>  src = aPositionService.getAllTrades();
    // ...and make a place to put the good ones
    Set<Trade>  retval = new HashSet();
    // now filter them on the trading parameter
    for (Trade t : src) {
      if (allowedUsers.contains(t.getUser())) {
        retval.add(t);
      }
    }
    return retval;
  }

While this works in all cases, it doubles the references used, and that's not a good thing when you get into a tight memory application and garbage collection is already an issue.

What they should have done isn't clear. Face it, they wanted this to fit into the standard Collections in Java. But they are all mutable by default. No... the problem here lies with the coder that uses these. You need to be more explicit on the return types and explicitly say they are ImmutableSets. Then, the user/maintainer of the code can see what was intended.

Alas, such was not the case for me today. I had a production problem because some of the code returned a standard HashSet to the method signature and some returned a Google ImmutableSet. Unfortunately, this has been in production for several weeks, but it's just now getting hit. Lovely bug to catch and fix in a hurry.

A Hallmark of a Good System – Clean Mid-Day Restarts

Wednesday, August 12th, 2009

Well... when I left last evening, I left my system in a less-than-perfect state: if you restarted the web app in the middle of the day, it's possible that you might not get correct firm totals for P/L, etc. Why? Because it was based on a table trigger, and if you didn't get new rows, they weren't being counted. That's not going to work when some data sources send data only during fixed times of the day. Crud.

And while it's true that I don't restart production mid-day very often, it's still not the sign of a really good app to restart poorly in the middle of the day. The problem was, properly restarting mid-day was a real problem.

First, the firm totals needed to be stored on each alert - thankfully, they already were. Why? Because the alert could be an 11-point moving average filter or a 31-point moving average. In these cases, the firm totals will be different because of the different smoothing employed on the raw data. So it's got to be "local" to the alert.

Second, the firm totals needed to be updated for all incoming data. Initially, I had first checked to see if the portfolio was one of the ones I was interested in, but that was a mistake. I needed to update the firm totals and then filter on the appropriateness of the portfolio. That was a simple fix, so no big deal.

Finally, it's how the data needed to be fed into the alerts in order to get them primed for action on the restart. If there are n alerts, then we need to push n copies of each portfolio. Doing this at the alert level means there's a lot of hits to the database, and while that's logically reasonable, it's not a good plan as the number of alerts grows. So we need a different plan.

This morning I came up with a plan that seems to be working quite well.

In the alert controller code, just after creating all the alerts, I'm going to look at all the portfolios that have sent in data so far today. I'll then look at all the alerts and ask them how many data points they need, add one to that, and know that this is the number of "recent rows" in the data table I need for each portfolio.

I'll then run through all the portfolios, get the maximum number of rows I'll need from the table, and then feed those into each of the alerts, one at a time. Of course, I'll limit the data I feed any one alert to be that which is needs, but there will be at least one that will need all the data I've obtained, and that's not bad. Then, I'll be able to use this to calculate the firm totals for each alert.

What's interesting is that this process is really quite fast. One database connection, and to the H2 in-memory database at that. Most of the pulls there are less than 150 msec. in duration (yeah, I timed them in the code), and then pushing the data to the alerts is really fast as it's all just a bunch of data structures.

In the end, I have firm totals that survive a mid-day restart quite nicely. I'm more than a little pleased that all this work took no more than three hours. I was expecting quite a bit more. Nice surprise.

Finally Started Using My Gitosis Server

Tuesday, August 11th, 2009

gitLogo.gif

After I had upgraded to Git 1.6.4, I decided to take a little bit of time and push up one of my projects to the gitosis server I have set up at home. It really was as simple as the gitosis docs say: you set up a project in the gitosis.conf file, commit that and push it back to the server, and then push your project to the server. From then on, you'll be able to sync with that server and you're done.

Simple.

OK, maybe not simple, and maybe not exactly easy, but with some notes, it's very reasonable to do and takes only a few minutes. First, get a clone of the gitosis-admin project:

  git clone git@git.themanfromspud.com:gitosis-admin.git

and add in the new project, and possibly new team to the gitosis.conf file witin the newly cloned project:

  [gitosis]

  [group gitosis-admin]
  writeable = gitosis-admin
  members = drbob@sherman

  [group myteam]
  members = drbob@sherman steve todd
  writeable = Spill MServer

where I've added the [group myteam] section with the users and projects indicated. Then I can simply put this all back up to the server with:

  git commit -a -m "Added new Team for Projects with Steve and Todd"
  git push

and then we're ready to push up the projects.

Simply create the project:

  git init

put all the files in there, commit them, do all that normal stuff, and then push it to the server with:

  git remote add origin git@git.themanfromspud.com:MServer.git
  git push origin master:refs/heads/master

and it's done!

At this point, you can pull and push changes to and from the server and you'll not have to hassle with this ever again. It's all linked in. Very nice.

SubEthaEdit 3.5 is Out

Tuesday, August 11th, 2009

subethaedit.jpg

One of the editors that I want to use, but just isn't there - yet, is SubEthaEdit. It's collaborative features and vision of being lightweight and fast are things I want to use, but there have always been a few things that BBEdit or Vim does better and I can't seem to get behind using SubEthaEdit over either one of those.

Well, this morning I noticed that SubEthaEdit 3.5 was out, and I certainly upgraded to see if any of the things I needed were included in the update. Sadly, the big feature improvement is code-folding, and while that's really neat, I've never really used it - regardless of the editor. I've had it in many editors, and I just prefer to see all the code as opposed to seeing just the highlights and then "pop open" a section to work on it. I can see the value, I just never got used to it.

Old School, I guess.

So it's nice to see the update, but it's still not there - quite yet.

Updated Boxes to Git 1.6.4

Monday, August 10th, 2009

gitLogo_vert.gif

Given that I've looked at Mercurial and decided against using it as my main distributed source control system, and my version of Git was 1.6.0.2, I decided to update to the latest version of Git which was 1.6.4 on my laptop, my Intel iMac at home, and my gitosis server at home. It was just time for an update. Plus, I've read that the newer versions have pulled quite a bit of the old wrapper code into the C codebase so that things are more compact and faster. All around, it looked like it was time to upgrade and get on the latest features.

The updates for my laptop and iMac were easy because there's a Google code project for a Git installer for Mac OS X. It's as easy as downloading the package, installing it and running with it. Sure, there are a few things that you might want to do if this is the initial install of Git, and I've documented these previously. But an upgrade is easy-breezy.

The update for my OS X 10.3.9 box, frosty, at home is about as easy. Again, since I've built 1.6.0.2 on it before, it was as simple as:

  curl -o git.tar.gz http://www.kernel.org/pub/software/scm/git/git-1.6.4.tar.gz
  cd git-1.6.4
  ./configure
  make
  sudo make install

and we're off an running. Yeah, the docs aren't updated, but that's documented in the previous install, and I use my laptop for docs more, anyway.

Getting gitweb updated was pretty simple - just like the original install and it's in the Git distribution. One thing I liked about this time is that I'm a lot more savvy about the use of CSS and layout, so when I updated the code for gitweb, I fixed up the HTML for the gitweb header to be:

<div style="margin: 0px 0px 0px 0px; width: 100%;
 background-color:#2f2f9d;">
  <img src="/icons/repository_title.gif" alt="Dr Bobs Repository"
   width="344" height="61"/>
</div>

so that I get a nice looking header that renders a lot faster than the old TABLE HTML.

In the end, it's been about 30 mins all-told, and we are up to the latest Git at 1.6.4. Not bad at all.

Mercurial versus Git – For Now, I’ll Stick with Git

Friday, August 7th, 2009

Mercurial.gif

I was doing a little reading this morning and I came across another developer that was talking about how nice Mercurial was. He was mentioning a nice book he'd read on the subject and how much he liked the book. So I decided to take a little time this morning and give it a quick once over. Compare what I knew of Git to Mercurial and see how they stack up for my needs.

First, it's clear that each has a strong following. The linux kernel is in Git, and Google has adopted Mercurial. Both are strong statements to be considered. Certainly, if you're going to work on a project and it's in Subversion, or Git or Mercurial, then you have to do as the Romans do. But if you have a choice, and specifically, if I have a choice, what would I pick?

So what do I need? Well, I work on a laptop a lot of the time. Frankly, most of the time. I want something that's going to be significantly better than CVS - and I know CVS pretty darn well. Thankfully, both Git and Mercurial are winners here. There are Mac OS X GUI clients, and command line clients as well. Check.

Xcode support is important, but with the changes in Xcode to use ASCII files as opposed to binary files for nibs, I don't have to worry too much about this. Also, they have removed the 'bundle' concept for most things in Xcode, so I don't have to worry about directories appearing and disappearing on me. To be fair, Apple did all this work on it's own. That the SCM of choice doesn't have to worry about it is nice, but not leaning one way or the other.

I'll grant you that Mercurial is more like CVS, in that it's file-based and simple to understand. There's even a built-in web server to make it easy to act as a server for remote users to 'pull' a repository. Not bad. But Git has Gitosis, and now that I have it running, it's a server that I don't have to think about. It just runs. Period.

I could try to get Mercurial up on my Mac at home where I have CVS and Gitosis, but then I need to get Python 2.5 on that box, and it's an old Mac - suitable for the serving of these files, but it's not got the up to date Python that Leopard has. So that would be a pain. Not impossible, but a pain.

I guess it comes down to inertia.

I have Git, I have GitX, and they work. I have CVS for the old stuff, and it's never going away. Never. In the end, I might do some Mercurial work, but I'm guessing that it's even money that I'll stick with Git. It's just as nice and I have it all set up.

Finally Getting my Alerts Working Properly

Thursday, August 6th, 2009

cubeLifeView.gif

I'm finally pretty happy that the alerts in my web system are working as they should. The aggregated values are being calculated properly, the alerts are being triggered correctly, it's all working as it should. This is a "Big Deal" as it's one of the important things this guy was supposed to do - alert people via "push methods" (email and chat for now) of problems in the data. With these alerts in place, it's easier to let people not look at the data all day long, and have it alert them to problems. With a good set of alerts in place, it'll be easy to have the right people informed of the proper conditions.

Sure, it's geeky, but this is what it often comes down to - being able to finally bend this code into what I want it to be and seeing the results play out in front of me.