What's it all about, Alfie?

I've been hip-deep in Hemlock again this afternoon - trying to track down a NullPointerException whose reported location is giving me absolutely no help in trying to figure out the problem. I spent about an hour trying to figure out the problem until I finally stumbled on the answer and got it fixed. But the problem was an enormous pain in the rear, and related to excessive use of Java's autoboxing feature.

The problem was that the original developer of Hemlock had decided to use Double objects as instance variables as opposed to using the elemental double. This is not because he wanted to use the ability to assign null to these values - no, it's just because he was being lazy, that's the best I can figure.

If you use Double objects and then mix in arithmetic operations, you're counting on Java's autoboxing to do all the real work for you. Whereas, if you used double you might have to think about illegal values, but in this case, that wouldn't be a problem.

What I had done to the code to simplify it was to replace the use of dozens of ivars with a map. Specifically, a ConcurrentHashMap<String, Double> where I would make the setters and getters use specific keys so that the API to the class appears as if the ivars as distinct, but they're all really being looked up in the map.

This was done because the data needed to be outputted in a JSP as an XML data stream. With the Map, I was able to make a very simple loop and the dozens of getters and output statements were reduced to a few in a loop. Massive simplification that was really a long time overdue. Everything was working fine until I ran into the following code:

  Double  newValue = (obj == null ? 0.0 : obj.getMyValue());

where:

  public Double getMyValue() {
    return (_values == null ? null : _values.get("myValue"));
  }

where the _values is the ConcurrentHashMap<String, Double> that stored all the values.

Because of the nature of the ConcurrentHashMap<String, Double> neither the key nor the value could be null. Still, I was setting the value to a Double so that shouldn't be an issue. Or should it?

When I added setting a new Double(0.0) for all values in the _values Map, the error went away. What's the lesson here? I believe it's that the conditional having the 0.0 first sets the expected datatype for the second part of the conditional, and when it's null, a NullPointerException is thrown, but since it's in the return call, the exception is actually reported on the line where the method returns.

What have I learned? That with autoboxing, you have to be very, very careful about the nulls floating around your code. But more importantly, their use should really be limited because the overhead of the conversions is hidden in the code, and problems such as these don't show up as easily.

Then again, I've never had problems with these guys, but then again, I harly ever use them. I'm far more interested in the explicit conversions than implicit conversions done by the compiler or runtime.

Today I've been finishing up on a lot of little things that I've been putting off because they just aren't that important, but today their time came. I was having a little fun making a few servlets spit out the same data as HTML, CSV, and XML. The first two were easy as I had already done that, but the XML was pretty simple too. The JSON format isn't all that different from a compressed XML format, so the same kind of data was pretty easy to reformat to XML. Nice. Now I'll be able to make that data available to other requesting clients. Nothing amazing, but it needed to be done sooner or later because I had one of those clients waiting in the wings.

I was able to spend some time monitoring my production processes and realized that one of my boxes has the memory configuration messed up. I did a little checking, and updated the start script (catalina.sh) to use the proper maximum memory setting. Tomorrow, it'll start up with the right limit. No idea who changed that, or why, but it was nice to have the time to monitor the apps and see that this was happening before it was a problem.

Plus there's the complete mental health angle... I've been running flat out for a long time, and now that I have a few things in Q/A awaiting certification, I have the time to just rest a bit. It makes a big difference on my outlook.

Not sure I'd like this every day, but it's nice to have a little breathing room now and again.

I'm continually amazed by my attitude towards the work I do. I really love the work I do - development, but I don't always see eye-to-eye with the people I work for, or with. This afternoon I was able to add in the parsing and storage for strike-level volatility data to be stored with every update from select sources. It's going to be a ton of data - I'm guessing about 720,000 rows/minute or roughly 432 million rows per day. Each row isn't that big - I'm guessing around 30 bytes a row, forget the indexing, etc. the data itself is over 12 GB/day. Very big.

Still, it's what they want, and they are willing to spend the money to get the database storage for it. I just hope that I can add enough RAM to my boxes so that I can hold at least two days of data. We'll see as things progress.

But it's exciting to write good code. It's frustrating to see others butcher it, and I had a little of that too, but there's nothing I can do about that. It just comes with the territory.

Still... as with kids, you have to hold onto those nice moments and make them last.

Today I've run across something that makes me cringe. I mean really... cringe. And while it's easy to lay blame at the feet of the developers, I'm really beginning to lay it at the feet of the development managers. If you allow it, they will do it. Period.

I've been having problems with a very poorly created API to a DatingService at The Shop. I've touched on this before, and I won't belabor the point further. No, today I was told that there were significant bug fixes in the latest version of the API, and that it was probably a good idea to upgrade to this latest version and the problems I've been having should go away.

I'm not surprised, it makes sense, but at the same time, if the developers, and managers of those developers, allowed such a bad API to be created, I was worried what the "upgrade" would mean. I had hopes that it was just a Java jar replacement, but I had an inkling that it wouldn't be that easy.

When I got the word that one of the jars was split into two, I had a bad feeling.

It got worse, when I put in the three new jars and tried to compile the code. Not even close. Hundreds of problems. Holy Cow.

So I asked the developer that was helping me "What changed?" Normally, I'd think "nothing", or at worst, a package name change, but again, I had a feeling.

The silence that followed was not reassuring.

He sat at my desk - because there were no docs, no examples, just the raw code, and as he looked at the Google Protocol Buffers code, he was guessing what the method names were. I was shaking my head the entire time. I simply could not believe that a developer who used this code was this unfamiliar with it.

It took far longer than it should have, and included a lot of changes - all for an API that really should have been less than five lines, but ended up being far more than that. The types of data were different, the arguments different. It seemed that they threw most of what was out in favor of something similar, but not really equivalent.

While I'm the first to understand that things need to change, APIs just aren't one of them. There's no reason they couldn't have made the same fixes into the old API. THey just were lazy. Or worse.

So everyone that used the old way had to re-tool their code to use the new. Why? Because a few guys (and their manager) were too bloody lazy to actually design a decent interface and then build it?

It appears that's about the size of it.

I shake my head at the sheer lack of common sense displayed in these interactions.

Today I was finishing up the additions to Hemlock and it while I expected to get the changes done by the end of the day, I was a little surprised to see that I was done by 2:00 pm. That's a little scary, to me. I mean, if I know this codebase this well, then what does that say about me? Am I close to being a Pod Person?

If I ever find myself saying Spring and AspectJ are good choices, then I'm going to throw myself off a bridge. Just sayin'...

Today I've been pushing on the Hemlock project to add in some new functionality before the Q/A Team gets backed up with other releases and the changes in Hemlock are delayed another month's expiration. All this is not really new, but it brings up a point that a friend made about parenting teenagers.

Such is the life of a parent, I guess. It's a challenge of character to not allow your children's idiocy/lack of appreciation ruin your relationship with them so that in 5 years when they're surprised by how much you have learned, you can start having a civil, adult relationship.

And as I was pushing very hard today for these new features in a project that I really dislike a great deal, I realized that this, too, is a test of character and commitment. It's easy to put in the "extra effort" when you're excited about the outcome. It's easy to be the parent of a grateful kid.

But show me how much "extra effort" you give to a project that you never want to use, and would send into The Daily WTF if it weren't for the fact that people at your place of work would recognize the horrible code and know it was you that sent it in. Show me how hard you're going to work on something that gives you a headache just keeping track of the useless unit tests and what goes where.

Yeah, my friend is right... being an adult is a lot about a test of character. I'm glad I can say that I was able to pass this particular test.

I just finished up on tracking down this nasty little bug in Hemlock that has been giving me a little grief with one of the desks. Sure, it was a little difference, and the changes to the code totaled about 15 lines, but those 15 lines were very important to the process.

It's nice to be able to end the day on a high note.

Maybe I don't need that bell.

I'm not a prima donna of systems design, I know I've put together a lot of things that were good for their intended purpose, been pushed to their limits, and then some, and shown the effects of time. Everyone does. But what I have a hard time dealing with is the kind of interaction I've had this morning about something as simple as a DatingService built by the guys here at The Shop.

It's pretty standard stuff in the industry - you want to have a single place where you can get the definitive word on the business dates for any given calendar. You need to use this to calculate times to expiration, maturity, etc. It's static stuff, so it doesn't make sense to have it sitting in a database for thousands of people to read, but that's been done other places I've worked - better to have it loaded in-memory on a box that answers requests for dates, and delivers them very quickly.

But that's about the last good idea the DatingService had. Some time ago, the developers here - or maybe the management, decided that Google Protocol Buffers was The Way, and while there's nothing wrong with using them as a serialization/de-serialization scheme, you really shouldn't force the user to know that you're using them because that exposes unnecessary implementation details to the user.

If you want to make a service - make a Client too.

Have that client take only those parameters needed to identify the version/type/etc. of the service and then have all the calls return either objects defined in the API, or language-standard data types. It's simple to use, your client code can include auto-reconnect, auto-failover, all the things that a client might want to have. You are in control.

If you leave too many details to the user, then if and when you need to change any one of them, you're sunk - all users have to update which is a major coordination issue. Not ideal.

Take this example of a client I wrote to my CacheStation - a Reuters-based ticker plant that I wrote at my last position:

  CSClient    svr = null;
  try {
    svr = new CSClient(CSClient.CHICAGO_PROD);
    if (svr == null) {
      // ...handle the problem
    }
  } catch (BKException bke) {
    // handle any exceptions like missing server
  }

All details about the communication are handled in the constructor. There are exceptions thrown for a missing server, but you don't have to configure anything - the "servers" are defined as parameters in the client code so that I (as the owner of the code) can choose to have the real connection parameters in the code, or in a file, or in a database - and there's no difference to the user of the service.

The client auto-reconnects, auto-retries, and has methods for blocking and non-blocking calls so that the user can choose exactly how they want to get data from the server. As simply, the data is returned in objects that are defined in the package as well:

  Vector     symbols = new Vector();
  symbols.add("IBM");
  symbols.add("GOOG");
  Vector     prices = null;
  try {
    prices = svr.grab(symbols);
  } catch (BKException bke) {
    // ...handle the possible exception
  }
 
  MMPrice   ibm = prices.get(0);
  MMPrice   goog = prices.get(1);
  System.out.println("IBM volume: " + ibm.getVolume());
  // etc.

The point is that the API/Client should be simple, simple, simple. The more complex it is, the worse it is. So let's look at what I'm trying to deal with today.

To be fair, the code presented here is not mine, and had I written it, I'd have broken it up into logical steps, logging the success or failure of each, which would have helped in knowing what failed, but not in the fact that it failed. So here's the code for connecting to the DatingServer written here at The Shop:

  private void initializeCalendar() throws Exception {
    boolean  shouldInitializeService = false;
    try {
      readWriteLock.readLock().lock();
      shouldInitializeService = !serviceInitialized;
    } finally {
      readWriteLock.readLock().unlock();
    }
 
    if (shouldInitializeService) {
      try {
        readWriteLock.writeLock().lock();
 
        this.rpcClient = new RpcMessagingClient(transportEnvironment,
                                                messengerFileLocation);
        HolidayService holidayService = HolidayService.newStub(
                         rpcClient.getRpcChannel(
                             HolidayService.getDescriptor().getName(), ""));
        rpcHolidayServiceProxy = new RpcSyncProxy(holidayService);
        HolidayCalcRequest.Builder holidayCalcRequestBuilder =
                         HolidayCalcRequest.newBuilder();
        holidayCalcRequestBuilder.setHolidayCenter(holidayCalendarName);
        holidayCenterData = rpcHolidayServiceProxy.rpcCall("getHolidays",
                         holidayCalcRequestBuilder.build());
 
        serviceInitialized = true;
      } finally {
        readWriteLock.writeLock().unlock();
      }
    }
  }

I don't think I could imagine a more Rube Goldberg scheme for connecting to something as simple as a DatingService and getting a list of holidays for a given calendar name. And this doesn't even include the dozen lines for parsing the output of the RPC call which is another serious problem.

Why do I need to know about the RPC? Why do I need to make builders that serve no other reason than to get the data? It's a mess, a joke, and that people think this is good is just depressing.

And so we have my continuing problem with this service - when I was starting this application at 5:45 am, it was fine. The users wanted to have it running nearly continually, so I moved the start time to 00:45, and there, it's not working. Sure, the code isn't that great - goes the RpcMessagingClient need to be shutdown if the constructor succeeds but the other calls fail? The code ends up getting run make times, making a lot of connections with the RpcMessagingClient and then we run out of sockets and the web server is useless.

But if I restart it at 5:45 am, it's all fine and there are no errors.

So clearly there's a problem in their end of things, but with the code written the way it is, I can't give them any more information than the sequence of steps failed. I would think that's enough. But it's not.

No, they are pushing back saying it's fine on their end and that I need to provide them with the detailed error logs and stack traces. WHAT? I have to provide them with the reasons their code is not working? What's wrong with this picture?

Clearly, this place has a lot of dysfunction and a good deal more issues. I am playing by the rules my manager gives me, and if that means that the early morning hours of the web server are shot because they can't fix their stuff... well... I can't help that.

I give up.

This morning I noticed that the 'dev' feed of Google Chrome is at 5.0.335.0, and so I decided to download it and give it another try. The reason I'm not using it more is that I've found it to be very unreliable in connecting to the sites/servers that I want it to. I wanted to use it as my back-up browser, which is Firefox 3.6 now, but it just wasn't as stable and useful as Firefox 3.6.

While I use Google Chrome all the time in my work at The Shop, and find it to be a very respectable browser for the work I do there, the Mac OS X port is just not up to that level of quality - yet. Well... today when I tried 5.0.335.0 it was acting much better about getting to the sites and moving between them. Not bad.

Given this, I may give it a day or two and see if it holds up to the same load that Firefox does. If so, I may switch over. Would be nice to see.

When I thought my cow-like appearance to my manager couldn't possibly get any worse, today happened, and I just feel like starting to wear a bell. No joke, I had no idea how bad it could get in the span of a day. It's stunning.

Recap: When Two Days is Too Long

Previously on The Farm... yesterday my manager stopped by and asked me to implement this new feature into my web app. It was a reasonable request, and I didn't think it'd take too long, but it wasn't just a configuration change, either. Basically, I'm using the H2 database as an in-memory database for a Tomcat web app and it's working pretty well for me. When I was asked a while back to add in alerts for conditions in the data stream, it seemed logical to use the database triggers capability of H2 to do the job. That way, I just deal with the stream of data regardless of it's source. Seemed pretty slick, and it is.

The problem was, this new request was to add the same kind of alerting to a different data stream - also in the H2 in-memory database, but a different table structure. Now as any database guy will tell you, converting a trigger from one table to another one with different columns, etc. is possible, and not too complex, but it's not something you can do in five minutes. Especially when you've got something as complex as the configurable and flexible alerting system I created.

So I said "Yeah, I can do that... sometime between two days and two weeks."

"Really! Come on... you can do it in a day, Right?" he said squinting and showing his displeasure with the answer I had just given him.

"Nope, that's a pretty decent time because you'll want me to have all the capabilities of the other, which means I need to generalize some things, and that's at least a few days."

He left, disappointed that I hadn't said "2 hours", and leaving me with the distinct impression that he thought I was sandbagging, or lying, or at least an ungrateful piece of meat (a.k.a. a Cow).

The Ole End-Around

This morning I was talking to a teammate and he said that our manager told him to ask me about adding this feature to the system. Immediately I was very defensive - did he seriously not believe me when I said it's take two days? Did he think so little of me to ask a guy that doesn't know this codebase one tenth as well as I did? What was going on here? But this wasn't my teammate's fault, so I tried to remain calm.

"Yeah, Steve, I know... He talked to be about it and told him two days. I guess he didn't like that number so he's asking you."

At this point, Steve clearly looks caught in the middle of things. I can relate. He had no idea what was going on as he wasn't involved in the previous conversation, but was hip-deep in it now. So we talk for just a minute or two, and then I decide to pull in our manager again.

"Ralph, I hear you're asking Steve about the feature we talked about yesterday."

"Yeah, we really need it - maybe he can throw it into Hemlock. Maybe he can put it into another page..."

I can clearly see what's happening, this feature isn't important to our group. It's important to another group, but by giving them something they don't have, Ralph looks like a RockStar. Good for Ralph. But he's doing it on the backs of us - he's not capable of doing this, and it's making me madder by the second to think that all he cares about is how "cool" he cal look to the other people here without giving any consideration to his people.

"Fine, Ralph, I'll put it in the other page, and it'll be a highlighted row." And off I went to start coding this up.

So Once Again, You Really Want This?

I code at a fever pitch to get this done because I'm also seeing emails start to fly around about another change coming, and while I didn't have a lot of time to respond to the emails, I did have a few responses to try and limit the work, but at each turn Ralph was saying we needed to do more, not less.

But I got the changes in and started testing them. Amazingly, I didn't have any real issues and it worked just exactly ass I planned. Guess there's something to this "experience" thing. So I then asked Ralph "Ralph, can you get the values from the business for the filter?" as I had in dummy values, but needed the real ones before I pushed the code to Testing.

"OK, I'll go get them." and off he went.

When he came back, he had a little sheepish look on his face and started to explain: "When I first asked Goose he liked the idea, but then Ice Man said it's not going to work because it's not on all the trader's desktops. Sorry..."

Basically, I had wasted half the day. Totally wasted for a feature Ralph was sure we needed to deliver to "look good", and it turns out, it's just a waste of my time.

Man, I was not happy.

But the day wasn't over, while I was doing this work, I saw in the email chain that Ralph had promised that I could get this other work done TOO!

Ralph's Looking Good: Part Deux

I couldn't spend any time worrying about the lost time, I had to get these other changes into two systems and get several things updated and then deploy a few things. It wasn't hard, but it took me several hours to make sure I got it right and that things would work.

When I was about half way done, and saw that I'd have to stay late, I asked "Hey, what's the rush on this? Why does this have to go in tonight?" And no one seemed to have an answer... but Ralph was conspicuously absent. So I had to trudge on...

As I was entering my 12^th hour of work, I finally got things done and checked out. I had wasted half a day on a useless feature, and then hurried through the addition ov several data sources to the systems just to make Ralph happy - more like make him look good to others.

As I was getting ready to leave, I asked Ralph if he had a few minutes... he tried to say he was sorry, but I said "Let's go have a talk", and I walked into an empty office.

What followed was my feelings about his actions of the last few days.

• the feeling of being massively manipulated about the feature addition with Steve
• the feeling that I was lying or sandbagging about the two day time required - and what's wrong with two days? Is it really that much to ask?
• the agreement "for me" as to what I could get done in a day - again manipulated
• the feeling of disrespect, trivialization of my work, and general "Cow-ness" by him

In the end, I didn't ask for his explanations, but he was trying to get them in anyway. I didn't want to hear it. I wanted to state my case and then leave. So I did.

I wonder if I should wear a bell tomorrow?