What's it all about, Alfie?

It's been 451 days, that's 1 year, 2 months, and 24 days since I walked into The Shop and started working. I had been working at my previous employer for 7 years, 8 months, and 9 days, so it's been a lot less, but it's been a far different experience here than it was there. In many ways, I like this place more - mostly the people, and in many ways, I like the last place more - mostly the business logic and drive.

But I've done a lot of good work here. I've done a lot of enabling as well, but that's part of the deal, and not my responsibility. As a worker, I'm supposed to carry out the wishes and plans of my management - so long as it's not criminal or business-threatening. It's management's job to make sure what they are asking me to do is what they need me to do, and will derive value from it. That's the contract.

So as I go through my last day today, I'll be thinking about all the things I might have done, and all the things I won't have to continue to do. As of right now, the advantages of leaving are far outweighing the disadvantages, and I'm honestly looking forward to being in a new group at the new job. The dynamics appear to be a lot more to my liking, and that hopefully bodes well for a better fit.

So it's the Last Day. The King is Dead. Long live the King.

This is really frustrating. This morning, out of nowhere, one of my Tomcat webservers was taking far longer to return data than normal. By 'far longer', I mean 5 to 10 seconds when yesterday it was sub-second. Really. It's a factor of 10x to 30x in response time. Horrible.

So I'm trying to figure this out. It's not a resource issue - got plenty of memory and CPU... it's not the code - it's running fine in London and NYC... it's not a simple data error - restarts and even different JVM GC parameters makes no difference at all.

It could be hardware. It could be networking. I seem to be getting no real help on either of these. It's pretty frustrating. My co-worker, Steve, who is taking over for me on this project, wants to put in another box to the mix and see if the problem clears up. If so, then it's the box. Or is it?

If they can't find the problem under load, are they going to fix it when it's isolated and not doing anything? Maybe. But the vendor isn't going to give us a replacement box "just because". It's got to be something. I'm just stumped as to what.

What's frustrating is that it appears to be a networking problem. I get the connection, but getting the contents from the Tomcat instance is exceptionally slow. I suppose, it really appears to be a software problem. Everyone says their stuff is working just fine, but still we have this delay in getting the data.

If it's not the code - and I haven't changed the code in weeks, then it might be the configuration. Nope, I checked that... it's configured exactly right. If it's not the code, and the data loads haven't changed that much, then how likely is it really "within" the box? Not a lot, in my book. But what else could it be?

Not fun.

UPDATE: a reboot of the box fixed it. Something went very wrong with the box. Oh well... can't do much about that.

[5/28] UPDATE: turns out, there's a problem with the stated defaults with Tomcat 6.0.18. In the conf/servlet.xml file there's a maximum number of acceptor threads for the incoming socket connections. It appears that the stated default of 200 was not, in fact, what the default was. Using jConsole, we were able to see that the actual value of maxThreads for the port 8080 connector was 40. We set it in the conf/server.xml:

  <Connector port="8080" protocol="HTTP/1.1"
             connectionTimeout="20000"
             redirectPort="8443"
             maxPostSize="0"
             maxThreads="400" />

and with this (and the automatic change in jConsole) the number of acceptor threads popped up from 40 to 166 and the speed was back. Amazing that the default wasn't what they said. Had that been the case, we would have been fine.

Good lesson to know.

To activate the monitoring for Tomcat assuming it's not already active, you need to add a few command-line options to the CATALINA_OPTS environment variable in either cataling.sh or the startup.sh script:

  export CATALINA_OPTS="-Dcom.sun.monitoring.jmxremote \
      -Dcom.sun.monitoring.jmxremote.port=7999 \
      -Dcom.sun.monitoring.jmxremote.ssl=flase \
      -Dcom.sun.monitoring.jmxremote.authenticate=false"

and then restart the Tomcat server. It's probably not a huge load, but you have to have it in order to connect with jConsole, so it's probably something you need to use unless you know it's not a great idea.

Honestly, I've been working at The Shop for about 14 months - the shortest job I've ever had outside college, and still I seem to know more than many of the "smart" folks that have been here for years and years. Why is it that people think they know so bloody much, and when they do, why don't they bother to ask as opposed to springing into immediate action and then showing the world how smart they are from averting certain destruction?

Had they asked, I could have told them that their ideas were good ones, but that they really weren't paying enough attention to the details of the situation and had overlooked the essential facts that made the situation not the end of the world. In fact, their change would have been the disastrous one.

I don't mind telling people this, but it's disheartening to say the least. On one hand, it's nice to see that people are checking things to make sure everything is OK. On the other hand, they aren't checking very well so their checks are causing more problems than if they just hadn't checked.

I've talked to a lot of my friends in this industry outside of The Shop they are amazed by what I've come to understand as "general practices" here. There's no need to list them all, it's one of the motivations for my leaving in about a week. But what's worse is that all these technical issues are being looked at by the same people that are checking the things I've been working on for the last year.

Because of this, I can't really honestly hold out a lot of hope for the technology decisions for this place. It needs to be gutted of it's primary technology folks, new ones brought in that understand what a global, multi-million dollar trading firm needs, and then staff up those people capable of achieving those goals. Because I promise you, the "stars" that don't catch the details in a start-up script are not going to be able to catch the nuances of a price feed or much of anything else, for that matter.

Business can be harsh... but it's not about harsh or friendly - it's about business, and if it's your money, do what you want. But if it's someone else's... well... then they have a right to deserve a little more.

This is an interesting experience. When I have given notice that I'm leaving at other companies, it's been "Thanks, let's get everything out of your head before you leave" and I'd go through a series of meetings - some with just a few folks, others with a much larger audience, and I'd present, they'd question me... You know - a typical debriefing. That's what I expected here.

I couldn't have been more wrong.

While there have been a few emails and a few conversations, the largest component of my transition period is doing work. I can see taking the time to finish up things that I can fit into the two weeks I have left. That makes sense - I'm spending a day or so on something that would take another developer several days to get up to speed on. It's just reasonable efficiency.

But Ralph is going much farther than that. He's asking me to start and finish new work. He also wants me to stay for three weeks. Well... following this logic, why don't I stay for another 3 years and do all the new projects?

Oh that's right... I quit!

This transition time is me being nice and professional, and in return, I'm being treated like a cow (again) - stay as long as I say... do all that I ask... Sorry Ralph, I don't think so.

So if it continues tomorrow, it's time to go have a talk with HR. This isn't what they want to have done, I'm sure. It's my final impression of this place, and they'd rather it end on nice, friendly terms, than have me walk out of here and tell everyone I know of the lunatic managers in this place.

This week marks the real beginning of the end for my work here at The Shop. Late Friday, I got word that my current CEO gave me his blessing to get an offer from another firm - a friendly one, and the Head of IT said that he'd be working on an offer letter for me. So this week I'm expecting some kind of letter which I'll respond to immediately and then put in my two-weeks notice here and work my way towards the new place.

It's nice to see a little progress on this front after so many weeks of interviewing with the new place, and talking about potential solutions to my reasons for leaving here. It's been very stressful. But hopefully now it's really starting to clear and I can see a little light at the end of the tunnel.

Lots more going on in my life these days, but one step at a time... one at a time.

They changed the designation of the Google Chrome (dev) stream from the 5.x to the 6.x stream today. I happened to notice when I decided to check for updates and there it was. Very interesting. I expect a big change on the "release" stream as well - jumping from 4.x to 5.x which would be a very welcome development for my work at The Shop.

Well... this morning I'm tinkering with Garbage Collection (GC) options for the JVM and I've seen two that look like they may have real promise to reduce these performance-robbing GC pauses in the current install on my development box. They are:

• -XX:+UseParallelGC - this guy is supposed to be a little more adaptive than the other, but it's got a limitation in that it can't function with some more advanced options. Alternatively, this guy is meant to work with very large heaps, which we have.
• -XX:+UseParNewGC - this guy is meant to be the successor to the other, and has no such limitations with more advanced features. The issue is whether or not it's really tuned to handle the larger Heaps, and if it's fixed algorithm is good enough.

What I'm hoping is just to get rid of the GC events where one CPU is pegged to 100% for more than a minute and everything else on the server just stops. If I can keep this guy running on a more even keel, it'll really help.

I'm running with the -XX:+UseParNewGC now and we'll see how it runs today. I can't believe it'll be worse, but the real question is will it remove the big pauses? We'll have to see.

UPDATE: I tried -XX:+UseParNewGC and I ran into the exact same issue. So I'm going to try -XX:+UseParallelGC and see if it's going to be any better. If not, I'll go back to the -XX:+UseParNewGC and see if I can try something else.

UPDATE: Yeah, the -XX:+UseParallelGC wasn't as good. I'm back to -XX:+UseParNewGC and Googling to see if there's something else I can try.

UPDATE: I saw this on Oracle's (Sun's) web site: it looks interesting:

  java -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=8 \
       -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31

with decent explanations of the options. The use of two different GCs is interesting, but they make the point that they are for different regions (young and tenured). Also, the target survivor ratio is at 90% - up from the default 50% making it more efficient on the long-term storage of which this app is heavily weighted.

More tests, hopefully these will look significantly better.

[5/14] UPDATE: In the end, the defaults were just as good, and while the memory footprint was a little bigger with the defaults, there were fewer pauses and the overall throughput of the application was better. In general, I guess they tune it about as well as it can be tuned. Shucks. But hey, it's "done", then.

I was working on a few problems this morning and ran into one that I thought I'd solved with a simple binary flag. Well... let's do the set-up.

We have a web page with a JavaScript timer on it, and every so often, it fires off a request to the server to get new data. It's a poor man's push notification system, but it works - expect when it doesn't. Case in point: what happens when the requests are taking longer to service at the server than the timer's interval? Well, they still get sent, and they stack up at the server, and slow things down even more.

It's a mess. We need to have some idea that the request is in process, and if so, we don't want to send anther to the server. It'd just make things worse. So my first cut looked a lot like this:

  var requesting = false;
 
  function refreshData() {
    if (!requesting) {
      requesting = true;
      ...
    }
  }
 
  function handleResponse() {
    ...
    requesting = false;
  }

And while it's not a good design, the idea is simple - have a global variable that indicates if there's a pending request, and if so, then just skip the current request as the timer will issue another in just a little bit. Should work great... and it does - except when it doesn't.

Where it fails is in the failure on the server. If the request fails, then the handleResponse() code might not be called. Some cases it will and some it won't - it depends on the library doing the call. In my case, that's the Google Visualization Query, and that seems to be about 50-50, depending on the server's condition.

When it fails, the flag will never get reset and from that point on, no requests will get sent to the server. That's not good. What I needed was a much more secure method of knowing when it's busy, with the ability to detect when it had failed.

What I came up with was the idea of a simple timeout. That, coupled with a global timeout limit should be able to do the trick:

  var requestTimeout = null;
 
  function refreshData() {
    var now = new Date();
    if ((requestTimeout == null) || (now > requestTimeout)) {
      requestTimeout = now;
      requestTimeout.setSeconds(requestTimeout.getSeconds() + ServletTimeout);
      ...
    }
  }
 
  function handleResponse() {
    ...
    requestTimeout = null;
  }

In this code sample, we're going to get the current date/time and compare the timeout to that value. If we've passed the timeout, or there is none, then we can hit it again - setting the new timeout to the appropriate distance in the future.

In my tests, this works great with servers that die on the page, and it also doesn't allow too many requests to buffer up on the server. Hopefully, there aren't any other gotchas that I missed with this guy.

I'm in the financial business, and as such, many people think I know something "special" about the markets or trading. I don't. What I know is how to build systems, and the little domain knowledge I need to do it well in this industry. However, the bounce that happened the other day was something totally unrelated to how the markets really work.

Something silly happened and programmed trades weren't smart enough to see that it was an isolated, very short term spike. Bad coders made the Big Bounce, nothing else. Maybe a little panic, but for the most part, bad coders.

Look at today:

All the losses are gone... things are back up, everything is working it's way back. It was a very obvious problem that didn't effect the fundamentals of any of the companies.

Well, this morning I got a nice shocker - several of my web tools were broken in London, but working fine in Chicago. Since the server is in Chicago, I started looking at the network as the issue. Everything here worked fine, but nothing there was working. Well... the basic page was loading, but nothing else.

What a pain. I was able to see the server was up and returning data. I was able to make it work for me. But they were dead. So I asked them to have a look at the JavaScript console log in Chrome and let me know what it said.

The answer was simply astounding: it couldn't find the Java applet on the page.

Now a bit of background for this one. My web app needs to have some sense of authorization. It's got P/L data for the entire organization for a long time. There's a desire to have this somewhat controlled. OK, fine by me, corporations can choose to filter their data as they see fit. I just needed to put something into the JavaScript to make it possible to read the logged in user running the web browser (Chrome or Firefox).

I solved this a while ago, but recently my teammate Steve decided that I didn't need to have the applet tag visible, and so he changed all the applet tags from:

  <applet archive="firecache.jar" code="Bridge"
   height=10 width=10>

to:

  <applet archive="firecache.jar" code="Bridge"
   height=1 width=1 style="display: none;">

He said I had made a mistake in leaving it visible, and too big. I remember doing tests on it, and thought I had tested this, but hey... maybe it's an update to Chrome, and it's working better now.

Yeah... I should have trusted myself.

It's important to realize that I have a multi-level process of getting the authenticated username: first, we check for a cookie, if that's not there, then we hit the applet, and if that's not available, then we throw up a login panel. The problem was, with the applet not accessible, the JavaScript stopped at that step and we never say the login panel.

So I changed the code to ignore the cookie, and quickly reproduced the problem. Now I knew what it was. Then I changed the HTML on the page to:

  <applet archive="firecache.jar" code="Bridge"
   height=1 width=1>

leaving it "small" is OK with me, but I have to have it "visible" so that it's accessible to the JavaScript engine.

When I did that, I had the page working again. Whew! All I had to do now was to fix up the thirty-some other pages and check it all in and put it in UAT.

It didn't take too long, but it was a pain to look stupid to the London users, but I took the bullet for Steve. It is my web app, not his, and while he didn't test his changes, I didn't either, and that's my fault. I'll be a lot more careful about Steve's work in the future.

I do not like looking stupid to the London users.