What's it all about, Alfie?

This morning I had the need of passing a completion function to a JavaScript function - I'm going to be doing an AJAX request, and when that request is done I want this function called. It's pretty obvious, really... when you're doing the request you might want a "Working..." message displayed, and when it's done, you want that removed. So here's the code I ended up with - pretty neat.

The calling function says:

  function showFitData(beg, end) {
    showFitsMessage('Working, please hold on...');
    var url = 'admin?cmd=fitData&out=json';
    // do the request and show the results
    doXMLRequestToTableWithFormat(url, fitTable, fitTableParams,
        function() { clearFitsMessage() });
  }

where we take the JavaScript function and encase it in another function definition. Pretty nice, but not as clean as I actually expected with JavaScript. Here nor there, this works and that's good enough.

The function called looks like this:

  function doXMLRequestToTableWithFormat(url, table, params, doneFcn) {
    var xhr = new XMLHttpRequest();
    xhr.open("GET", url);
    xhr.onreadystatechange = function() {
      // if we're not yet done, skip doing anything
      if (xhr.readyState != 4) {
        return;
      }
      // ...otherwise, get the tag for the answer
      var rawJSON = eval('(' + xhr.responseText + ')');
      var answerData = new google.visualization.DataTable(rawJSON);
      // let's get the 'negative as red' formatter for the columns
      var colorFmt = new google.visualization.ColorFormat();
      colorFmt.addRange(null, 0, 'red', null);
      // now let's process each numeric column, in turn
      for (var c = 0; c < answerData.getNumberOfColumns(); ++c) {
        if (answerData.getColumnType(c) == 'number') {
          colorFmt.format(answerData, c);
        }
      }
      // now we can draw it
      table.draw(answerData, params);
      // clean up what we no longer need
      delete answerData;
      delete rawJSON;
      delete colorFmt;
      // if we have a 'done' function, call it
      if ((doneFcn != undefined) && (doneFcn != null)) {
        doneFcn();
      }
  }

With this, it works like a charm. Nice to have worked this out. I'm sure I'll need it again and again.

Today has been one of those days that I didn't see coming, and while there have been a lot of good parts to it, there have been a few that really make me want to head for the door and never look back. A long time ago a good friend said to me that one day he had a talk with his manager that used to be a really good coder. This manager said to my friend: "The toughest thing about management is learning to accept other people's code."

Amen, brother!

Today I was once again hit in the face with the reality that even when you give "marginal" coders examples, they're as likely not to even understand that these are the examples they are to follow. No... most of the time they'll google something and see a new library, or something, and read the magic words in their mind, and lock down on that as the solution.

Now it would have been easy to ask the other folks in the group - or the creator of the system: "Hey, I need to be able to do this... is there anything like that in the code?" And get the hint as to where to look. But even that is more than the "marginal" coder will do.

So I had these XML/URL-based data pollers in the code. They worked fine. Easy, clean, no problem. And yet this wasn't enough for a teammate to use. Nope. What we really needed was a Java-based crontab system. Yup. That's it.

Meanwhile, there's nothing these add that the other doesn't - other than code bloat, more dependencies, and a larger memory footprint. But what the heck? Why not, right? I'm sure there's no question of the "best decision" in this person's mind.

Meanwhile, I'm having to write more code to allow for some truly horrible data sources that are sending malformed XML, and poor socket handling on the part of a really flakey web server based on another technology (not Apache or Tomcat, that's for sure).

So once again, I realize that there's nothing I can do. Not one blessed thing. This teammate is not about to understand that what they've done is weaken the project. Less efficient, less compact, harder to understand. They just don't get it - and that's something I can't change, either.

I had a user that liked the pages I'd created in my web app, but they really needed to be able to export the data to Excel so they didn't have to transcribe it. Makes sense, and given that I know Excel takes CSV data, and the DataTable I wrote in Java has an option to output CSV, it should be easy.

Well... that was a good idea... but it really wasn't too hard once you get a few things cleared up. First, the CSV format. I had my CSV output method, toCSV(), formatting the numbers which was a terrible mistake. I needed to stop that. Then I needed to escape the CSV data according to these basic CSV rules:

• if the data has a comma in it - surround it with double-quotes
• if the data has a double-quote in it, double it, and then surround it with double-quotes

The first one is obvious. If I have data like this:

then the CSV needs to look like:

  32.5,Reggie,"Indianapolis, IN",5

and if I had data like this:

then the CSV needs to look like:

  32.5,"Reggie ""Dog"" Johnson","Indianapolis, IN",5

OK, these were new little wrinkles for me, but it didn't take too much time to make these fixes to my DataTable Java class. Now I have the right format, but how do I set up the rest of the response so I have Excel automatically handle it without a lot of grief?

Well... it turns out it's the header and content type. Each of my servlet requests has a parameter associated with them called out. If it's json, then the output of the request is JSON. If it's html, then it's HTML. This really helps in the debugging of the servlets as I can add &out=html... to the URL and see the results in the browser. Nice.

Well... let's make another 'type' of out - excel. Then, in my servlet I can do the following:

  String   out = request.getParameter("out");
  if (out == null) {
    out = "html";
  }
 
  if (out.equalsIgnoreCase("json") || out.equalsIgnoreCase("html")) {
    response.setContentType("text/html");
  } else if (out.equalsIgnoreCase("csv")) {
    response.setContentType("text/csv");
  } else if (out.equalsIgnoreCase("excel")) {
    response.setHeader("Content-disposition", "attachment;filename=\"myDump.csv\"");
    response.setContentType("application/vnd.ms-excel");
  }

I was lucky to get the header and content type from a few google searches, and then it was just a matter of putting it all together. Once I get my DataTable I can then ship it back to the caller with the code:

  if (out.equalsIgnoreCase("json")) {
    response.print(ans.toJSON());
  } else if (out.equalsIgnoreCase("csv") || out.equalsIgnoreCase("excel")) {
    response.print(ans.toCSV());
  } else {
    response.print("<center>" + ans.toHTML() + "</center>");
  }

When I put this all together, the client doesn't need to know a thing. The request comes in for the URL, and the data comes back, intercepted by Excel, and it fills in the cells. Pretty neat.

I've got some massive web pages - lots of AJAX requests feeding thousands of data points back to the client for inclusion into some Google Visualization widgets - it's as heavy-duty as many visualization apps I've created. But therein lies a problem - memory usage.

One of my users wants to have a dozen of these pages open and hammering away all day long, and while that's nice, it's not a good idea from the resource consumption angle. It would be ideal if there were a JavaScript event that told me when the view was completely hidden. But I've looked for that and there's nothing I can find that is going to tell me when to shut down the updating, or when to start it back up again.

So I need to do some memory profiling of the pages and try to figure out why they are getting out of hand. So I started digging into what Google Chrome had, and was pleased to see that it was really WebKit that had what I needed. Well... close, anyway.

Within Chrome, there are a lot of nice tools to show you what's happening. Not a lot of help when dealing with Google Visualization widgets as their code goes through an 'optimizer' and the objects are then pretty useless for help, but it tells me that the problem is most likely in their stuff and the Strings that are being created in the moving of data back and forth between the server.

None of this is really Rocket Science, but it's nice to have quick access to the tools to be able to see the difference in a few memory dumps, and even a little code profiling. Not bad at all.

Today I spent the entire day cleaning up a few things that I forgot about the data from the Hemlock database, and then writing the Java loader of that data so that it's all loaded when it's needed, updated on a decent interval (love that database polling) and then available in a reasonably decent way to the servlets that will need it.

Lots of heads down coding today.

Today was spent digging into the SQL of the Hemlock database and trying to find the data the users are asking for, then trying to organize it in the servlet so that it's possible to access what we need very quickly and with a minimum of fuss. Given the way these things are laid out, I was stunned that I got as far along today as I did.

I was able to get the stored procedures written to pull the data for today, and the last week (to give you a hint, I have to go to one database for one, and another database for the other). I was also able to get the product data and the month/strike data. Really... I was pretty impressed with what I was able to get done.

Heads down... it can pay off.

Wonderful quote - horrible feeling if there's nothing to be gained. In this case, there wasn't. I had to once again dig into what I'm now calling Project Hemlock, the project I was given very early on at this job, and once again the lack of any documentation either written or comments in the code makes this a challenge I'd not wish to have.

There's something broken in a part of the Hemlock web app, and I need to fix it, but it's not a simple page - it's got business logic in the Flex client as well as the servlet gathering the data for the Flex client. It's not got a clear layout as to what it's doing or why, so the only way to follow the logic is to read each line of the code, understand what the variables are, why this might be happening, and then infer the function.

This logical single-stepping is very tiresome, and today I leave with very little done - more a point of a little more understanding, but still not enough to know where the problem is, or how to fix it. The only way I'm going to get this figured out is to pull in the original developer tomorrow, and have him tell me what's going on.

It's almost funny, but it's not. It's sad.

I've worked in a lot of different circumstances, and while I can function in just about any of them, there are some that certainly make it harder to excel and take personal initiative. Look at the military - it's not about personal initiative in the chain of command. It's about following orders, and while there may be room for personal interpretation in the execution of the orders, the fact that they are orders makes it pretty clear that the person following them is not expected to take responsibility for all of the consequences of the thought and planning that lead up to those orders - after all... he was given them.

Take another extreme, the entrepreneur. He's on his own. All decisions are his, all consequences are his. He's it. If there's a problem in the execution, or planning, or anything, he's all there is, and will live or die by those decisions.

When in corporate life, I've found that many times people want to believe they are in a military system ('...just follow my orders...') when what they might really need is a little more of the personal creativity and initiative that comes from some sense of ownership in the process.

No doubt - there are times that unpleasant things have to be done. They need to be done, and hopefully, the management will see that their role in this is just as much as those that might ultimately end up executing these unpleasant tasks. I've seen this handled very well, and I've seen it handled very poorly. I've seen the "boss" be there... after hours... walking around getting pizza and helping keep folks drinks filled and running to the printer for output. It's not a lot, but it shows that the manager believes he's part of this group and will be there pitching in however he can.

I've seen folks hand out the assignments and then walk away. Period.

One of the most frustrating situations I've been in lately was micro-management in a culture whose primary tenets include the entrepreneurial culture of self-empowerment. It was a classic case of the manager not "getting the memo" on how he needed to handle assignments and responsibilities in the larger scope of the organization.

It's hard to feel that it's going to be worth you going the extra mile, or coming up with a unique and interesting way to solve a problem when there's very little chance you'll get this done before management is over your shoulder asking you to explain this approach, and to compare it to this idea they had "last night". It's possible, that this manager is really good at the job, but most often times, that's not the case. It's possible that this employee is really playing off the reservation too much, but most often, that's not the case, either.

What I've found is that it's a misplaced sense of self-worth, or value to the company. A new manager will often think they need to be "hands-on" to help "their people" get the job done. But the opposite is often far closer to the ideal solution: back off... let the people come to him with questions, problems, clarify points... all those things that a only a manager can do. An employee typically doesn't need help doing their job, they need help getting their job done within the organization.

This is the "paper-pushing" that most managers hate. Well... guess what guys? That's what you're supposed to do! You're supposed to make each member of your team's like a little better. A little less full of paperwork and crud. That's your "value add" to the organization. They don't need you to type for them... or design for them. They need you to be their voice in the organization - their advocate.

And I guarantee that there's nothing that will crush creativity and initiative more than you, the manager, standing over their shoulder and asking them to explain every last detail of their work. If you wanted to know that badly, then grab a chair, check-out the code, and start coding yourself. But if that's not what you wanted to do, then realize that what you wanted is what you got. Accept it and live with it - or change, that's up to you.

I am not a manager now for exactly that reason. I enjoy the doing not the managing of the doers. I can do it, but it's just not as fun. So I choose not to. My choice. My consequences.

Just wish more managers understood this as well.

Like a lot of the Mac Faithful, I've been watching the discussions about Flash on the iPhone, and now the iPad, with interest and a little bit of a foot in both camps. There's a strictly technical viewpoint to think about, and then there's the reality when multi-billion dollar companies get involved.

From a technical standpoint, Wil Shipley said it best in this series of tweets:

Honestly, I don't understand Flash-haters. It's a content type. It's prevalent. The iPhone and iPad should understand it.

Want to know what we did at Omni when there wasn't a flash player for Rhapsody? WE WROTE OUR OWN FLASH PLAYER. It's not that hard.

Apple: Get off your ass and write a minimal Flash player. Everyone else: Stop acting like not being able to see video is cool.

And yet, the counter argument is valid as well: that anything controlled by one company (Adobe) can't be a real "standard" for the web as it's possible that it could be modified/licensed/changed by the company for their own commercial goals, and that's not something that should be supported and encouraged.

Wil is right - Apple could solve all this by putting in a Flash player into QuickTime. But then is that really in Apple's best (commercial) interests? Probably not. Then when Adobe changes, Apple has to change to keep "compatible". That is putting Apple in a constant game of catch-up. Not a great place to be. And then there's the legal angle. Say they did this, and then Adobe came after them. It's Adobe's property, after all, and while there's an open source Flex compiler, there's a lot that's missing from that project as well.

It's a battle of Titans, and we're pawns in this game. They are far more interested in control than they are in the customers. If Apple really wanted to make folks happy they'd work with Adobe to make Flash supported in QuickTime. Easy. But that means someone has to "flinch" first. Apple? Not likely. Adobe? Can't see that, either. But it would be pretty easy to clear this up.

On the other hand, it wouldn't bother me a bit of Flash became a niche player. Sure, there will be folks that will use it, but again, if it's below 80%, then it's just another plugin system. There will have to be others supported by the developers. Like YouTube's HTML5 video. Things like this will move in to create open alternatives to Flash.

So it wouldn't be horrible if the stand-off continued and HTML5, and it's successors, made Flash something less than a defacto standard. But then again, it's just a content type, after all... just support it, Apple... Or Perian... Someone.

I have been working today to try and solve a problem that a new datasource to the main web app I've been working on since I arrived at the Shop. All the other data sources I've had to use sent me a complete, consistent, state in one packet. This meant that I was able to put the data into multiple database tables with foreign keys and have the 'acquired time' be a link from one table to another. This made it easy to see what arrived when. But the latest data source is more like an event-driven system, and it's not being so considerate.

This new guy sends packets of information at each level - and they are totally dissociated from one another. It's like they are destined for cells in a spreadsheet, where the user is supposed to know that the goal is the most recent data, even if it's not directly comparable and consistent.

For instance, there are about a dozen groups, and each group has from less than a dozen to upwards of 75 symbols in it. Each of these pieces of information is arriving at different times, and there's no guaranteed consistency between them. This includes the elements of the group and the group totals. It's meant to be "the best I know, with the least bandwidth". I can see the reason for the design, but it's very, very different from what we have for the remainder of the data sources. And this is causing me grief.

No longer can I look at the arrival time of the group totals, and know that this represents the arrival time of all the elements in the group. Nope... they have their own arrival times. So in order to get the complete state of this data source, the SQL to fetch it out has become far more complex, and time-consuming. It was really getting to be annoying.

And then a co-worker said something that made it all click. "Just buffer it up".

Wonderfully simple. I don't know why I didn't think of it before. Well... probably because I was thinking of processing the data as it arrived, and we can't do that. What we can do is to take the data as it arrives, buffer it saving the arrival time as the 'generation time', and then when the group totals packet arrives, use that as the 'acquired time', and save it as well as all it's components to the database.

This makes it possible to see the individual arrival times (generation time), as well as link the data all together with the 'acquired time'. It made the "shotgun" data source fit into the existing mold that I'd created for the other data sources. Sure it was simple, but it was something that hadn't occurred to be, and I was better for having listened to him and understood how to synchronize things up.

In the end, I took about an hour to fix things up and it is working wonderfully. Very neat.