What's it all about, Alfie?

I had a user that liked the pages I'd created in my web app, but they really needed to be able to export the data to Excel so they didn't have to transcribe it. Makes sense, and given that I know Excel takes CSV data, and the DataTable I wrote in Java has an option to output CSV, it should be easy.

Well... that was a good idea... but it really wasn't too hard once you get a few things cleared up. First, the CSV format. I had my CSV output method, toCSV(), formatting the numbers which was a terrible mistake. I needed to stop that. Then I needed to escape the CSV data according to these basic CSV rules:

• if the data has a comma in it - surround it with double-quotes
• if the data has a double-quote in it, double it, and then surround it with double-quotes

The first one is obvious. If I have data like this:

then the CSV needs to look like:

  32.5,Reggie,"Indianapolis, IN",5

and if I had data like this:

then the CSV needs to look like:

  32.5,"Reggie ""Dog"" Johnson","Indianapolis, IN",5

OK, these were new little wrinkles for me, but it didn't take too much time to make these fixes to my DataTable Java class. Now I have the right format, but how do I set up the rest of the response so I have Excel automatically handle it without a lot of grief?

Well... it turns out it's the header and content type. Each of my servlet requests has a parameter associated with them called out. If it's json, then the output of the request is JSON. If it's html, then it's HTML. This really helps in the debugging of the servlets as I can add &out=html... to the URL and see the results in the browser. Nice.

Well... let's make another 'type' of out - excel. Then, in my servlet I can do the following:

  String   out = request.getParameter("out");
  if (out == null) {
    out = "html";
  }
 
  if (out.equalsIgnoreCase("json") || out.equalsIgnoreCase("html")) {
    response.setContentType("text/html");
  } else if (out.equalsIgnoreCase("csv")) {
    response.setContentType("text/csv");
  } else if (out.equalsIgnoreCase("excel")) {
    response.setHeader("Content-disposition", "attachment;filename=\"myDump.csv\"");
    response.setContentType("application/vnd.ms-excel");
  }

I was lucky to get the header and content type from a few google searches, and then it was just a matter of putting it all together. Once I get my DataTable I can then ship it back to the caller with the code:

  if (out.equalsIgnoreCase("json")) {
    response.print(ans.toJSON());
  } else if (out.equalsIgnoreCase("csv") || out.equalsIgnoreCase("excel")) {
    response.print(ans.toCSV());
  } else {
    response.print("<center>" + ans.toHTML() + "</center>");
  }

When I put this all together, the client doesn't need to know a thing. The request comes in for the URL, and the data comes back, intercepted by Excel, and it fills in the cells. Pretty neat.

I've got some massive web pages - lots of AJAX requests feeding thousands of data points back to the client for inclusion into some Google Visualization widgets - it's as heavy-duty as many visualization apps I've created. But therein lies a problem - memory usage.

One of my users wants to have a dozen of these pages open and hammering away all day long, and while that's nice, it's not a good idea from the resource consumption angle. It would be ideal if there were a JavaScript event that told me when the view was completely hidden. But I've looked for that and there's nothing I can find that is going to tell me when to shut down the updating, or when to start it back up again.

So I need to do some memory profiling of the pages and try to figure out why they are getting out of hand. So I started digging into what Google Chrome had, and was pleased to see that it was really WebKit that had what I needed. Well... close, anyway.

Within Chrome, there are a lot of nice tools to show you what's happening. Not a lot of help when dealing with Google Visualization widgets as their code goes through an 'optimizer' and the objects are then pretty useless for help, but it tells me that the problem is most likely in their stuff and the Strings that are being created in the moving of data back and forth between the server.

None of this is really Rocket Science, but it's nice to have quick access to the tools to be able to see the difference in a few memory dumps, and even a little code profiling. Not bad at all.

Today I spent the entire day cleaning up a few things that I forgot about the data from the Hemlock database, and then writing the Java loader of that data so that it's all loaded when it's needed, updated on a decent interval (love that database polling) and then available in a reasonably decent way to the servlets that will need it.

Lots of heads down coding today.

Today was spent digging into the SQL of the Hemlock database and trying to find the data the users are asking for, then trying to organize it in the servlet so that it's possible to access what we need very quickly and with a minimum of fuss. Given the way these things are laid out, I was stunned that I got as far along today as I did.

I was able to get the stored procedures written to pull the data for today, and the last week (to give you a hint, I have to go to one database for one, and another database for the other). I was also able to get the product data and the month/strike data. Really... I was pretty impressed with what I was able to get done.

Heads down... it can pay off.

Wonderful quote - horrible feeling if there's nothing to be gained. In this case, there wasn't. I had to once again dig into what I'm now calling Project Hemlock, the project I was given very early on at this job, and once again the lack of any documentation either written or comments in the code makes this a challenge I'd not wish to have.

There's something broken in a part of the Hemlock web app, and I need to fix it, but it's not a simple page - it's got business logic in the Flex client as well as the servlet gathering the data for the Flex client. It's not got a clear layout as to what it's doing or why, so the only way to follow the logic is to read each line of the code, understand what the variables are, why this might be happening, and then infer the function.

This logical single-stepping is very tiresome, and today I leave with very little done - more a point of a little more understanding, but still not enough to know where the problem is, or how to fix it. The only way I'm going to get this figured out is to pull in the original developer tomorrow, and have him tell me what's going on.

It's almost funny, but it's not. It's sad.

I was talking to a co-worker yesterday about several things that are scheduled for the coming year, and we got around to a very significant re-write that I've advocated for about as long as I've been at the Shop. It's something that's needed very badly, but the problem is that it would take several months to get a re-write to the point of the existing application (which I support). This delay of months for "nothing" new, while it's still possible to add things to the old application, makes it a very hard sell. So hard, in fact, that I've become convinced that it'll never happen.

Really? You ask.

Really.

Why? You ask, and I'll be glad to tell you.

'Good enough' is the Enemy of Great. - Anonymous

Doesn't sound right, and in fact, the original saying is French:

'Better' is the Enemy of 'Good Enough'

and that is true, but anything - taken to an extreme, is bad. And this motto is no exception. Imagine this: a company starts off small, hungry, willing to take risks to get things done. They succeed, they grow, and life is good. Now the company is doing "good enough". It could certainly be worse - many people are still there that remember the lean days. So the willingness to risk like they did in the past is lost. They don't want to risk the "good enough" for the possibility that things get worse.

There's nothing wrong with this... it's up to the people involved. If you have achieved something, and wish to protect it, there's nothing wrong with that. But don't fool yourself. Someone out there is going to be in the same place you once were, and they will be ready, and willing, to risk a little in order to gain a little more. If you can protect your gains, that's great. But if you're still in a competitive environment, you've just become the target, and you're standing still.

Dead meat.

So rather than grow, you stagnate. Maybe that's OK, and you get out of the business altogether. Lots do. Nothing wrong with it. But if you expect to stay a leader, then you need to act like a leader, and that means you need to continue to improve and take the same kind of (appropriate) risks that you did to get to this point.

Our conversation got to this point on many fronts of the Shop. It seemed very puzzling to me why this would be the case with the original founders still with the company. Why would they stop? And as I was thinking about it, it might be because they (and others) are still with the Shop.

They all remember the "bad days", and so are fearful of returning to them. They aren't thinking that today is a new baseline for the future. They're probably thinking that today is 'good enough' - compared to not too long ago. And that's not a good philosophy to run a business by. Pretty soon, your competitors are going to make your 'good enough' a lot more like the 'bad old days' because you're too fearful of the kind of risk that got you to this place in the beginning.

As a university professor, I didn't understand why a university would not hire someone that got all their degrees from that school. I mean, who better to lead the school than someone that's been through it? But that's exactly the person you don't want. Get some new blood in there... see the world from a different perspective... refine the vision. If you're managing people in an international business and the only place you've ever worked is the one employer, then you're doing everyone a grave disservice. You need to move on.

Of course, this isn't welcome opinion, but it's good advice. You are a product of the environment, and as it has grown, you have grown, but you have no basis of comparison for your actions. When people come in to a business, and you only know how this one place functions, you have no idea what they are expecting. Even if you ask them, the myopic views you'll have will end up causing the 'good enough' to slip in and degrade the business.

There was a question I didn't understand when taking all those entrepreneur tests - What's your exit strategy?. Mine was always: Retire, but that's a bad plan, and it misses the reason for the strategy: an exit. If it gets really successful, you are most likely going to have to take yourself out of the day-to-day operation as it's going to outgrow you. I see it now, and I'll be sure not to make that mistake, but it's a lesson I think several people here should take to heart.

I've worked in a lot of different circumstances, and while I can function in just about any of them, there are some that certainly make it harder to excel and take personal initiative. Look at the military - it's not about personal initiative in the chain of command. It's about following orders, and while there may be room for personal interpretation in the execution of the orders, the fact that they are orders makes it pretty clear that the person following them is not expected to take responsibility for all of the consequences of the thought and planning that lead up to those orders - after all... he was given them.

Take another extreme, the entrepreneur. He's on his own. All decisions are his, all consequences are his. He's it. If there's a problem in the execution, or planning, or anything, he's all there is, and will live or die by those decisions.

When in corporate life, I've found that many times people want to believe they are in a military system ('...just follow my orders...') when what they might really need is a little more of the personal creativity and initiative that comes from some sense of ownership in the process.

No doubt - there are times that unpleasant things have to be done. They need to be done, and hopefully, the management will see that their role in this is just as much as those that might ultimately end up executing these unpleasant tasks. I've seen this handled very well, and I've seen it handled very poorly. I've seen the "boss" be there... after hours... walking around getting pizza and helping keep folks drinks filled and running to the printer for output. It's not a lot, but it shows that the manager believes he's part of this group and will be there pitching in however he can.

I've seen folks hand out the assignments and then walk away. Period.

One of the most frustrating situations I've been in lately was micro-management in a culture whose primary tenets include the entrepreneurial culture of self-empowerment. It was a classic case of the manager not "getting the memo" on how he needed to handle assignments and responsibilities in the larger scope of the organization.

It's hard to feel that it's going to be worth you going the extra mile, or coming up with a unique and interesting way to solve a problem when there's very little chance you'll get this done before management is over your shoulder asking you to explain this approach, and to compare it to this idea they had "last night". It's possible, that this manager is really good at the job, but most often times, that's not the case. It's possible that this employee is really playing off the reservation too much, but most often, that's not the case, either.

What I've found is that it's a misplaced sense of self-worth, or value to the company. A new manager will often think they need to be "hands-on" to help "their people" get the job done. But the opposite is often far closer to the ideal solution: back off... let the people come to him with questions, problems, clarify points... all those things that a only a manager can do. An employee typically doesn't need help doing their job, they need help getting their job done within the organization.

This is the "paper-pushing" that most managers hate. Well... guess what guys? That's what you're supposed to do! You're supposed to make each member of your team's like a little better. A little less full of paperwork and crud. That's your "value add" to the organization. They don't need you to type for them... or design for them. They need you to be their voice in the organization - their advocate.

And I guarantee that there's nothing that will crush creativity and initiative more than you, the manager, standing over their shoulder and asking them to explain every last detail of their work. If you wanted to know that badly, then grab a chair, check-out the code, and start coding yourself. But if that's not what you wanted to do, then realize that what you wanted is what you got. Accept it and live with it - or change, that's up to you.

I am not a manager now for exactly that reason. I enjoy the doing not the managing of the doers. I can do it, but it's just not as fun. So I choose not to. My choice. My consequences.

Just wish more managers understood this as well.

Like a lot of the Mac Faithful, I've been watching the discussions about Flash on the iPhone, and now the iPad, with interest and a little bit of a foot in both camps. There's a strictly technical viewpoint to think about, and then there's the reality when multi-billion dollar companies get involved.

From a technical standpoint, Wil Shipley said it best in this series of tweets:

Honestly, I don't understand Flash-haters. It's a content type. It's prevalent. The iPhone and iPad should understand it.

Want to know what we did at Omni when there wasn't a flash player for Rhapsody? WE WROTE OUR OWN FLASH PLAYER. It's not that hard.

Apple: Get off your ass and write a minimal Flash player. Everyone else: Stop acting like not being able to see video is cool.

And yet, the counter argument is valid as well: that anything controlled by one company (Adobe) can't be a real "standard" for the web as it's possible that it could be modified/licensed/changed by the company for their own commercial goals, and that's not something that should be supported and encouraged.

Wil is right - Apple could solve all this by putting in a Flash player into QuickTime. But then is that really in Apple's best (commercial) interests? Probably not. Then when Adobe changes, Apple has to change to keep "compatible". That is putting Apple in a constant game of catch-up. Not a great place to be. And then there's the legal angle. Say they did this, and then Adobe came after them. It's Adobe's property, after all, and while there's an open source Flex compiler, there's a lot that's missing from that project as well.

It's a battle of Titans, and we're pawns in this game. They are far more interested in control than they are in the customers. If Apple really wanted to make folks happy they'd work with Adobe to make Flash supported in QuickTime. Easy. But that means someone has to "flinch" first. Apple? Not likely. Adobe? Can't see that, either. But it would be pretty easy to clear this up.

On the other hand, it wouldn't bother me a bit of Flash became a niche player. Sure, there will be folks that will use it, but again, if it's below 80%, then it's just another plugin system. There will have to be others supported by the developers. Like YouTube's HTML5 video. Things like this will move in to create open alternatives to Flash.

So it wouldn't be horrible if the stand-off continued and HTML5, and it's successors, made Flash something less than a defacto standard. But then again, it's just a content type, after all... just support it, Apple... Or Perian... Someone.

I have been working today to try and solve a problem that a new datasource to the main web app I've been working on since I arrived at the Shop. All the other data sources I've had to use sent me a complete, consistent, state in one packet. This meant that I was able to put the data into multiple database tables with foreign keys and have the 'acquired time' be a link from one table to another. This made it easy to see what arrived when. But the latest data source is more like an event-driven system, and it's not being so considerate.

This new guy sends packets of information at each level - and they are totally dissociated from one another. It's like they are destined for cells in a spreadsheet, where the user is supposed to know that the goal is the most recent data, even if it's not directly comparable and consistent.

For instance, there are about a dozen groups, and each group has from less than a dozen to upwards of 75 symbols in it. Each of these pieces of information is arriving at different times, and there's no guaranteed consistency between them. This includes the elements of the group and the group totals. It's meant to be "the best I know, with the least bandwidth". I can see the reason for the design, but it's very, very different from what we have for the remainder of the data sources. And this is causing me grief.

No longer can I look at the arrival time of the group totals, and know that this represents the arrival time of all the elements in the group. Nope... they have their own arrival times. So in order to get the complete state of this data source, the SQL to fetch it out has become far more complex, and time-consuming. It was really getting to be annoying.

And then a co-worker said something that made it all click. "Just buffer it up".

Wonderfully simple. I don't know why I didn't think of it before. Well... probably because I was thinking of processing the data as it arrived, and we can't do that. What we can do is to take the data as it arrives, buffer it saving the arrival time as the 'generation time', and then when the group totals packet arrives, use that as the 'acquired time', and save it as well as all it's components to the database.

This makes it possible to see the individual arrival times (generation time), as well as link the data all together with the 'acquired time'. It made the "shotgun" data source fit into the existing mold that I'd created for the other data sources. Sure it was simple, but it was something that hadn't occurred to be, and I was better for having listened to him and understood how to synchronize things up.

In the end, I took about an hour to fix things up and it is working wonderfully. Very neat.

For the last two days I've been busy just slugging it out in the trenches. Nothing fun, nothing exciting, just lots of drudge work that needed to be done, and I'm faster at doing it than anyone else, so I might as well just shut up, suck it up, get my head down and push through it.

Nothing exciting and glamorous about the last two days. Nothing.