What's it all about, Alfie?

One of the things that's really quite limiting about the current demand forecast data from another system at The Shop is that it's not a month-by-month demand projection. They took the year's sales data and then extrapolated for the next month. But in doing so, they have removed any monthly/seasonal trends so that the demand for Boat Tours in Chicago is the same in July as January. Clearly, absurd. Unfortunately, there doesn't seem to be the will to push a change in and fix the Demand in a reasonable timeframe. So I had to add in some kind of demand adjustment based on the month and some sense of the time to close a deal.

What I came up with was a series of factors - one per month, that could be individually adjusted in a GUI - like an equalizer, and then we'd take each of these and multiply the given demand by this factor and divide by 100 (converting it into a percentage). In this way, we can ramp things up, and shut them down, a month at a time.

If they want to boost something up, they can pretend it's seasonal, and use move everything up to 200%… if they want to suppress something, then move all months to 25% - simple. It's a great plan, and we can "seed" the data with the sales data by service, by month.

What I need to do is to start with the code and then work back as I'm not about to do the GUI first - I'm no artist, and that's what this is going to take to make it look really nice.

This morning we once again had a series of production problems caused by Salesforce.com - this time, it was that a class had been deployed to the production instance and the permissions hadn't been set properly. How I figured this out - I don't know. I mean, I know how I figured it out - I looked, but to look was a stroke of luck.

I certainly believe in things we can't see and touch, so I'm not against divine intervention, but I just have no idea why He'd want me to figure this out so quickly. In any case, all the calls to updating the merchant data were failing, and the error was not the same one as I'd seen previously, so seeing no one in the Salesforce support group in at 7:00 am, I decided to start digging and see if I could find the problem.

Thankfully, I remember the support crew checking these in the last go-round, so I thought I'd give it a go. Turns out I was right. Still took me until 9:00 am to get everything re-run for the day. Nothing I could do about it, but without checking on their part, and them coming in later, there's not much else I could do.

Today my day started out with a rather early meeting for my work here at The Shop - 9:30 am, and it was about an interesting topic - the seasonal adjustment of the demand data we're using in our project. The problem is that the demand data is based on the previous 12-months historical sales data - all annualized. This means that boat tours should sell as well in July as December. Nope. So how to do this?

I'd already suggested in an email that we make a simple web app that allows the right people to add the important location and taxonomy data as well as a simple 12-month segmented graph (or even sliders or dials) so that the users could control the seasonality for their division and their sales reps. It makes sense - boats trips in Arizona are not down in December, so we need to really look at all the factors that might contribute to the seasonality of the demand.

Then we need to overlay this on the incoming demand and we're in business.

There's also the need for a manual demand entry screen where the users can input demand that they anticipate, and let them run with that. All sounds pretty decent. Very do-able.

But the proposal was that we base it on some Google Doc and parse out the data from a spreadsheet.

That's a horrible idea. I've had to do it so many times, and it always becomes a nightmare. The project manager thought it would be "easy" and "fast"… but all it really does is move the development effort from building the UI and data integrity checks to the data parsing and processing. There's no savings here. But Holy Cow! That was a 30 min meeting that took 90 min because this guy could not get the idea out of his head that this wasn't the way to do it.

I consider myself a decent communicator, but I've come to a loss with this guy. I think he's just not a good listener, but who knows. In any case, at the end of the 90 min meeting I was able to get some support from another developer manager and I think we're going to do the "Right Thing". But it was not easy.

Now I think I'm going to have to come up to speed on Rails, as I think we're going to build this in Rails, but at least it's going to be done Right.

Today has been one of those days… it started out with some wonderful steel cut oatmeal at the cafeteria… I had no idea they had that! It was wonderful. But from there on, it was a steep descent downhill.

We are coming off the horrible morning and then trying to get a good number of features and fixes into the code for the afternoon release to production. I like to keep releasing something off master to production each day as that allows the users and project manager to see some visible progress each day. Big or small, it makes little difference to be able to point out the changes we have made based on their feedback and requests.

Often times, though, this means doing a lot of nasty work to follow up with people that dropped the ball (but doing so in a way to make them feel like they are doing you a favor), cleaning up problems and messes left by others, and all the little ick work that comes with software development.

Today has been that day for me.

I'm glad it's over.

On the up-side… we are releasing something far better than yesterday, and the really great ones never loose sight of the fact that it's all in the details.

This morning I've been fighting off a few things and while I'm doing better at being able to handle the slings and arrows these days, I was given a wonderful reminder from twitter:

In 100 years I won't be here. My kids won't either. Their kids? Probably, but that's only if they live well. There may not be a soul on this planet that remembers what I'm doing in this life, and that's OK.

Who was working in the machine shop (the high-tech equivalent of today) in GE's plant 100 years ago? No idea. He worked hard, tried to raise a family, be a good husband, and some did well, others not so much. But today, they are long forgotten.

Carpe Diem. 'Nuff said.

I love it when I'm able to work with people that see the same Big Picture as I do. It doesn't happen often, but when it does, it's almost magical. The next best thing is to work with someone that can see some Big Picture - even if it's different from mine, and then we can hash out the differences and come to some accord with how to get to that endpoint.

Some of the most frustrating people to work with are those that simply are incapable of seeing the Big Picture. Maybe they don't think in those abstract terms. Maybe they don't think there is such a thing. Maybe they aren't looking at what's being done as much as how it's done. For whatever reason, I'm in the midst of trying to make someone see the Big Picture, and it's something that before too long, I'm just going to give up on.

This isn't a critique of the person… it's the old adage:

Don't try to teach a pig to sing - it only frustrates you and annoys the pig.

if a person isn't going to see what you want them to see - for whatever reason, then it's time to just stop and let them be the person they want to be. If they have the capability of seeing it, and just don't want to, then maybe, someday, they'll change their mind and come to you seeking out your advice then.

If they can't see it at all, you're annoying them, and if at some point in the future, they want to see try again, they will again seek you out, and try again.

But until then, it's just a problem - for you and for them. Better to accept them as they are and move on. No amount of cajoling, pleading, arguing is going to make an adult change their mind. They have to come to that decision themselves.

So I'm trying to convince myself that the right time to let go is now.

Right now.

This morning I spent all morning struggling with some production issues. The runs didn't complete, and I had to dig into the logs to find out why. Here, again, the way a lot of the Ruby devs function really hurts maintenance. This optimistic coding is something I've fought for a great number of years, and it seems that it's really systemic, or maybe endemic to the industry. People want to think "This works… and if it doesn't then it's not my fault". This might be true, but that doesn't make it right.

So first thing was figuring out what was wrong with the data. It seemed to be a data problem, so that's where I started digging. Pretty soon, I realized that the source of the data - Salesforce.com, wasn't returning the data - saying that the HTTP GET was invalid, but a POST was acceptable. I looked at the code, saw where we were doing GETs and figured out that we had the ability to do POSTs as well - changed them, retried, and still no good.

Got onto Campfire to explain the situation and try to find help. Clearly, something with Salesforce.com changed overnight and it was now no longer accepting the calls that were working yesterday.

After a lot of failed attempts, I was finally able to convince myself that there was nothing wrong with our code - that it was Salesforce.com that was simply refusing the API calls we had made yesterday. I was able to confirm this with one of our Salesforce support guys, and he thought he knew the problem, but not the solution. So off he went to figure it out.

In the end, Salesforce requires that when you deploy code, you have to manually recompile everything - or manually run all the tests to activate all the URLs in the code. Interesting.

Once that was fixed, the calls worked and everything was able to run. I finished the production runs at about 11:00 am.

What a morning.

Today I did a little digging on the idea of exporting a Google Visualization Table to CSV all in javascript. Face it - the table is already there… it's got the data… the trick is how to get it all up and going for the CSV export. Well… as it turns out, it's not all that hard. I was pretty surprised.

The core of it is really the Google Visualization DataTable. Since that's the core of most of the Visualizations, that's a great universal starting point. What we're really doing in the code is making a simple javascript method that will make a URI and encode it, such that when it's opened, it'll appear as a download to the browser and be kept as a file.

The first part is to save the DataTable when you render the Google Table on the page:

  // this is the Google DataTable we'll be creating each time
  var dtable = null;
 
  // This method looks at the selected data set and loads that into
  // a new table for the target div and redraws it.
  function render(tbl) {
    // save this data table for later
    dtable = tbl;
    // now create a Google Table and populate it with this data
    var dest = document.getElementById('table_div');
    var table = new google.visualization.Table(dest);
    table.draw(tbl, table_config);
  }

At this point, we have the DataTable, and then we can place the button anywhere on the page, I happened to place it, centered at the bottom of the page:

  <p align="center">
    <input type="button" id="toCSV" value="Click to download data as CSV"
     onclick="toCSV()" />
  </p>

So that when the user clicks on the button the following code will be run:

  // this downloads the current data table as a CSV file to the client
  function toCSV() {
    var data = dtable;
    var csvData = [];
    var tmpArr = [];
    var tmpStr = '';
    for (var i = 0; i < data.getNumberOfColumns(); i++) {
      // replace double-quotes with double-double quotes for CSV compatibility
      tmpStr = data.getColumnLabel(i).replace(/"/g, '""');
      tmpArr.push('"' + tmpStr + '"');
    }
    csvData.push(tmpArr);
    for (var i = 0; i < data.getNumberOfRows(); i++) {
      tmpArr = [];
      for (var j = 0; j < data.getNumberOfColumns(); j++) {
        switch(data.getColumnType(j)) {
          case 'string':
            // replace double-quotes with double-double quotes for CSV compat
            tmpStr = data.getValue(i, j).replace(/"/g, '""');
            tmpArr.push('"' + tmpStr + '"');
            break;
          case 'number':
            tmpArr.push(data.getValue(i, j));
            break;
          case 'boolean':
            tmpArr.push((data.getValue(i, j)) ? 'True' : 'False');
            break;
          case 'date':
            // decide what to do here, as there is no universal date format
            break;
          case 'datetime':
            // decide what to do here, as there is no universal date format
            break;
          case 'timeofday':
            // decide what to do here, as there is no universal date format
            break;
          default:
            // should never trigger
        }
      }
      csvData.push(tmpArr.join(','));
    }
    var output = csvData.join('\n');
    var uri = 'data:application/csv;charset=UTF-8,' + encodeURIComponent(output);
    window.open(uri);
  }

You can see the entire page here:

The downside of this is that the file will have an unusual name. On Mac OS X with Safari 6.0.1, it's "Unknown". On other platforms, I'm sure it's something nearly as odd and useless, but that's the name of the game. There's seemingly no way to get the name of the file in the URI or the window.open() method.

Still… I'm pretty pleased. We're looking at a 100% client-side, javascript solution to the CSV generation problem. That's pretty nice. If you look at the code, there's really very little that's exclusive to the Google DataTable - it's really just the means to get the headers, and the row and column data. We could have easily built this from any regular data source and made that work as well.

Sweet.

While we were messing with getting merchant data from Salesforce.com, we had another little story in Pivotal Tracker about replacing the write-back of demand gap analysis with a simple report on our Metrics Site - a series of web pages we've put together for viewing the metrics of the runs.

Given that we've had a lot of timeouts in these specific write-backs, it made a lot of sense to me to get rid of the write-backs and just have a report for the merchant researchers to use. It was a pretty simple page to throw together - I had to update the CouchDB view to change the key of the view from:

  ["2012-09-24_14:51:21.232_cleveland", "cleveland", "Food & Drink"]

to the simpler:

  ["2012-09-24_14:51:21.232_cleveland", "Food & Drink"]

because we really didn't need the division in the key, but we really needed to be able to specify the key as well as have a nice reduce set of the counts by date/time and category of merchant. That took a bit to regenerate, but when it was done, I had everything I needed - or so I thought.

The code for generating the URL was pretty simple:

  var et = run_opt.value + '_' + div_opt.value;
  var cat = cat_opt.value;
  var url = svr_opt.value + '?key=' + JSON.stringify([et,cat]);

and it wasn't obvious to me, but the ampersand in the Food & Drink was going to be a real pain in the rump for me because it's a valid JSON code, but it's also the argument separator in the URL. So I had to do a little modification:

  var et = run_opt.value + '_' + div_opt.value;
  var cat = cat_opt.value.replace("&", "%26");
  var url = svr_opt.value + '?key=' + JSON.stringify([et,cat]);

to get the ampersand into a hex value for sending to the server.

With this, I was all ready to go. I pushed the view changes to the prod and UAT CouchDBs, so in the morning, first thing, I'll be able to check it all in and then push the new pages to the server.

I sent a note to the project manager asking him if he really would prefer this, and why it'd be great for us to not have the write-back to Salesforce.com… I'm hoping this is going to work out really well.

Today we've been once again fighting Salesforce.com's APEX APIs for getting Merchant data from their servers. It's a key part of our processing, and they have an API, but it's got some major limitations, and one of the problems is getting all the merchants for a division (like Cleveland, Cincinnati, etc.) of The Shop. As we were kicking around ideas, one of my co-workers came up with the idea to fetch all the IDs first, and because there's not a lot of data in just the IDs, it's possible to get all the merchant IDs in one hit.

Then, once we have the IDs, we can make a series of parallel requests where we chop this up into requests of 100 or 1000 merchants a piece. There are a lot of benefits in this plan - first, we know the IDs right off the bat, and if we don't get valid data for them, at least we know what to clear if we need to. Secondly, by knowing all the IDs up front, we can make a series of async, parallel requests that will make the data loading a lot faster.

It's days like this, with pretty smart guys that I get a giggle. This is going to take a few hours to do, but in the end, it'll be a huge improvement in the processing of the data.

Fantastic!