What's it all about, Alfie?

Today I was looking at the check-ins and saw something that was a good idea, but it wasn't really completed. I knew the guy on the team that did it, and I could easily see why he'd stopped half-way through: he didn't see what he'd really started as something more than a little hack to the problem he had. This is one of the things I'd like to be able to teach people, but I'm worried that it's really something that people have to want to see.

Initially, we had one source for merchant data, and one source for demand data. To be fair, we had two: fixtures (static files that make offline testing nice), and a "live" source. We could switch between these with a simple config option:

  # control whether to use the real data sources
  use_fixtures: true

or:

  # control whether to use the real data sources
  use_fixtures: false

But we're moving forward, and it is very reasonable to think that we have multiple sources for each of these. Right now, we have the need to think of multiple sources for the merchant data, while just two for the demand. The coder implementing the need for the multiple sources for merchants added a config option:

  # control whether to use the real data sources
  use_fixtures: false
  # select from 'fixtures' | 'sf' | 'm3'
  merchant_from: 'sf'

to indicate that the merchant data is coming from Salesforce. We can add other options in the code, and this is very nice and extensible.

The problem is that this really doesn't address the logic inconsistency of having those two options in the same file. If we're using Salesforce for the merchant data, what does "using fixtures" mean? Why not just add:

  # select from 'fixtures' | 'sd'
  demand_from: 'fixtures'
  # select from 'fixtures' | 'sf' | 'm3'
  merchant_from: 'sf'

so that we're specifying all the inputs the same way?

This is the finished idea. What was done was only really half the problem solution, and in leaving it half done, it really lead to a lot of confusion that just didn't need to be there. It's this insight that I'd love to be able to teach coders that don't see it.

But given that this coder is way past 20, I'm guessing he doesn't consider himself "junior", and therein lies the other problem: He's not thinking he needs to learn this. Sad, but true.

Anyway… I have to check these changes because of just this level of incompleteness.

This is something that I find very interesting about people in the business world - their almost complete reliance on Excel. It's as if they can't even function without it. Case in point, JSON data. Here's a section of a file that I sent to a business person today:

  {
    "cleveland": {
      "Academic Tutor": [100.0, 100.0, 100.0],
      "Airplane": [100.0, 100.0, 100.0],
    }
  }

and there are clearly more lines in the file than this, but they all follow this same pattern. I thought it was pretty easy to understand: City, then Service, and then monthly values (12 versus the 3 I show).

But once again, I'm surprised.

The response was "Can you put it into something I can export in Excel?"

Forgetting that they don't know the difference between import and export, I find it really hard to look at the file above and see how it's different from Excel - save the graphs. But you know… that's the tool they know how to use, and that means that's the tool I need to give them data for.

So I modified my code to spit out CSV or JSON.

Worked just fine, but I'm still wondering what the real limits are for a lot of business people.

This morning I saw that the Google Chrome team has upped the major version number of Chrome to 24.0.1284.2 with a few nine updates, and several bug fixes - notably a few Flash issues, crashes, and a new version of WebKit and the V8 javascript engine. Nicely done. I'm curious why this qualifies as a major release number, but I think it has more to do with when the version gets stable enough to deploy out to the masses, then the dev version has to up it's number to be unique. Sure seems like it anyway.

Well… nicely done to the Chrome team!

In the original code of the seasonality adjustment to the raw demand data we're getting, I had added in a fixed 6 weeks lead time for getting and closing deals. This was given to me by the project manager, and I used it because I didn't know any better number to use. What this allows us to do is to anticipate the demand rise (or fall), and adjust who to call today. Sounds like a good idea, but it could always be better.

This afternoon, it got a lot better. Ryan looked into Teradata at the time (in days) between the first call to a merchant, and the closing of the deal and came up with an average number of days for a deal cycle - bucketed by the category of the business. He sent these to me, and I was able to use this - along with a more general global average number of days for all deals, and use this in the code.

Now we have a solid default deal time, as well as better values for specific categories of businesses. This is great news. It took him less than a few hours, and I implemented it in about 15 mins. Fantastic work.

Ever better. Sweet.

After looking at what I had done for the seasonality adjustment of the demand data - something that should be handled upstream of us, but at this time it's not, the project manager decided that he wanted me to add in the reasons to the solution. Something like: Increased demand for Boat Tours in Oct (150%). And then to carry these through to the client so that they can see why the demand is, what it is.

I told him that I thought it was a bad idea because it's not going to last. Specifically, when the demand service is really finished, it's going to be broken out into at least four components:

• Inventory Replenishment - any decent inventory control system can see what's being sold (rate), the inventory on hand (quantity), and then project the need to acquire more. This has nothing to do with seasonality, but will naturally take care of a lot of demand issues because the more that's bought, the more we'll acquire. Simple.
• Manual Demand Insertion - there will always be the need for a system to accept manual commands from a wise and thoughtful operator, and in this case, there are folks that will anticipate demand for things that simply can't be put into code.
• Demand Forecasting - this is where you want to look at the month-by-month sales, see what might be needed - see the lead time for it, see what we have, and then try to plan what's needed based on past experience. As well, it'd be nice to have this part of the system capable of detecting demand much in the same way the manual input is done.
• Mixing Board - these previous three components need to be "mixed" together with an aggregator/balancer that allows the user to adjust the "signal strength" of each input independently, and then also adjust the output. Very much like a sound board, you need to be able to mix in all this demand carefully and then feed it into the main engine we've developed.

and he knows all these pieces, and knows they are on the way.

So I say: "The final demand system will have seasonality built-in… we won't be able to get at that data because it'll be baked into the operation and data."

"Yeah, but maybe we need introspection on that…" he replies.

"Think about it… it's not going to happen… what if there's a natural 3 month cycle to some service… we'll see it repeated, but we won't know the reason"

"Oh…"

So he presses - despite my objections, and I implement the feature. Then we talk about this afterwards, and he says "Listen, you tell me if you think it's a bad idea, or hard to implement". I about scream. But I keep my cool.

We talk for about 15 mins about this, and I point out to him that I have been saying the exact words he asked me to say to stop this process. He just wasn't listening. And this isn't the first time for him, either. Not by a long shot. It's one of his more annoying features - he just doesn't listen.

It's in there, and it'll be going away, and the users will wonder where it went, or we'll make a complete mess of the system and try to include it in the final demand values. But I think I'm going to say "No" to that. It's just not possible to really do a decent job of that.

One of the nice things about using JRuby is being able to use Warbler to package up all the files and scripts into a single jar for easy deployment. One of the problems with that is that some of the common coding statements don't work the way you expect them to - but they do work.

When I had an issue with reading a CSV file in a jar, my solution was to "go to the metal" and work it out in Java. This worked, and it was OK, but it wasn't something that was transportable to a non-JRuby environment, and I wanted to have that. So this morning I tackled just that.

The thing that's been bugging me was that Configulations works just fine in our code - jar or no-jar, so why was that working and the file reads of CSV files not? What I needed to do was to look seriously at how they were being used.

We were using Configulations in the following snippet:

  require 'configurations'
  require 'date'
 
  AppConfig = Configulations.new(File.dirname(__FILE__)+'/../config')

I know that JRuby 1.7.0 doesn't like the use of the '/../' in the path in a jar as it tries to "walk" the structure, so I changed this to a slightly less round-about way:

  require 'configurations'
  require 'date'
 
  AppConfig = Configulations.new(File.dirname(File.dirname(__FILE__))+'/config')

This allows JRuby to handle the parsing of the path as it sees fit and I don't have to worry about moving around within the jar file like I can in a filesystem.

And then it hit me… how we're using the filenames in the reading of the CSV files:

  require 'csv'
  require 'json'
 
  module FileUtility
    DATA_FILE = 'lib/sales/histData.csv'
 
    # ...
    if File.exists?(filename)
      File.open(filename) do |file|
        contents = file.read
      end
    else
      # ...
    end
 
    # ...
  end

and it stood out as clear as can be - the path was wrong. We were looking for these CSV files in a directory relative to the existing directory. How was JRuby to know that we wanted to look within the jar? Impossible.

The Configulations example worked because it used the dirname() method relative to the existing file - which is in the jar. That means that we were the ones telling JRuby to look in the jar (or on the filesystem), and it was all about the path we prepended to the beginning of the file we wanted to load.

There was no need to have the Java solution - we just needed to be more careful with the location of the CSV files. What we have now is far simpler:

  def self.read_file(filename)
    contents = ''
    File.open(project_root + '/' + filename) do |file|
      contents = file.read
    end
    contents
  end
 
  def self.read_csv(filename, key)
    res = {}
    CSV.parse(read_file(filename), :headers => true).each do |rec|
      k = key.map { |c| rec[c] }
      res[k] = rec
    end
    res
  end
 
  def self.read_json(filename)
    JSON.parse(read_file(filename))
  end
 
  def self.project_root
    @root ||= File.dirname(File.dirname(__FILE__))
  end

where once again, we use the "double dir" method chain to get the parent's directory, and use that to know that it's the root of the project - based on the location of this file in the project. This is far, far simpler than we had in the past, and it's removed all the silly Java code that wasn't really necessary in the first place.

It's important to realize you solved a problem, but not in the way it was intended. I'm glad I went back and fixed this. Very glad.

One of the things that's really quite limiting about the current demand forecast data from another system at The Shop is that it's not a month-by-month demand projection. They took the year's sales data and then extrapolated for the next month. But in doing so, they have removed any monthly/seasonal trends so that the demand for Boat Tours in Chicago is the same in July as January. Clearly, absurd. Unfortunately, there doesn't seem to be the will to push a change in and fix the Demand in a reasonable timeframe. So I had to add in some kind of demand adjustment based on the month and some sense of the time to close a deal.

What I came up with was a series of factors - one per month, that could be individually adjusted in a GUI - like an equalizer, and then we'd take each of these and multiply the given demand by this factor and divide by 100 (converting it into a percentage). In this way, we can ramp things up, and shut them down, a month at a time.

If they want to boost something up, they can pretend it's seasonal, and use move everything up to 200%… if they want to suppress something, then move all months to 25% - simple. It's a great plan, and we can "seed" the data with the sales data by service, by month.

What I need to do is to start with the code and then work back as I'm not about to do the GUI first - I'm no artist, and that's what this is going to take to make it look really nice.

Today my day started out with a rather early meeting for my work here at The Shop - 9:30 am, and it was about an interesting topic - the seasonal adjustment of the demand data we're using in our project. The problem is that the demand data is based on the previous 12-months historical sales data - all annualized. This means that boat tours should sell as well in July as December. Nope. So how to do this?

I'd already suggested in an email that we make a simple web app that allows the right people to add the important location and taxonomy data as well as a simple 12-month segmented graph (or even sliders or dials) so that the users could control the seasonality for their division and their sales reps. It makes sense - boats trips in Arizona are not down in December, so we need to really look at all the factors that might contribute to the seasonality of the demand.

Then we need to overlay this on the incoming demand and we're in business.

There's also the need for a manual demand entry screen where the users can input demand that they anticipate, and let them run with that. All sounds pretty decent. Very do-able.

But the proposal was that we base it on some Google Doc and parse out the data from a spreadsheet.

That's a horrible idea. I've had to do it so many times, and it always becomes a nightmare. The project manager thought it would be "easy" and "fast"… but all it really does is move the development effort from building the UI and data integrity checks to the data parsing and processing. There's no savings here. But Holy Cow! That was a 30 min meeting that took 90 min because this guy could not get the idea out of his head that this wasn't the way to do it.

I consider myself a decent communicator, but I've come to a loss with this guy. I think he's just not a good listener, but who knows. In any case, at the end of the 90 min meeting I was able to get some support from another developer manager and I think we're going to do the "Right Thing". But it was not easy.

Now I think I'm going to have to come up to speed on Rails, as I think we're going to build this in Rails, but at least it's going to be done Right.

Looks like we have a new version of Google Chrome dev this morning - 23.0.1271.10 with a sparse, but informative list of release notes. With this update, it appears they are about to jump to 24.x - but we'll have to wait and see. They may all be working on getting iOS Maps out 🙂

In any case, fixes for Flash are always good (I really dislike the implementation of the Flash interpreters I've seen from Adobe), and while I'm not a Windows 8 fan, it's nice to throw them a bone once and a while.

Today has been one of those days… it started out with some wonderful steel cut oatmeal at the cafeteria… I had no idea they had that! It was wonderful. But from there on, it was a steep descent downhill.

We are coming off the horrible morning and then trying to get a good number of features and fixes into the code for the afternoon release to production. I like to keep releasing something off master to production each day as that allows the users and project manager to see some visible progress each day. Big or small, it makes little difference to be able to point out the changes we have made based on their feedback and requests.

Often times, though, this means doing a lot of nasty work to follow up with people that dropped the ball (but doing so in a way to make them feel like they are doing you a favor), cleaning up problems and messes left by others, and all the little ick work that comes with software development.

Today has been that day for me.

I'm glad it's over.

On the up-side… we are releasing something far better than yesterday, and the really great ones never loose sight of the fact that it's all in the details.