What's it all about, Alfie?

I'm working with a vendor's product where they have kindly created the possibility of adding customer-specific fields to each object in the database. It's a nice touch, and if done correctly, it makes the system very flexible and expandable, but if it's done wrong, it can make the system horribly slow and difficult to deal with.

Yes... I know... it's easy to see which this one is.

But to the point at hand. Each of these Customer Bits is tagged with a datatype and what object (both type and instance) it belongs to, etc. The interesting thing is how to add a lot of data to the system very fast. If we need to add in the base object and then, say, a dozen of these bits, then we need to do it all very fast - if it's going to be on something like tick processing, or trade processing. You get the picture.

So today I learned through the School of Hard Knocks, that when adding these bits to an object, you can add them one of two ways: individually, or in bulk. If you add them in individually, you're basically adding the base object, committing it, and then one by one, adding the bits, and committing each as you go. If you add them in bulk, you're creating the base object and all the bits and then committing just once.

What I learned was that if you take the individual approach, the data you place in the bits doesn't matter - specifically, you can place an empty string in a bit and it's all OK. But if you try to do it in bulk, you get a very cryptic error message that it's clearly tied to the bit you're setting to an empty string.

So it's an easy fix - once you know the problem. Simply don't allow setting bits to empty strings. But you'd think that this shouldn't be a problem in the first place. But there you go thinking again.

UPDATE: I heard from a developer deep in the Vendor's shop and he said that there was a bug report about this already and the answer was to disallow all empty string usage in the bits. That's certainly one way to handle it.

I was monitoring one of my price injectors this morning and several weeks ago I had worked up an interesting little addition to monitor the app that's processing the prices I'm injecting. I worked up the log-based monitoring, and at the time I was a little concerned that it was too loosely coupled, and that we'd get into trouble because of this. Because of this, I worked very hard to try and anticipate a lot of the conditions that might occur, and as of this morning, I'm very glad to say the efforts have certainly paid off.

Sometimes, it's good to be loosely coupled. I had no real choice, but even if I had, this was a vendor's closed package and what was I really going to do? I have done pretty much all I can. In this case, it was really thinking about what might happen that did the trick. I can see that my app detects that their app is frozen and simply stops sending. I can see it detect that the app has been restarted and starts sending again. Little things, I know, but it's going to make it possible to have my app proactively restart theirs.

That would be a great kick in the pants, wouldn't it? "Hey guys... your app freezes on me -- a lot, so I put in code to detect it and restart it. You might want to work on that stability someday." Man, that would be s...w...e...e...t.

It's just what the doctor ordered.

While I have no idea if this phrase is really Chinese, it's certainly true of working in the Banking industry these last two months. The Dow has taken a beating that I haven't seen in a very long time: (image from Google Finance)

I worked in the Chicago Board of Trade on the morning of 9/11 and remember everything about that morning and how we ended up leaving the building and the masses of people in downtown Chicago. It was something I hope I don't experience again - for a great many reasons, but especially because of what it did to the collective conscience of the American people.

So here we are, many years later, and greed - plain and simple got us, as a nation, and therefore, a world, into this pickle. I'm glad to see that the markets are working. I have faith in the smarts of the traders I work with, and others like them. They are going to price things reasonably again and be a little smarter in the future about these grouped debt instruments.

I do not support McCain's statement at the second debate about having the government buy up the mortgages, work with the owners to re-value the homes down and then re-finance at this new level - all so they can stay in their homes. Good if you're in a house you could not afford, and now you'll be able to. But bad for the rest of us that didn't get into debt trouble because we lived within our means. Bad idea, McCain.

Let the markets work. It'll get us through this. It always does.

I've been looking for a good (decent) desktop client on Windows to keep my work blog (journal) up to date. I've tried w.bloggar and it's OK, but when I try to edit an existing post I get a time-shift on the posting time by five hours - exactly the same as my timezone offset from UTC. It's OK, but what I have to do is to post in w.bloggar and then edit any posts in the WordPress web interface.

So I looked again... and this morning I tried Deepest Sender a Firefox add-on and it's pretty nice. I'd like it if they had the ability to set the font in the 'Source' view to something fixed-width, but I can live with that as a minor annoyance. The biggie is that when I save an edited post it's not off by 5 hours, it resets the post date to epoch!

OK, that's a little extreme... I just wish these guys would allow an option to leave the posting data/time alone. So I sent in a bug to Deepest Sender as I think I'd like to use that one more as it's cross-platform with Firefox. We'll see what happens.

The Adium team has released 1.3.2 that fixes two bugs that I've been having - problems with MSN (they updated to the latest libpurple) and the frequent disconnects from Yahoo!. There were a few cosmetic changes as well, and that's nice that they're working on the "little things" as well as the biggie like tons of daily disconnects from Yahoo!.

So... pick it up, it's the very best IM client I've ever seen.

I'm coming off being exceptionally angry with a Vendor. I got a Diet Coke, had a few sips, and tried to calm down, when it hit me - It's amazing what these people didn't lean in Kindergarten. No really... from the poster/book All I really Needed to Know I Learned in Kindergarten I'm amazed that these people didn't learn these things.

Maybe they did, but when they deal with us, they casually "forget" the important things: Don't lie... Be Honest... Take responsibility when it's yours to take, and help people even if it isn't. These are simple things. We expect them from our kids - we tell them to tell us the truth, and we honestly go easier on them when they are honest and tell us the truth. It really does work.

So why do people dealing with a 'customer' act differently?

If it simply because they want to look good? I don't think so. I don't think these contacts at the Vendor really even care if they are seen as nice and helpful, or mean and unhelpful. Does it effect their pay? No. They do a job for a company, and unless it costs the company money (like a client leaving) they're not going to get involved. So I don't think they care about being seen as nice. They just don't want to get fired for losing a client.

Is it because they need to protect the company's reputation? That probably happens sometimes, but in a big global corporation, they are more interested in who recommends them and not necessarily likes them. Face it... Oracle might not be liked but they are going to be used by people because of their reputation. So, I don't think it's for the company's image they do this - again, most of the time.

I think it's because they know their product has faults - everything does, and when they are confronted by it, their first instinct might be to confess the short-coming, but then a call from here is made to there, and CTOs are involved, and all of a sudden honesty evaporates. All because someone else, higher up in the chain of the vendor, starts to scream.

They're being a bully to their people. Nothing more, nothing less. These people dared to tell us the truth, which was uncomfortable for them and for us. But if it was the truth, then we should know that, do our best to deal with it and move on. This isn't the time to bring out the rubber hoses and beat people on the feet... it's a time to recognize the limitations of something and work within those limitations to try and make something that might work.

There's two ways to handle disappointment - suck on the sour lemon, or make lemonade. I'm all for the latter. Let's try to remember what we learned in Kindergarten - take turns, share, work together... all these things are going to make the situation better not worse.

I was formally introduced to Python in the context of a vendor's application. They used it as the glue that binds and controls the individual components of their system as well as the internal language of operation. It's a gigantic python collection. And the experiences I've had with the product, and therefore by extension, the language have been bad. Miserable, in fact. But I'm trying to overcome them.

Python is a very simple language with great power because, as in perl, people have taken a liking to it and have written tons of libraries and add-ons to it to provide all the power the original author(s) and designer(s) didn't have the time or vision to write. Both languages are impressive in that way.

I have to admit that I'm not a fan of a language where control rests in the level of indentation, but I can see the reason for it - uniformity of the look of the code. You can't have the religious arguments about tabs/spaces or where to put the bracket if you have to indent to create the structure of your application. But I have to admire a simple language where the creativity is in finding and using those additional libraries and putting them to good use.

So I'm trying to warm up to Python, and I think I'm going to make it. It's just got this major handicap right now - it's connected with a product and a vendor that I really can't stand, and getting over that is hard.

Starting on Dec 9, 2005, I was asked to keep a journal here, at work. I'd done this before - which was the genesis of this blog, but I hadn't done one specifically for this job. So I started one. Just like the first one, I started with simple HTML. That's all I needed. Then I got WordPress, and I realized that I needed to move everything over.

Well, finally, today I migrated over the last of the old entries from the HTML file to the WordPress install for work. It's a great feeling of relief to know it's now all in WordPress and that if I have to move, I can back everything up, take it with me and keep it as a historical record of what I've done since that day.

I still need to figure out how to backup a complete MySQL database, but that can't be hard, probably a single command would do it. But... whew!... what a relief.

UPDATE: pretty simple to dump/load MySQL databases. To dump a MySQL database from the command line:

  mysqldump --user <em>username</em> --password=<em>password</em> <em>database ></em> <em>dump_file</em>

and from there you can compress that bad-boy and you're good to go. Likewise, to restore a database simply use the MySQL client as the dump file is just SQL commands:

  mysql --user <em>username</em> --password=<em>password</em> <em>database</em> < <em>dumpfile</em>

This, and a simple tarball of the complete WordPress directory and I've got golden backups of the blog - wonderful!

Yesterday I ran into a roadblock on this one little project I was working on, and I realized that the docs were useless, the data dumps weren't helping, and in the end I needed to talk to the tech support folks. So I wrote up a little message saying in very clear terms what I was looking at in the client, and that I needed to have programmatic access in their development language to those values.

Pretty simple stuff.

I was not expecting what I got back.

The first person responded saying that the values I was referring to were wrong, and why would I want them? Answer: because the user wants me to get them.

But those numbers are wrong... Here... let me show you how to set them properly (with screen shots, etc.) OK... but that didn't tell me how to get them even one they were 'right'

To this last exchange I wrote back saying "Hey, right or wrong, I just need to know how to get the data." Clearly, I wasn't getting through. I thought I'd get the answer I needed. Silly me...

The next reply was from the second level support asking why I needed that data, and what was I going to do with it? Holy Cow, people... is this an Inquisition, or what? I just want to know how to get at the data that I can see with my own eyes in the Client. I can get all the other pieces I can see, why not this one?

"Bob, if you'd just let us know the background, I'm sure we can help you."

Finally, I gave up. I told them I was making a report for the users and needed the data as they have highlighted it as the data they want to see. To this, I finally get "OK, we'll see what we can do".

I was stunned. Why it matters to them what lead me to this place, or what on earth I wanted to do with it was, and is, so totally beyond me I can't even begin to put it into a common frame of reference. I know for a fact that the first and second levels have no idea how to get the data I need, so maybe they are just stalling for time. But to be as belligerent about this as they were is something I never expected from a vendor - Attitude.

So now I know... when I want a piece of data, I have to ask for it in the right way or they're just going to keep at me to ask it in the "right" way until I do. Only then will I get the information I asked for. Needless to say, most of my co-workers thought I showed considerable patience with this interchange. Many were using not-safe-for-work language after the second email.

Just amazing.

A developer stopped by today asking me about a problem he was having using the BKBaseGraph as it was saying it couldn't lock the dataset for the change - a nasty exception. I got his code which had a section like this:

  private void initGraph() {
    try {
      curveGraph = new BKBaseGraph(0, 1, BKBaseGraph.LINE);
      graphPanel.add(curveGraph, BorderLayout.CENTER);
 
      int pointCount = 10;
 
      // display the points in the graph
      curveGraph.resizeData(pointCount, 1);
 
      for (int row = 0; row < pointCount; row++) {
        curveGraph.setData(row, 0, (Math.random() * 100.0));
        curveGraph.setRowLabel(row, "" + row);
      }
    } catch (Exception e) {
      e.printStackTrace();
    }
  }

The problem comes in line 3 when you are trying to create a zero-row dataset. That's no good. You have to have non-zero rows and columns - default to nothing and let the graph figure it out. For example, it's better to simply say:

      curveGraph = new BKBaseGraph(BKBaseGraph.LINE);

and let it go at that. Alternatively, if you know the numebr of rows, as it appears is the case here, you can put that in and skip the resize alltogether:

  private void initGraph() {
    try {
      int pointCount = 10;
 
      curveGraph = new BKBaseGraph(pointCount, 1, BKBaseGraph.LINE);
      graphPanel.add(curveGraph, BorderLayout.CENTER);
 
      // display the points in the graph
      for (int row = 0; row < pointCount; row++) {
        curveGraph.setData(row, 0, (Math.random() * 100.0));
        curveGraph.setRowLabel(row, "" + row);
      }
    } catch (Exception e) {
      e.printStackTrace();
    }
  }

Either way works, but don't put the zero rows or zero columns.

He came back to me and asked: OK, but how do I display no data on the graph? Well, there are methods on the BKBaseGraph that handle that:

      curveGraph.hideGraph();
      // ...
      curveGraph.showGraph();

the first hides the graph behind a blank white JPanel, and the second brings it to the front. This is useful if you have a lot of updates to do, or you simply want to say "no data - no graph." Simple as that.