What's it all about, Alfie?

Today I did a little coding on my fast tick server because it was giving one user in London the head fake, and rather than ask them to understand that it was just that, I decided to fix the problem so it didn't misrepresent the data to the data maintenance team.

The problem was really that I worked hard to make sure the data in the server was maintained properly, and didn't spend any time thinking about making sure it was visually updated properly when it corrected itself. So if the user made the change, everything was fine, but if the system corrected itself, the user might think it hadn't. And then think that it was wrong when it really was just fine.

The code change was minor - maybe 10 lines, and the same code was used in at least one other place, but re-factoring it wasn't really necessary due to the locking constraints, so I just updated the code and it worked like a charm. I didn't have the heart to tell the user that they changed nothing, so I let them believe that I had found a bug in the way the data was moved around, and I did, but the effect is solely for the human's benefit.

But hey... it's always fun to code. Right?

I'm still on the trail of the problems in the message bus API. I stayed a little late last night to try a few things while my price injector wasn't injecting ticks. Turns out, I learned a bit, but then this morning I learned quite a bit more. It's getting downright interesting.

First, last night, when I wasn't injecting prices, I would see the poll() working as it should. This morning, I noticed that as soon as I started sending price messages, I started getting tons of poll() hits on the incoming message socket. Why on earth are they doing that?

More to the point, the poll() loop was working perfectly and I could cross that off my list. It was certainly a possibility that the polling loop was just stopping - an error, an exception, something could have stopped it and that would explain it. However, this morning, I saw the polling loop running like a champ, but the messages I sent this morning were not getting delivered to my code.

Interesting fact #2: the act of sending a message in this API causes the system poll() to return true even when there's no data waiting at the socket. I can imagine a few reasons why this might be the case, but all of them are pretty bad. The criteria for a positive result from poll() should always mean that there's data there on the socket waiting to be read. But clearly, this is not the case with this API.

So one good, one bad. What I'm left with is that I needed to add in more logging on the processing of the message data stream and hope that it's a trap-able error in the code I can control. If not, then there's no hope for a solution on my end, and I'll have to football all this information back to the vendor and hope that they can figure this out with the data I provide them.

I'm not holding my breath.

Today I got an email from a tech support guy for a vendor we use, asking if I'd tried their latest version to see if it fixed the problem I was having. Basically, it's a proprietary message bus that's very simple, and at the same time, fast. The problem is, it's not really fast enough to warrant being on it's own, but it's so old, that when it was new, it was probably something pretty useful - in the context of their system.

He asked a bunch of questions, all things I'd gone over with the last guy that contacted me on the 18 month old bug, but I wrote back a detailed message saying what I'd tried and what I believed to be the issue. However, because of the way they wrote their library, there's really no way I can know what's happening under the covers.

I suspect that it's in the socket handling - specifically the use of the poll() system call, and what it returns in different conditions. I know from experience that it can be tricky on linux if the socket gets in a weird state. My friend who wrote our C++ wrapper to the C API from the vendor did the simple poll() system call to see if there was a pending message on the socket, and if so, he'd call their handling method. But he used the simple poll() system call.

I decided that I'd see if the improved poll() I had written for CKit would help. It's got a lot better error handling and maybe the issue is with that. Certainly a good place to start. So I added that code, and we'll see what happens.

If that's not the case, then I'm going to start logging the activity around the poll() call to see if the vendor's handler function is hanging, or if there's even data at the socket to read. I'm not sure what's going to come of this, but it's an interesting diversion. Maybe I can come up with a work-around and fix this horrible problem.

Today I was trying to clean up a script we use at the Shop for equalizing perl on the different platforms around the place. We all need some variable of perl 5.8, but if it's 5.8.3 versus 5.8.8, it's not that big a deal. However, all perl implementations need to be able to have CPAN, plus database access which means that they have been linked against the right libraries for each version.

The problem is that this sounds easy, but in reality, there's a disconnect and not every build of perl available on the network mounts is built similarly. Some have database access, others don't. It's a bit of hit-n-miss.

Yesterday I was working on a perl script to monitor one of my tick injectors, and realized that in order to get perl working on these new boxes I had to enumerate all the hostnames. That is just far too brittle, and I didn't like it - but I couldn't see a nice pattern in the basic machine set-up to decide which one to use.

Today I decided to find out, so I made a matrix of all the machines, their architecture, OS version, and name, and then looked at the perl builds available to them. Turns out that there is a pattern, but it was obscured by the fact that not all versions of perl were available on all machines. I was able to create a solid filter based on the machine architecture (x86_64 or not) and then have a primary and a back-up.

It wasn't rocket science, but then again, very few things are. But this was a fun little excursion into what was there and what pattern fit. Interesting stuff.

The Shop is a heavy user of chat - be it IRC-based (as it was for so many years) or the 'secure' chat that we now use, it's chat. It's very useful as a conduit to the users for all kinds of information - especially the monitoring of servers and processes. This morning I did a little 5 min job to convert back one of the server monitoring bots from the 'secure' chat back to IRC chat as we have one of the latter, and the global messaging group which controls the former is still not allowing bots back on their network after a particularly bad server meltdown caused by a few bad bots.

I wanted to move this one guy back to straight IRC because it's a very nice example of a perl-based bot that can monitor all kinds of interesting things. I then spent a few hours crafting a monitoring bot out of this starting point to replace the java-based monitoring tools that had caused me so much trouble yesterday. I asked the guy that created them to turn them off - save one development box (his choice), until such time as we can be assured that there's no conflict in the communications.

His response was that these monitors are providing vital data on the status of the ticks flowing from my injectors into the system(s). I told him I totally understood his position, and if he'd just let me know what the processes were monitoring, I'd be glad to give him that same functionality, quickly, in a less intrusive monitoring framework.

I didn't hear from him, but I started one anyway with the likely candidates. I also talked to the head support guy and he had a simple little monitor as well. I included his test in with my code and started banging on it.

The first cut was nice, but I also wanted to add a little additional chatting when a stalled log restarted so that the users monitoring the chat channel would know that the problem corrected itself and they didn't have to worry about restarting anything. In this particular system, it's common for an updating process to get stalled and then restart without any intervention. I wanted to make sure that this was being passed on to the operators so they didn't worry about a problem that's corrected itself.

I sent out an email explaining it, and how to stop/start/restart it along with where it chats, and what it chats about. Basic information. I haven't heard from the group that put the other monitoring tool, but I'm guessing they are not going to be happy about what I've done. Not in the least. I hope I'm wrong, but in the past there has been more than a little animosity between my group here, and the group that did the other tool. Sad, but there's nothing that would have kept them from writing the same thing. Nothing at all.

The MacVim guys had a problem with Snapshot 41 - the iconv library wasn't included in the package. So they quickly released Snapshot 42 this morning. I'm not exactly sure what uses this library's functionality, but I didn't hit it in the two days running it.

Nevertheless, update I did, and here we go.

I've had quite a day today... seems when it rains it pours, just like the salt. The latest little nugget was a possible interaction with a Java-based monitoring toolset that was written by another group in the Bank. I'm sure it's got a lot of potential to monitor Java-based applications and possibly even system processes and scripts, but we ran into a really nasty little side-effect today with a program I wrote that uses a third-party API message bus.

I've had a lot of problems with this particular message bus API - it will seemingly hang on input of a message. I believe that it's related to the socket communications the API is using, but I can't prove that because the vendor won't let me have access to their protocol. But that's not the biggie.

The biggie was that this new monitoring app was put on the production and UAT machines with my software without any word to us as a "heads up". Things started failing and when I couldn't understand why they weren't working, I saw this monitoring package on the box. I killed the monitor and Shazam! everything worked.

While I think monitoring is good, and automated monitoring is really nice, it's got to be zero impact. You can't have a monitoring package that interferes with the normal operation of the application - and given that you can't know how the application works, you need to be very careful about how you communicate with your clients, etc. This package might not be really good, but in this case, we need to be a lot more careful about implementing it in our environment.

The twist in the system is that the people that put this monitoring package on the machines didn't bother to tell us what they were doing. They're supposed to be part of the Team working on this system, but they acted more like a bunch of little kids about this issue. They wanted to be able to do something "on their own" and so rather than talk it out and so it safely, they decide to go "joy riding" with the UAT and production systems. If they had done their homework and really checked things out, then this would be fine with me. But they didn't, so it's just another example of their unwillingness to really follow-through and do a good job.

I came in this morning and watched something unfold that reminds me of highschool - and not in a good way. This morning a co-worker came in and saw that his cube had been filled with all kinds of stuff - boxes of tickets, trash can, all kinds of stuff. Making it clearly impossible for him to even enter his cube - let alone work there. Clearly, this was a prank. It happens quite often around here - for a business office. If this were a movie theatre and we were all still in highschool, then it'd happen more often, but we're not, and so it really shouldn't happen at all, but it happens every couple of months.

This time, it was too far.

The guy who was pranked took all the stuff out of his cube and started filling up the alleged prankster's cube. I asked him in the process of this "re-pranking" how he knew this was the guy. He was angry and pointed out to me that this guy had hassled him all day Friday after he asked someone to move some boxes because they were in the way of him getting to a few machines he had to work on. Not an unreasonable request, and handled very politely.

The prankster thought differently. Or so the prankee thought.

When the alleged prankster arrived to see his cube filled with crud (some of which was new to the re-pranking), he walked over to the re-prankster and said "It wasn't me. I know you think it was, but it wasn't me. Can you help me clean this up?"

"Nope. I don't believe you." he replied.

This continued for about 15 mins each one trying to convince the other that they were in the right. That's the thing with pranks - you think it's nothing, so why get all upset about it, but when someone gets upset, as they have every right to, the praknster gets offended and makes it about the intolerance of the prankee.

Yup, highschool, all right.

In the end, I have a feeling this is going to escalate until management gets involved. These guys have been at this for years and years, and it's not stopped yet, so I have no reason to believe that it will stop with this re-pranking. Who knows... maybe it will. I sure hope so because I have no desire to be collateral damage in this fight.

This past weekend I took an online C++ test and received a score of 90% - which was in the 60th percentile. I also took a programming test that ran, but they may take off points based on the assumptions I made for the source of the data stream. In the end, I think I did OK, but the tests got me a little bummed out. I was thinking "Who's going to want to hire me when all I can do is a 60th percentile on the C++ test?" I then got to thinking about all the things I'm missing - all the things I wanted to do with my career.

I was pretty bummed out.

I wasn't looking forward to coming to work either.

But all that changed about an hour into my day when I chatted with a co-worker in London. He's taking over the duties as the lead developer on this project that I didn't want to be involved in. His statements were, without a doubt, the defining moment for me and why I do deserve to be paid what I'm paid.

It started out with an email from him about changing the settings in your .vimrc file to change the tab stops and expand tabs so that all the files you edit are spaces, and spaced properly. This is because his project relies heavily on Python scripts, and in Python the spacing is critical.

I wrote back to the group saying that the same thing could be accomplished within the file by using the Vim tags as part of a comment. Then, no matter what the individual's preferences are, the file in question is going to be formatted in the proper way.

His response was that "no one uses Vim, and this is what you should have as the standards anyway - for Perl, Python, Java, C++, Shell Scripts..." Amazing.

There's a need for Python file formatting, so let's modify Vim for everything to make it easier for this one set of scripts? When I asked him if that was his professional opinion that Vim should be modified as opposed to using the in-file tags, he said "Yes".

Now I saw why I can draw the salary I do. People who don't learn from history are doomed to repeat it. He made the statement:

You need to use some frameworks... and to be honest, development in vim is probably not as efficient as it used to be

Let's set aside that the frameworks questions is idiotic as it has nothing to do with the point at hand, but how on earth can a professional developer say something has become less efficient with time? Beats me.

So I'm sitting here laughing to myself about this exchange. He believes that HTML needs to be written by an optimizer, and IDEs are the way to do things. I have to laugh. I remember my High School Trigonometry teacher teaching us the half-angle/double-angle formulas for the trig functions - the 30/60/90 and 45/45/90 triangle - so that we could calculate all the trig functions without the calculators was all carried to class each day. Why? Because in order to learn what's going on, you need to understand it. Trust a machine too much, and you're liable to get burned.

Forget that the best development tools are the mind, a pencil and a pad of paper, and you're going to be just like this guy. A poser, thinking they know what's really going on when all they are doing is following the latest development trends. It's all shiny objects to them, and no substance. That's while I'll always have a job, and not have to worry about how much I've forgotten about the syntax of C++. The real work is done between the ears, not in a window.

Today I saw that MacVim Snapshot 41 is out and they have put in some really nice new fixes and features. The first one is that the Sparkle support is working which means that I won't have to download it directly each time - very nice. The next is that they have changed the default font loading for the initial window load, and since I don't use the default font, adding the following default:

  defaults write org.vim.MacVim MMLoadDefaultFont 0

I can significantly improve the opening speed of the editor windows. I will admit it looks a little snappier. I'm glad to see that, as well.

Once again, this is the superlative implementation of Vim on the Mac OS X platform. If you have any use for Vim, get it. You won't be sorry.