What's it all about, Alfie?

I've decided this afternoon that I haven't been able to really give Python a clear chance. I've been using it for over a year as part of this vendor's application - it's the embedded language that virtually everything is done. That, in itself is not bad, as one of Python's strengths is the ability to embed it easily in C/C++ applications. No... it's what they have done to it that makes it hard to get a really good read on Python.

For example, you should be able to run a python script, and upon proper loading of libraries, get all the added functionality of the loaded module - like sybdb. Standard stuff. In fact, you should be able to use any python of the same version on the same box, and if you can load those libraries you should be good to go

The problem is, they have fiddled and monkeyed with the language to the point that this isn't really possible. You can get some things but others are only half-working and others still are completely broken. This means that you need to run their python and set up a very complex environment to get this running.

This represents a huge initial cost to running a python script. No such thing as a quickie... no sir. You have to really want to run a new python script. It's a pain. They are not really flexible. It's a system that makes python look bad. And I'm only just coming to realize what part of this train wreck is the vendor's stuff and what part is the python.

As I get more experience with the system I realize that there are a few things I'm not a fan of that are indeed python. However, they are completely overshadowed by the vendor's mistakes and limitations. It's amazing.

So I'm trying to give python the benefit of the doubt and realize that 99% of all the problems I'm seeing with this system is not the fault of python, but the implementation they slapped around it. Too bad. I'm sorry, python.

UPDATE: case in point: the difference between the '=='/'!=' and 'is'/'is not' operators in an if statement. For example, consider the two code samples:

  if value == None:
      print 'Value is not defined.'

and:

  if value is None:
      print 'Value is not defined.'

The difference is that the equality ('=='/'!=') operators call a method on the objects to do a value comparison, and the instance operators ('is'/'is not') do a instance comparison (pointer comparison) on the two instances. This means that the latter is significantly faster than the former, and in the case of None, it's the preferred method as well as there is one and only one None object in the python runtime. This makes the latter test preferred, and faster. That's pretty cool.

I was pulled in this morning to a problem with trade committals using a Python server that interfaces to a certain vendor's application. The python server is pretty nice - uses the built-in XMLRPC server and it's as clean and easy as pie to work on. Python - Good. The system it feeds - not so much.

I was getting an exception in committing a new trade to the system. I was getting a very unhelpful exception message, and as a result I had to run this guy several times with the same inputs and different levels of logging statements to see where in the code this was throwing the exception. It's python 2.3, so the really nice exception stack trace printing wasn't available to me, and the custom logging package in use wasn't my pick and I had no idea if it re-directed stdout/stderr.

What followed was about 30 mins of print debugging and about 5 mins of input data conditioning to make sure that none of the incoming data values were illegal and causing the problem. Good stuff to do, but given that this is all system-to-system interaction, this shouldn't have been strictly necessary. Nevertheless...

So I kept looking and finally got it down to the one field that was causing the problem. It was a string that was 23 characters long and the data description in the vendor's docs said it was limited to 20. OK, that's understandable in some systems like old-style client/server stuff. But in this day and age why are we limiting ourselves to 20 chars when we know any decent database has varchar fields and by their very nature, they are variable in length up to a point. Make that point 256 chars, or even 1k - what's the harm? Yes, it might not all fetch back in one packet, but given gigabit ethernet, is this still really a concern?

But even if it is, how about giving me a really useful exception like "Data value out of range" or something like that. Then you can use that for integers that are too big, strings, etc. It is pretty universal and then you are really helping out the guy trying to debug the problem.

As it was, I was forced to check "Why?" by searching around and finding in the docs the limit. But wait... there's more.

This vendor publishes a limit of n, but the limit is really n-1. Why? Good question. If the limit is 20, make the field in the database 21 or something. In fact, most varchar fields can hold up to their maximum, so why the offset? I can only imagine it's something from the designers/developers that is so silly as to be laughable.

I'm not laughing. I'm shaking my head.

So I finally put in the code to clean up the limits on the strings and log that data was getting truncated. Then I passed it off to the guys that were supposed to have figured this out, and they ran the tests and things worked. But we're still not done because with this truncation they have to make sure that it's not going to break anything moving forward. At this point, I don't know and don't really care. It's a messed-up system with horrible exception messages, pitiful documentation, and tech support that's virtually non-existant. I don't like it, and hope soon to be rid of it.

This morning I was looking at the logs of one of my price injectors and I got an exception on the CKStopwatch saying that the number of time events and time structs didn't match, which is a serious data integrity problem. I'm never gotten this before, and so I had to dig into it right away.

When I looked at the code, I saw something I knew was there, but it hadn't hit me in all the months this thing had been running. You see, in this injector, I need to have a few (configurable, based on load and number of processors) threads that take the prices and send them to the destination - inject them into the message stream. In these threads, I need to have an idea of the elapsed time they have been running so that I can print out statistics on their operation. Nothing big, but I need to have the number of prices they have injected over the time interval they have been working. Ticks per sec.

Since I needed to have this persistent over several loops of this thread's main processing method, I decided to declare it static at the top of the run loop:

  int SPPoller::process()
  {
      bool        error = false;
 
      static int         totalSent = 0;
      static CKStopwatch interval;

and then it checked to see if this was the first time through and reset the stopwatch. But all this was a horrible mistake waiting to bite me.

The static reference is going to give me one and only one value for this guy regardless of the number of threads using this code. The fact that the int was thread-safe was lucky for me, but the CKStopwatch wasn't. That's where I got the exception from the other day, and a different one this morning.

The reason I did this was to try and keep the variables close to their location of use. And had I gone with thread-local storage, I'd have been in good shape. But I didn't. I debugged this with a single injector and it was fine. I only have two in most configurations, so it's a little more dangerous, but still not as bad as if I had 50 threads.

The solution was easy - make the total sent count and the interval timer instance variables of the class and then in the constructor, reset the timer and zero out the count and everything will be just fine. This was only effecting my logging, and so it's not a horrible problem, but the exception caused the poller to die, and that was a serious issue. Had to be fixed. Easy to do.

Whew! That one caught me by surprise.

Today again, I was reminded that I don't work in a friendly environment. I can't open up my journal, I can't tweet... there are people literally looking at me waiting for me to make a mistake so they can yell at me - or worse. I'm not so dumb as to think that the issues with the weblog are over and forgotten with. I know there are at least a few people that would love to have me give them a reason to have me fired.

I have not made any friends in the current crop of co-workers here.

Those that I did get along with are all gone. And with each one, a little part of me knew that this place was getting a little more hostile towards me every day.

There are folks that come by and ask me to help them because for the last year they haven't taken the time to learn a new system and how to find things in it. They think they only need to know what they know, and don't believe that they need to stretch themselves. I was busy, but I had to make time to help this person find their problem in the code. Why? Lazy.

But I'll be seen as the Bad Guy... I have in the past, and will again. I'll be seen as short-tempered and unhelpful, but he will not be seen as lazy or directionless.

I read an email by another 'developer' saying that there was a limitation in Python that would not allow it to reload modules when embedded. I knew this to be wrong because a friend was doing it here in a C++ program before he left. He's still doing it where he is. I privately emailed this 'developer' to say it's not a limitation of embedded Python, it's a limitation of the embedder. He said I didn't understand, and that it was dangerous and not standard Python. I knew better.

I double-checked with my friend that had escaped a while back, and he verified that indeed it works just fine, and that he did it, and is still doing it. No questions. Sure, there are a few issues and limitations, but those are true in any reloading scheme. I tried to explain this to the 'developer' but finished off the email saying that I was very unsure if I should have sent this email exchange.

So often these things degenerate into an ugly exchange where one person is convinced that they are right - and really aren't. Their limited knowledge of the subject matter is not relevant to their belief that they are without question the authority on the issue. The person that attempts to correct them - right or wrong, is in for a dogmatic argument, not a learned discourse.

Yet this guy is considered the "expert"? You gotta be kidding me. He's not an idiot, but if he's making arguments like this, it's not a matter of knowledge, it's a matter of character. You either are open to the idea that you might be wrong, or you're not. I try to stay firmly in the former category. He is clearly in the latter.

It's exchanges like this, and others that I've had today, that make me feel once again, like I'm really better off just nodding my head and letting people believe what they want. THey clearly aren't interested in the truth, why should I bother them with it?

I have worked with some very unusual people. I've worked with some very smart people as well. Sometimes they are the same people, and when that happens you need to deal with them very carefully. Why? Because they are smart enough to be curious, and smart enough to be dangerous, and not careful enough to stay out of trouble. Many people don't realize this until it's too late - they give a little too much information to someone like this and then they are constantly cleaning up the carnage that the smart, unusual person creates.

This happened today, and the key to controlling the situation is early detection. Make sure you see that this is a potentially dangerous situation as soon as possible and take steps to mitigate the damage. For instance, if the new linux user asks for root on their box, the answer doesn't have to be "No", it's better to say "What can I do for you?" Make it seem that you're more interested in doing this for them than having them waste their precious time on such mundane things.

The effect is the same. You don't need to let this person have complete control of a machine and then have to rebuild it because they decided to remove a few "offensive" libraries and the machine becomes unusable. You can help them, do the things for them that are liable to cause problems, and then all they need to do from there is to use the programs - editors, compilers, etc. that you have placed there. Simple. Disasters averted.

If you fail to do this, then it's either a constant game of catch-up and clean-up, or you have to be firm and say "We're taking this away because it's not safe at this time for you to do these things." This, unfortunately, never goes over very well. There are hurt feelings and problems, and then there's the turf wars... it's ugly and it takes months to recover.

Best to be proactive and simply avert the problem. If you find that someone who doesn't recognize the danger inadvertently "spills the beans" you need to step in quickly and offer to "help" this person - kindly explaining the problems with, say, RedHat package management. Then there's the inevitable "scary talk" about what could go wrong, and lots of offers to do things for them.

I've found this plan to be quite effective in the vast majority of cases. Today was no exception.

I've been carrying a decent laptop for a lot of years. I've come to rely on it not only for electronic documentation (which was the first use I had), but also for staying connected, using online docs, keeping my calendar... just about everything. And my backpack is like a purse. There's everything in there I'd possible ever need.

I have a flashlight... and spare batteries... and a spare batter for my cell phone... and a PSP for games if I get bored on the train... and my Kindle... and a magnetic screwdriver... and the train schedule... and band-aids... and my iPod Touch... and my paper documents from work. You name it, I've got it. If I need it, I can reach into that guy and pull it out.

The same is true for my laptop. Just this afternoon, a friend chatted me asking me what I had planned for the long weekend. Long weekend? I had no idea what he meant. Why? Because I hadn't put in the holiday calendar for the year (2009) yet. When I pulled that up on the corporate web site and checked - sure enough, Monday is Martin Luther King Day, and we have off! Sweet!

So I put in the Holiday calendar and away I go - good for another year.

Of course, iCal is about the best little free calendar app I can imagine. I can move individual events in a series around if I need to - without interrupting the series. That's great for holidays on Monday when I need to send in my weekly hours on Friday because that Monday I'll be out. It's just a fantastic little tool. Love it.

It's just funny to realize I would have come in without knowing this. Silly me.

Today I did a little coding on my fast tick server because it was giving one user in London the head fake, and rather than ask them to understand that it was just that, I decided to fix the problem so it didn't misrepresent the data to the data maintenance team.

The problem was really that I worked hard to make sure the data in the server was maintained properly, and didn't spend any time thinking about making sure it was visually updated properly when it corrected itself. So if the user made the change, everything was fine, but if the system corrected itself, the user might think it hadn't. And then think that it was wrong when it really was just fine.

The code change was minor - maybe 10 lines, and the same code was used in at least one other place, but re-factoring it wasn't really necessary due to the locking constraints, so I just updated the code and it worked like a charm. I didn't have the heart to tell the user that they changed nothing, so I let them believe that I had found a bug in the way the data was moved around, and I did, but the effect is solely for the human's benefit.

But hey... it's always fun to code. Right?

The Shop is a heavy user of chat - be it IRC-based (as it was for so many years) or the 'secure' chat that we now use, it's chat. It's very useful as a conduit to the users for all kinds of information - especially the monitoring of servers and processes. This morning I did a little 5 min job to convert back one of the server monitoring bots from the 'secure' chat back to IRC chat as we have one of the latter, and the global messaging group which controls the former is still not allowing bots back on their network after a particularly bad server meltdown caused by a few bad bots.

I wanted to move this one guy back to straight IRC because it's a very nice example of a perl-based bot that can monitor all kinds of interesting things. I then spent a few hours crafting a monitoring bot out of this starting point to replace the java-based monitoring tools that had caused me so much trouble yesterday. I asked the guy that created them to turn them off - save one development box (his choice), until such time as we can be assured that there's no conflict in the communications.

His response was that these monitors are providing vital data on the status of the ticks flowing from my injectors into the system(s). I told him I totally understood his position, and if he'd just let me know what the processes were monitoring, I'd be glad to give him that same functionality, quickly, in a less intrusive monitoring framework.

I didn't hear from him, but I started one anyway with the likely candidates. I also talked to the head support guy and he had a simple little monitor as well. I included his test in with my code and started banging on it.

The first cut was nice, but I also wanted to add a little additional chatting when a stalled log restarted so that the users monitoring the chat channel would know that the problem corrected itself and they didn't have to worry about restarting anything. In this particular system, it's common for an updating process to get stalled and then restart without any intervention. I wanted to make sure that this was being passed on to the operators so they didn't worry about a problem that's corrected itself.

I sent out an email explaining it, and how to stop/start/restart it along with where it chats, and what it chats about. Basic information. I haven't heard from the group that put the other monitoring tool, but I'm guessing they are not going to be happy about what I've done. Not in the least. I hope I'm wrong, but in the past there has been more than a little animosity between my group here, and the group that did the other tool. Sad, but there's nothing that would have kept them from writing the same thing. Nothing at all.

I've had quite a day today... seems when it rains it pours, just like the salt. The latest little nugget was a possible interaction with a Java-based monitoring toolset that was written by another group in the Bank. I'm sure it's got a lot of potential to monitor Java-based applications and possibly even system processes and scripts, but we ran into a really nasty little side-effect today with a program I wrote that uses a third-party API message bus.

I've had a lot of problems with this particular message bus API - it will seemingly hang on input of a message. I believe that it's related to the socket communications the API is using, but I can't prove that because the vendor won't let me have access to their protocol. But that's not the biggie.

The biggie was that this new monitoring app was put on the production and UAT machines with my software without any word to us as a "heads up". Things started failing and when I couldn't understand why they weren't working, I saw this monitoring package on the box. I killed the monitor and Shazam! everything worked.

While I think monitoring is good, and automated monitoring is really nice, it's got to be zero impact. You can't have a monitoring package that interferes with the normal operation of the application - and given that you can't know how the application works, you need to be very careful about how you communicate with your clients, etc. This package might not be really good, but in this case, we need to be a lot more careful about implementing it in our environment.

The twist in the system is that the people that put this monitoring package on the machines didn't bother to tell us what they were doing. They're supposed to be part of the Team working on this system, but they acted more like a bunch of little kids about this issue. They wanted to be able to do something "on their own" and so rather than talk it out and so it safely, they decide to go "joy riding" with the UAT and production systems. If they had done their homework and really checked things out, then this would be fine with me. But they didn't, so it's just another example of their unwillingness to really follow-through and do a good job.

I came in this morning and watched something unfold that reminds me of highschool - and not in a good way. This morning a co-worker came in and saw that his cube had been filled with all kinds of stuff - boxes of tickets, trash can, all kinds of stuff. Making it clearly impossible for him to even enter his cube - let alone work there. Clearly, this was a prank. It happens quite often around here - for a business office. If this were a movie theatre and we were all still in highschool, then it'd happen more often, but we're not, and so it really shouldn't happen at all, but it happens every couple of months.

This time, it was too far.

The guy who was pranked took all the stuff out of his cube and started filling up the alleged prankster's cube. I asked him in the process of this "re-pranking" how he knew this was the guy. He was angry and pointed out to me that this guy had hassled him all day Friday after he asked someone to move some boxes because they were in the way of him getting to a few machines he had to work on. Not an unreasonable request, and handled very politely.

The prankster thought differently. Or so the prankee thought.

When the alleged prankster arrived to see his cube filled with crud (some of which was new to the re-pranking), he walked over to the re-prankster and said "It wasn't me. I know you think it was, but it wasn't me. Can you help me clean this up?"

"Nope. I don't believe you." he replied.

This continued for about 15 mins each one trying to convince the other that they were in the right. That's the thing with pranks - you think it's nothing, so why get all upset about it, but when someone gets upset, as they have every right to, the praknster gets offended and makes it about the intolerance of the prankee.

Yup, highschool, all right.

In the end, I have a feeling this is going to escalate until management gets involved. These guys have been at this for years and years, and it's not stopped yet, so I have no reason to believe that it will stop with this re-pranking. Who knows... maybe it will. I sure hope so because I have no desire to be collateral damage in this fight.