Archive for the ‘Cube Life’ Category

Chasing Elusive Performance Bottlenecks

Tuesday, November 3rd, 2009

Today I've spent a lot of time trying to find performance where there was no guarantee that it should be found. I had a strong feeling that there had to be a better way, but I had no proof. I have this application that needs to talk to a legacy system through a DataSource connector (yeah, Windows) to get data out and into the app I inherited. Doesn't sound too hard. Yeah, right.

Well, this is complicated by the fact that there are a few ways to get at this data, and I was looking at different access schemes and thinking "If I could get the speed Excel has in getting the data, I'd be OK". So it became a quest for how Excel did it, and matching it's speed.

I started with the existing code - using an OleDbAdapter. This was taking a horribly long 130 sec. to get the data that Excel could return on my desktop box in about 12 sec. That factor of ten was killing me. I needed to run it on the VMs, but hey, they're supposed to be virtual machines and match the performance of my desktop - right?

So I tried an ODBC connector. That didn't get me anywhere because the methods to populate the .NET DataTable weren't implemented in the ODBC driver. After I passed this back to the developer that pointed me in this direction, he agreed that yeah, this wouldn't work. Nice.

Then I tried a more "bare bones" approach - getting the data at a low-level array and then pulling that apart to make a DataTable out of it. WHen I got that one into test, I realized that the time was even worse. Not the right direction to go in.

So finally I started running all the tests on all the machines I had at my disposal: my desktop, my Test VM, and the Production VM. The results were shocking.

Machine Excel
Query
Code
Query
Desktop 12 s 11 s
Test VM 20 s 75 s
Prod VM 26 s 130 s

so while I might see that the difference between a VM and a desktop is somewhat fixed for the Excel query, it's hardly fixed for the programmatic query.

Why the difference?

My best explanation is that VMs are horrible for anything that needs to use a disk. The two VMs have different disk subsystems and neither is really good. We are getting a new SAN, but that's going to be a few weeks away, still. The performance should then even out as all the VMs will be moved to the new SAN, but until then, we're stuck with what we have.

Additionally, the desktop seems to have consistent performance in either case. While this seems logical, it also seems to point to the fact that the Excel query is much less memory-intensive. I can see it pulling in the data in a stream. The code query has to get it all in a DataTable and then process it. I can see this being a big difference in the table sizes we're looking at.

What I'm going to try tomorrow is to get a plain desktop and try to set it up as the compute box to replace the production VM. This is not going to make me any friends in the support group as they hate having production apps on desktop boxes, but hey... this is an order of magnitude difference. We can try to get XP-compatible servers and replace the desktop with a server-room solution when we can get it in, racked and powered. But for now, the users are screaming and I need to give them a solution.

Amazing how things work out.

The Ups and Downs of Programming

Monday, November 2nd, 2009

Today has been a real up-and-down day. It started out very nicely when I was able to come back in after a day off helping friends in Indy and see that the work I'd done for the persistent state of the alerts I'd been working on last week was really rather good. I had a few typos in the code, and one logical mistake of not returning the value from a map look-up, but that was it. I was really very happy with the way it was working out.

But that happiness didn't last long.

I got an upset call from one of the bigger users and they were very unhappy about the state of the performance of the updates from the system I inherited. There's not a lot I could think to do about the issue, but I had to dig in and give it a go.

What ensued was a day long realization that not a single soul had solid understanding of several key components of data-access code that I was forced to use. It was frustrating to say the least, and because I made very little headway today, I know I'm not going to be out from under it tomorrow.

Yeah, it went from great to crappy in no-time flat.

Development Isn’t Always Glamorous Work

Wednesday, October 28th, 2009

Today I was doing a lot of little things to my web app. Nothing really amazing, but in the end, they added an amazing bit of functionality to the alerting system. I added in the thresholds for each desk so that they can be used as variables in the alerts which means that we can make the system a lot more proactive than it is now when it comes to desks getting near their limits. Adding the variables to the code was easy, and not really amazing in and of itself, but the larger effect is pretty nice.

Anyway... lots of work and not a lot of interesting things to show for it.

Contemplating the Ideal Structure for Application Development

Monday, October 26th, 2009

Professor.jpg

I've been thinking a lot lately about the ideal structure for application development. It's more than just what makes the developer productive, it's also what makes a great application. There are components of infrastructure, user-interface design, support, reliability, stability, responsiveness. All the things that work together to create truly exceptional software.

I've been thinking about this because recent jobs have had vastly different models for how things should be done. Is there a QA department? Is there an Operations group that must take ownership of the developed product? Who supports the app with the end users? How does feedback make it's way to the developer from the users?

There are a lot ways to do this - probably as many as there are companies on the map. But I'm looking at the few examples I've been exposed to up-close, for an extended period of time.

The Cube Farm - Stick to Your Assignment, Developer

Probably one of the most Dilbert-esque, and therefore, most common, forms of organization is to function under the belief that management knows best. That people can really be modeled as ants in an ant farm - interchangeable, fungible, assets that can be told what to do and they will execute on those instructions as well as if they had any input or control in the decision-making process. It's born out of the idea of the old Peter Principle:

In a Hierarchy Every Employee Tends to Rise to His Level of Incompetence.

Developers that are really good developers must be good managers, so they are put in charge of a small team. Do that well, and he must be a good manager, and so put him in charge of a few teams. This continues until he's no longer really good at his job, and the promoting stops. However, the damage is already done.

You now have a great developer in management, and so he believes that he knows how to do this development, and he's right - he knows. But only about how he does it. If there's a person that's really effective because they write the docs first and then use that as a template for the system, that's great. But it's not a universal imperative. Not everyone will work best that way. Maybe some people see a design unfold best when they start writing code and see where it takes them. This might be along the lines of the Extreme Programming movement. No two people are really interchangeable. Not really.

In this kind of environment, there are Teams for everything. There's a Team for requirements gathering. Another for design. Another for development. Another for documentation. Another for Q/A. All these groups are designed to be very good at their jobs, and individually, they are. But that doesn't alter the fact that the line of communication between the users and any one individual is very complex, and error-prone. It's like the school-kids game of "operator" - each kid whispering the word they heard to the next, and at the end of the class what comes out is often times not at all like what went in.

To get the users the very best application, every person in that complex system needs to be aware of what is going on in all the different groups. If they hear of something that should be told to another group, they need to be able to get that information to that group. Unfortunately, that's not how these groups typically work. The larger the group the less effective the communication. When in order to work, it needs to be just the opposite.

So while this seems like a good idea for development, it's never been really good at delivering a great product to the users. It can get useful stuff out there, but it's at the cost of time. This kind of structure runs very slowly.

The Tiny Shop - Jack-of-All-Trades Developer

One could think that the other end of the spectrum is the indie developer. Let's surround him with a few partners and call it the start-up or small business developer. This guy has to do everything. This includes doing the sales, marketing, support, requirements gathering, design, development, deployment, etc. There's just no one else to help. In my opinion, there's a lot to like about this situation, having been in it before - there's no one else to worry about dropping the ball. You have to understand the problem domain in order to write the system, but that means you have to really understand your customers - never a bad thing. You have to be able to balance the infrastructure "cool" with the practical and the user-interface - it has to perform acceptably, but it's got to look nice as well. You have to be good at almost everything in the pipeline.

There are plenty of downsides to the person that's a little weak in one or more of these areas. If you're horrible at UIs, then you're going to be very uncomfortable with the fact that there's no one else to lean on for a UI for your back-end. The same would be true for sales or support or domain knowledge. If you're not willing to come up to speed on these areas - or shop them out to someone who is, then you're going to be uncomfortable. But for those people that are used to this, it represents a wonderfully freeing opportunity.

You can write your own code because you understand the domain. Be it a drawing app, a photography app, a simulation of some kind, or something as focused as a cellular image processing app. If you understand the domain, and can code, this model really allows you to control the entire project - much like the old masters painters. They sat in a room, conceived of a painting and then did it. Period. There's a lot to be said for those that can do this, to enjoy this more than working as a single member of a much larger team.

But it's not without risks as you're only as good as your weakest link, and if you're doing it all, your product is only as good as your weakest subject. This isn't for everyone, that's for sure. But it's something I've had experience with, and found very satisfying.

The Partnership - A Hybrid Approach

In engineering, there's always the continuous spectrum where an optimization is done to find the most effective solution to the problem. Given that this is my background, I have to look at this same problem with this same bias. Where is the most effective balance?

My current thinking is that the best balance is a partnership with the right partner. You need to complement each other. I know how important this is as I've been in partnerships where this wasn't the case, and it ended horribly. You have to be willing to take control and take over for your partner when they are down, and likewise, you have to be able to trust your partner to do the same for you.

This doesn't mean that you need to identical skill sets, or even complementary ones. It's enough to overlap in some areas and be weak in others. It's part of the dynamics of the partnership. Have someone to lean on, but not someone that's going to ask you where the cover letter is for your TPS Report.

While I think there is value in a larger organization, and there's considerable freedom in the indie developer, an appropriate balance of the two is really where you have to live. If I had a bunch of money and never needed to make another dime, I'd be an indie developer and that's it. But if I need to actually sell something to someone, I think I'm shooting for the partnership.

Problems are Solved by People that Show Up

Friday, October 23rd, 2009

It's been a really awful morning today. When I got into work there were problems with one of the apps I inherited, and it was ugly. There were data issues that caused problems with the code changes I'd had to put in at the request of the users. It was my fault, but I'd been pressured to get the changes out as soon as possible by the users. I didn't get to do all the tests that I would have normally done and it got me. Period. No excuses. I had to make several emergency fixes and get things up and going, but in the end things were going OK.

It's not a fun way to start the day... no fun at all. But it happens when you rush things. I'm learning from this mistake and I'm not going to push things into production. Period.

The next problem was that there was a problem with one of the data feeder apps, but it wasn't at all clear where the problem was. I tried restarts... no good. I tried re-installing the little controlling apps that manipulate the feeder code... no good. I tried re-installing the feeder app itself... no good. I'm clearly getting desperate now because I have more than ten other boxes feeding good data - but there's this one that's not working and I'm running out of options.

But hey... problems are solved by those that show up, and I'm here and no one else is. So I have to find the solution.

I keep trying to reset things and in the end, I get a question from one of the users asking why the feeder was dead. I told him I was working on it, but it was dead. He mentioned that this wasn't the only problem - other apps were having problems this morning.

Bing!

There was nothing I could do, and there was nothing I had done wrong. Some bad data had gotten into the database and it was messing a lot of things up. When they fixed up the data, I was able to restart the feeder and all was OK. But for that first 90 mins of the day, I was in a near panic mode.

No fun.

But hey, problems are solved by people that show up.

Sometimes I just wished more people showed up. I could really use the help.

Just Plain Working Smarter Not Harder

Thursday, October 22nd, 2009

I was looking at the changes I'd made to the SQL in my web app to speed things up, which was really nice, but it ended up still being an issue. You see, the speed was nice when there were no INSERT statements hitting the database, but when I got into the thick of things today, I realized that the real issues were classic - locks, amount of data moved, etc. No way around the simple facts.

What ended up happening was the time climbed to the point that we got back to the 400 - 500 msec range. What I realized was that there's no "free lunch", and to get the times down to reasonable values I needed to look at what I was doing not just messing with the engine and tables.

What I had been doing was re-fetching the same data for the entire timespan and invalidating the cache when any data comes in from any of the sources. It's foolproof, it's just making me do the same work over and over and over again. The best I was able to do was a constant access time that increases during the day. But in reality, it's easy to think of a better solution.

Let's leave the table in-place and then "refresh" it by looking at the last acquired data point and then using that as the start time for the same request. I then could use that data to make an identical table, and then merge that table to the bottom of the existing table.

What I end up with is updating the last data point and then if there's more data, it's going to be added to the end of the table. If we have multiple requestors for the same table, the data will be added for each one. We put a nice little synchronization on the table being refreshed so that we don't have multiple edits at the same time, and all is OK.

What I've seen when I implemented this was a dramatic increase in speed. More than a factor of 10x on all the tables of data in the web app. It's going to make the load on the server a lot less, and we'll be able to handle a lot more users on the same hardware. Nice.

Really, this isn't a real surprise - you can work harder and get a little bit more, or you can work smarter and really make a huge difference. Should have done this earlier, but I didn't need it, so I didn't do it.

The Black Magic of Optimizing SQL Queries

Wednesday, October 21st, 2009

OK, so maybe it's not really Black Magic but putting yourself in the place of the query planner/optimizer is a really big help. For instance, today I was working on trying to get a little more speed out of the following H2 (JDBC) SQL query:

  SELECT acquired, portfolio, leadFuturePrice, pnlSumNet
  FROM RAT_FirePortfolio
  WHERE acquired >= '2009-10-20 00:00:00'
  AND acquired <= '2009-10-20 23:59:59'
  AND portfolio NOT IN ('ED_NRML')

and the requests were returning in excess of 120,000 rows of data in about 600 - 775 msec. Not bad for an in-memory database, but I was thinking that the use of the "not in ()" was going to slow down the execution of the query because it would have to check each of the row's portfolio fields against each of the elements of the list. Not ideal.

So I got the idea to change it to a more direct version:

  SELECT acquired, portfolio, leadFuturePrice, pnlSumNet
  FROM RAT_FirePortfolio
  WHERE acquired >= '2009-10-20 00:00:00'
  AND acquired <= '2009-10-20 23:59:59'
  AND portfolio <> 'ED_NRML'

So I coded that up. At the same time, I realized that the exclusion of 'ED_NRML' was really a transitional artifact and was no longer needed. There are times when we drop data sources, and this keeps the new version from falsely showing the existing, but old, data. Now that we have completely transitioned off that, we didn't need it and the query simplified to:

  SELECT acquired, portfolio, leadFuturePrice, pnlSumNet
  FROM RAT_FirePortfolio
  WHERE acquired >= '2009-10-20 00:00:00'
  AND acquired <= '2009-10-20 23:59:59'

The times for this guy are in the 50 - 57 msec. Amazing!

We're looking at a factor of ten by removing the test. Now I had suspected that the "not in ()" clause was a little bit of an issue, but I had never guessed it was 90% of the time! That's amazing.

I'll never doubt again. I need to really hit these queries and make sure they are as clean and simple as possible. Simply amazing.

Trying to Assert More Control Over a Wayward Application

Tuesday, October 20th, 2009

Today I spent most of the day trying to check the fix I made yesterday afternoon. I believe it's right, as I've stepped through the code and I can't see why it might be wrong, but I wanted the users to check the data. I was very hard to get them to verify the data.

The primary problem was that the data is sampled from one system, pushed into another system and then shown in a client. It's a bit of a travel, and the question is: How old is this data?

The problem I kept running into was that I could not prove the data right because there was much I could not tell about the processing. The original developer used Aspect logging, and while that did what he wanted (kept the logging out of the processing code), it meant that there were limitations on the logging because of that same fact.

I decided that it had to change, and started putting in the logging at all levels of the application so that when I was done, there would be one log file that would tell me:

  • what machine gathered the data for the server (by hostname or IP address)
  • what type of request was done
  • when it was sent to the extraction process and how long he took
  • time required to parse the packet received at the server
  • time required to incorporate the data in the packet
  • total time from start of grab to the successful incorporation of the data

This is the kind of logging this application needs to have. It's got to mean something and be useful.

In a way, I'm a little glad. After thinning out the old Aspect logging, it's starting to look like something I might have done from the start. I can quantify the delays so that I can tell the users exactly what they are seeing. It's a much better place to be.

They may not like it, but that's a different issue.

Nothing Like Getting Blindsided on a Problem

Monday, October 19th, 2009

Today I was trying to make headway on a problem I was struggling with and got word from the users that there was a data error in the web app I inherited - in production. Not good.

The problem was that they believed that the data they were seeing was not correct, and while I had changed a lot in the code, I hadn't changed that. Assuming they were right, it would mean that the code has been wrong for 9 months! Yikes.

Nothing like getting blasted from something you never expected.

I started digging into the code (no comments, of course) and realized what I thought was the problem. I spent the afternoon putting in a fix for this problem, and got it running before the end of the day. Unfortunately, there's not enough time to really check what it's doing, so I'll have to check it tomorrow.

I sure hope this works.

Problems with Large Java Web Apps

Friday, October 16th, 2009

java-logo-thumb.png

I've got two large Java web apps that I'm working on - one I wrote from the ground up and another I inherited. Both have the problem that they are getting very large for the boxes we have, and I need to be doing something about their memory footprint. The problem is, in both cases, the values are relatively important and thinning the data footprint out is not as trivial as it seems.

Multi-Level LRU Caches

In one of the apps, I had recently put an LRU Cache so that I would not have to save every value, only those that had shown themselves to be of use to the users. Sounds like a good idea. But there's a limit to that. Each operation on the BKLRUCache is going to re-order the aging list, and if a lot of updates come into the system in short order, there's a possibility that they will purge the cache of "good" values.

Sure, you can increase the size of the cache, but that defeats the purpose. You can also buffer up the changes and treat them as a single batch, but that has negative effects as well. No, the idea is to somehow "flag" the requested entries so that they stay over the "untouched" ones in the cache. But what's the easiest way to accomplish that?

What I came up with, after much hassle and arguing with myself, was a two-level LRU cache. The primary level is the same, fixed size. When the user asks for a value, that value is first looked for in the secondary LRU Cache, and if it's not there, the primary cache is checked. If it's there, it's put into the secondary cache so that it's not competing with the other entries that may never be touched by a client.

I don't have to worry about the size issues - they are independent caches, and I can size them to suit my needs. I also don't need to worry about the flood of updates wiping out the one previously asked for. It only has to compete against the other requests. Very nice.

H2 File System Database Tests

In my other web app I've got a very large H2 in-memory database. This is getting to be a real problem. So much so that I'm starting to think about alternatives to the in-memory database. There's MySQL, PostgreSQL, SQLite, even H2 has a file-system database mode. So as I looked at each of these I realized that there's no way I would be able to tell the difference until I started running some tests.

Given that H2 has good specs and I already had it in the project, it seemed like a good enough place to start. What I did was just to make it not in-memory, but based on the local filesystem (simple disk), and run the application and see what the results were.

I was stunned to say the least.

Location Query Format
In-Memory 300 -> 500msec 8 -> 21msec
File 3200 -> 3300msec 9 -> 12msec

I tried a lot of things, but I kept coming back to the factor of 6x to 10x in speed. It's just not possible to have a 3 second response time to the queries that I need to process. No way. So I must come up with something that makes it possible to get better speed while still not taking up a hundred GB of RAM.

No solution yet.