Launch Day!
Well, I didn't realize what today was, so I was a little surprised to see that today was Launch Day for the project I've been working on at The Shop. It's nothing that I'm worried about, but it's been a very busy day considering what's going on.
The day started out with problems in the production runs, but I was in early enough to make sure that things worked out, and then we're off to the races. There were a lot of things still to do, and today I started a few of them and passed off a few to some of the guys in the group.
First, we need to look at the performance. It's horrible. I mean really bad. It's taking an hour to run a division on the production "box" at EC2. Compare that to running it on my MacBook Pro at about 5 min. and the problem is evident. We need to fix the I/O issues. Maybe it's all in the resources of the EC2 machine, but maybe we have what we have and need to change the code around to make it more efficient.
Right now, we need to up the resources, which I did, and let the runs go again and see what the performance differences are. I'm hoping to see significant improvements, but I'm guessing we won't see real improvements until we get the dedicated hardware at the SNC1 datacenter when that arrives. Still… cutting down the hour to 30 mins or so would really be nice.
Second, we need to have a lot better handle on the performance metrics. We have NewRelic wired in, but not nearly enough. We need to make it a lot more interwoven. So we took the time today and added a lot more instrumentation in the code so that come tomorrow we will be able to see a lot more of the breakdown of where the time is being spent.
Third, we needed to fix a bit in the code regarding merchants with closed deals that haven't run yet. In the existing code, these were considered merchants that should be contacted again, but in reality they shouldn't. So I needed to modify the Salesforce endpoint, and then put that to Staging and write the code to pick it up and do the math that was needed. It's not hard, but considering that we're live, it's something that needs to be done today.
So I spent a bit of time this afternoon getting the Salesforce changes in, checked into git and tested and going in UAT. Then I felt with the Salesforce guys to get it pushed to production. I needed it there in order to make sure it was going to run well tonight at midnight. I'm glad to see that it all got done.
Finally, there were a ton of little issues from the initial user feedback. Nothing major, but lots of little things that needed to be cleaned up before tomorrow. It's been a heck of a day!
Hopefully, it's going to work well tonight. But I'll be sure to be in good and early (like usual) to make sure that things are ready for the day.