This morning I feel sick - I mean really sick! I know it'll pass, it was dinner last night, but it doesn't make it any better. I feel like crap, and there's nothing I can do to feel better other than give it time.
I'm sure by later this afternoon, I'll feel better, but right now my guts feel like someone stabbed me with a saber. It's nasty. But I know it'll pass… so I go to work and get on with life.
Crud.
This afternoon I had to make a lot of tools for supporting the app I've been working on. I'm getting bug reports, and I need to look into the data we're generating, and the only place it really all sits is in the CouchDB. So I had to write some scripts to hit some views and then load up the values I want with some key restrictions. None of it was all that hard, but when I needed to make a change to one of the views on the production database, I had to wait upwards of a couple of hours for Couch to regenerate the view due to the amount of data we have, and the limitations of the EC2 instance we're currently running on.
Soon enough, we'll be on our own hardware in our datacenter, but it's hard to beat the spin-up time of an EC2 box. It's pretty nice. But the performance isn't all that great, and that's what we're really running up against - I/O to the disk and CPUs to process. I'm hoping that as soon as we get the new machines, things like this will be a lot faster.
All told, I got built what I needed, and it's just a matter of time to be able to use it.
[10/9] UPDATE: worked perfectly! Once I got in this morning, I was able to use these views and tools to look into the bug reports and prove that they aren't bugs - the code is working exactly as designed.
OK, I've got to believe this is Photoshopped, but either way, it's pretty bloody funny:
There have been a very select set of political images that make me laugh, mostly because they all typically make me upset, but this is a rare gem. Really quite funny.
Thanks for the laugh!
UPDATE: it's not Photoshopped! I saw similar pictures from a campaign stop, and it's legit. Now, of course, the girl isn't reacting to The Romney Toot, but it's not faked! Too funny!
OK… I was just watching twitter this morning, and this came across, and I just started giggling:
Wil is a character. Smart, funny, witty, but a character that is totally unafraid of calling out someone or some company. It's nice to see it, as so many of the tech community has enforced biases, and that's not really doing anyone a service.
Funny, funny, tweet.
Today I was surprised to learn that cron jobs are not run in a shell. That was really a surprise. The reason it shocked me was that I had the following in a crontab:
15 05 * * mon-fri $HOME/bin/startQL | $HOME/bin/campfire -p
where I'd written a nice little bash script to send a piped message to our Campfire room, and I wanted to pipe the output of the startQL command to that room so we'd be able to see what happened without having to check our email.
The script takes stdin and sends it on as a pass-through, but with a copy being send to Campfire. It's a pretty nice little script that just uses the Campfire curl API:
What I found was that the pipe wasn't working because we weren't in a real shell. It was easy enough to fix - just put the commands in a new file and call that, or use a subshell:
15 05 * * mon-fri ($HOME/bin/startQL | $HOME/bin/campfire -p)
Learn something new every day!
So I've been working at home this weekend, and I have to say, it's an amazing experience. I can take a little time Saturday morning and fix up a new data import table for the business user and not sweat about it Monday morning. Liza and the kids are asleep, and I'm awake, and after a nice bowl of Raisin Bran, I can work for an hour and get some stuff done without having to worry about what I'm missing with the family, or making someone angry that I'm too involved with work.
I can pop open the laptop, write a little code, do a little magic, check it in, do a run, and then call it done. That's pretty sweet!
This is what I'd hoped for when I thought about developing. Something as effortless as just doing the work. Work a little when you have the time, and then during the week, pound out some major hours. But always be flexible enough to take a trip to the grocery store if needed.
When I think of the joys of self-employment, that's it. If I could do that with the place I am now, I'm not sure I'd ever leave. Ever.
It's just that great to me.
Today I was trying to get some work done (from home!) and I got a chat from one of the Data Sci guys I work with in Palo Alto. He was saying that he couldn't find the data I'd just generated into CouchDB. Because Couch is a NoSQL database, we needed to enforce some kind of structure to the documents, and what we chose was the idea of an execution tag. Currently, it's the date and time of the run - to the millisecond, and the name of the division it's running. This is calculated once at the start of the run and then used in all logging and persistence stores so that it's easy to correlate all the data for a given run.
But today, it seems, the execution tag wasn't constant after all. That's a big problem.
So I started to track this down, and I very quickly came to the realization that the massive refactoring that a co-worker had done to try and streamline logging in the application really wasn't preserving the essential nature of the execution tag. It was generating several of them - all a few seconds off.
Now, don't get me started on the need for logging. It's what I think of as essential to serious, commercial, production apps. You just gotta have it. But there seems to be a group of folks that wants to get back to the APL days where there's no logging, no comments, and the strong belief that the code speaks for itself.
I can't possibly disagree more with all these points, but that's another day, as I said. But not respecting a critical thing like the immutability of the primary correlation factor in the data is kind of a big deal. But it's really a detail, isn't it?
The tests all passed - and the code ran, so it's got to be good, right?
Unfortunately, that's far too often the case with a lot of the younger developers I'm meeting these days. Nice guys, and they are capable of writing good code, but I think they just don't want to put forth the effort it takes to pay attention to all the details. After all, there are a lot of them in even the most simplistic of business apps.
I've talked to a lot of younger coders, and they are self-confessed "Lazy coders". Now I know they aren't really lazy, they work hard, but they aren't interested in paying attention to the details, and that's where everything of importance is to be found.
I just wish I could teach them of the need, but I think the only one that can teach them that, is Time. She has a funny way of catching up to even the smartest of folks.
So today, for the first time while being at The Shop, I did a Work from Home. There have been a lot of times in the last three months that other guys in the group have done it, but today was a first for me. I have to say, the IT guys at The Shop know how to do it right. I've been at a lot of places with VPN access, and it's always the same: you either control a desktop with RDC/VNC or you have full VPN and it's Cisco and locks you out of all other network activity - save that that goes through the VPN.
The Shop does it right - all SSH tunneling and it's all automatic. I open up my MacBook Pro at home and it's exactly the same as being at work. There's no configuration changes, there's nothing to do, or worry about. Email is clean, calendar is there… it's all working just like I was sitting at my desk at work.
Of course, this means working at Starbucks or Panera is also as nice.
Holy Cow! I'd dreamed that this could be possible, but I've never really been at a place that was done this nicely. It's effortless and seamless. You can't ask for anything better than that.
Today I was looking at the check-ins and saw something that was a good idea, but it wasn't really completed. I knew the guy on the team that did it, and I could easily see why he'd stopped half-way through: he didn't see what he'd really started as something more than a little hack to the problem he had. This is one of the things I'd like to be able to teach people, but I'm worried that it's really something that people have to want to see.
Initially, we had one source for merchant data, and one source for demand data. To be fair, we had two: fixtures (static files that make offline testing nice), and a "live" source. We could switch between these with a simple config option:
# control whether to use the real data sources use_fixtures: true
or:
# control whether to use the real data sources use_fixtures: false
But we're moving forward, and it is very reasonable to think that we have multiple sources for each of these. Right now, we have the need to think of multiple sources for the merchant data, while just two for the demand. The coder implementing the need for the multiple sources for merchants added a config option:
# control whether to use the real data sources use_fixtures: false # select from 'fixtures' | 'sf' | 'm3' merchant_from: 'sf'
to indicate that the merchant data is coming from Salesforce. We can add other options in the code, and this is very nice and extensible.
The problem is that this really doesn't address the logic inconsistency of having those two options in the same file. If we're using Salesforce for the merchant data, what does "using fixtures" mean? Why not just add:
# select from 'fixtures' | 'sd' demand_from: 'fixtures' # select from 'fixtures' | 'sf' | 'm3' merchant_from: 'sf'
so that we're specifying all the inputs the same way?
This is the finished idea. What was done was only really half the problem solution, and in leaving it half done, it really lead to a lot of confusion that just didn't need to be there. It's this insight that I'd love to be able to teach coders that don't see it.
But given that this coder is way past 20, I'm guessing he doesn't consider himself "junior", and therein lies the other problem: He's not thinking he needs to learn this. Sad, but true.
Anyway… I have to check these changes because of just this level of incompleteness.
This is something that I find very interesting about people in the business world - their almost complete reliance on Excel. It's as if they can't even function without it. Case in point, JSON data. Here's a section of a file that I sent to a business person today:
{ "cleveland": { "Academic Tutor": [100.0, 100.0, 100.0], "Airplane": [100.0, 100.0, 100.0], } }
and there are clearly more lines in the file than this, but they all follow this same pattern. I thought it was pretty easy to understand: City, then Service, and then monthly values (12 versus the 3 I show).
But once again, I'm surprised.
The response was "Can you put it into something I can export in Excel?"
Forgetting that they don't know the difference between import and export, I find it really hard to look at the file above and see how it's different from Excel - save the graphs. But you know… that's the tool they know how to use, and that means that's the tool I need to give them data for.
So I modified my code to spit out CSV or JSON.
Worked just fine, but I'm still wondering what the real limits are for a lot of business people.