For the second day in a row I'm leaving on the last express out of the city. Man... these are long hours. I can't remember working this hard, but it's the weekend, and I can't work at home because I can't take it with me, and I have no remote access (yet). So it works out that I have a little 2-day vacation.
I need to get some sleep...
We're making progress, but it's still a long way to go. This "Developer Day" project is turning out to be a lot more than a day. No question. But hey... with this part done we'll be ready to really deliver something really great. I just have to make it happen. Gotta hold out.
It's been a lot of work, and we have a little more than a week to go, but I think we can make it. I'll be close.
This morning, once again, I'm getting timeouts hitting the mongoDB replica set in staging at The Shop. I know mongoDB is supposed to be this amazing piece of software, but I'm seeing it right now as a major pain in the rear. I'm looking at a database whose size right now is less than 10GB, and it's a major source of grief. We are having replication issues, and the erlang drivers had to be changed, and then retrofitted. The Java drivers had to be updated to get document sizes greater than 16MB.
But those are the known issues. It's these gotchas that are really annoying. I'm trying to get a single document from it and it's timing out. I get no idea why. One box it's fine, another box it times out. Same mongoDB, but different boxes. This worked yesterday, so what's changed overnight?
Arrggghh... I'm really beginning to hate this software.
UPDATE: it turned out to be the socket buffer size on the default socket for erlang. It was at something like 1kB and I was moving MB through. Very inefficient. When we got the socket options set right, the speed wasn't too bad. Not fantastic, but not bad.
Today was spent in a few meetings, and helping get a few developers on the right track, but when I had the time to do a little coding, I was working on getting everything glued together for the greek engine. Specifically, the first part of the engine will be a Broker service that will do all the what if scenarios for the users. This is something that the folks have wanted for a long time, and it makes a lot of sense to deliver this first, with the large-scale, center-state, reliable multicast, greek engine put together from these same components once we get this guy up and running so we know how much hardware to spend on the problem.
This phase of the project is nice in that it appears to be making a lot of progress in a very short time, but in reality, it points out where you made mistakes in the design up to this point. If you have a good design, then it indeed goes together pretty quickly. But if you missed something, then you have to go back and fix or retro-fit that, and then put things together.
Interestingly enough, I had a little of both today. My general Source and Sink objects needed to have locks provided on their lists, but that's because I was starting to violate the assumptions that had been made in their original design. I could go back and implement a lockless list, but that's a little overkill, I think at this point. If we need it, I know how to do it, I just need a little time to actually do it.
It's not done by any means, but it's getting there. Slowly, but surely.
This afternoon I've been helping the team get things straight for the next few days, and starting to pull things together in how we'll present this to the clients for their consumption. One way would be to have another ZeroMQ PUB/SUB reliable multicast system, but that's really overkill. What we really want is to have a point-to-point, one-on-one conversation with a client so that they can tell us what they want, and we can supply it for them.
Things are starting to take shape in my mind, but it's always better if I noodle on them for a while.
This morning OPRA, the Options Price Reporting Authority, is changing their distribution from a 24-line multicast feed to a 48-line feed to make it a little easier to balance the load and allow for receivers like us to actually do something with the data we get. So this morning when I got in I changed all the configuration data from the 24-line set-up to the new 48-line set-up. In truth, I'd had the configuration for a while, I just didn't update the config files as I didn't need them. Today is the first day I need them, so they'll be live and ready for me.
Progress... there's no stopping it.
So I've been working on this little service for my greek engine - it's not a major component, but its' something that finds use in the Shop, so I was replicating it's functionality in the new codebase. One of the things that the legacy messages had was the OPRA Message Type for the trade messages. This is a one-character field that says what kind of trade this message describes. Is it a cancel? Is it electronic? There are a lot of meta-data you could have about a trade, but typically, you want to take it out of the exchange-specific realm, and put it into bit flags, etc. Make it source-independent.
Which I had.
Then I came upon this legacy message and saw that it had this OPRA message type as a field. I asked around, and was told that I needed to have that in the message. That's odd. Very odd. This means that every app will have to have the same logic for what this "means" to the trade. This doesn't make a lot of sense at all. In fact, I think it's silly.
But it's a requirement, so it's in. Silly. Totally silly.
UPDATE: after another meeting, it was the consensus that this wasn't such a hot idea, and that we should try to live without it. OK with me. Simple git revert.
This afternoon I'm onto another problem with The Broker, and this time it's really difficult to figure out because there have been a lot of changes made to the codebase, and none of them are documented in the least. The problems include the immeditate unregistration of services after they have been registered, as well as not accurately identifying those services that aren't available to the client.
I think I was getting close to the answer, but the erlang code is just too functional, and it's hard to know where something is called from if you don't have a complete stack trace. In this case, I don't - or at least don't recognize it if I do. I'm about a 5 or 6 out of 10 in erlang, and that's not enough to really be able to dig all this out of the code without some form of documentation to help me know what role each module plays in the overall scheme of things.
In the end, I was able to document what I saw, what I think the problems were, and how I'd go about fixing them, and sent that off to the guy who wrote all the code and is far far better at erlang than I am. I'll have to wait and see for tomorrow when he returns.
This morning I finished up on a nice simplification that I saw in the Time and Sales service: the temporary structure I was using to hold the data from the Option and the Print was really a new message - the OptionPrint message. This is exactly what we need to send out to the new clients, and a version of this can easily be made to send out to the legacy clients.
So I gutted all the code in the service and made the new message in the old messaging codebase, and then retrofitted it into the new service. In all, it took me about half a day - finishing up this morning. But it's worth it.
Now, all the pieces fit again - if we wire up a transmitter to the service, it'll automatically send the messages out the proper channel - be that legacy or new. It also makes it very memory-friendly as the same structure that's hold the temp data is the message we'll be sending out. That means there's no conversion to a message - we just ship what we have.
It takes no more to create, and it's far more efficient to use. Sounds like a win to me.
This morning for the second time in about as many weeks, another group in The Shop decided to update their Mongo database, and it's going to cost me most of the day to fix my code because of the change. They say their java client to mongo was not allowing them to use larger than 4MB documents, but I've been storing 16MB+ docs in the database with the erlang driver. But they didn't ask me before they did it. They just told me they were doing it.
In going from 1.6 to 1.8, it turns out that there's now a hard and fast rule about docs being less than 16MB - so I get messed over. I am going to have to look at how all my documents are created and make sure that no one gets too big because the failure will hang my process. It's ugly - really ugly.
I think when I'm done with this, I'll try to put something into the configuration service that errors out if you send it a payload that's more than 16MB. This way, it can't let you save something too big, and erroring out is better than locking up every time.
Still... it's these "detours" that are really getting annoying - and quite avoidable. They could put up a dev environment and we can test, and then work out solutions before doing it to the staging install. That's what they should be doing. I've no reason to believe they will, however.