Adding Transactions to a Servlet-Based System
Today I had the difficult task of trying to add in transactional integrity to a web system where the data is coming in as posts to the web server. If each post was a transaction, that wouldn't be so bad, but if I needed to have transactional integrity over multiple posts, then we get into a lot of trouble. Face it, web servers aren't noted for their state-maintenance - that's something you add on top of the web server in order to create the illusion of saved state for the user.
But in this case, we had one program feeding another. The sender wasn't passing in real markers for the beginning and end of a transaction, and that was the first thing that needed to change. We then needed to do something with this information, so I added that in. Now I had a place to handle the meat of starting and ending a transaction - but I needed to know what to put there.
The next problem I attacked was a little simpler, the market data service wasn't handling the blocks of data in a unit. Rather, as it was parsing data, it was sending it to it's cache. This would allow for inconsistent data as the instrument prices move but the greeks were in a buffer until the entire block was read. Also, the market data cache had no locking on it. So I added the locking and the buffering so that the market data was "clean" for each block.
But that still didn't solve the real problem - how to put multiple blocks in a transaction?
After thinking about all the alternatives, I came to realize that the only good way to do this is to have the sender "mark" each block of a transaction - including the BEGIN and END, so that the receiver can buffer the data by transactionID, and then release all that data and update the reports on the END. So I needed to figure out a way to get these transactionIDs.
From the UUID work I've done in the past, the IP address turned into a long was a good start. I didn't need to add in the time, as I wanted to have a transactionID per sequence, and not per transaction. So no need for the time. I did add in a simple three-digit sequence number on top of the IP address, and that should do it just fine.
Now the transactions needed to be tagged with this number. That meant modifying the payload format. Not something I was fond of doing, but it could not be helped. Did that, and decoded it on the receiver (web) side. Then I had to modify the buffering of the market data and the greeks to buffer by transactionID. I had to thread the transactionID into the code - passing it from the decoding through all the method calls all the way to the market data methods and greek cache methods. It wasn't more than three levels, but it required a ton of changes to the unit tests.
I then had the transactionID to the right places. I added that to the buffering in the market data and greeks cache, and then was ready to update the way the update events were sent. Previously, after each update of a block, the values would be recalculated. This almost guarantee problems with updates as the values would be recalculated within a "transaction". Bad. So I changed all that to only update if there was no transaction active - or at the end of a transaction.
The results are really impressive, but I want to do more testing tomorrow. It was really pretty simple once I had decided the best approach.