The Tech Trenches is a Complex Place to Be
I'm continually amazed by my attitude towards the work I do. I really love the work I do - development, but I don't always see eye-to-eye with the people I work for, or with. This afternoon I was able to add in the parsing and storage for strike-level volatility data to be stored with every update from select sources. It's going to be a ton of data - I'm guessing about 720,000 rows/minute or roughly 432 million rows per day. Each row isn't that big - I'm guessing around 30 bytes a row, forget the indexing, etc. the data itself is over 12 GB/day. Very big.
Still, it's what they want, and they are willing to spend the money to get the database storage for it. I just hope that I can add enough RAM to my boxes so that I can hold at least two days of data. We'll see as things progress.
But it's exciting to write good code. It's frustrating to see others butcher it, and I had a little of that too, but there's nothing I can do about that. It just comes with the territory.
Still... as with kids, you have to hold onto those nice moments and make them last.