Adding Email Send Messages
Today I got news that the email send log messages were now including the deal UUID so that I could generate the daily_send messages in the unified message stream. This is something we've been waiting a long time for, and as I dug into this, I realized very quickly that there are a number of really significant issues in adding these messages:
- There's a whole lot of them - like 10k msgs/sec just for the sends, and each send has anywhere from 6 to 12 deals - which translates to upwards of 100k msgs/sec just for the sends. That's a lot.
- There's no user identifier - there's a UserId, but it's not really what I need because it's coverage is not really complete - missing in 3.7 million messages in 4 hours today, and it's only useful for US and Canada. We need to move to the UUID identifier that's good for international locations as well.
So I looked into ways to solve this, and it's possible, but it's not easy. I could map all the email addresses to UUIDs for the users, but then that's a map of 110 million - plus - emails. That's a lot. And keeping it fresh is going to be a challenge due to the size and the frequency of additions/changes.
And this is only for the sends. The email opens are a different stream, and that was going to require that we cache the data from the sends to know what to send for an open. Thankfully, there was a ticket in the system already to add the deal UUIDs to the email open message, and I created a ticket to either get better coverage on the UserId and/or add the user UUID to the sends and opens so that we can use the data in the message(s) and not have to have any lookups.
I have no idea how long this will take, but until I get some support from them, I really can't go a lot further. Nearly a million malformed messages (missing the UserId) every hour is not a good way to establish a dataset. So things have to get better upstream.
Hope so.