Great Way to Skin a Cat
Today we've been once again fighting Salesforce.com's APEX APIs for getting Merchant data from their servers. It's a key part of our processing, and they have an API, but it's got some major limitations, and one of the problems is getting all the merchants for a division (like Cleveland, Cincinnati, etc.) of The Shop. As we were kicking around ideas, one of my co-workers came up with the idea to fetch all the IDs first, and because there's not a lot of data in just the IDs, it's possible to get all the merchant IDs in one hit.
Then, once we have the IDs, we can make a series of parallel requests where we chop this up into requests of 100 or 1000 merchants a piece. There are a lot of benefits in this plan - first, we know the IDs right off the bat, and if we don't get valid data for them, at least we know what to clear if we need to. Secondly, by knowing all the IDs up front, we can make a series of async, parallel requests that will make the data loading a lot faster.
It's days like this, with pretty smart guys that I get a giggle. This is going to take a few hours to do, but in the end, it'll be a huge improvement in the processing of the data.
Fantastic!