|
I would imagine what he's saying is that it'd be a whole lot more efficient for each of these crawlers to hit different subsets of data and for all of that data to be synchronized up to a central store which could be cloned out much more easily/efficiently than individual character queries.
|