In addition to requesting the specifics on third-party data sources to which Panjiva “connects the dots”, i.e. Customs data, I wanted an under-the-hood explanation of their normalization processes. Frankly, I was dubious of Josh’s claim to have a 70% success rate linking or resolving Manifest records to specific companies. In response, again courtesy of their “go to” PR lady, Katelyn, they recite the following:
“Panjiva takes great pride in its patent-pending normalization procedure, which was developed by a team of MIT-trained computer scientists. Although the exact details of the algorithm cannot be shared, the fully automated process involves natural language processing, machine learning and clustering technologies. Panjiva first takes shipping-level data for all companies, then combines it with company-specific data and those from Panjiva’s full data set. Together, these paint a more thorough picture of the companies than one particular data source could do alone.
As part of this process, Panjiva keeps entities separate if they are operating at different locations. This allows the user the option to look at a specific location of a factory or supplier. However, there are also automated and manual mechanisms that can roll these into super profiles to view related entities in aggregate. Essentially, users can group them depending on how granular – or not – they want to get.
Because Panjiva relies on an algorithm, it is not always perfect and there are some errors due to the roughness of the data. However, the technology analyzes enough attributes to make decisions that are robust. A few other things to note:
70% of Panjiva records are bound to entities using our proprietary algorithms for identifying when multiple shipping records are actually describing the same company/location entity. The remaining 30% of the records are opted-out, so those are not bound to companies, but these records are available in Panjiva’s Trends interface and via a raw customs search on the platform.
Panjiva does flag companies that are purely shipping companies, as many buyers are not interested in evaluating these companies as trading partners. This is one of the few manual processes conducted by Panjiva.”
Having engineered similar technologies at CenTradeX, I can tell you that it is not easy. I will also state, based upon the results set that I evidenced during the Panjiva demo, that they do a rather remarkable job in the normalization process. From what I have witnessed thus far, no other TI provider, with the possible exception of PIERS, compares.