Consultation, Application Development & Project Management I am available on an hourly, daily or project basis, to consult and lead during the many facets of developing, integrating and applying trade intelligence. By the day : Contact me directly via email. Rate: $1,000 to $2,000 per day ($125 to $250 per hour), plus expenses (billed in 1/2 […]Read more
- World Trade Daily Commercial Services: Consultation, Application Development & Project Management
- World Trade Daily Commercial Services: Research & Reporting, Publishing & Social Marketing and Sponsorship
- CenTradeX: Development and Overview of Available Trade Intelligence (T.I.) Applications
- World Trade Daily Commercial Services: Artificial Intelligence Engine Connects the Dots
- World Trade Daily Commercial Services: Database Repositories for License or Sale
- WTD Commercial Services and Advanced Trade Technologies Available
World Trade Daily Commercial Services: Research & Reporting, Publishing & Social Marketing and Sponsorship
Research & Reporting Services Over the last decade, we have performed a plethora of research and reporting services for clients in many industries. The scope and focus have ranged from industry and sourcing analyses conducted for Disney, Wal-Mart, ECRM and the WTCA… involving months of work and a handful of researchers, writers and analysts… to […]Read more
The following provides a cursory outline of processes employed by CenTradeX in crafting innovative trade intelligence applications. Readers are invited to (mouse over for explanation) click on the hyperlinks below to expand their understanding further through graphic illustration and related articles. Additional information can be made available upon request. From March 13th WTD article, “The Holy […]Read more
CenTradeX was known to be on the cutting edge of developing innovative trade applications. Over a decade we focused our resources in 3 general areas: Access, Integration and Delivery. Access is in terms of providing user-friendly interfaces with common sense terminology and easy to understand processes. Integration is connecting the dots, i.e. associating many types […]Read more
Licensing /Sale of CenTradeX Data Repositories Data is THE fundamental building block used in constructing Trade Intelligence. When I founded CenTradeX in the Spring of 2000, I endeavored to offer a missing value added component to readily available, inexpensive but obtuse trade statistics and reports. My initial premise was that there was tremendous inherent value […]Read more
We are offering several commercial products and services to the trade community, which I will briefly outline in this article and further develop and expand upon in follow-up posts. Consulting I am available on a hourly, daily or project basis, for the many facets of developing, intergrating ad applying trade intelligence. By the hour (by […]Read more
Last year, I was contacted by a New York based consulting firm on behalf of one of their clients: a major equity firm interested in investing in the Trade Information field. This multi-billion organization, who shall go unnamed for confidentiality reasons, focuses on investing in existing, profitable businesses (primarily within information technology) not start-ups. Seems they were looking for a trade information company to buy or heavily invest in.
As the expert du jour, I consulted with the principal – first providing an overview of the trade intelligence industry: history, sources and types of data, business applications, current challenges, and areas of potential opportunity.
Concurrent with his intention, we spent a goodly amount of time discussing the major players (along with their respective strengths and weaknesses). Of course, I gave him a run down of the “Usual Suspects” – those I call the “Top Tier” TI providers: PIERS, Datamyne, Zepol, Panjiva and Import Genius. I avoided the recent “Gang of Twelve”: newly minted domestic and foreign companies offering access to U.S. Customs Waterborne Import BOL data via an off-the-shelf BI intelligence software utility.
I started with TI companies offering the U.S Customs Data, because it is inherently the most complex to deal with and potentially most valuable data source available. Notwithstanding, we expanded the list of candidates to include statistical, company and reference based information providers (non-governmental). Included on the short list was GTIS, WISER, Kompass, FITA /GlobalTrade.net and a handful of others.
What became obvious over the course of our conversation was that there really wasn’t any one company out there that I would put on the “A” list for one reason or another. The top choice would be PIERS, but since they are a member of a London-based publicly traded media conglomerate, it would be a complex deal to get done. Another problem is that, they are still laboring to overcome a decade of inertia left over from previous management. It is more of a challenge to stop and change the direction of a huge oil tanker, than it is a cruiser. The others really don’t integrate data (connect the dots) to any significant degree.
Really THE KEY, the Holy Grail of Trade Intelligence, is about INTEGRATION: connecting the dots, dimensionalization and visualization of interconnected layers of disparate sources and types of data.
In the movie world, from whence I came a couple of decades ago, a company called In-Three defines dimensionalization as “a method developed by In-Three of converting 2D content to stereoscopic 3D content.” They employed the technique in the movies Alice in Wonderland, The Transformers and G-Force to give depth and dimension to an otherwise flat celluloid world, pixel by pixel.
How would dimensionalization be applied within the Trade Intelligence field? Imagine an interactive, visually oriented dashboard designed for fund-managers and financial analysts whereby any given publicly traded company’s vitals could be displayed along with dynamic drill down and reporting capability including (for instance) real-time analytics, risk assessment and competitive analysis partially constructed from U.S. Customs Waterborne Import BOL records as well as other datasets.
Many pieces of the puzzle are there already. Few (actually NONE) have invested adequate resources, creativity, intelligence and vision to put them together.
Next along its path of transformation and enlightenment, Customs data is enhanced with a number of ancillary but related (and connectable) data. Once the company location and name has been successfully resolved, its location can be assigned to a respective county, city, MSA, CSA, CBSA, congressional district, area code, time zone or even latitude-longitude. Thereafter it can be integrated into a dynamic mapping application or be used in detailed geo related analysis and reporting. Any number of “groupings” or consolidating factors can thus be applied.
Expanded information about the vessels (ships), containers (types, sizes), container owners, ports, carriers (SCAC reference) and any number of “connectable” relevant databases can also be linked during this step of the process. The possibilities are as infinite as imagination and business requirements dictate.
At this point, the U.S. Customs data has been imported, organized, cleaned, groomed and dressed. It is now ready for “prime time”. It’s show time in the data world. The data is now ready to move into an “exportable” mode. Therefore, it is organized (and refreshed) within a distinct database of its own.
All the processes, detailed over the last several days, tens of thousands of individual BOLs, have been completed in a number of hours. These routines run habitually every day. All databases and processes – through the creation /update of the “AMS Trade Data Export” DB – occur internally and securely (behind the veil).
Now four processes, involving six databases have been completed. Thereafter, two additional processes are initiated on a weekly basis. One integrates various client’s proprietary data with our completed Customs Data and updates the databases upon which their respective web applications sit. The second process refreshes the reporting repository from which the host of commercially available web applications draw.
Transformed Customs data, integrated with other statistical, company trade and economic data sources can be a powerful tool to navigate and succeed within the multi-trillion dollar international trade marketplace. A plethora of applications and services can be enhanced by the skilled utilization of Trade Intelligence.
The collection of programs, procedures and referential databases with which we transform raw data into usable business intelligence we refer to as our “A.I.” (Artificial Intelligence Engine). It, along with our huge data repositories of statistical, company and transactional data collected over the years, together represent the primary assets (along with the human intelligence and experience by which to integrate, develop and deploy them) that we are offering for license, sale or joint venture consideration.
Please refer to our Commercial Services Menu on the top navigational bar of this site for information on Application Licensing, Research & Writing Services, Database Repositories, Artificial Intelligence Engine as well as other Consulting, Project Management and Application Development Services.
Two of the most important normalization processes are accounting for the many iterations of company names and establishing an accurate company location. See the previously published article, “The ABCs of U.S. Customs Data- Issues & Shortcomings“. There can be many dozen iterations of the same company name. This wreaks havoc with the veracity of the data under analysis. The problem is evident is a cursory review of trade intelligence applications offered by most data vendors.
In order to resolve these issues, the name and address fields contained on the bills of lading (for both shipper and receiver) are broken down into “tokens” and compared with a dynamically evolving referential database of “resolved” names and addresses. Actually, accurately “geo-locating” the entity is the simplest of the two tasks. Zip codes, for the U.S. at least, follow a predictable pattern and typically occur at the end of the text string in the ”address” block of the flat file.
The two diagrams below are tables utilized within the fourth database involved in the third step of the transformation. The first diagram shows elements that are utilized to resolve company location. The second shows those necessary to resolve company name.
A separate, complimentary and very important utility – called the company-location resolver – is THE essential cornerstone of the A.I. (Artificial Intelligence) Engine and is required to dynamically evolve and “educate” the system. More on that later.
The location – company match utility is a very nifty accessory and vital component of the A.I. Engine. Although the system is set up to quickly, accurately and automatically normalize U.S. Customs data, it also has the capacity to “learn” and improve its performance over time. Some of this learning takes place automatically over time as it gains more and more experience performing its daily processing rituals. Adjunct education is interjected manually.
For instance, perhaps during the last several days/weeks/months processing routines, our A.I. Engine encountered some company name iterations that it hadn’t handled before and wasn’t in its library of established “tokens”. Conveniently, it would display these unresolved iterations, ranked by the number of occurrences along with likely matches. With one stroke an operator could resolve and match all particular aberrations or variations on a particular supplier or importer name or location… sometimes representing several hundred or thousand individual BOLs.
Thus the A.I. Engine learned something new. And unlike its human counterparts, it will never have to ask the same question again.
The location – company match utility also can be used to link unlinked branch locations to their respective parent company or regional/ divisional headquarters. Furthermore, it can process and link a proprietary client’s database of customers as well. In this fashion, one can monitor customer’s trading activity and supply chain operations on a daily basis! This information can be incorporated into a web application which is distributed within the secure company intranet or protected proprietary web site. An example is Panalpina, one of our previous (CenTradeX) clients wherein we integrated their proprietary information into a customized web application for distribution to their regional sales offices.
It took us several years (at CenTradeX) to develop an intelligent system by which to quickly and seamlessly assimilate the daily Customs feeds.
Over time we developed and incorporated automated procedures and administrated them under an umbrella control panel. Statistical data update processes from U.S. Census and U.N. Comtrade were initiated from this centralized control panel. U.S. Customs data, initial processing and normalization as well as company, parent and location matching, were also conducted from the same control panel.
A detailed diagram of the individual components that make up the control panel (as a constituent part of the A.I.Engine) can be downloaded from Google Docs by clicking this link.
Company data collections (from sundry vendors because each contained its own unique non-standardized characteristics) were initially processed utilizing different arrays of queries and procedures. They were then integrated into the combined company repository which, in turn, were correlated with the U.S. Customs and statistical data. See U.S. Customs Data Primer Part 4: Enlightenment Through Graphics & Diagrams for illustrative diagrams.
U.S. Customs Data that we referred to as AMS – automated manifest system – went through six distinct processes which are depicted below. An illustrative diagram of all six processes and eight sequential databases (or collections) can be viewed by clicking this link.
Customs data is received and processed on a daily basis, but the final, resultant databases utilized to serve up web reports were refreshed weekly to allow for enhancements (beauty treatments) and interconnectivity with other data collections.
Let’s look at the steps from the beginning. Roughly speaking, the first task is to import all the data properly – correctly parsing all the elements contained in the original “flat file” and organizing them within a relational database. Every data element and every permutation and aberration must be accounted for. The diagram below depicts the second of seven databases (the first “DB” is really just a collection of the all the raw AMS or Customs data itself). This database is resultant and refreshed daily from the first processing step.
A high(er) resolution depiction of the above diagram can be obtained from our Google Docs site, by clicking this link.
Next comes the “normalization” process, wherein each element of parsed data is refined and standardized. For instance, a simple Port code, whether foreign or domestic, has its corresponding state, province /region, country and normalized name. Each container code is translated into presentable information about its type such as refrigerated or non, height, length and particular identifying number. Within this normalization process company name, address, and contact iterations are resolved as well.
Below is a diagram depicting the third of eight databases after the second step along the Customs data transformation journey. A high(er) resolution image is available for download from our Google Docs site.
The original CenTradeX Trade Intelligence platform was developed over a three to four-year time frame from Spring 2000 to Spring 2004. Initially, we focused on bringing a host of value added features to statistical U.S. import and export data from U.S. Census. Thereafter, we layered and connected this U.S. centric data with global import and export data (from U.N. Comtrade) on approximately 190 countries. In time, we also incorporated U.S. state export data into the mix. Surprisingly, no one had ever layered these data sets together before.
Atop these statistical data collections, we crafted a graphic, interactive interface wherein users after selecting an “X” product vector (one of the 6,000 Harmonized System product classifications) and a “Y” location vector (one of 200 countries listed) were (almost) instantaneously presented with dozens of dynamically created reports representing many perspectives pertaining to the intersection of their choice. Over a billion unique reports could be potentially generated by the system. These included historic analyses spanning 20 years, trends 1, 3, 5 years into the future, contextual reports for the respective region and industry, competitive analyses, product/industry segmentation and trending, etc. One economics /international trade professor remarked that the system made “data dance”.
Market testing and resulting feedback, compelled us into the task of finding and incorporating company data – both foreign and domestic – toward the objective of uncovering the actual traders behind the statistics. Economics and trending with numbers is one thing… pinpointing buyers and sellers is quite another. Consequently, we apprehended and assimilated the best known company sources (at the time) including; Kompass, Harris Info, Hoovers, D&B, PIERS and others.
One of the most challenging aspects was that statistical data is organized under one (HTS) classification schema while company information is organized under other (unrelated) systems (SIC, NAICS, or a vender’s particular proprietary taxonomy). However, after successfully tackling that enigmatic brainteaser, we were able to incorporate other data sets, tariffs (for all countries and products), estimated shipping costs (from four U.S. port regions to any /all countries), foreign exchange as well as our clients’ proprietary data collections and others… with relative ease.
By far the most arduous of our data transformation and enhancement endeavors was to understand, normalize and intelligently incorporate U.S. Customs data into this dynamic mix. ALL OTHER purveyors of Customs data started the other way around. Some (the best) have consequently connected some other data elements. Most notable among the few is PIERS, who several years back contracted with D&B to “tag” their data collection of U.S. Importers & Exporters. Datamyne is presumably undertaking a similar process now. Panjiva has connected with reasonable success many other vendor’s data pertaining to foreign sourcing. To my knowledge, at the present, no others have made those necessary connections. Zepol has begun offering statistical data, but not connected to Customs data. They remain in separate unrelated silos. Suffice it to say, there are still significant vistas to explore and develop.
The 45 second video (slide show) below is an irreverent depiction of what many users have reported experiencing when trying to find solid “trade intelligence” amidst the seemingly endless sea of Customs data obscurity.
In the past we published a series of 5 articles; “U.S. Customs Data Primer”, Parts 1-5, about the particulars of understanding, processing and enhancing the daily transactional inbound shipping records published by DHS/Customs. This article will expand upon the fourth article in that series, “U.S. Customs Data Primer Part 4: Enlightenment Through Graphics & Diagrams” which provides a visual guide for the processes we at CenTradeX employed in transforming raw data into trade intelligence.
The 90 second video (slide show) below portrays the original Trade Intelligence vision and mission that fueled the innovative growth and development of CenTradeX.
Developing innovative and powerful trade intelligence applications involves attending to three major areas: Access, Integration and Delivery.
- Access: Helping the target audience(s) understand, find and use the data and application.
- Integration: Normalizing of base data and connecting it with other relevant data sources.
- Delivery: Enhancing the speed, efficacy and beauty with which the combined data is organized and presented.
For the purposes of this series, we will only focus on the second aspect, Integration. Furthermore, having provided a foundation of understanding through the above referenced (linked) article, we will proceed to explore the more technical (under the hood) facets involved.
I refer to U.S. Customs waterborne import manifest data as the “base” data because it is considered (by myself and many others) the most intrinsically valuable, if challenging, international trade data set available. It’s daily. It’s transactional. The U.S. is considered the easiest market to access. It contains a wealth of detailed information about the global supply chain. It represents $1 trillion dollars of trade a year.
U.S. Customs data is #1. It’s THE KING of the international trade jungle. However, a powerful Kingdom is more than just one regal personage. It must include a capable entourage as well. Thus the need for complimentary data sets.
The first, primary step in building a powerful trade intelligence “kingdom” is attending to the King. PIERS has an easy to understand graphic portraying the processes of normalizing “base” Customs data layer.
In performing the seven steps highlighted above, we at CenTradeX developed and refined many sophisticated procedures. Over time, and through much scrutiny and evolution, we constructed a reliable, interconnected system of transforming data into intelligence.
- It involved an array of automated queries and stored procedures for importing new data on a regular basis.
- It involved created programs and “scripts” that would parse, tokenize and reference selected data elements, compare and contrast them with its expanding library and referential databases as well as “learn” better ways of matching and connecting.
- It involved scouring the planet for the best, most reliable, accurate and timely ancillary databases to enhance and expand the KING and the Kingdom.
This week we went under the hood to look at nature and application of Customs data that tracks U.S. Waterborne Import Shipments from Overseas Suppliers and Sellers.
- U.S. Customs Data Primer Part 1: You Can’t Always Get What You Want… BUT
- U.S. Customs Data Primer Part 2: “Holes” in the Data & Other Frustrating Anomalies
- U.S. Customs Data Primer Part 3: The Devil (or a Worthwhile Treasure) is in the Details
- U.S. Customs Data Primer Part 4: Enlightenment Through Graphics & Diagrams
There are a number of previous articles wherein I have referred to other shortcomings and challenges inherent with the understanding and applying U.S. Customs data. Please note the following:
- TI Transformation: Data into Information into Knowledge into Intelligence into Application. Current trends and commoditization of U.S. Customs data.
- Trade Intelligence or TI: IT all depends upon how you define “IT” and “TI”. The plethora of current “Trade Intelligence” suppliers of U.S. Customs data.
- Understanding Data: Normalization Procedures with U.S. Customs Data. Normalization procedures for U.S. Customs data with accompanying charts
- U.S. Customs Waterborne Import Data: Perspective is Everything. Need for a broader understanding and application of trade data.
- U.S. Customs (AMS) Waterborne Shipping Manifest (BOL) Import Data. BOL fields listed with an excellent chart depicting relevant trade flows. A must read.
- The Use and Application of Trade Intelligence Can Be a Matter of Life and Death. Case Study for application of U.S. Customs data in “tainted toy” (lead paint) fiasco.
- Three part series: The ABCs of U.S. Customs Data – Issues & Shortcomings.
We also published several dozen articles focusing on the current Trade Intelligence purveyors of Customs data. The links provided below will pull up a handful of articles each – for a particular company, group of companies (in cases where they are “minor, second tier” providers) and summary evaluations. You can also find these articles, and others grouped by various categories, on the top navigation menu of this site.
- Overview of T.I. Providers
- UBM Global Trade /PIERS
- Import Genius
- Other Providers
Please refer to our Commercial Services Menu on the top navigational bar of this site for information on Application Licensing, Research & Writing Services, Database Repositories, Artificial Intelligence Engine as well as other Consulting, Project Management and Application Development Services.
So now that we have addressed a few of the issues relating to understanding the inherent limitations contained within the U.S. Customs data, let’s look at the processes we (at CenTradeX) employed in parsing, normalizing and enhancing this data. Every Trade Intelligence provider has their own approach to processing the data, along with their own particular brand of “spice” they add as well as the tools utilized to search through and display the data. Notwithstanding, the best few have many things in common.
Therefore I believe the following explanation may be both enlightening and educational, whether or not it is precisely mimicked. First of all, let’s take a look at the “big picture”. As reflected in the illustration below, Customs data contains detailed records – bills of lading – of the particulars of each and every transaction between foreign shipper and U.S. receiver for waterborne freight.
As I’ve mentioned, Customs data is distributed (as a “flat file”) on a daily basis as the BOLs for various arriving “vessels” are cleared at the respective U.S. ports. The first, rather arduous process, is to “normalize” data into usable, organized elements contained in a relational database.
We found that the best, most efficient method to add accuracy and value to Customs data, after the initial normalization process had been completed, was to connect it with our other comprehensive company and referential databases. After going through many elaborate transitions, this enhanced customs data was ready for “show time”.
We found that the more relevant ancillary databases we were able to connect to the Customs data, the more dimensionalized and powerful the individual portraits of trade and the underlying traders became and the broader the business applications and potential.
Trade Intelligence begins with data. It is the fundamental building block from which dynamic business applications are crafted. To make delicious, even digestible Trade Intelligence you must adhere to some basic steps.
- Get good, accurate, timely data.
- Scrub it up, remove the “dirt”.
- Mix it with other good data.
- Cook it well.
- Serve it with style.
Several other Trade Information providers have developed some very powerful and cool applications incorporating U.S. Customs data. Check out the top navigational menu of this site under T.I. Providers Links> Transactional, Article Categories> Suppliers as well as Trade Blogs> T.I. Providers and Video Library> T.I. Providers for more information on these companies and the products and services they offer.
Also, please refer to our Commercial Services Menu on the top navigational bar of this site for information on Application Licensing, Research & Writing Services, Database Repositories, Artificial Intelligence Engine as well as other Consulting, Project Management and Application Development Services.
Let’s go back to the intrinsic nature of the U.S. Customs Data itself.
U.S. Customs data is gathered electronically through the AMS (Automatic Manifest System), for sea, air and rail. However, only waterborne manifests are available publicly. Each daily tally contains detailed records of the tens of thousands of shipments that arrive at U.S. ports, many millions of shipments each year. Since we are a country of consumers and most imports arrive via ship, U.S. Customs Waterborne Import data represents MOST of our trade activity… to the tune of $1 trillion annually.
In spite of its inherent shortcomings, pause to appreciate the fact that detailed records of virtually every waterborne shipment, every foreign seller, every corresponding U.S. buyer, every product and component, every carrier, every port, for every day is made available publicly. The potential value contained therein is staggering. Most countries (perhaps wisely) don’t publish such information. In some countries releasing such information would be /is a capital offense.
First of all, U.S. Customs data comes as a “flat file”. It is not conveniently delimited for easy assimilation. For each and every Bill of Lading (BOL) the respective data fields have a reserved number of characters rigidly assigned; some fields are filled with interesting data, others remain completely empty. Analogous to a train hauling rail cars of varying lengths, each BOL must have its fields carefully unloaded and organized. One mishandled BOL field can wreak havoc with accurate assimilation and analysis of the data.
For instance, there is a very important single character field contained in the data string that signifies whether the proceeding data for the BOL is original and new or whether it represents a revision of an already processed BOL (from a previous day!). I have seen cases where there are several dozen revisions published to a single record occurring over a span of several months!
If not accounted for, you have multiple (and inaccurate) shipments counts. The difficulty is that the only way to adequately correct the problem is to go back into already processed and published data to completely erase and replace the previous record or retain the inaccuracy.
Another significant problem is that there are many times multiple containers for one BOL AND/OR multiple BOLs for one container (LCL –less than container loads). If not prudently accounted for there will be huge discrepancies when calculating TEUs (the standard measure for shipment volumes).
Implications? If you are evaluating whether or not to construct a new distribution center, expand port capacity, open up a freight forwarding office, evaluate economic development, perform competitive analyses on a particular U.S. buyer or foreign seller, or deconstruct and improve supply chain logistics, what you don’t know or what you think you know (but is really fictitious) can kill you.
Once you know where the holes are, many times you can fill some of them. U.S. Census (statistical) data – which is published on a monthly basis – can give you an accurate measure of the value, number of units (and thus by computation the average cost per unit), country of origin and U.S. port for a particular product grouping (arranged within the Harmonized Tariff System) and method of transport (air, water, and again by computation “other” which would typically be rail /truck from Canada or Mexico).
Unfortunately, U.S. Customs data and U.S. Census data are asynchronous in many important ways. For reasons beyond the scope of this article, it is impossible to take a record of all waterborne shipments for the month of January from U.S. Customs and seamlessly overlay it with the aggregate statistical record of imports provided by U.S. Census. Further, the HS product categorization system is many times either too specific or too broad to apply.
Another problem is that several thousand U.S. importers and their corresponding foreign suppliers have been “suppressed” from appearing in the U.S. Customs data publications through the “trade secrets” exclusion to the Freedom of Information Act. This “suppression” results in about 1/7 of all shipment records having blank fields where the “foreign shipper” and “U.S. importer” identification would normally be.
Again, once you know where the holes are, there are ways to work around them. Wal-Mart is an obvious entity that attempts to mask its supply chain activities and valuable suppliers. Notwithstanding, in a landmark report done on the “tainted toy” fiasco several years ago, we were able to extract 40,000 imported shipments of toys by Wal-Mart (of the 400,000 we retrieved) over an 18 month period of time.
How? Several methods. Although presumably “suppressed”, tens of thousands of transactions slip through the filtering methods applied by U.S. Customs technologies. Port pairs (matching foreign port with U.S. port) for a particular product also yield significant dividends. The “product description” and “marks and numbers” fields contained on the shipping manifest sometimes contain references to either Wal-Mart or one of its known suppliers. Product identification information – SKUs, trademarks, etc.- are also sometimes found.
It’s all a matter of sleuthing: trying to put together a complex puzzle from the resources at hand. In the end, it’s an imperfect world with incomplete data. However, with some effort, technological tools, multiple data sources along with intelligence and knowledge, you can discover an amazing amount of very valuable trade /business intelligence. You just need to increase your awareness in order to align your expectations to what’s real and possible.
Most people seem to want what they don’t have. I guess it’s human nature. It’s that way with trade intelligence. Folks want to extract more information from it than what is intrinsically possible. You just can’t get soda pop from milking a cow.
You aren’t going to get a complete picture of importers, supply chain, current inventory, shipment valuations and intrastate transport patterns from U.S. Customs data. However, just because you can’t have everything, doesn’t mean you can’t get a lot. As Mick Jagger sang, “You can’t always get what you want, but if you try sometime, you may find, you get what you need.”
To understand what you can and can’t get from U.S. Customs data… we need to dig into what it is and why it is… what it has and what it doesn’t have… what current T.I. providers are doing to enhance the base data… where the holes are and how best to fill them.
U.S. Customs, now under the auspices of Homeland Security, requires detailed documentation of all waterborne shipments entering into the United States. This information must be filed 24 hours before the shipment disembarks from its originating foreign port.
Once the carriers dock at their respective domestic port, each day’s documentation of shipments (midnight cut off point) is published and distributed via FTP (used to be sent via overnight on a DVD) to awaiting subscribers (of which there are only a couple handfuls). This is made available through the Freedom of Information Act.
First of all, with the exception of UBM Global Trade /PIERS, who has special reciprocal information exchange deals with many ports and carriers as well as a cadre of data gathers assigned to many U.S. ports, only daily transactional data on U.S. IMPORTS is available, NOT EXPORTS. Thus, as a U.S. manufacturer, you aren’t going to find a list of foreign buyers for your particular product within the confines of U.S. Customs data.
Secondly, only commodities and products that enter the U.S. via SHIP (waterborne freight both containerized and non-containerized) are accounted for. Shipments that come via TRUCK or RAIL, let’s say from our North American neighbors – Canada or Mexico – are invisible. Also absent are shipments that come via AIR.
Therefore, if you’re looking for U.S. Customs data to provide information on shipments of high-tech components, you’re going to be very disappointed (because they are mostly shipped by air). If you want competitive intelligence on a company who largely imports from suppliers in Mexico, again, you’re going to become very frustrated. If you’re after an accurate analysis of all foreign suppliers and U.S. importers for a particular component that may have originated from several countries (including our NAFTA neighbors) and shipped by multiple means (air, rail, truck, ship), it just isn’t going to happen. There are going to be huge, gaping holes in your report.
- Check out our commercially available technologies, products and services. Contact: Robert@WorldTradeDaily.com
- ©WorldTradeDaily.com® All rights reserved. Robert Thompson, Editor-in-Chief. Louisa Avery, Editor. Isaac Thompson, Research Writer.
- For discounted subscriptions to PIERS Prospects and Stats Plus contact: Aliet Martinez: email@example.com and mention “World Trade Daily”.
PIERS has an excellent graphic that explains the process by which a reputable TI Provider should handle the U.S. Customs Waterborne Import Manifest (bill of lading) data. Many other TI providers, particularly the newcomers to the market, jump from collection to publishing. After all, if you eliminate cleansing, standardization, verification, validation and enhancing you’re bound to save time and money. That’s why there are some data providers offering access to the U.S. Customs data for as little as $30.10 per month.
During my tenure as founder/CEO at CenTradeX, we worked very hard to make sense of data, connect it in innovative ways and provide easy access and graphic delivery. We had a lot of smart and creative people working a long time toward those objectives. We spent over two years working out the bugs before the first interface integrating and utilizing the U.S. Customs data could be launched. It’s complex and obtuse. It’s also perhaps the most valuable single source of Trade Data available.
The inherent treasures buried within the data have only begun to be unearthed. PIERS has gotten the furthest, particularly with their acquisition of key CenTradeX applications and technologists, but even they have a long, long way to go. It is my hope that beyond succumbing to the recent and base marketplace inertia that has led to the commoditization and devaluation of the data, that necessary capital and creativity will be applied to the task of furthering innovation in the trade intelligence field by those with the vision and resources enough to carry it further which may or may not be PIERS.
The following illustrations address one of the necessary aspects in the normalization, integration and enhancement processes involved with U.S. Customs Data. The first two diagrams (which can be clicked upon to display full size) relate to the identification, normalization and enhancement of the Foreign Shipper and U.S. Importer of record.
The respective U.S. Customs data fields containing Shipper and Importer names and addresses need to be normalized (including standardization of the many iterations of those names) and broken down into separate “tokens” such as zip code, state, phone, city, etc. These tokens are matched against a refined and dependable company data repository derived from third-party sources (we used Hoovers, D&B, Kompass, PIERS and others) as well as the perfected names collected over time from the Manifests themselves. The number of tokens matched are then scored on a reliability or veracity scale that is internally developed.
The same procedure can be utilized to normalize and match several “silos” of disparate data held between different companies or divisions of the same company such as marketing, operations and finance. CenTradeX was once consulted by Maersk for a project in which they wanted to normalize, standardize and match their own internal company information, after which time they then could connect it to the individual shipment manifests for themselves or their competitors.
Foreign sourcing is one of the predominant applications that many trade data users seek. PIERS and Datamyne are the only TI providers whose products integrate daily shipping manifests from DHS/Customs with statistical and company data from the Census Bureau and D&B respectively. Each boasts of their particular brand of easy-to-use interfaces.
U.S. companies have long turned to overseas sources for cheaper raw materials and components. Utilizing AMS inbound shipment manifests (BOLs) – detailing shipper (exporter), consignee (U.S. importer) along with specifics on the product(s) transported – U.S. manufacturers and distributors can easily identify potential foreign suppliers.
Vetting the list of candidates is another matter. Here, technology can aid to some degree. Various filters can be applied such as preferred country source(s), minimum shipment thresholds, number of other customers (importers) that a particular seller(s) maintains, source history /track record, and diversity of products sold. All these criteria can assist a U.S. company in sifting the list of potential suppliers and is inherent within the BOL data.
In addition to the expected normalization and refinement procedures that respectable TI providers apply to the Customs Data, Datamyne* and PIERS employ supplemental data sources to further enhance the value of the data and application in the sourcing arena.
Connecting D&B company information like credit score, annual sales, key personnel, financials and current contact information to daily (15,000,000 annually) shipment manifests helps narrow the prospect list considerably. D&B (and other vendors) provide a reputable base by which to normalize the dozens of iterations that exist on shipping manifests for any given company, thereby making the vetting process more accurate and meaningful.
Statistical data can be employed to identify which countries are the most popular sources for a particular product. Aggregated values (cost of goods) can be compared to see which countries export said product cheapest. One can also chart overall sourcing and pricing trends (on a monthly basis). Further investigation of trade flow statistics can reveal tariff considerations, transportation costs and preferred port-to-port routes.
Once again, in the end, the best that data and technology can provide is a partial solution to the objective of foreign sourcing. The remaining gaps of data interpretation, contextual understanding and appropriate application require Trade Intelligence of the third kind. Notwithstanding, an informed utilization of TI products such as Datamyne’s new Datamyne 2.0 interface can get a U.S. manufacturer or distributor a long way down the road in identifying new overseas sources for their raw materials or product components.
*Although Datamyne doesn’t currently incorporate the full extent of available D&B data such as credit scores, personnel records or detained financial data, they do currently use it in their normalization processes… and have plans for further integration in the future.
The commoditization (and devaluing) of trade data and trade data based products is accelerating, perhaps inversely proportional to the quality and number of suppliers and products in the marketplace.
Let’s look at one of the most treasured (and perhaps useful) types of trade information – U.S. Customs Waterborne Import Manifest Data. Automated Manifest System data, sometimes called “AMS”, is information collected daily by DHS (Department of Homeland Security) U.S. Customs for and about each and every ship and shipment bound for the U.S.. Thousands of imported shipments are logged everyday. Each manifest contains information about the foreign shipper (exporter), the receiver (importer), details on the product shipped and various logistical specifics on routing.
“Back in the Day” when I founded CenTradeX in the summer of 2000, only PIERS offered such information. Primarily, it was distributed to customers via a stack of CDs each and every month. The user hunted through an Excel type interface for specifics on a particular product, shipper or importer. For elite customers, PIERS offered a plethora of prepared reports. (One of their customers once showed me a CD containing over 57,000 such reports). Slowly, PIERS converted such customers to an on-line system which also served to reduce rampant piracy.
In the last 5 years, available technological resources have grown exponentially. Correspondingly, vendors offering access to and products based upon the U.S. Customs Waterborne Manifest Data have proliferated like bunnies. On the one hand, this has led to increased competitive pressures which have driven innovation forward, quality upward and prices downward. On the other hand, there is a widening gap between data and intelligence.
I stayed awake until 2:00 a.m. one evening recently, trying to catch up on all (that I could find) of the NEW vendors offering AMS data… WHEW! The data has gotten incredibly cheap. The quality of companies /products are mixed. Some look like they are solo operations run out of someone’s garage. Others are incredibly slick.
The first competitor (not including my company CenTradeX) on the scene was Zepol. Started by two young fellows out of Minnesota – they had a simple business plan – improve on PIERS’ search utility and undercut PIERS’ price by 20%. Slowly they made headway. They were followed by Datamyne, Import Genius and Panjiva, which appeared in the last several years. Now, add to the list Manifest Journals, Cybex, Info Drive India, IE Intelligence, Trade Intelligency, Data Trade, Trade Mining, Import Intel, TradeKey, Vanguard…
At the peak of the pyramid in price and quality /value is PIERS (of course) – a handful of “G Notes” will buy you the best. Second tier providers include Datamyne, Zepol, Import Genius and Panjiva whose prices range from a few Benjamins to a couple of Clevelands (the President on the now defunct $1,000 bill). Cascading down the food chain are the bottom feeders, like Manifest Journals, Cybex, Info Drive India and the others, which offer access to AMS data for as low as $30.10 a month.
The current or prospective user of “trade intelligence” products and resources must decide upon which “values” he values most. It’s cheaper to harvest the wheat yourself… you can purchase the raw AMS data directly from U.S Customs for $100 per day. Maybe even start your own company. Heck, the last three CEOs of PIERS did just that. They either run or have founded companies in the list above.
*This post was originally published during the first week of May, 2011.
TI providers such as PIERS, Datamyne, Zepol, Import Genius, Panjiva (and a growing host of others) depend upon BOL data as the basis of their product offering. They all receive base shipping documents from U.S. Customs/DHS (Department of Homeland Security) via direct FTP feed or delivered on daily DVDs. Each follows their particular processes of collection, cleansing, standardization, verification, validation, enhancement and publication.
There are many names used to refer to this data. Among the identifiers I’ve heard (and used) alone or in combination are: AMS, BOL, Customs, Waterborne, Manifest, Shipping, Import, and others. Let’s clarify. Not less than 24 hours prior to an inbound shipment from a foreign port to the United States, a handful of documents must be filed with the U.S. Government (Customs and Border Protection), namely bill(s) of lading. By and large, these BOLS are filed electronically utilizing the AMS (Automated Manifest System).
For the purposes of this article we are limiting our discussion to U.S. waterborne (by sea) imports. There are also air and rail AMS as well as documents collected for trans-border shipments via truck to/from Canada or Mexico. Also excluded is transactional data collected on U.S. (waterborne) EXPORT shipments. At this juncture only PIERS, with an on-the-ground staff of reporters stationed at over 80 U.S. ports is able to collect and disseminate such information.
Thus, we address the roughly 15,000,000 annual waterborne (cargo carried by ship across a deep blue sea) shipments (BOLS) filed and collected via the AMS (computer) system by our friendly Customs officials, gathered and disseminated by your neighborhood TI provider via their particular product /interface (or purchased directly from DHS/Customs @$100 per day).
It’s all the same base data. There are different refinement processes and “value added” flavors added to the stew. The data is served in a plethora of fashions. Some TI providers dish up the data (or offer it on a self-serve basis) on paper plates while others have a cadre of courteous, well trained, superbly tailored trade experts to serve you.
There are only two dozen fields of data on any particular bill of lading that can be made publicly available, through the Freedom of Information Act. No TI provider or entity, despite size or age, has the right to more or less data. The basic available data elements are: (* signifies that the information is sometimes but not always listed):
- Consignee: (Name, Address, *Phone, *Email) Essentially the U.S.importer or buyer.
- Shipper: (Name, Address, *Phone, *Email) Essentially the foreign exporter or seller.
- Notify Party: (Who gets to know when the shipment arrives. There can be multiple “notifies”.)
- Product Description: (Sometimes extremely detailed including 10 digit HS identifiers, invoice #, product #, etc.)
- Marks and Numbers: (Notations on the boxes or containers, trademarks, product identifiers, etc.)
- Port Info: (Foreign, U.S. along with transfer points, plus a couple of other items.)
- Shipment detail: (BOL #, TEUs, weight, quantity, measurement, container #, container type.)
- Carrier detail: (Ship name, ship code, voyage #, Carrier /sub-carrier.)
- Misc: (A couple of other minor fields I won’t bother to mention.)
Call it what you want. The data reflects daily trading activity for U.S. import shipments by sea… end of line… end of story.
Buyers beware. Users of U.S. Customs Waterborne Import Manifest (Bill of Lading) data need to be aware of the major shortcomings & pitfalls. Part 3 of 3.
In addition to the plethora of potential iterations for each U.S. importer and corresponding foreign shipper identified on the shipping manifests, there are other significant problems.
There exists the Master versus House Bill of Lading (BOL) issue, which leads to many duplicate container counts. The same shipment may appear under both filings. Unless the TI provider has developed the technology to address this issue, accurate container counts for both shipper and importer will be impossible.
Further, there may be numerous – sometimes dozens – of revisions made to a particular bill of lading. These revisions may be published days or months subsequent to the original filing. Unless said roadside TI provider makes provision for ongoing corrections by going back and deleting all previous entries for a particular shipment whenever a new revision shows up, transactional profiles will be greatly skewed.
Many bills of lading contain multiple containers. Some containers contain multiple shipments that have been aggregated together. There are many types and sizes of containers. A 40 foot container is 2 TEUS (Twenty Foot Equivalent Unit) and a 20 foot container is 1 TEUS. A 45 foot container is 2.25 TEUS. There are many other variations. Therefore, the number of containers alone is not a dependable measurement. Neither is shipment count. One shipment may contain 20 containers or may represent 1/5th of one container.
Although it is against the law, and more stringent measures have been employed since 9/11, many times the real shipper and importer of record do not appear on the BOL. Instead, the Freight Forwarder, NVOCC or some trade agency (middleman) may be listed.
Several thousand U.S. importers have petitioned CBP to have their identities suppressed on the publicly distributed BOLs under the trade secret provisions of the FOI (Freedom of Information) act. Around 14% of all BOLS (millions of shipments a year) are thus suppressed. It has been called “The Walmart Effect”.
In addition, some U.S. Importers and Foreign Suppliers seek to hide their identities by providing the required identity information but listing it within the product or trademarks area of the BOL instead of the name fields.
Despite legitimate tactics of name suppression or other more dubious methods employed to conceal the details involved in one’s trade activity, with a little clever sleuthing much can be revealed.
For the tainted toy study we conducted several years ago, we were able to identify over 40,000 toy related shipments over an 18 month period by Wal-Mart alone, despite their obvious efforts to mask their import activity.
Trade Intelligence is a lot more than data and a search/reporting tool.
Buyers beware. Users of U.S. Customs Waterborne Import Manifest (Bill of Lading) data need to be aware of the major shortcomings & pitfalls. Part 2 of 3.
Although shipping manifests contain valuable information about the trade transactions of the U.S. Importer and Foreign Supplier, they can’t and don’t provide a complete picture. One of the most predominant shortcomings is that they only document U.S. Waterborne Imports: products and commodities transported by ship, not air, not rail, not truck, not camel.
Therefore Canadian and Mexican cross border trade is all but invisible. Export transactions are not listed. (Re-exports in some cases are). Air freight shipments are not available. And although a majority (around two-thirds) of the products we, in America, import from overseas comes to us by ship, an important minority don’t. Particularly high value, just-in-time, perishable and fragile components or merchandise are not transported by water.
So, U.S. Customs Waterborne (BOL) import data is not a great place to look for foreign suppliers of such things.
Furthermore, specific product identification is not easily uncovered within the data. Yes, sometimes the respective Harmonized Tariff codes will be buried within the product or trademark fields on the BOL, if your particular trade provider has developed the algorithms to accurately parse an HS code from among other numerical data such as invoice numbers, quantities, phone numbers, addresses, reference numbers, etc.
Many times there are no specifics. Toys, Furniture or Glassware may be the full extent of the product description. Other times there may be extensive descriptions including 10 digit HS codes, trademarks and even SKU numbers. There is no uniformity. Your typical TI provider does little to help in this regard. Most simply offer a blunt search tool which plods through the millions of products descriptions contained on individual BOLs.
In order to put an estimated price tag on a shipment, you must first know the specifics of the product, down to at least a 6, preferably 10-digit Harmonized code. Then, using statistical information you can roughly infer an estimated value which is a very crude measuring stick.
PIERS, who has been at all this the longest, is the only company I know of that has even attempted to attribute an estimated shipment value. To do such, they first had to assign a specific product identifier (much of this is still done by hand) to each BOL and then attach a gross average based upon aggregated statistical data from U.S. Census.
Many times even the resulting calculations have been flawed. However, as we say in the data world: “Bad breath is better than no breath at all”.
Note: For those who want to give shipment valuation a spin, greater accuracy can be achieved by disaggregation of the statistic (overlying) data by its respective (foreign & U.S.) port and foreign (source) country.
Buyers Beware! Roadside TI Vendors May be Selling you a Pig in a Poke
Users of trade intelligence, in particular U.S. Customs Waterborne Import Manifest (bill of lading) data, need to be aware of the major shortcomings and pitfalls. It’s important to learn the ABCs of the data.
With the recent proliferation of TI Providers offering access to Customs data via off-the-shelf BI software packages – some with subscription plans costing less than 99 cents a day – the veracity of the resulting reports needs to be seriously considered. If a company saves a few thousand dollars by buying cheap data from a roadside TI vendor, and thereafter depends upon errant reports to base million-dollar global trade decisions, what is profited?
Yes, the base data comes from the same source: DHS/CBP (Department of Homeland Security /Customs and Border Protection). However, a veteran TI Provider, namely UBM Global Trade /PIERS, being the most reputable in the field, has invested significant resources over decades in various refinement and value added processes that help ensure quality, dependability and usability. Whether these value added enhancements justify the pricing differentials involved is a matter for the market to ultimately decide.
What are some of the pitfalls? Let’s take a simple hypothetical example: How many containers did ABC, inc. (American Business Corporation) import last from foreign supplier DCF, ltd. (Decent Chinese Factory)? The answer is theoretically contained within the U.S. Customs data.
The first, most basic problem is that the names are not standardized on the bills of lading. Consequently, there are literally dozens upon dozens of iterations for each importer name:
- Amer. Bus. Corp.
- American Business Corporation, Inc.
- American Business Corp.
- American Business Corporation, Incorporated,
- ABC, Inc.
Multiplied by the variances in naming conventions appearing for the corresponding trade partner:
- Decent Chinese Factory, ltd.
- DCF, Inc.
- DCF, limited
- Decent Chinese Factory
Depending upon the parsing and refining algorithms employed by the respective TI Provider (if any), there are also matters of matching location. There may be 10 or more “ABC” corporations in the U.S.
- American Business Corporation
- Advanced Banana Clockworks
- Aerospace Ballistics Controls
- Atlanta Baseball Company
Then there are matters of several divisions or locations with divergent business operations under the same conglomerate name. Sometimes an NVOCC may be listed as the importer of record which is a violation of law that occurs thousands of times a day. Some 14% of importer names are suppressed and thereby appear blank on the BOLs. The list goes on. Yet, naming conventions are one of the relatively easiest problems to address!
Many companies label themselves as trade INTELLIGENCE providers in some fashion or form. Each and every organization, foreign or domestic, that I list below utilizes the DHS/U.S. Customs Waterborne Import Manifest (bill of lading) data as THE primary basis for their trade intelligence.
First, there is the 500-pound gorilla and industry leader, PIERS: “The STANDARD in Trade Intelligence” with products such as PIERS TI and a handful of others. They’re the standard. The (big) measuring stick.
Occupying the next rung on the food chain, you’ve got:
- Datamyne: “The best VALUE in Business Intelligence”
- Zepol: “Global intelligence that MOVES your BUSINESS” – with products like Trade IQ
- Import Genius: “A LEADING provider of intelligence”
- Panjiva: “THE leading intelligence PLATFORM”
These suppliers boast of value, motion, leadership and having a firm footing when it comes to their brand of trade intelligence.
Lastly, let’s look at the folks I label “bottom feeders” (which upon reflection I should rename in a more complementary or at least neutral fashion because, who knows, one of them could be my next business partner) who have their particular shtick.
- ImportIntel: “CUSTOM intelligence reports”
- Trade Data Channel: “US IMPORTS Trade Intelligence”
- Manifest Journal: Call Michael to discuss your “TRADE Intelligence IQ”
- Cybex: “ONE STOP SHOP for research and business intelligence”
- InfoDrive India: “Export Import Business Intelligence Information in the most USER FRIENDLY & cost-effective (CHEAP) manner”
- Trade Intelligency: “A TOP provider of Trade Intelligence”
- IE Intelligence: “World INTEGRATED import export intelligence solutions”
- IBIS Trade Intelligence: “METALS, CHEMICAL, PLASTIC (and other industry) trade intelligence”
- Trade Mining: “QUICK Business Intelligence for Trade”
- OTHER “bottom feeders” that somehow slipped out of my (inter) NET (search).
Yup, you can drop in on one of the “top”, “user-friendly”, “one-stop-(TI)shops” for some “quick”, “cheap”, “US import” trade intelligence either for a specific industry like metals, chemicals or plastics or get it “integrated” (all together) or “custom made” (sliced and diced how you like it). Or, you can just call Michael at Manifest Journal to discuss your Trade IQ personally. Maybe he’ll give you a trade IQ test. You may even qualify for Trade Mensa.
So, back to Trade Intelligence, what is it?
On one level it is simply DATA with a little tech magic:
- Presumably sifted like grain (to remove the rocks, sticks and foreign objects). Everyone gets said “grain” from the same farmer, CBP.
- Prudently classified, sorted and deposited into neat little silos (database objects called tables) hopefully guided by geeks with know how.
- Craftily shaked, baked and served up using whatever “off-the-shelf” or custom-made business intelligence software and graphics reporting solution that is “tech du jour”.
Since the same data is publicly obtainable and relatively cheap and with sophisticated BI software packages within easy reach, definitions and expectations of what constitutes Trade Intelligence is likely to change…rapidly.
What, then, is Trade Intelligence? Many, if not most, data providers like to link their company and products to that term in one fashion or another. After all, it sounds a whole lot sexier than “trade data”. Data is about as sexy as dirt. Wouldn’t you pay more for “trade intelligence” than you would for “trade data”? I imagine that it evolved from the popular use of the term “business intelligence”. One should be discerning though; a pig dressed up like a princess and given a noble title, is still, after all a pig.
As founder /CEO of CenTradeX, I (and my team) spent a decade trying to build better bridges between trade data and trade data users. The initial premise was that there was tremendous inherent value locked away in publicly available trade data that had been hitherto ignored or undervalued. Our journey led to many innovations in the field of “trade intelligence”.
The first was combining many sources of data together to provide a more complete picture, 3-D versus flat, for example: U.S Trade Flows with Global Statistics with State Exports. Thereafter, we integrated several sources of company data with statistics… a project that often kept me up late into the night, and that I called a “Beautiful Mind” endeavor because it about drove me crazy. Marrying huge sets of data organized under very different, asynchronous code systems is not a simple task.
Another aspect of bridge building was presenting the data in graphic form with user-friendly reporting tools. One of our first customers, Howard Cochran, IB professor at Belmont University, used to comment that we “taught data to dance”. Our goal was to dress up data to look like Disney… trade data dancing and singing alluring melodies that would woo potential customers. All these innovations were elements of the bridge building process to transform trade data into trade intelligence.
However, at the end of the day, we learned a very important lesson: No matter how much creativity and technological wizardry we brought to the table, we could not build a bridge that would reach to the other side. The last section of that bridge is people. No interface or TI product can, or ever will, stand alone as THE consummate solution. Suppliers, Marketers, Consultants, Analysts, Trade Reps, Teachers, Matchmakers, and the trade data users themselves, ultimately provide the “intelligence” behind TI products.
Therefore, the core, primary, essential element of Trade Intelligence is first and foremost people: people are the intelligence behind “trade intelligence”. And thus the reason why WorldTradeDaily.com focuses on all the key players in the industry from CEOs to database engineers, marketers and clients.
*This post was originally published during the first week of May, 2011.
PART 1 of “Trade Intelligence” looks at the real “smarts” required to make TI products useful and profitable. If vendors claim that their respective product can provide the “magic” that will make your business succeed, they are lying. No matter how sophisticated and technologically slick the “inter-face” is, profitable end results will only be extracted with considerable “face” time invested in the process by you and trusted trade “sages”. Machines (or TI products) that can out think (or do the thinking for) their human creators (or data users) still only exist in the realm of science fiction.
From WIKI: “Intelligence derives from the Latin verb intelligere which derives from inter-legere meaning to ‘pick out’ or discern.” Other references note that intelligence is also “The aggregate or global capacity of the individual to act purposefully, to think rationally, and to deal effectively with his environment.” ”The unique propensity of human beings to change or modify the structure of their cognitive functioning to adapt to the changing demands of a life situation.” Also from WIKI: ”Business Intelligence is a set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information used to enable more effective strategic, tactical, and operational insights and decision-making.”
In my opinion, one of the most distinguishing characteristics of intelligence is curiosity: the hunger to know, to learn, to discover. Another aspect of intelligence is to see patterns, connections, and find meaning in seemingly unassociated fragments of information. Finally, intelligence applies these discoveries adaptively in order to reach its objectives.
I’ve often remarked during my presentations that the “the genius is in the question”. Questions drive innovation and application in the Trade Intelligence field. Behind every impactful report (in whatever product /application you may use) are queries. Queries are questions, crafted by ubiquitous database /tech guys, to fetch, arrange and deliver data in response to your questions. On a deeper level, the need to develop more meaningful connections between various types of data drives every respectable company in the industry in response to customer demand.
In coming up with profitable answers that will support and assist in making important business decisions in the realm of world trade, one must be careful to craft the question correctly… and to know what questions need to be answered. Then, a particular TI product can be evaluated on the basis of how well it can address those respective questions. Be careful to look beyond the price tag. After all, you’re going to be investing your most valuable commodity into the process of extracting any substantive value out of it…your time.
Part 2 of this article will examine the process of creating “Trade Intelligence” products along with practical “answers” that translate into bottom-line profitability.
*This post was originally published during the first week of May, 2011.
The official language of international merchandise trade exists within a Harmonized System of product codes; “a tariff nomenclature (which) is an internationally standardized system of names and numbers for classifying traded products developed and maintained by the World Customs Organization (WCO)… [used by] more than 200 countries, customs and economic unions, representing more than 98% of world trade.”
The HS (sometimes called HTS for Harmonized TARIFF System) is organized in 21 sections and 96 chapters. The 96 top-level two digit categories (let’s call them “parents”) form a hierarchical lineage containing over 1,000 four digit children and some 6,000 six digit grandchildren. ”To ensure harmonization, the contracting parties (countries) must employ all 4- and 6-digit provisions and the international rules and notes without deviation, but are free to adopt additional subcategories and notes.”
A familiar idiom states, “Give the devil his due”. The primary reason behind the HTS system of product identification, besides providing a common reference point for trading purposes, is taxation. It’s the all important middle “T” in HTS. Although Tariffs & Duties can be applied to both imports and exports, “Tariffs are usually associated with protectionism, a government’s policy of controlling trade between nations to support the interests of its own citizens. For economic reasons, tariffs are usually imposed on imported goods.”
Each country further applies additional sub-categories beneath the universally agreed upon 6 digit HS provisions in order to track more precisely the commodities and products they are most interested in “protecting”; i.e. monitoring and taxing. These subcategories (great-grandchildren, great-great grandchildren, etc.) can generate branches of 8, 10, 12 or more digits. ”The Harmonized Tariff Schedule of the United States (HTS) is the primary resource for determining tariff classifications for goods imported into the United States (and can also be used in place of Schedule B for classifying goods exported from the United States)”. There are over 17,000 unique ten-digit HTS classification code numbers.
One of the very first tasks that many Trade Consultants undertake in working with a newbie importing or exporting company is to help them assign the appropriate HTS codes to their respective product(s). Most U.S. companies don’t have a clue. In China, I think they learn the HS system in elementary school along with English. Improperly classified products are likely to have a very difficult journey through Customs and incur additional costs.
One of the biggest problems in the process of transforming trade data into trade intelligence is inherent in the HTS nomenclature. It is extremely obtuse. The descriptions are almost impossible to understand. The classification “logic” varies from chapter to chapter, category to category. The most commonly used term is the ever endearing and everlasting designation: NESOI. No, it’s not a type of plastic, article of clothing or technological component. It means, NOT Elsewhere Specified Or Indicated. World-wide imports and exports of NESOI have experienced exponential growth during the last 10-20 years during the information age.
By and large it’s all a form of Trade Technocracy, a malady which I have spent a sizable portion of my tenure as a Trade Intelligence Professional attempting to remedy.
In addition to the primary TI providers much referred to within the articles published on World Trade Daily, there are a number of valuable statistical sources that are useful when trying to develop a complete picture of International Trade. These include sources that provide Global, U.S., State and Metro trade statistics. They include the following:
- Metro Export Statistics
- U.S. Small to medium sized businesses exports
- State Trade Statistics (Government Source)
- State Manufacturing and Trade (National Manufacturers Association)
- U.S. Agricultural Trade
- Interactive U.S. Statistical Database (One of the best resources. You must register and login, but it’s free)
- Jump site for the International Trade Administration with links to many data resources.
- Global Trade Statistics – World Trade Organization
- Global Trade Statistics UN Comtrade (free, one of the best sources)
- Good supplemental source for Global data – The World Bank
Beyond the resources referenced above, which are mostly places to get hard facts, data and charts, it still remains a matter of interpreting the data, making sense of it, seeing patterns, making observations, and “spicing” /dressing it up a bit with some Google research as well as your own intuition (based upon intelligence, knowledge and experience).
Some other interesting articles I ran across recently as an adjunct or primer to the above are the links below. They are particularly helpful when trying to prioritize which cities, companies, states to select from when conducting research and developing analyses.
- Forbes articles on top 10 U.S. importers and top ten exporters
- Top 50 U.S. importers (2009) from Zepol
- List of Top 2000 companies in the world
- Leading Exporters (Countries) wiki
- Leading importers (countries) wiki
- Leading U.S. Trading Partners wiki
- Article about top ten countries that buy stuff from U.S.
- Sister article to above naming top ten countries that sell stuff to us.
- Very good PDF listing top exporting Cities in the U.S. Useful as reference material. Also see the Original article.
- U.S. government source (international trade administration) for top exporting cities .
- Another (old list of top U.S. Importers).
- Nice overview on worlds largest exporting counties (top 10).
- Wiki on international trade (primer).
What can you do with Global Trade Flow Statistics? What can the numbers tell you? Think of it like getting to know a person. Each person has a story. Each product that is bought or sold has a story too. The big picture of international trade is composed of many millions of individual stories woven together into a huge, ever-changing tapestry. Global Trade Flow Statistics are like “vitals”.
When you go to a doctor, he/she checks weight, blood pressure, temperature, etc. These pieces of data provide the foundation for understanding your particular situation. After the preliminaries are completed, you will be asked additional questions in order to assess your individual condition. Perhaps x-rays, blood tests or another specialist examination will be called for. Each vital statistic is evaluated in combination and in context (with the aid of experts) in order to render a proper diagnosis and treatment plan.
Global Trade Flow Statistics (about what has been, is currently and what is expected to be bought or sold by a particular country of your respective product) are vital pieces of information. They not only establish a fundamental knowledge of the situation, but also provide the necessary clues about what other data is required. Collecting, comparing, analyzing and reporting this and accompanying data are called Market Research.
No would-be (intelligent) exporter or importer of products or raw materials should attempt to conduct business without it. It would be similar to dispensing a prescription without a thorough examination. You might guess right more times than not, particularly if you consider yourself to have “street smarts”, but the cost of not knowing what you don’t know can be considerable.
Many times, the most important information that can be gleaned from statistical analysis is more questions. I remember stumbling upon a 10,000% jump in annual shipments of auto parts from the U.S. to an obscure African country. Further investigation revealed that, to circumvent the trade embargo imposed upon South Africa, U.S. shipments were being rerouted through an adjacent nation.
Another anomaly I remember was the exponential increase in imports by the U.S. of Chilean Salmon. Market research conducted for an entrepreneurial friend of mine, revealed that 98% of Chilean salmon shipments came into Miami, took several days to clear customs and then several more days to be trucked to destinations north. His plan was to establish direct flights into Nashville, reroute distribution and save almost a week. In the world of a fresh, perishable commodity like fish, that’s an eternity.
At CenTradeX, I developed dynamically generated reports that illustrated the trade balance (comparing exports with imports) for any particular product between any chosen trading partners (countries) over a 20 year period. With 6,000 product categories and 200 countries that generated over 1,000,000 snapshots, it was easy to observe, if “trade war” is a metaphor you are comfortable with, exports representing money (green) and imports representing (red) how the U.S. has been faring, economically speaking, (soldier by soldier/product by product) with China over time.
It all starts with the data. Data is THE fundamental building block used in constructing Trade Intelligence. Trade Intelligence, in one form or another, is the guide map used to navigate trillions of dollars of exchanges in goods and services by every year. WorldTradeDaily.com is dedicated exclusively to the matter of Trade Intelligence.
Let’s take a look at Global Trade Flow Statistics particularly merchandise (versus services) trade. First of all, it’s easy to forget when looking at the numbers, that beneath all the digits and commas are real people, companies, jobs, business relationships, political agendas, and the prosperity (or lack thereof) of countries and regions.
Specifically, Global Trade Flow Statistics, organized within the hierarchy of roughly 6,000 commonly used and agreed upon product (HS) codes, track the imports and exports between each of 200 countries. The respective governing (and taxing) authority for each government collects, aggregates and (usually through an associated entity) disseminates statistical information on their trade activities at least once per year. Many, like the U.S. and Europe, release data monthly.
The numbers reflect the total import or export value for each of 6,000 (six digit HS) product categories, which of course is easily aggregated to the four-digit parent grouping (around 1,200) and the 90+ (two digit) grandparent classes. They also document the number of units sold, whatever type unit, item, case, container, pallet, or barrel is being referred to. Obviously, cost per unit can be derived from dividing the value by the number of units bought or sold.
You can obtain this information on virtually every country on the planet with a few exceptions. For instance, Taiwan is not officially recognized by the United Nations as a separate country for publication purposes. You won’t find Palestine’s imports and exports either. The U.S. releases trade flow statistics every month trailing around 45 days. For handfuls of developing countries you may have to wait a year or two before you see trade figures.
Some countries maintain their own special rules for what they consider a documentable (and therefore reportable) export or import. Of particular note are China and several Middle East countries. China doesn’t report products manufactured and exported by foreign-owned companies as exports. Therefore, you will quickly discover HUGE discrepancies between what China reports they export to –let’s say the U.S. -versus what the U.S. proclaims they import from China.
Consequently, the truth about burgeoning trade deficits and trade imbalance along with the associated political and economic bantering about such things, needs to be reviewed in the light of underlying definitions. It is said, “Information is power”. Many of the specifics about trade transactions are carefully guarded government secrets. China, Inc. is closer to true than not.
The reality is that International Trade is the indisputable foundation for economic growth and prosperity. Global Trade Flow Statistics aren’t just obtuse and academic, they are historically relevant, currently pertinent and provide clues to future opportunities and trends that can inform and prosper the wise.
While taking a much-needed vacation in the Dominican Republic, I ended up being “commissioned” by a fellow vacationer to conduct a research study into the scooter (little motorcycles under 50cc) market. Although my girlfriend and I dispensed with computers, cell phones and all other electronics, I guess it’s harder to totally turn off the business side of the brain. I do confess though that it is somewhat recreational to engage in entrepreneurial deliberations over excessive inebriation. Anyway, I thought to share some steps that can help any newbie researcher to analyze a product for import. We’ll start with getting some basic data.
Step One: In order to retrieve data about a product, you must find the appropriate harmonized classification for that item. The Harmonized code for Mopeds is 871110. Remember, 6 digits is the most specificity you can uniformly retrieve for either exports or imports. Beyond 6 digits there are variations from country to country on classification. You can try to find the code via Google. Typing in the words: “Harmonized Code, scooter, under 50cc” brings back a set a results from which it is easy to lift the harmonized code.
You can also try a more formal approach such as visiting the U.S. Census Bureau tariff search site. The words: scooter or moped return a screen that asks the user to choose between the following categories:
- <= 50 cc (871110)
- > 50 and <= 250 cc (871120)
- > 250 and <= 500 cc (871130)
- > 500 and <= 800 cc (871140)
- > 800 cc (871150)
The International Trade Commission offers a comprehensive catalog by which users can drill down into the fine details of a particular harmonized code by chapter and verse.
To continue, each of the 6,000 six digit harmonized product classifications are organized within a parent – child – grandchild hierarchy. Each “family” has its own set of rules and criteria under which subclassifications are created. In the case of motorcycles 8711, its parent – the grandfather of the family – is 87 Vehicles. Motorcycle’s offspring are organized by size from the littlest of the brood: 87.11.10 (equal to or less than 50cc) to big bikes: 87.11.50 (the hunkiest of the gang). Although not displayed in the Census results, there is another grandchild, which is also in most of the families designated by the xx.xx.90 designation: 871190: other.
Step Two: Getting the export values from Census is simple. Basically we export about $24-25 million in scooters (mopeds) annually.
Step Three: Getting the import values, through the ITC DataWeb interface is a real pain. You first have to register and then go through an agonizing process to define and retrieve a report. UGH! So painful and laborious. It’s free at least.
Step Four: So, what does the data tell us? Well, for one thing scooter imports dropped drastically from 2008 to 2009 – by 60%, then by another 50% between 2009 to 2010. Wow. Plummeting from $140 million to $29 million is rather significant. YTD (January through May 2010-2011) reveals that imports are rebounding though, seeing almost a 400% increase over the same period last year.
Just so happens that our friends over at Zepol have produced a much friendlier, easier to get and easier tounderstand recap. Click this link for an updated chart.
Zepol’s rendition beautifully illustrates that the increase in imports gathered steam in recent months. It also goes to show that the instruments offered for free by the U.S. Government are world’s behind those available through reputable TI providers.
The most accessible and inexpensive source for global trade flow statistics is the United Nations. Through their COMTRADE database, users are free to search for and download data on the imports and exports of products classified in over 6,000 categories (in the Harmonized System) between almost 200 reporting countries. The same data is also available in other product classification systems (like the ISIC and SITC) for some countries.
The statistics are gathered and disseminated, as they are received, on an annual basis. With some countries the time lag is only several months following year-end, while others take a year or two to report. The dataset depicts trade value, number of units bought or sold and trading partner (corresponding country) for each given year. The U.N. maintains (and makes available to the public) historic records of each country’s trade activity up to several decades back.
Users are limited to the number of records they are able to download at one time without cost. However, the U.N. offers an inexpensive paid subscription option that provides unlimited search and download capability of all their trade databases. The corresponding interface allows users to save their search queries for later use as well as set up alerts with automatic download of updated data.
The only negatives are the lack of specificity and recency. The U.N. data only reflects aggregated annualized figures. Updates are sometimes spotty and infrequent. The data contains only the basest attributes of value, flow (import or export) partner (country) and unit of measure as well as other derivative statistical information therein contained (category sums, cost-per-unit, etc).
If you want greater specificity and frequency, you will need to turn to other sources. GTIS (Global Trade Information Services) and WISER Trade (World Institute for Strategic Economic Research) (They used to be called MISER.) These sources have gone through the sometimes complicated processes of obtaining trade data directly from the individual countries as soon as it is made available – sometimes monthly. The countries where they don’t get “special” data, they fill in with U.N. /ComTrade data, FYI.
Of course, individuals and organizations have the option of apprehending the same information (that GTIS or WISER sells) fairly easily, at least on 50% -60% of the countries. U.S. Census sells import/export merchandise trade flow statistics for a couple hundred dollars. EuroStat, releases similar information at no or low cost. Obtaining Japan trade data is a simple matter as well. Therefore, 80% of the worldwide merchandise trade, conducted by the countries referenced above, can be obtained and analyzed on a monthly basis at minimal cost.
What one abandons by such efforts are the technologies and convenient user search and reporting tools that have been developed by GTIS, WISER and several other Trade Intelligence providers, many of which are finally integrating aspects of Global Trade Flow Statistics into their particular product interfaces. On the other hand, for maximum flexibility, versatility and veracity in utilizing data for specific analyses, reporting and applications, one may be best served by going directly to the sources.
The best source for U.S. Trade flow statistics, if you want them in the purest, rawest form, is the U.S. Census Bureau – Division of Foreign Trade (USCB-FT). Trade Statistics are the bread and butter of Trade Intelligence. Several TI Providers, namely GTIS and Wiser Trade, have made a business from superimposing their particular brand of searching/reporting engine atop of said data, but for the greatest versatility and analytical capability, one is best to start at the source.
USCB-FT collects, aggregates, slices, dices and disseminates data collected on and about U.S. international import and export transactions. As they state, “The United States Code, Title 13, requires this program. Participation is mandatory. The Treasury Department assists in the conduct of this program.” Yup, if you’re going to buy or sell anything valued at $2,000 (imports) or $2,500 (exports) or more overseas you must pay homage to the Feds. Paperwork makes the world go around.
In whatever form the resulting (aggregated) transactional trade data is presented, USCB-FT takes special care to prevent anyone from being able to link the statistics to the underlying companies. It is one, if not THE, primary objective of your neighborhood Trade Intelligence Supplier to disaggregate this data and reconnect the dots obscured by the U.S. Government. It takes sophisticated technology, other data sources, lots of hard work, clever sleuthing and a bit of luck, but it can be done. But I do digress.
Back to basics. USCB-FT serves up a yummy variety of statistical delicacies in several schedules and venues (available for either imports or exports) including:
- USA Trade Online - a paid subscription service that provides access to most of their trade flow databases.
- U.S. International Trade in Goods and Services - monthly with annual summary and revisions.
- U.S. Merchandise Trade: Selected Highlights - imports of merchandise, available monthly.
- U.S. Exports of Merchandise – Monthly - you can get it back to 1989 in some fashion or form.
- U.S. Exports by State - monthly, quarterly or annually by 6 digit HS or 4 digit NAICS.
- U.S. Exports by Port - monthly, quarterly or annually by 6 digit HS.
The above list of Data are compiled in terms of commodity classification, quantities, values, shipping weights, methods of transportation (air or vessel), customs district, customs port, country of origin (or destination).
- In the case of exports – state of (movement) origin and whether contents are domestic goods or re-exports.
- In the case of imports – market share, unit prices, import charges and duties collected.