'Investigative journalism has never been this effective!' - Publico

The WIKILEAKS Public Library of US Diplomacy (PlusD) holds the world's largest searchable collection of United States confidential, or formerly confidential, diplomatic communications. As of April 8, 2013 it holds 2 million records comprising approximately 1 billion words. The collection covers US involvements in, and diplomatic or intelligence reporting on, every country on earth. It is the single most significant body of geopolitical material ever published.

The PlusD collection, built and curated by WikiLeaks, is updated from a variety of sources, including leaks, documents released under the Freedom of Information Act (FOIA) and documents released by the US State Department systematic declassification review.

We are also preparing the processed PlusD collection for standalone distribution. If you are interested in obtaining a copy, please email: [email protected] and put 'Request' in the subject line.

If you have unclassified or declassified US diplomatic documents to add to the PlusD collection please contact: [email protected] and put 'Submission' in the subject line. Please note that for inclusion in the PlusD Library we are generally unable to consider submissions of less than 1,000 documents at a time.

• The Kissinger Cables
• Cablegate
• Preparing the Kissinger Cables for the WikiLeaks Public Library of US Diplomacy
• Preparing Cablegate for the WikiLeaks Public Library of US Diplomacy

The Kissinger Cables

The Kissinger Cables comprise more than 1.7 million US diplomatic records for the period 1973 to 1976. Dating from January 1, 1973 to December 31, 1976 they cover a variety of diplomatic traffic including cables, intelligence reports and congressional correspondence. They include more than 320,000 originally classified records, including 286,000 full US diplomatic cables. There are more than 12,000 documents with the sensitive handling restriction "NODIS", 'no distribution', and more than 9,000 labelled "Eyes Only". Full cables originally classed as "SECRET" total more than 61,000 and "CONFIDENTIAL" more than 250,000.

The records were reviewed by the United States Department of State's systematic 25-year declassification process. At review, the records were assessed and either declassified or kept classified with some or all of the metadata records declassified. Both sets of records were then subject to an additional review by the National Archives and Records Administration (NARA). Once believed to be releasable, they were placed as individual PDFs at the National Archives as part of their Central Foreign Policy Files collection. Despite the review process supposedly assessing documents after 25 years there are no diplomatic records later than 1976. The formal declassification and review process of these extremely valuable historical documents is therefore currently running 12 years late.

The form in which these documents were at NARA was 1.7 million individual PDFs. To prepare these documents for integration into the PlusD collection, WikiLeaks obtained and reverse-engineered all 1.7 million PDFs and performed a detailed analysis of individual fields, developed sophisticated technical systems to deal with the complex and voluminous data and corrected a great many errors introduced by NARA, the State Department or its diplomats, for example harmonizing the many different ways in which departments, capitals and people's names were spelled. All our corrective work is referenced and available from the links in the individual field descriptions on the PlusD text search interface: https://search.wikileaks.org/plusd. For more information on what WikiLeaks did to prepare the Kissinger Cables please see here.

Not all records from the period 1973-1976 have been obtained. NARA claims diplomatic records for the period 1973 to 1976 chosen for content deletion were of a ephemeral character. These records were identified by the "TAGS" that were attached to them. TAGS ("Traffic Analysis by Geography and Subject") refers to the content tagging system implemented by the Department of State for its central foreign policy files in 1973. There are geographic, organization and subject TAGS. This system was developed to standardise search terms for departmental uses and was not static - TAGS were added and deleted as necessary over time. At review, all cables that only contained "temporary" TAGS, such as embassy logistical or staffing requests, were permanently destroyed.

Tens of thousands of documents were irreversibly corrupted in this data set due to technical errors when the documents were moved as computer systems were upgraded, or so the US Department of State claims. This caused the content of the document to be lost, though the metadata is still available. These are often noted by a error message in the content of the document. The documents lost in this manner are most documents from the following periods:

• December 1, 1975 to December 15, 1975
• March 8, 1976 to April 2, 1976
• May 25, 1976 to July 1, 1976

You can see the absence of these weeks by constructing a Timegraph of "TAGS" as this term occurs in the content of nearly every document: http://search.wikileaks.org/plusd/graph#q=TAGS

Top Secret documents are also not available. During a migration of records the Department of State printed out all Top Secret documents for "preservation purposes" and the electronic versions were destroyed permanently. These documents now only exist as hardcopies and so are unavailable online in any form, even if declassified.

The documents not deleted either remained classified (or were deemed unreleasable for other reasons), or were declassified and publicly released. For the former, a "withdrawal card" was provided giving some limited metadata about the document, the fields of which that were decided as releasable vary from document to document. This metadata provides some information about the document, for example the date and destination, that can be used for research purposes and also allows a detailed FOIA request to be made for the document. These FOIA requests can be directed to NARA's Special Access and FOIA staff. For more information about this, please see their online guide here. You will need the document number and the To and From information.

There are nine different "Types" of document included in the Kissinger Cables. The majority are of type "TE" - telegram (cable), which are official diplomatic messages sent between embassies and the US Secretary of State conveying official information about policy proposals and implementation, program activities, or personnel and diplomatic post operations. From 1973 onwards diplomatic cables were mostly electronic, therefore most cables made releasable include the body (content) of the cable. However, the other types of documents are paper records, including airgrams and diplomatic notes. These are stored on microfilm (from 1974 onwards, as the Department of State did not microfilm documents until then) and so were not released with the full content of the documents, even if marked for public release. Although the body of the message is not available online the full index (metadata) is provided for those "P-reel" documents that were marked for release. Even though the whole document has not been digitised the metadata is still useful for research purposes and the documents can be requested under the Freedom of Information Act. For those documents on P-reel that were not declassified and released a P-reel "withdrawal card" is provided giving limited metadata. To access P-reel documents that have a withdrawal card you should follow the same FOIA procedure as for Telegram withdrawal cards. For the content of P-reel documents which have been released, the process depends slightly on which year the document you are requesting was created, but all requests should be directed to: [email protected].

Cablegate

Cablegate is a set of more than 250,000 US diplomatic cables originally published by WikiLeaks from November 2010 and over the following year. These documents were released after being anonymously leaked to WikiLeaks and detail modern United States foreign policy over the last decade. Whilst the earliest document in Cablegate is from 1966, with the set including documents up to Febuary 2010, the majority of documents in Cablegate are from 2000 onwards.

All documents in Cablegate are diplomatic telegrams (cables), as opposed to other types of diplomatic records and correspondence. Although the documents were all made available, uncensored, by WikiLeaks, they have not all been declassified. Unlike the Kissinger Cables in the PLusD collection, these were not declassified and so their current classification can be presumed to still officially be as it was, with more than 15,000 of them being classified Secret. Although not one of the most difficult data sets in the PlusD collection to prepare, there were still additional technical procedures we had to go through to add Cablegate to the PlusD collection. For full information on what WikiLeaks did to prepare Cablegate please see here.

Over the year of its release Cablegate had a huge effect all over the world, being widely acknowledged as a key factor in igniting the Arab Spring and altering United States diplomatic relations across the globe after the activities of the US, its component corporations, its allies and its enemies, became public.

Preparing the Kissinger Cables for the WikiLeaks Public Library of US Diplomacy ("PlusD")

Initial Analysis

The metadata in the Kissinger Cables is far more extensive than any other data set in the Public Library of US Diplomacy. The data set as a whole was also the first to contain "P-reel" documents, where the document only has headers (metadata) and no body or content, and "withdrawal cards", where there is only limited metadata available as the document was not declassified. An initial analysis of the documents was conducted to establish the meanings of the fields and which would be the most beneficial to researchers.

Field Extraction

After identifying which fields were of interest, we extracted the entries from each State Department PDF record. As these documents are rather old and created when computer systems were new they are not as standardised as more recent diplomatic documents, requiring the construction of a complex natural language parser.

For a number of the fields we had to extract the field not only from the affixed printed metadata, but also from the content of the document, where available. There were often disparities between these two elements. It is more complex to extract the information from the content as the start of the field is not clearly defined, however in many cases the content had more information, for example multiple different destinations listed whereas the formal metadata contained only, say, "LONDON, OTHERS". In other cases it was the formal metadata that contained more information, for example additional TAGS were added as needed (e.g., for 'Vietnam' following the victory of Ho Chi Minh), making it necessary to extract the field from both places and then remove any duplication later.

After extracting the fields we had to break up the field information into its different elements, for example the field entry "PFOR PREL" into two separate TAGS, "PFOR" and "PREL". The method for distinguishing between TAGS differed not only for each field, but within a field: often the Destinations are listed as one per line, yet a significant number of times they are listed on one line separated by a space. This is then further complicated as a space might also be used within individual TAGS names, for example "WARSAW PACT". We therefore developed complex algorithms, supplemented by a number of manual tasks, to create the individual field entries for each field.

To ensure that we had extracted and separated the field entries correctly we error-checked each one. For this we built a system for displaying a random document and checking the source document against our extraction from it that was to be used in the search and display functions of PlusD. We checked 800 documents randomly for each field for an error rate of less than 5% (40/800). If it failed this, we examined the errors and corrected our processes. We then repeated the 800 document check. Once this had an error rate of 5% or less, we checked 400 documents. If the result of the 400 document check was more than 5% error rate we corrected our processes again and restarted the error-checking process from the 800 document check. We did this for each field in sets of random samples of 800, 400, 200, 100, 50 and 25 documents until each one passed with an error rate of not more than 5%.

Resolving acronyms, spelling variations and frequent errors

After extracting the individual fields we assessed all possible entries for each field. Some fields have as many as 150,000 unique possible entries for the field. To complicate matters further, there are many different spelling variants of individual entries in a number of fields. These range from abbreviated forms and different acronyms with the same meaning to spelling mistakes. For the fields where this was most prevalent we have grouped spelling variants under the correct name for the entry. This means it is not only easier to understand the field but also permits complete searches. For example, to search all documents sent to Henry Kissinger you need only search for "Kissinger, Henry" once, and not all ten different spelling variations we found in the "Destinations" field one by one. The fields where these types of errors and variants occurred most often were "Office", "To" and "TAGS". We have grouped every single variant we could decipher in the "Office" field, and all those occurring 100 times or more in both the "TAGS" and "To" fields (that we were able to decipher), and many more as well. Each variant is listed under the 'true name' for the entry, that is, the one that we were able to officially reference as the correct way to refer to that entry. It is these canonical 'true' names that are listed in the dropdown menus of the PlusD search interface and in the synopsis at the top of the document display page.

For all acronyms in the data set we have looked for an official source as a reference to the meaning of that acronym. Where we have found one, we have expanded the acronym and given a URL to the citation source.

You can find full information on every spelling variant grouping we have made, our 'true name' assigning and our references for acronym meanings in the link supplied for each field in the search interface of the Public Library of US Diplomacy, or by clicking the 'more information' button in the summary of each document.

Document IDs

For document IDs within PlusD we have created a canonical and truly unique document ID. We created this by taking the original document ID and suffixing it with a '_'. We then added the letter 'b' to denote that these documents are from the Kissinger Cables rather than the other data sets within PlusD. Where the original document ID was duplicated within the source data set, we have added a number to the end of the document ID suffix. This number denotes which number duplicate the document is, for example '1973CAIRO02512_b2' is the second document within the Kissinger Cables with the document ID '1973CAIRO02512'. We provide links to all other documents with the same original document ID in the synopsis of each one.

References

A number of documents in the Kissinger Cables have a field in the metadata, "References", that gives the document ID of any other documents that are referred to by that document. We have extracted this field and matched up the references. On each document we have created a 'Reference' section that provides links to all the documents referred to in that document, and to all other documents which refer to that one. Not all document IDs in the Reference field are written in exactly the same manner, and a number of other documents could be matched with it. In these cases we have listed all possible matches.

Preparing Cablegate for the WikiLeaks Public Library of US Diplomacy ("PlusD")

Although Cablegate was originally published by WikiLeaks further work was needed to integrate it into the Public Library of US Diplomacy. For search purposes, the fields available in the short metadata of the Cablegate data set needed to be extracted and error-checked to confirm the extraction had read the fields correctly. Whilst this is one of the easier sets to achieve this with in the PlusD collection, because the majority of documents are reasonably standardised, it still had a number of issues to overcome.

There were a number of spelling variants for many field entries, which needed to be grouped together to create a reliable search function for such entries, and hundreds of acronym meanings to research and reference.

Due to the fewer metadata fields orginally provided by the Cablegate corpus, we needed to assign entries to the fields that the Cablegate corpus did not provide. There were two fields we were able to assign correctly through knowledge of the field and the data: the "Type" and the "Locator" fields. All Cablegate documents, unlike other data sets in the PlusD collection, are of type "TE" - telegrams, and all are (since being published by WikiLeaks) located in the form of "TEXT ONLINE". For the other fields which Cablegate did not have, we did not have the necessary information to provide these. We therefore gave all other fields not originally provided in the Cablegate data set the entry 'Not Assigned' in order that Cablegate could be fully integrated into the PlusD collection and search interface.

For document IDs we have created a canonical and truly unique document ID number. We created this by taking the original document ID and suffixing it with a '_'. We have added the letter 'a' to donate that these documents are from Cablegate rather than the other data sets within PlusD.

References

A number of documents in Cablegate reference other documents in the header of the cable by stating "Ref:" and then the relevant document IDs of the cables to which it refers. We have extracted this field and matched up the references. On each document we have created a 'Reference' tab that provides links to all the documents referred to in that document, and to all other documents which refer to that one. Not all document IDs in the Reference field are written in exactly the same manner, and a number of other documents could be matched with it. In these cases we have listed all possible matches.