Top of The Roof Gang

...I was just over at Castle Argghhh, and I noticed that John has a post reminding us of the USS Pueblo.. being a former Crypto/ELINT guy myself, his mention of "Surveillance Ships" got me to reminiscing... heh... I've seen a few of the Russian versions too, but alas, that is for another blog.. oh, and if you haven't read his post and comments, then get over there and read the story...

...I remember once watching a Master Gunnery Sergeant get inducted into the "Top of The Roof Gang"... at the time, he was ending his career in the Corps, and moving into a cushy civilian job as a Network Security Manager in Atlanta.. being a newbie at the time, I had never heard of the "Top of the Roof" boys... but, at his ceremony, the Captain gave a speech about the birth of Naval Cryptography... he spoke about the original "gang"... and how, those talented few Men had helped to change the course of the War in the Pacific... since then, Sailors and Marines who had added something to the Naval Crypto community, were honorarily inducted into the "gang".... so, it was quite an honor for his name to have been added to that list of notable cryptographers...

I just did a quick search for "Top of The Roof Gang", and didn't come up with much...except this little article from a book review...here is a small sample... click below to see the whole article....

"The JN-25 code appeared first on the radio circuits carrying the Japanese fleet's traffic on 1 June 1939. Long before this the U.S. Navy had established a network of listening posts across the Pacific--at Guam, Honolulu, and Cavite. (The Marines operated a fourth post, at Shanghai, that concentrated on Japanese diplomatic traffic.) Since 1928, the Navy had been training special radio operators in the esoteric craft of intercepting and taking down the unique Japanese Morse code. They soon became known as the "On the Roof Gang." Trainees sent to Washington for the course found that their first assignment each day was to climb a ladder to a concrete blockhouse where the classes took place, in great secrecy, atop the sixth wing of the old Navy Department Building. "

"Too Late for Pearl Harbor

By Stephen Budiansky

Few genuine mysteries remain from what is probably the most exhaustively probed event in U.S. history--the Japanese attack on Pearl Harbor. One that especially has nagged historians--until now--is the extent of U.S. Navy codebreakers' ability to read the main Japanese naval code in the months preceding the attack.

Almost immediately after World War II, congressional investigations broke the secret that the United States had been reading Japanese diplomatic codes before the outbreak of hostilities. Through these "MAGIC" decrypts, the U.S. government had definite knowledge on the night before 7 December 1941 of Japan's intentions to break diplomatic relations. For several weeks before, the decrypts also had contained strong hints that Japan was preparing to initiate hostilities.

When Frank Rowlett, a senior cryptanalyst of the Army's Signal Intelligence Service, arrived at his office in the old Munitions Building on the Mall in Washington, D.C., at noon on 3 December, he found the latest MAGIC decrypt on his desk. Minutes later he found himself seized with astonishment: The Japanese embassy in Washington was being ordered to destroy its codebooks and one of the two precious machines with which it had been entrusted for coding and decoding diplomatic traffic.

Colonel Otis Sadtler, who was in charge of distributing MAGIC, showed up at Rowlett's office at that moment and began peppering him with questions about what this latest communication could mean. Had the Japanese sent anything like this before? No, Rowlett said, and it was hard to see how the embassy people could handle even their normal flow of traffic if they destroyed their codes and machine. By this point, Sadtler had become so agitated that he pulled himself to attention and barked out, "Rowlett, do you know what this means? It means Japan is about to go to war with the United States!"

Decrypt in hand, Sadtler took off, literally running down the corridor to alert the head of Army Intelligence. Secretary of State Cordell Hull recalled later that when he saw this decrypt, he felt that "the chances had diminished from one in a thousand to one in a million that war could then be avoided."

On the night of 6 December, President Franklin D. Roosevelt received MAGIC decrypts that revealed the final instructions to Tokyo's embassy that it should break off relations. Roosevelt read in silence; he then turned to his aide, Harry Hopkins, who had joined him in his bedroom at the White House as the news came in, and said that this meant war.

Conspiracy theorists have mined the MAGIC decrypts for all they are worth. But while they show that Washington had strong indications of Japan's intentions to strike, they contain not a hint of where that strike might come. To know that would have required intelligence from a very different source. Historians have known since the 1960s that the U.S. Navy's brilliant decryption of the Imperial Japanese Navy's Fleet General Purpose Code in early 1942 turned the tide of the Battle of Midway that June. Also well established is the fact that the Navy began working on this code, which U.S. codebreakers dubbed AN-1, and later JN-25, from the time it first appeared in 1939.

What has not been well established, because of continuing security classification of key documents, is just how much of JN-25 was readable in the critical months before the Pearl Harbor attack. Perhaps inevitably, this secrecy has fueled speculation of something to hide in all of this; those out to prove that President Roosevelt deliberately concealed warnings of an attack on Pearl Harbor to bring the nation into war first have to prove that something was there to conceal. The mystery surrounding JN-25 offers fertile ground for speculation.

Several authors have attempted to weave a circumstantial case that British and U.S. codebreakers were indeed reading JN-25, and therefore that specific intelligence must have been available regarding Japan's plan of attack. The most recent and the most vehement of these authors is Robert Stinnett, whose new book, Day of Deceit (New York: The Free Press, 1999), contends that JN-25 and other Japanese codes were read throughout 1941. Stinnett argues that because documents show that U.S. Navy codebreakers were close to cracking JN-25 in October 1940, they surely were reading it a year later.

In testimony before the many investigating bodies that scrutinized the U.S. intelligence failure at Pearl Harbor, the Navy codebreakers recalled that only 10% to 15% of JN-25 was being read in November 1941. But those statements all were based on memory, not documentary evidence. And so enough doubt has lingered to keep the conspiracy theories alive.

In March 1999, while conducting research at the National Archives at College Park, Maryland, I discovered several heretofore unreleased documents detailing the Navy's work on JN-25 during this crucial period. These were declassified several years ago but had not yet been processed by the Archives staff; nor were they listed in the finding aids of materials available to researchers. The documents include contemporaneous, month-by-month, date-stamped progress reports on how many code groups in JN-25 had been deciphered. They also include the first declassified account of exactly how JN-25 was broken.

What they show beyond all doubt is that by 1 December 1941--when Japan changed all of its codes and call signs in preparation for the Pearl Harbor attack--the U.S. Navy had succeeded in identifying the meanings of only a minuscule fraction of the currently used JN-25 code groups. The documents show that in the year leading up to the Pearl Harbor attack, the Navy codebreakers failed to read a single JN-25 message sent at any time during that period. And the documents show why they failed: A year earlier, on 1 December 1940, the Japanese threw out their codebook and introduced an entirely new book for JN-25. It was a huge setback for the codebreakers. This new documentary evidence, of which Stinnett and other conspiracy theorists are completely
unaware, decisively refutes the claim that JN-25 or any other high-level Japanese codes were being read in the months leading up to the Japanese attack.

These newly available documents also show, ever so painfully, what might have been. The Navy's codebreakers had broken the basic key to JN-25 months before in a truly brilliant feat of cryptanalysis. They had developed pioneering methods of using IBM punch card sorters and printers in that precomputer age to automate much of the needle-in-a-haystack searching that is the stock in trade of codebreaking. All they lacked was the manpower to get the job done in time.

How different history might have been is heartbreakingly revealed in the JN-25 traffic from those fateful months that was only much later--in 1945 and 1946--broken out. One of the most striking of these messages, read four years too late, was an order sent by Carrier Division 2, on 4 November 1941:

YUUZUKI-DD will pick up and take to Kagoshima the torpedoes which CARDIVs 1 and 2 are to fire against anchored capital ships on the morning in question.

None of the messages mentions Pearl Harbor specifically, but their cumulative weight as certainly suggestive. Dozens of key messages give an unmistakable indication of preparations to initiate hostilities shortly after 1 December. Several make explicit references to a surprise air attack--several refer to practice drills for an "ambush"--to be launched from carriers against the U.S. fleet.

The New Code Appears

The JN-25 code appeared first on the radio circuits carrying the Japanese fleet's traffic on 1 June 1939. Long before this the U.S. Navy had established a network of listening posts across the Pacific--at Guam, Honolulu, and Cavite. (The Marines operated a fourth post, at Shanghai, that concentrated on Japanese diplomatic traffic.) Since 1928, the Navy had been training special radio operators in the esoteric craft of intercepting and taking down the unique Japanese Morse code. They soon became known as the "On the Roof Gang." Trainees sent to Washington for the course found that their first assignment each day was to climb a ladder to a concrete blockhouse where the classes took place, in great secrecy, atop the sixth wing of the old Navy Department Building.

At each of the Navy's monitoring outposts, operators tuned into the stream of dots and dashes and copied down messages that were, to them, completely incomprehensible--nothing but a series of meaningless Japanese syllables, each of which was represented by a unique string of dots and dashes in the Japanese Morse system. It was a mind-numbing task that demanded total concentration; a single slip could render a coded intercept worthless. Rumors circulated constantly about trainees who had gone "code nutty" under the strain of listening to nothing but beeps for hours on end.

Once a week, the message sheets were bundled together and turned over to a captain of one of the Dollar Line's "President" ships that plied the Pacific. The captains all held commissions in the U.S. Naval Reserve and so had authority to serve as couriers for top-secret documents; upon reaching the West Coast they would forward their packets of intercepts to the Navy Department via registered mail. A small amount of urgent traffic could be dispatched by way of the "Clippers" of Pan-American Airways. A small strongbox had been built into the hull of each aircraft just for this purpose, with the keys held by Navy officers.

The Navy radio operators who listened in on their future enemies noticed immediately that they were dealing with something new when the JN-25 messages began to appear. Unlike earlier Japanese codes, these were sent in groups of five numbers. The U.S. codebreakers would, in short order, become all-too-familiar with this type of code. It was a type used for all high-level Japanese naval and military communications, and it was based on a system that, to any normal human being, would seem impenetrable. Words, numbers, place names, punctuation, Japanese syllables, and various abbreviations were each assigned a distinct, five-digit code number. JN-25 initially used such 30,000 code groups. To encode a message, a clerk would look up each word or
character in a book and write down its numerical equivalent. Then came the devilish complication: The clerk would open a second book, a 300-page volume that contained 30,000 random five-digit numbers, 100 to a page. This was known as the "additive" book. He would open the book at random and copy out as many five-digit additives as there were code groups in the message he was preparing. He would then add each code group to each additive in turn. For example:

message text:
FROM KAGA: ESTIMATED TIME OF ARRIVAL 2130
code text:
21936 48322 01905 38832 87039 11520 38832
from upper case Kaga stop ETA 2130 stop follows additive:
02923 41338 00989 15861 28959 23693 18229
enciphered text:
23859 89650 01884 43693 05988 34113 46051

To simplify matters and to keep each resulting sum of enciphered text a five-digit number, the addition was done digit by digit, without carrying (e.g., 9 plus 4 equals 3). When the clerk was finished, he had a string of five-digit numbers that, to any casual or even not-so-casual observer, would seem random and meaningless. Although the code group "stop" appears twice in the message text above, it appears in the final signal as 43693 in one place and 46051 in another. In another message it would appear as an entirely different number. That was a mighty armament against cryptanalytic attack.

To tell his recipient how to decipher this otherwise meaningless string of numbers, the code clerk's last step was to include in the message an additional five-digit group that served as a "key" or "indicator"--a number that, when decoded, would indicate on what page and line number of the additive book he had started. The recipient would turn to that page and line number, subtract the additives from the enciphered code groups he had received, and then finally look up the meanings of the recovered code groups.

The security of the system depended above all on not reusing any one stretch of the additive book too often. Japanese code clerks were under strict orders to pick a different starting point for each message. To spread the traffic even more thinly across the entire additive book, different clerks each were ordered to begin using additives in a different part of the book.

Only through the laziness of Japanese code clerks did the Navy's cryptanalysts make their first crucial break. Throughout the summer of 1939 the codebreakers at the Navy's OP-20-G in Washington, under the direction of Commander Laurance Safford, punched every intercepted message onto IBM cards and began groping for even the slightest irregularity that would give them a toehold. After searching every way they could imagine, they found one vague unevenness, so slight as to be almost invisible. If the code clerks really had done their jobs, the indicators would be random. They were not. When the codebreakers printed out a complete catalog of the indicators in each day's
traffic, they found that the numbers tended to bunch up. In other words, the clerks were tending to use the same pages over and over. (Not surprisingly, these pages corresponded to the front of the additive book, the easiest place to flip open a book.)

That was a small toehold indeed. But to a codebreaker it meant everything; it meant the theoretical possibility of beginning to tease apart the underlying code groups from the additive encipherment that concealed its true value. The trick was to find, among the thousands of messages, two that overlapped, two that had been enciphered with the same stretch of additive. If it was the cryptanalyst's lucky day, a pair of these overlapping messages might contain identical pairs of code groups that had been enciphered by one additive in one spot, another additive in another. From such slender reeds the cryptanalysts of OP-20-G--one year and hundreds of thousands of IBM cards later--had identified the numerical values of a few dozen code groups and a few dozen additives.

The First Break

The real break in JN-25 came on a single day in early fall 1940, and when it came it proved a remarkable blend of absolute brilliance, combined with sheer doggedness and just a touch of thievery.

To start, IBM runs had found another curious bunching. The only place where enough overlaps occurred to allow additives to be recovered were in the first four groups of messages. The IBM searches revealed that the same code groups were being used at the start of some messages. That led immediately to the hypothesis that these code groups stood for numerals: It was natural to begin a message by saying something such as "Reference your message 1234."

See much more http://www.usni.org/Proceedings/Articles99/PRObudiansky.htm

by Eric on February 07, 2004 | Comments(0) | Military Stuff

Comments so far: