Information theory

Bibliography

Prior work

• Harry Nyquist, "Certain Factors Affecting Telegraph Speed", Journal of the AIEE, Vol. 43 (1924), p 124; repr. in Bell System Technical Journal, Vol. 3 (April 1924), pp 324-346. Presented at the Midwinter Convention of the AIEE, Philadelphia, February 1924. Shows that a certain bandwidth was necessary in order to send telegraph signals at a definite rate. Considers two fundamental factors for the maximum speed of transmission of 'intelligence' [not information] by telegraph: signal shaping and choice of codes. Used in Shannon 1948.
• Harry Nyquist, "Certain Topics in Telegraph Transmission Theory", Transactions of AIEE, Vol. 47 (April 1928), pp 617-644; repr. in Proceedings of the IEEE 90:2 (February 2002), pp 280-305. Presented at the Winter Convention of the AIEE in New York in February 1928. Argues for the steady-state system over the method of transients for determining the distortion of telegraph signals. In this and his 1924 paper, Nyquist determines that the number of independent pulses that could be put through a telegraph channel per unit time is limited to twice the bandwidth of the channel; this rule is essentially a dual of what is now known as the Nyquist–Shannon sampling theorem. Used in Shannon 1948.
• Ralph V.L. Hartley, "Transmission of Information", Bell System Technical Journal 7:3 (July 1928), pp 535-563. Presented at the International Congress of Telegraphy and Telephony, Lake Como, Italy, September 1927. Uses the word information as a measurable quantity, and opts for logarithmic function as its measure, when the information in a message is given by the logarithm of the number of possible messages: H = n log S, where S is the number of possible symbols, and n the number of symbols in a transmission. Used in Shannon 1948.

Primary documents

• Norbert Wiener, The Extrapolation, Interpolation, and Smoothing of Stationary Time Series, NDRC Report, MIT, February 1942. Classified (ordered by Warren Weaver, then the head of Section D-2), printed in 300 copies. Nicknamed "Yellow Peril". Published in 1949 (see below). [1], Commentary.
• Claude E. Shannon, A Mathematical Theory of Cryptography, Memorandum MM 45-110-02, Bell Laboratories, 1 September 1945, 114 pages + 25 figures; repr. in Shannon, Miscellaneous Writings, eds. N.J.A. Sloane and Aaron D. Wyner, AT&T Bell Laboratories, 1993. Classified. Redacted and pubished in 1949 (see below). [2]
• Norbert Wiener, Cybernetics: or Control and Communication in the Animal and the Machine, Paris: Hermann & Cie, Cambridge, MA: Technology Press, and New York: John Wiley & Sons, 1948, 194 pp; MIT Press and Wiley, 2nd ed., 1961, 212 pp; 1965; 1980. Reviews: Dubarle (1948, FR), Littauer (1949), MacColl (1950). In the spring of 1947, Wiener was invited to a congress on harmonic analysis, held in Nancy, France and organized by the bourbakist mathematician, Szolem Mandelbrojt. During this stay in France Wiener received the offer to write a manuscript on the unifying character of this part of applied mathematics, which is found in the study of Brownian motion and in telecommunication engineering. The following summer, back in the United States, Wiener decided to introduce the neologism ‘cybernetics’ into his scientific theory. According to Pierre De Latil, MIT Press tried their best to prevent the publication of the book in France, since Wiener, then professor at MIT, was bound to them by contract. As a representative of Hermann Editions, M. Freymann managed to find a compromise and the French publisher won the rights to the book. Having lived together in Mexico, Freymann and Wiener were friends and it is Freymann who is supposed to have suggested that Wiener write this book. Benoît Mandelbrot and Walter Pitts proofread the manuscript. [3]
  • Cibernetica. Controllo e comunicazione nell’animale e nella macchina, Milan: Bompiani, 1953. (in Italian)
  • N. Viner (Н. Винер), Kibernetika, ili upravlenie i svyaz v zhivotnom i mashine [Кибернетика, или Управление и связь в животном и машине], Moscow: Sovetskoe radio [Советское радио], trans. G.N. Povarov, Moscow: Sovetskoe radio, 1958, 216 pp; 1963; 2nd ed., 1968. (in Russian)
  • Cibernética, trans. Miguel Mora Hidalgo, Madrid: Guadiana, 1960, 314 pp; 1971. (in Spanish)
  • Kybernetik. Regelung und Nachrichtenübertragung in Lebewesen und Maschine, rororo, 1968; Econ, 1992. (in German)
  • Cybernetyka, czyli sterowanie i komunikacja w zwierzęciu i maszynie, Warsaw: PWN, 1971, 261 pp. (in Polish)
  • Cibernética ou controle e comunicação no animal e na maquina, São Paulo: Poligono, 1970. (in Portuguese)
  • Cibernética o el control y comunicación en animales y máquinas, trans. Francisco Martín, Barcelona: Tusquets, 1985, 266 pp; 1998; 2002, 150 pp. (in Spanish)
• Claude E. Shannon, "A Mathematical Theory of Communication", Bell System Technical Journal 27 (July, October 1948), pp 379-423, 623-656. Part 2
  • Klod Shennon (Клод Шеннон), "Statisticheskaia teoriia peredachi elektricheskikh signalov" [Статистическая теория передачи электрических сигналов; The Statistical Theory of Electrical Signal Transmission], in Teoriya peredakhi elektrikheskikh signalov pri nalikhii pomekh [Теория передачи электрических сигналов при наличии помех], ed. Nikolai A. Zheleznov (А. Н. Железнов), Moscow: Izdatelstvo inostrannoi literatury, 1953. (in Russian). The editor rid the work of the words information, communication, and mathematical entirely, put entropy in quotation marks, and substituted data for information throughout the text. He also assured the reader that Shannon’s concept of entropy had nothing to do with physical entropy and was called such only on the basis of "purely superficial similarity of mathematical formulae". [4]
  • "Matematicheskaya teoriya svyazi" [Математическая теория связи], trans. S. Karpov, in Raboty po teorii informatsii i kibernetike [Работы по теории информации и кибернетике], Moscow: IIL (ИИЛ), 1963, pp 243-332. (in Russian)
• Robert M. Fano, The Transmission of Information, Technical Reports No. 65 (17 March 1949) and No. 149 (6 February 1950), Research Laboratory of Electronics, MIT. A similar coding technique like Shannon's, only deducted differently. In 1952 optimised by his student, Huffman (see below).
• Norbert Wiener, The Extrapolation, Interpolation, and Smoothing of Stationary Time Series with Engineering Applications, New York: Wiley, 1949; MIT Press, 1964. Earlier printed as a classified NDRC "yellow peril" Report, MIT, 1942. Reviews: Tukey (1952). Uses Gauss's method of shaping the characteristic of a detector to allow for the maximal recognition of signals in the presence of noise; later known as the "Wiener filter."
• Warren Weaver, "The Mathematics of Communication", Scientific American 181:1 (July 1949), pp 11-15; repr. in Basic Readings in Communication Theory, ed. C. David Mortensen, Harper & Row, 1973, pp 27-38.
• Claude Shannon, Warren Weaver, The Mathematical Theory of Communication, Urbana: University of Illinois Press, 1949, 117 pp; 1963; 1969; 1971; 1972; 1975; 1998. Reviews: Hockett (1953, EN). Consists of two texts: reprint of Shannon's 1948 paper and Weaver's "Recent contributions to the mathematical theory of communication". [5] [6]
  • コミュニケーションの数学的理論, trans. Atsushi Hasegawa, 明治図書, 1969; 2001. (in Japanese)
  • La teoria matematica delle comunicazioni, trans. Paolo Cappelli, Milan: Etas Kompass, 1971. (in Italian)
  • Mathematische Grundlagen der Informationstheorie, Munich and Vienna: Oldenbourg, 1976. (in German). Contents.
  • 通信の数学的理論, trans. Uematsu Tomohiko, 筑摩書房, 2009, 231 pp. (in Japanese)
• Claude E. Shannon, "Communication Theory of Secrecy Systems", Bell System Technical Journal 28 (October 1949), pp 656-715; repr. as a monograph, New York: American Telegraph and Telephone Company, 1949. Unclassified revision of Shannon's memorandum A Mathematical Theory of Cryptography, 1945, which was still classified (until 1957). [7]
  • "Teoriya svyazi v sekretnykh sistemakh" [Теория связи в секретных системах], trans. С. Карпов, in Raboty po teorii informatsii i kibernetike [Работы по теории информации и кибернетике], Moscow: IIL (ИИЛ), 1963, pp 333-402. (in Russian) [8]
• Norbert Wiener, "The Machine Age", [1949]. Unpublished. Written for The New York Times.
• Norbert Wiener, The Human Use of Human Beings: Cybernetics and Society, Boston, MA: Houghton Mifflin, 1950; 2nd ed., 1954; London: Eyre and Spottiswode, 1954; New York: Avon Books, 1967; New York: Da Capo Press, 1988; London: Free Association Books, 1989.
  • Cybernétique et société, l'usage humain des êtres humains, Union Générale d'Éditions, 1952; 1971. (in French)
  • Mensch und Menschmaschine, Frankfurt am Main: Metzner, 1952; 4th ed., 1972. (in German)
  • Cibernética e sociedade: o uso humano de seres humanos, trans. José Paulo Paes, São Paulo: Cultrix, 1954; 2nd ed., 1968, 190 pp. (in Portuguese)
  • Kibernetika i obshchestvo [Кибернетика и общество], trans. E.G. Panfilov, Moscow: IIL, 1958, 200 pp. (in Russian)
  • Cybernetyka i społeczeństwo, Warsaw: KiW, Warszawa 1960, 236 pp; Cybernetyka a społeczeństwo, 2nd ed., Warsaw: KiW, 1961, 217 pp. (in Polish)
  • Introduzione alla cibernetica. L’uso umano degli esseri umani, trans. Dario Persiani, Torino: Bollati Boringhieri, 1966, 229 pp; 3rd ed., 1970, 240 pp; 1982; 2012, 234 pp. (in Italian). Review.
  • Ihmisestä, koneista, kielestä, trans. Pertti Jotuni, Helsinki: WS, 1969. (in Finnish)
  • Cybernética y sociedad, Buenos Aires: Sudamericana, 1969. (in Spanish)
  • Cybernética y sociedad, México: Fondo de Cultura Económica, 1984. (in Spanish)
• Dominique Dubarle, "Idées scientifiques actuelles et domination des faits humains", Esprit 9:18 (1950), pp 296-317. (in French)
• Louis de Broglie, La Cybernétique - Théorie du Signal et de l'Information, Paris: Edition de la Revue d'Optique Théorique et Instrumentale, 1951. (in French). Collection of a lecture series given in April and May 1950.
• David A. Huffman, A Method for the Construction of Minimum-Redundancy Codes", Proceedings of the I.R.E. 40 (September 1952), pp 1098–1102. Fano's student developed an algorithm for efficient encoding of the output of a source. Became popular in compression tools.
• Colloques Internationaux du Centre National de la Recherche Scientifique 47 (1953). (in French). Proceedings of a congress held in Paris in January 1951. Paul Chauchard: the congress was "the first manifestation in France of the young cybernetics, with the participation of N. Wiener, the father of this science." For this congress, organised by the French scientists Couffignal and Pérès who had visited the U.S. laboratories and sponsored by the Rockefeller Foundation, a number of foreigners were invited, including Howard Aiken, Warren McCulloch, Maurice Wilkes, Grey Walter, Donald MacKay and Ross Ashby, along with Wiener who was staying in Paris for a couple of months at the Collège de France. 300 people attended; 38 papers were presented; 14 machines from six different countries were demonstrated. [9]
• Stanford Goldman, Information Theory, Prentice-Hall, 1953, 385 pp; New York: Dover, 1968; 2005.
  • S. Goldman (С. Голдман), Teoriya informatsii [Теория информации], Мoscow, 1957. (in Russian)
• Pierre de Latil, La Pensée artificielle, Paris: Gallimard, 1953. (in French)
• Benoît Mandelbrot, Contributions à la théorie mathématique des jeux de communications, Institut de Statistiques de l'Université de Paris 2, 1953. (in French). Ph.D. dissertation in mathematics making a connection between game theory and information theory. He showed for instance that both thermodynamics and statistical structures of language can be explained as results of minimax games between ‘nature’ and ‘emitter’. He also made the connection between the definitions of information given by the British statistician Ronald A. Fisher in the 1920s, by the physicist Dennis Gabor in 1946 and the already well-known definition proposed by Shannon. Mandelbrot was at MIT from 1952-1954 and later at the IAS in Princeton. [10]
• Raymond Ruyer, La cybernétique et l'origine de l'information, Paris: Flammarion, 1954. (in French)
• G.-th. Guilbaud, La Cybernetique, PUF, 1954. (in French)
• Aleksei Liapunov, Anatolii Kitov, Sergei Sobolev, "Osnovnye cherty kibernetiki", Voprosy filosofii [Problems of Philosophy] 4 (August 1955). (in Russian). The first Soviet article speaking positively about cybernetics and non-technical applications of information theory, authored by three specialists in military computing—Liapunov, a noted mathematician and the creator of the first Soviet programming language; Kitov, an organizer of the first military computing centers; and Sobolev, the deputy head of the Soviet nuclear weapons program in charge of the mathematical support. They presented cybernetics as a general "doctrine of information", of which Shannon’s theory of communication was but one part. The three authors interpreted the notion of information very broadly, defining it as "all sorts of external data, which can be received and transmitted by a system, as well as the data that can be produced within the system." Under the rubric of "information" fell any environmental influence on living organisms, any knowledge acquired by man in the process of learning, any signals received by a control device via feedback, and any data processed by a computer. [11]
• Claude E. Shannon, "The Bandwagon", IRE Transactions on Information Theory 2 (1956), p 3.
  • "Bandvagon" [Бандвагон], trans. S. Karpov, in Raboty po teorii informatsii i kibernetike [Работы по теории информации и кибернетике], Moscow: IIL (ИИЛ), 1963, pp 667-668. (in Russian)
• Claude E. Shannon, "Information Theory", Seminar Notes, MIT, from 1956, in Shannon, Miscellaneous Writings, eds. N.J.A. Sloane and Aaron D. Wyner, AT&T Bell Laboratories, 1993.
• Léon Brillouin, Science and Information Theory, New York: Academic Press, 1956. A bestseller rewrite of physics using information theory.
• A.N. Kolmogorov (А. Н. Колмогоров), Teoriya peredachi informatsii [Теория передачи информации], Мoscow, 1956. (in Russian)
• Information and Control journal, *1958. Founding editors: Léon Brillouin, Colin Cherry, Peter Elias.
• Igor’ A. Poletaev, Signal: O nekotorykh poniatiiakh kibernetiki, Moscow: Sovetskoe radio, 1958. (in Russian). The first Soviet book on cybernetics.

History and interpretation

• Louis H. M. Stumpers, "A Bibliography on Information Theory (Communication Theory - Cybernetics)", I.R.E. Transactions on Information Theory 1 (1955), pp 31-47.
• William Aspray, "Scientific Conceptualization of Information: A Survey", Annals of the History of Computing 7:2 (April 1985), pp 117-140.
• E.M. Rogers, T.W. Valente, "A History of Information Theory in Communication", in Between Communication and Information, Vol. 4: Information and Behavior, eds. J.R. Schement and B.D. Ruben, New Brunswick, NJ: Transaction, 1993, pp 37-56.
• Hayles, N. Katherine (1999). How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics. University of Chicago Press. pp. 350.
• David Mindell, Jérôme Ségal, Slava Gerovitch, "From Communications Engineering to Communications Science: Cybernetics and Information Theory in the United States, France, and the Soviet Union", in Science and Ideology: A Comparative History, ed. Mark Walker, London and New York: Routledge, 2003, pp 66-96.
• Slava Gerovitch, From Newspeak to Cyberspeak: A History of Soviet Cybernetics, MIT Press, 2004, 383 pp.
• Libb Thims, "Information theory", Hmolpedia, 2008-.
• Axel Roch, Claude E. Shannon. Spielzeug, Leben und die geheime Geschichte seiner Theorie der Information, Berlin: gegenstalt, 2009. (in German)
• Andrew Pickering, The Cybernetic Brain: Sketches of Another Future, Chicago University Press, 2010, 526 pp.
• James Gleick, The Information: A History, A Theory, A Flood, Knopf, 2011, 496 pp.

External links

• History of information theory at Wikipedia
• A Mathematical Theory of Communication at Wikipedia
• Information theory at Wikipedia