How Do I Learn DSP?

What follows is one person's response to the perennial question "How do I learn DSP?" It was written by Dale Grover, with input from folks including Rick Lyons, Randy Yates, and Grant Griffin, though their participation is not an endorsement of every opinion expressed here--they're just good guys.  A disclaimer:  I am a co-author of one of the books mentioned.

1/10/07

Introduction:  What do you want to know?  What do you know now?
The three necessary elements
    1.  Books
        Non-traditional
        Online Tutorials
        Classics
    2.  Software
        Octave, Matlab, & relatives
        DSP-specific
        Symbolic
        General purpose
    3.  Hardware
Formal courses
A brief overview of DSP topics
About this document
 

What Do You Want to Know?  What Do You Know Now?

The best way to learn digital signal processing (DSP) depends on your goals and background.  Why is this? In the past--in fact, until a few years ago--DSP was taught (and books were written) almost exclusively for people with an electrical engineering background.  If that's your background, you have plenty of options in learning about DSP, and the resources noted in this document will give you plenty of choices.  But if you don't have an electrical engineering background, there are now some books (and courses) that can bring you up to speed on basic DSP topics, and even on implementation issues.  More about these options in a moment, when we discuss books.

There are some folks--myself included--who claim that we should treat DSP in a similar way with how statistics is taught.  Instead of claiming that every person who wants to know a little about statistics has to learn Everything There Is To Know about statistics, there are instead statistics courses for humanities majors, engineers, mathematics majors, and so on.  Why?  Because statistics is useful in many fields, and a useful subset of statistics can be taught to and used by folks even if they haven't studied all the theory behind statistics.  In much the same way (actually, in a very deep sense), DSP has these same characteristics.  DSP is, of course, useful, and indeed most anyone can learn the basics of DSP--and successfully apply these tools in their field.  This document outlines some resources and approaches to do just that.

Like statistics, there are trade offs between the size of the toolbox (and your skill in using those tools) and the amount of study and background required. At one extreme, you can end up with just the step-by-step "recipe" without any need to understand what's going on. At the other extreme, you can reach the point where you can select techniques from a wide array of possibilities, or even create your own, though this will require a good background in mathematics and so on.  Your interests and backgrounds will be different from everyone else, and there is no single path to "DSP enlightenment"--or even "DSP familiarity."

Two (or Three) Necessary Elements

Any approach to learning DSP outside a formal classroom setting will eventually involve some combination of books, DSP/math software, and DSP hardware, since each can offer insight and motivation into DSP. We discuss each in the sections that follow.  We also mention some formal learning situations.  It is our hope that this document can provide you a framework for determining an appropriate path, and useful tools, for you to learn more about DSP.

Books

"Non-traditional"

DSP texts are now available that address a range of backgrounds and goals. As we mentioned above, until recently, DSP texts assumed you had a "traditional" electrical engineering background. However, there are now some books and courses that take a subset of DSP techniques and present it in a way that is accessible to people without this background. For example: (See the comp.dsp FAQ and/or dspGuru.com for bibliographic information.)

These books are aimed, for example, at scientists, engineers, computer science/programmers, and researchers who want to learn basic DSP for use in their work. Generally speaking, these books downplay the mathematics, and emphasize intuitive presentations.  (Stein's book, addressed at computer science students, doesn't shy away from the mathematics and theory, but like the others doesn't make assumptions about electrical engineering topics.)  Some also emphasize practical implementation issues. Digital filters and spectral analysis (e.g., the FFT--the fast Fourier transform) are topics common to all these texts; other topics can include synthesis of signals, implementation on real hardware, signal detection, etc. Generally speaking, the core topics common to all these texts are useful in understanding and using more specialized DSP techniques such as are used, for example, in creating special music effects.

Online Tutorials

There are several online tutorials available--see for example the list of tutorials given at http://www.dspguru.com/info/tutor/other.htm. Not every tutorial starts with the same assumptions, however. So, just as with books, don't despair if your interests and background are not addressed; just seek alternatives.

There are also books aimed at traditional electrical engineering students who do not yet have the traditional DSP prerequisites; these are an experiment in rearranging the order in which topics are taught to electrical engineering students, rather than targeting a non-engineering audience. (See, for example, McClellan, Schafer, and Yoder's DSP First.)  However, since the intended students do not have an extensive background, these books might be useful to other readers as well.  Bear in mind that while the background that is assumed may be similar to the other books, the goals are different.  (In this case, the goals include preparing students for further electrical engineering coursework, so some additional foundations are laid.)

Where Will This Leave You?

The "non-EE" books and resources above provide a (mostly) self-contained, often "low-math" path to basic DSP theory, and in some cases, implementation.  However, as you might guess, there are some trade offs for books that take this route.  In particular, for those books that downplay the (mathematical) theory, some connections can be lost between topics that otherwise are easily seen.  If the basic DSP topics you've learned are sufficient, then this isn't a big loss.  But to tackle more complicated DSP, whether that's understanding some specialized technique, or striking out on your own to tackle some new problem, it will be necessary to have a deeper background, including Many DSP texts start assuming roughly this type of background, corresponding to perhaps a third year undergraduate engineering curriculum (see below). It is entirely reasonable to initially explore DSP using less rigorous texts (some of which are listed above), but it is difficult to understand the more advanced DSP theory without the techniques and language of these background areas.  The fact is, most DSP papers, books, presentations, and even software, will assume a certain familiarity with concepts from the areas above, and it will be hard going to fill these in "on the fly."

Classics

Some classic (and popular texts) are listed in the comp.dsp FAQ (see below), which include A recent text that covers linear signals & systems through basic signal processing is Lathi's Signal Processing and Linear Systems (1998), though there are many, many books on linear signals and systems that should be accessible to readers who have had calculus, differential equations, and an introductory electronics course.

The more traditional DSP texts do not necessarily omit practical issues, and some include applications such as digital audio effects (see Orfanidis' Introduction to Signal Processing, for example). However, you should be aware of the level of mathematical comfort the authors assume, even if the topics covered are appealing.

"But all I wanted was to do some neat audio effects..."

Does this mean that you'll have to master all of basic DSP theory to understand, for example, some neat digital audio effect?  Not at all--but there are two caveats.  First, depending on the field, these neat algorithms are often expressed using terminology and mathematics that assumes a traditional DSP or analog electronics (especially analog filter design) background.  You might have to dig around for explanations that are aimed at folks without a deep DSP background.  (For example, Circuit Cellar Ink often publishes articles on signal processing that are aimed at non-DSP folks.)  Second, realize that you won't have as much flexibility in using these techniques--it might be difficult to know how to tweak or change the algorithm without running afoul of some problems.  But this is true with many tools we use, and it is not an excuse to give up.  In fact, playing with some algorithm you are motivated to use and understand can give you real life experience with the issues textbooks cover (for example, how sampling rate affects a signal).

Once the basic DSP background is in place, many application-specific DSP areas are accessible. This includes speech processing (recognition, synthesis, compression), image processing, music processing (analysis, synthesis, etc.), all types of biomedical signal processing, and on and on. All draw upon a common set of tools, but you'll find each field also has its set of favorite tools, many of which might not be used extensively in other areas of DSP.  (In part, this is may be due to historical influences, but it is also the case that the signals being processed in different fields can have strikingly different characteristics that make them more or less amenable to different DSP techniques.  Thus, speech processing often uses very different techniques than image processing, though they also share many techniques.) In addition, a number of advanced DSP techniques (multi-rate filters, various stochastic techniques, etc.) have wide application, but are difficult to approach without a solid background in basic DSP.

If you're getting the impression that folks active in DSP end up with quite a few books, you're right! No single book can take into account the wide backgrounds and interests of everyone who needs to use DSP. Not to mention the many areas within DSP that have their own set of tools and techniques, like image processing, speech processing, etc. In addition, not every author places the same importance on every topic--so it's not uncommon to try a couple of books before finding, for example, a particular algorithm you want discussed in nitty-gritty detail. Though DSP is many decades old, some cutting edge techniques might be best found in the professional journals, trade magazines, or conference proceedings. A good web search may turn up these more recent publications.  The divide, mentioned earlier, between theory and implementation also means some additional books, since the overlap is often minimal.

Web Resources

If you're reading this, you've likely come across at least some web resources for DSP. A good starting place is the comp.dsp FAQ at http://www.bdti.com/faq/dsp_faq.htm. The FAQ contains a list of popular DSP books, online resources, useful and classic articles, and a host of other information. You will also find an interesting mix of people on the comp.dsp newsgroup with a wealth of experience. However, most basic questions can be answered in any good DSP book, and more specific questions reserved for the newsgroup. Another resource is http://www.dspGuru.com, which contains a number of articles on basic and advanced DSP techniques, pointers to online tutorials, sources of software, and other information you might find useful.  Another site you may wish to check is http://www.dsprelated.com/.

Software

PC-based software helps in several areas of DSP.  First, software can allow you to experiment with DSP.  Second, you can do (generally speaking, non-realtime) signal processing on PC's using a variety of software.  Third, PC-based software can be used to generate the realtime software that will run on a dedicated DSP chip.  The same software might do all three pieces, or just one.  But for a person learning DSP, the ability to experiment is most important, and the one that this section will emphasize.

Whether you want to acquire a few DSP techniques, or make DSP your main area of study, it is immensely helpful to have a software environment where you can experiment with signal processing. This environment should allow you to create (or at least import) signals such as audio signals and images, process them using simple mathematical operations, and play or display (graph) the processed signals and their frequency domain representations. University students in the US will find little trouble locating relatively inexpensive but powerful mathematical packages that support DSP very well. The following are some possible environments.  (See the comp.dsp FAQ for more details.)

Octave, MATLAB, and Relatives

Octave is a very powerful environment for numerical computation.  Programs can be written in a simple language, but the true strength of Octave is in the powerful matrix and vector operations, along with an extensive library of functions including some for signal processing.  Plotting is available using the gnuplot program.  Octave is available in source code, and in binary for Mac OS X, Linux, and Windows.  Octave uses a language that is largely compatible with Matlab (see below), though offers additional features such as zero-base indexing.  Actively maintained.  Installation can be a little more complicated than commercial packages.  Highly recommended.  Mac OS X users may find the High Performance Computing for Mac OS X page of use, as they often provide recently compiled versions of Octave.

MATLAB is a very popular commercial software package with strengths in numeric processing of arrays and matrices, but also has a number of "toolboxes" including one for DSP. Programs (script files) can be written or commands entered interactively, and the graphics are very good (including 3-d plotting). Supports audio output (and input?), but does not do real-time processing.  Simulink is an add-on product that allows you to simulate systems using graphical building-blocks, and there are toolboxes for this product as well that apply to DSP.  For non-students, MATLAB can be extremely expensive.  There are many books that feature MATLAB code (much of which can be ported to Octave fairly easily).

There are several other software packages, freely available, that are similar to Octave and MATLAB.  These include Scilab and Rlab.  See the discussion at http://www.dspguru.com/sw/opendsp/mathclo.htm for details.

DSP-Specific Software

Many packages are available that are specific to creating and analyzing digital filters (and in many cases, other more advanced DSP operations). In some cases, code can be generated automatically, targeting either specific DSP chips (in assembly or C language), or generic C code. The trade off is flexibility vs. turnkey operation. As discussed below, the low cost and high function of generic mathematics packages with DSP support probably makes them a better value for US students and those interested in learning more about DSP as a subject, while the DSP-specific packages will require less learning in order to generate viable digital filters.  (Both comp.dsp FAQ and dspGuru.com contain more information.)

MathCAD, Mathematica

Mathematica and MathCAD are also popular mathematical environments, historically with an emphasis on symbolic mathematics. However, DSP is possible in both packages, and to a large extent comments applicable to Octave, MATLAB, and relatives apply to these packages as well.  It is also very handy to have a symbolic mathematics package to verify derivations and other tasks common in traditional DSP courses and their precursors. Good to excellent plotting capability is present in these packages.  DSP-specific extensions to Mathematica are described in the comp.dsp FAQ.

General Purpose Programming Languages

BASIC, C, Java, and other languages can all be used to design, demonstrate, and implement DSP. For learning DSP, however, it is very useful having good plotting capabilities--the simplicity and power of the plotting routines in the mathematical packages is hard to beat.  (Of course, you can always export your data and use gnuplot or similar for plotting.)  In addition, the interactive nature of the mathematical packages is nice to have, as it allows more of a exploration of the processing than a batch-mode environment.  While the design and implementation of many digital filters is fairly straightforward, it is very convenient having ready-made functions to both design and implement DSP operations, as are available in the mathematical packages.  Likewise, other tools such as wavelets or the FFT are fairly easy to write if performance is not an issue, but are very sensitive to coding if speed is very important.  Numerically intensive operations like the FFT are optimized in the math packages.

In some cases you may want to create a standalone program that implements your DSP task.  For example, a filter that despeckles an image using a median filter.  In this case a useful strategy is to prototype the program using Octave, where it is fast and easy to make changes and see what is going on, and then to write the code using C, Objective-C, or what have you.  You can then compare the output or intermediate results with the Octave code to verify operation, but you'll end up with a smaller and likely faster program, especially if you are able to take advantage of operating system calls that implement DSP operations using vector processing abilities that are now fairly standard on modern PC's (e.g., PowerPC's AltiVec, Intel's SSE).

Spreadsheets

Spreadsheets can offer both good plotting and simple data manipulation abilities. However, they are fairly constrained when compared to mathematical packages, or even general purpose languages (though the interactive nature of the spreadsheet is a positive).  Environments like Octave would likely be much more productive.

Software Used in Texts

A recent trend is for text books to integrate computer-based exercises and examples. By far the most common software used is MATLAB. While this is quite beneficial to the average US engineering student (who not only may purchase the software relatively inexpensively--on the order of $100--but will probably use it for many classes), the high cost of the full version of MATLAB is a barrier to the non-student reader.  As discussed above, Octave may be a reasonable replacement as the language is quite similar, however some functions might need to be written.

It should also be noted that if you have a specific processing need--such as signal analysis with the FFT, speech compression or synthesis, or processing audio files--there are often ready-made programs that are freely available. The source code is not always public, but if the function is sufficient, this is by far the easiest method to perform very specific DSP jobs. Even if you do avail yourself of these prepackaged solutions, you will probably benefit from doing a little additional reading on DSP, to get a basic idea of what's going on "under the hood" and what limits there are to the processing that is occurring. The comp.dsp FAQ lists a number of such programs; a web search on appropriate keywords would also be productive.

Real-Time Hardware

Most DSP algorithms are not very complicated.  It's true.  In fact, some of the most important DSP functions are implemented by a simple repeated multiplication and summation operation, and could be written out in a BASIC program that nearly anyone could understand. However, the speed with which the algorithms must operate often requires specialized microprocessors ("DSP chips") in order to run in real-time. While DSP chips have many similarities with general-purpose microprocessors, they have an architecture tuned to common DSP operations, and the programming of such hardware (sometimes in assembly language) for maximum performance is a skill that must be acquired.

The good news is that most major DSP manufacturers (e.g., Motorola, Texas Instruments, Analog Devices, etc.) now provide low-cost evaluation platforms for their DSP chips. Generally $100-$500, these kits typically include the DSP chip on a board with memory and analog input/output sufficient to do some real time processing of audio and other low bandwidth signals. Software almost always includes assemblers, linkers, and simulators. A C compiler is often available. (The fact that a C compiler can be used does not invalidate the need for the programmer to understand the architecture of the DSP chip, though if the application represents a very light load on the processor, the programmer can postpone detailed study during earlier stages in the design process.)  A major factor in choosing a DSP chip is whether it employs floating or fixed point math. Fixed-point chips are generally much cheaper, but floating point chips are easier to program (since one does not have to worry about certain effects the fixed-point math can produce).

These kits are an excellent way to explore the implementation issues of real-time DSP. However, they are typically not a great way on their own to learn the theory of DSP--this is best accomplished using the texts noted above and software environments that allow better debugging, visualization, quick modification, and less complicated programming environment.

In addition, it should be noted that if the processing needs are not great, it is possible to do DSP using a general-purpose microprocessor. DSP need not require a DSP chip; it is (usually) a series of rather simple mathematical operations on a set of numbers representing a signal. It is entirely possible to do DSP using a PC in real-time for many types of signals (such as audio).

Lest the wrong impression be given, it should be noted that many people will never require their particular digital signal processing occur in real time. For example, their data might be processed in batches, or be data that has no time component (such as images). Implementation of DSP on DSP chips is an additional skill, and one which will be entirely optional to many people who require DSP in their work.

Formal Training

What if you prefer to learn in a more structured way? Universities and companies offer a variety of short and/or evening courses on DSP. (See the comp.dsp FAQ for one list.) The quality of this training, according to some people who have attended, can vary from good to bad. University-based courses may still follow tradition and begin their classes with so much mathematics that the fledgling student may have trouble learning the simplicity and beauty of DSP. As Rick Lyons likes to say: "DSP training that's too rich in mathematics is hard for beginners to digest." Note also that many technical conferences are beginning to provide tutorial signal processing lectures. (DSP World, ICSPAT, and the Wireless Symposium are examples.)

We should also begin to see more DSP courses offered as part of the standard course offerings that package DSP up for the non-engineer (recall "DSP First" mentioned above, and Stein's text for computer science students). Though this may involve a much larger time commitment than alternatives, it may be worthwhile in that you would be able to ask questions of the instructor, and there may be labs with real-time DSP hardware set up and ready to go, along with the equipment to generate and analyze signals.

Summary

Digital signal processing is not impossible to learn, it doesn't require a PhD in mathematics, and it really can be useful even if you only ever learn some basic tools. DSP can be done on almost any hardware and using almost any software--it is just a question of how fast you need the processing done. You can learn DSP using a variety of software, including some excellent free programs. There are now texts and other resources aimed at the non-engineer, and it has never been easier to experiment with signal processing--even in real time. So get a book, download some files, and process some signals.

A 60-Second Overview of DSP

Start your clock. Here is very brief overview of some of the basic tools and concepts in DSP.

Why use DSP?

Digital Signals

Sample the analog signal at some regular interval (fast enough to accurately describe the signal, with enough resolution to keep the noise level low), and in doing so convert the signal into a long list of numbers that represent the amplitude (e.g., voltage) of the signal at these points. This is usually done with an analog-to-digital converter.

Processing Digital Signals

The key to DSP is that instead of using op-amps, resistors, and other analog electronics to process an analog signal, you can use a microprocessor (or specialized microprocessor, as in a DSP chip) to perform mathematical operations on that list of numbers (the digital signal) to achieve the same (or better) effect.  For example, this might filter out high frequencies, or translate the signal into another, more useful form (e.g., using the "FFT"). Optionally--though usually--you can convert the resulting list of numbers back into an analog signal using a digital-to-analog converter.

(Digital) Filters

As a generalization, digital filters pass certain frequencies while stopping others. Taking the average of the last 7 daily temperature readings filters out temperature changes on a daily level. This is a bona fide digital lowpass digital filter (of the "FIR"--finite impulse response--variety). Another simple way of filtering a signal would be to take a weighted combination of the last average and the current reading, and to use this combination as a new average.  This new average, in combination with the next reading, is used to compute the next average, and so on. This is another example of a digital lowpass filter (a type known as "IIR"--infinite impulse response). Most basic digital filters are just variations on these two themes.  The hardest part is in specifying the desired behavior of the filters and determining how to combine past input values and past output values to get this desired behavior (i.e., determining the weights or coefficients).  Often, you just punch in numbers to a software package to calculate the relevant coefficients, which you plug into very simple algorithms (a fact that some texts seem to treat like a state secret!).

Spectral Analysis

What frequencies are present in a signal? That is, if you had a filter that you could tune to just let a narrow band of frequencies through (kind of like tuning a radio), at what frequencies would parts of the signal come through? Or, alternatively, what sine waves (i.e., at what frequency and with what shift) should be added to create a duplicate of a given signal? The FFT and other related techniques give a picture of a signal in terms of frequency--somewhat like the display on an audio equalizer shows the energy at different frequencies of an audio signal.
 

This Document

This is not intended as a comprehensive review of DSP, but instead as a framework for pointing the DSP newcomer to some useful approaches to learning DSP.  If there are issues that you feel should be addressed that would be helpful to the DSP newcomer, comments are welcome--send them to dgrover@redcedar.com.  The most up-to-date version of this document can be found at www.redcedar.com. Nonprofit distribution of this document is permitted as long as this document is not altered and this notice remains. Specific permission is granted www.dspGuru.com and the comp.dsp and comp.speech.* newsgroups and associated FAQ's to include this document.
MATLAB, Mathematica, dspGuru, MathCAD, and others are registered trademarks.
Copyright 2002, Red Cedar Electronics.