To use FPGAs for communications applications, you often need to do some system modeling to understand the problem beforehand. This ensures that the correct approach is taken. You often must rely on expensive tools such as the MatLAB Signal Processing Workbench and Simulink, but low-cost alternatives are becoming available for the hobby user to use on less demanding projects.
Consider, for example, the SciCosLab and SciLab tools from INIRA Labs in France. The freely available SciCosLab may be thought of as a MatLAB clone. Jeremy Clark, an engineer, has put together a PDF/video course on communications theory using SciCosLab. The 80-plus-page course starts by introducing open-source tools like SciCosLab. It covers text-based tools, block diagram editor-oriented tools, and the Modnum toolbox used to create the communications modulation waveforms using SciLab.
The mechanics of using the simulation tools take up the first two chapters. Once that is out of the way, Clark gives a treatment of telecommunications. Chapter 3 gives an overview of telecommunications. That's followed by a chapter introducing baseband signals, including single tone, multi-tone, WAV, NRZ, and Manchester Data. Each chapter is followed by exercises that are completed using the simulator (SciCos or SciCosLab).
The next chapter introduces superheterodyne analog, and the various baseband signals are mixed using superheterodyne techniques. Chapter 6 presents the topic of amplitude modulation (AM), along with the modulation of various baseband signals using AM techniques.
Chapter 7 lets students visualize frequency modulation in the simulator, along with the modulation of differing signal formats. This chapter also covers phase modulation.
Chapter 8 covers quadrature amplitude modulation and orthogonal frequency division modulation formats. I should note that this just covers the basics -- there are other ways to perform these types of modulation. Chapter 9 discusses code division multiple access (CDMA).
The real power of simulation is that it allows you to visualize what happens to the signals before your design is realized. It also allows you to simulate the effects of the communications channel on the signal and to optimize the design before implementation.
Though they may not be as powerful as MatLAB and the Signal Processing Toolbox, these open-source simulators are very useful educational tools. So is Clark's excellent course, which comes at a very reasonable cost (mostly free except for the last few sections). Do you know of any useful open-source tools for communications or other applications?