It is very daunting for a beginner. The maths is literally complex (i.e. involving imaginary numbers, usually denoted i in maths, but j in electronics to avoid confusion with current). You need to have a basic understanding of transfer functions. You should be aware of what a Laplace transform is but you don't need to work one through. The same goes for Fourrier series (you may have heard of FFT) and Nyquist plots. You need to know about Bode plots and be able to draw them. You need to know the Nyquist stability criteria for feedback loops as well as logs for dB etc.
Briefly (hopefully), the Nyquist stability criteria for a feedback loop is that every frequency with gain over 1 (0dB) must be negative feedback for a system to be stable and not oscillate, i.e. have a phase margin (to 180 deg) of >0 deg, or put the other way, every frequency with negative phase margin (i.e. positive feedback) must have a gain of >0dB for any ringing to die out rather than turn into oscillation The 0dB crossover frequency is the most important frequency to look at.
This phase margin also determines the step response since the time and frequency domains are tied together (Laplace transform). Too little PM causes overshoot and ringing and the system is overdamped, too much phase margin and the system will take longer than necessary to settle in a step response and the system is underdamped. The ideal PM for critical damping is around 72 deg IIRC. In practice, you need at least 45 deg PM at crossover, 60 deg is a typical target value.
You can assess stability on a Bode plot. A pole (e.g. as used in a RC low pass filter), will add a gradient of -20dB/decade and -90deg phase to frequencies above it. A zero (e.g. as used in a RC high pass filter), will add a gradient of +20dB/decade and +90deg phase to frequencies above it.
There is also a right hand plane (of the Nyquist plot) zero which sometimes needs to be accounted for (e.g. in all boost and buck-boost derived converters, where in the time domain both the on and off time of the duty cycle are important and since more on time means less off time, a deficiency of off time can cause instability). In the frequency domain, a RHP zero has the gain effect of a zero but the phase effect of a pole. More gain and less phase margin is a bad combination and usually the only hope to stabilise a system with a RHP zero is to make sure it only happens where gain is <0dB.
There is also a special case for LC filters as both components are primarily reactive rather than resistive they form a complex conjugate double pole (or zero) which can have a resonant spike in the gain plot and can change phase 180 degrees in a very short span of frequency, depending only on how well it's damped. There is an LC filter in a Voltage mode controlled SMPS which can be difficult to bring under control. The main advantages of current mode control come from it splitting this complex conjugate double pole into 2 real poles separated by some distance and much easier to control (the high pole is usually well above the 0dB crossover frequency.)
Other typical targets for the design of your feedback loop are high DC gain for good DC regulation and a high crossover frequency for a fast step response. The theoretical limit for crossover frequency is half the switching frequency Fs (Nyquist's coming up again, this time in terms of sampling theory), and typically it'd be around 10-20% of Fs. Optocouplers and RHP zeros can push this much lower though.
Another thing that helps when drawing Bode plots is that the changes associated with a simple real pole or a zero are generally gradual from about a decade before to a decade after the pole/zero.
You can (and should) use SPICE to similate feedback loops and generate Bode plots, the hardest thing about it then is knowing whether your frequency domain model matches the time domain model, since non-linear components such as PWM duty cycle dissolve into the transfer function equations. If you have access to it, you can also use MATLAB to draw a Bode plot from a transfer function but it can be complex and only really worthwhile if a generic transfer function is all you have to work with.
Hope that helps,
Matt