As with RF connectors, there are many different types of RF cable suitable for use in HAB, whether in the payload or at the ground station. Here will discuss the factors that differentiate different types of coax, what factors are most important in a payload or on the ground, and finally which types are best for those applications.
Each type of cable has a Characteristic Impedance, and this is almost always 50 or 75 ohms. Ideally we want transmitters, receivers, aerials, cables and connectors to all have the same characteristic impedance, and the standard for that is 50 ohms.
Don't even think about using TV coax as (a) it's 75 ohms, and (b) it's usually of very poor quality anyway.
Any cable will attenuate the signal passing through it, so what comes out it always less than what went in. This loss can be considerable with long lengths of cable, which is a shame as when receiving you want to get the aerial as high as reasonably possible, possibly resulting in a long cable down to wherever your receiver is sited.
Attenuation is a function of:
Loss is proportional to the cable length, so long cables should be avoided if possible. As an example, a 20 metre length of RG58 cable will attenuate 434MHz by 13dB, which is to say that the signal coming out is at about 1/20th of the level of what went in. In other words, not good.
The cable type also makes a big difference. In general, thicker cables have lower losses though there are other factors. As an example, RG213 cable which is 10.3mm diameter (vs 5mm for RG58) has a loss of just 3.2dB for a length of 20 metres at 434MHz. 3dB is a perfectly acceptable loss but 13dB isn't.
To help decide which cable to use, use a Cable Loss Calculator.
For the payload, lengths are short so losses are tiny and can be ignored. Instead, we want a lightweight and flexible cable for which purpose RG174 is ideal. This can be used to connect GPS and UHF aerials, and can also be used to make the UHF aerial.
Here the priority is to have something flexible but not too lossy, so RG58 (less than 1dB for 3 metres) is a good choice.
Typically, mag-mount aerials come with 3 metres or so of RG174 cable. RG174 is a good choice as it is thin and will easily fit through door or window seals. Remember though that 3 metres of RG174 has about 2dB loss, which is OK, but if your aerial has a longer cable length then consider cutting it short and replacing the plug.
The priority here is to get the aerial up as high as reasonably practical, and then have a low loss down to your receiver. Some options are:
As well as the resulting cable loss, also consider the cost of the installation, and how easy it is to run the cable - some cable types can be very stiff. Also, if you are using an aerial rotator, you will need some flexibility in the cable.
Something else to consider is the option of adding a pre-amplifier to your ground station. This limits use to receiving only, but can allow you to use fairly lossy (and therefore cheaper and easier to run) cable, and even then have an overall gain rather than loss. The amplifier should be fairly close to the aerial, to reduce the effect of any noise it generates, but should still be accessible in case you need to replace it. A good compromise might be to have the amplifier in the loft, with the aerial outside.