Debug A Poor Link

Pardon the lack of organization here. This page will need polishing.

How to characterize a link

The neighbor on the other side of the link should show on the routes nodes list. Run the R R command.
See also G8BPQ R command.

SPHEN:KA4QRT-2} Routes
> 1 K1QSY-2   200  14! 442  236  33% 0 0 19:41  6 200
> 2 WD1QRS-2  200  14! 221   17   7% 0 0 19:41  0 200
> 3 K2CC-2    200  12! 562    5   1% 0 0 19:42  1 200
  4 K7QTH-2   200   0!  55    0   0% 0 0 21:02  0 156

In the example above, port 1 K1QSY-2 has a bad retry rate. Averaged over the last 442 information frames (messages with text) the node has had to transmit 236 extra copies of the information frame in order to complete the message. That has resulted in a 33% figure. We like to keep the percentage number down to about 9% in an acceptable link. 3% is considered excellent. The higher the percentage number, the slower the link will seem.
Also important in K1QSY's line is the 2nd to last figure. The 6 in this column means there are 6 messages waiting to go across this link. We'd like to see that figure at 0, or nearly 0, meaning little or no traffic waiting.

Further down in the R R result, port 4 really has a problem. The column before the ! shows 0 nodes sourced. That means the local node, SPHEN, is not hearing the routing table "nodes" broadcast from K7QTH-2. Note that the percentage rate has been excellent. My guess in looking at this is that K7QTH-2 suffered a catastrophic failure, like somebody turned off the node entirely.

Your next move to solve K7QTH-2 bad link problem would be to see if K7QTH-2 show on the MHEARD 4 list and how long ago K7QGH-2 was heard.
See also G8BPQ M command.

Looking back at K1QSY-2. Connect across the link and see what the R R table on the other end shows.

MIKEY:K1QSY-2} Routes
> 1 KA4QRT-2  200   8! 806    4  2% 0 0 19:41  0 200
> 2 N8DDK-2   200   4!1221  170  7% 0 0 19:41  0 200

This shows us that the failure is directional. Either the transmission from SPHEN:KA4QRT-2 is too weak, has bad audio, or maybe MIKEY:K1QSY-2 has an incorrectly set receive volume level?

Even if the R R table shows a good link to one of your neighbors, you should obtain the performance results from the other end of the link periodically. One way to do this is to connect across to the neighbor and do the R R command from their end. In our current environment we can't tell the entire story about a link without looking at both ends, though if you have a radio and antenna that can hear both ends, you can do a pretty good analysis just listening to a busy channel.

R R on both ends of the link at the same time
Every 15 to 18 minutes, the TARPN software grabs the R R results on your node and, for each link, transmits the value to the neighbor in a [TARPNstat] beacon. Also on each node is a listener for the [TARPNstat] messages. When a [TARPNstat] is received, the node grabs the R R for that link and stores both the local and remote-end figures into a file in the Raspberry PI. As of this writing, that file is /usr/local/etc/tarpn_home_linkquality.dat.
TARPN HOME reads that file and creates a graphical chart for each link presented under the Info tab.
TRR (called from QTterm or Node pane) also reads that file and performs an analysis of that file.
You can also examine the linkquality file using Linux text commands like cat, tail, or a text editor like Geany called from the Raspberry menu on the Raspberry PI desktop. Click on the Raspberry, then down to the Programming menu item, and then Geany. In Geany, use the File menu, pull down to Open, then chose Other Locations, then usr, then local, then etc, then tarpn_home_linkquality.dat.

Required for a good link

20db quieting - the stations must be strong to each other with no buzzing or clicking
transmit modulation just below saturation of the transmitter
transmitted signal must match the bandwidth of the receiver (commercial radio wide vs narrow mode setting can screw this up)
audio must be relatively flat. The high tone can actually be a very different volume than the low town, but only if the other characteristics are spectacular
If you are connecting into the Mic and Speaker connectors on the radio at one end of the link, you should probably be doing the same at the other end. The aggressive audio roll-off on the Mic circuit would mostly be matched by the audio recovery in the speaker circuit of the other end. If using the data connector on one end, make sure the radio on the other end also has a data connector. This isn't a showstopper, but it could make the difference between, for instance, 25 watts on each end vs 100 watts on each end
During any packet transmission, the modem tones must be audible at the receive TNC's modem before the important bits of the packet start. This can be foiled by a slow squelch or too-short TXdelay
Radio should have low audio distortion. Some radios will distort the data enough that 15% is the best rate we'll get with big packets

See also QTterm.
See also TNC and Radio Adjust and Test
See also Configure NinoTNC

Things that can hurt the link

Weak signal
too long a path or too much in the way
Local interference to the receiver on one end or the other
Bad radio on one end or both ends
In-band desense from another local transmitter
multipath
TxDelay is too short
Receive audio level from the radio to the TNC is too high almost non-existant
Transmit level from TNC to radio is too high or too low
RFI (from one of your transmitters) affecting the modem
Receive squelch taking too long to open
Radio shutting off its receiver to save power
Automatic Gain Control messing up the modem (this was a problem affecting DireWolf)
radio off frequency
Wiring error
Not enough voltage during transmit
Radio turned off
Temperature affecting something
Hidden transmitter causing collisions
Hidden transmitter causing your receiver to start receiving a packet before the desired link starts
Some radios will distort the data enough that 15% is the best rate we'll get with big packets

The way packet radio links over FM behave at the receiver is disconcerting in that the digital link can be on the threshold of failure, yet to the node the link appears to behave perfectly. Then when they cross the threshold into failure, the link appears to fail completely, only to recover suddenly shortly after. This can happen on a several‑second cycle; or minutes; or hours; or entire seasons.
To a human brain this is akin to having a great big suspension bridge across a river be there one day, having it missing completely for a couple of hours the next day, then suddenly re-appear after lunch.
Well, we're talking about radio here. If this was voice radio, we'd be able to understand signals that were rough to hear but were still useable. They'd dip into being really rough once in a while. Being gone suddenly and completely for a short period is just not natural.
So.. when debugging a packet radio link, falling back to listening to the link with our ears is a good plan.

Steps to debug a link

Verify the routes and port numbers on each side.
With the receiver driving a speaker on one side, have the other transmit a signal. It should be full quieting. Repeat with the other direction.
Set up a monitor receiver to listen to both ends at the same time. The receive signal from the distant station need not be full quieting to the monitor receiver, you just have to be able to hear the entire packets from the other end.
Verify that the TXDELAY leaves an audible lead-in tone from each station to your monitor receiver. Set the TXDELAY long, just to make sure.
Run BPQterm on each end.
Set the transmit audio on each end such that the audio is not distorted on a receiver at the transmit site. Do this with tarpn linktest <portnum> while listening on a local receiver. To set the transmitter, turn on a receiver on that frequency and set the volume on the HT so you can hear it. Adjust the transmit level across its entire range and verify that you can tell the changes in transmit level on the HT. Set the transmitter TNC so it's transmit level is just below the maximum (from the HT's receive perspective) that you can set it to. There is a point in the transmit audio level control, where the level will top out and everything beyond that is "overdriving" which will just be distorted and useless at the receive end. Set the transmitter audio level control to just below the point where the level tops out.
Send packets from the transmit end, using Linux command tarpn linktest <portnum> . Alternatively you can connect over the radio to the other end and request stat.
If you are using a Mic/Speaker connector radio, you should open up the squelch all the way and then adjust the audio until the left-most light on the NinoTNC (labelled CRC) just barely flickers when no packet is being received.
Observe the TNC's Good Packet light. You'll want to see that light turn on for every packet you hear on the monitor receiver. Run QTtermTCP (g8bpq program installed on the Raspberry PI Desktop) and turn on monitoring. See the page under builders for QTterm for configuration instructions.
Repeat on the other end
No receiver method: Observe the QTterm monitoring the link. On the other end of the link start a transmit cycle connecting to a non-existant callsign, as above, but now observe the timing of the transmits. At the receive end watch for packets which do not show on the QTterm by looking for gaps in the expected traffic. You should be able to listen to the other end transmit and know if there are any changes in the period between packets. Fine tune the receiver's volume knob to optimize until no packets are missed.
Repeat in the other direction
Connect to both ends of the link and grab the current numbers from R R for the link you are working on.
Now connect to the far end, connect back, connect out, back, out, back. Now do a stat command.
Listen to the exchanges and verify that there are no times of dead air until the stat is fully received at your end
Use QTterm's monitor window to verify that every transmit has a response. See whose messages are being sent but not being received at the other end.
Observe the INFO-frames sent value vs the INFO-frames-retried value. Compare against the starting value. You can use these values from both ends and identify which side is having difficulty sending to the other.
There are still several things that can go wrong, including multi-path distortion. The clue that there is multipath is that the received signal from the other end of the link sounds distorted, but when listened to at the transmit end, is not distorted. Multipath can be caused by having a reflection which is as good as the direct path. It can be exacerbated by badly aimed directional antennas, or a major RF attenuator or hill directly in line between the two sites. Horizontal directional antennas are a good thing to combat multipath. If the antennas can't be moved or improved, the best thing to do is to change frequencies or even bands between the sites.
If you do get it to work well, set the TXDELAY down to where it the lead-in is noticeable but not finicky short and not tedious.

---

to be continued