Question / Problem
The customer informs you, that he has increased amounts of FAX transmission- or FAX reception errors with the KCS Line server.
This article should help you with:
- Performing the necessary steps to find out details about the problems.
- Verifying if the problems are related to the customer’s environment (e.g. an ISDN line with a high Error rate) – and if this is the case - provide suggestions to the customer to detect and solve these problems.
- Setting up proper tracing and get an idea on how to check the resulting traces.
Note: This document assumes that the connection between the KCS Line server and customer’s (ISDN) environment is principally working (sending and receiving is possible). It does not cover problem situations where the connection is not working at all caused by configuration mismatches between customer’s environment and the KCS Line server.
Step 1: Get the necessary Tools for analyzing FAX line quality problems:
- A text editor, which is capable to list all lines matching a specific search string (Windows Notepad can only navigate to the next search result). If possible, the text editor should accept also regular expression search terms. Additionally the text editor should be able to handle multiple files and search also in multiple Files (if possible also using regular expressions). We recommend to use ‘UltraEdit’ (a commercial product, see http://www.ultraedit.com or – if you prefer a free product use for example PSPad (http://www.pspad.com/en ))
- ‘Microsoft Excel’ to analyze/filter outbox exports done with the KCS Communication server client (TCFW)
- The Binary Tracer tool BTrc.exe as shipped with KCS 9.0 (TC/SP 7.88.04) or any later version of BTrc.exe. This tool is necessary to convert TCOSS binary trace files (.btr). Previous versions of BTrc.exe do not have the capability to extract the FAX T.30 protocol data from the binary trace files. You can use the BTrc.exe of KCS 9.0 to convert binary trace files, which were captured with an older KCS TCOSS version.
The BTrc.exe Tool can be found within the folder C:\TOPCALL\LSD, if you install the KCS TC/SP Setup option “Administrative Tools (Group) – Line server diagnostics tool”. We recommend that you apply a file association for .btr files in Windows Explorer, to convert these .btr files, if they are double-clicked.
- A visual audio file Editor to view the wave output of Line server binary trace files. We recommend to use the free open source Tool “Audacity” http://audacity.sourceforge.net
Step 2: Check KCS TCFW Communication server Client Outbox and the TCOSS Journals
To get an impression about the problems you should first check the KCS outbox of all users. Apply filter settings to also include the Send attempts. Export the outbox to CSV format (using menu option “Folder – Export fields”) and import it into Excel to have better Filter possibilities: Check especially the columns “Error”, “Response”, “Reception Error”, “To:”, “Local address”, “Channel number” and “Pages”.
After this step you should be able to answer the following questions:
- Do they have transmission errors or reception errors or both?
- Which types of errors are reported (XT, XQ, XO, XY, or others)?
- Which error rate (amount of errors compared to the total number of send orders) do you have for each error code?
- Are the problems related to specific counter parties (senders or recipients), for example – do you face a higher error rate for send orders addressed to countries in the Far East?
- Are the problems related to a specific (ISDN) line or group of lines (check the used Channels)?
- Do they mainly have problems for large documents (check the file size and the page counter)?
In addition to the KCS outbox, the KCS Journals can help you to identify sending or reception errors:
Journals should be configured to write at least 3 line journal entries (Configuration line 38 of the FAX channels set to ‘0A 03’), they are located in the +MAIL5V folder (KCS System folder) and are typically named as follows:
AJyymmdd: outbound journal for one specific day, e.g. AJ100917 is the outbound journal of 17th Sept, 2010
AIyymmdd: inbound journals for one specific day, e.g. AI100824 is the inbound journal of 24th August, 2010
Step 3: Check the Windows Application Event log of the KCS TCOSS Server
- Are there any event log entries in the Windows application event log indicating communication problems between TCOSS and the line servers? For example Link errors (e.g. EventID 16020, Timer resync error errors), latency issues?
- Are there any event log entries indicating ISDN clock synchronization issues?
Step 4: Get details about the customer’s environment
- Which call route is taken exactly for those messages showing the problems?
- Is a PABX involved, which supplier and which model is used?
- Which trunks are used on the PABX, is there any relation to the problems?
- Is the ISDN clock of these trunks synchronized with the public telephone network (PSTN)?
- Are there any VoIP components (VoIP Gateways) involved before the message goes to the public telephone network? VoIP Gateways may alter the FAX signal!
- Any WAN connections or PABX inter-connections?
- Are there any Voice specific features used in this routing path, e.g. Echo cancellation, Voice compression methods, Voice activity detection?
- Did they change anything in this environment or in the method how the messages are routed in the near past?
Step 5: Setup proper Traces for the KCS TCOSS Server
Setting up proper traces can be quite difficult:
- If the trace levels are set too high, it is very likely that the traces are already overwritten before the customer even realizes that there’s a problem or before you have the possibility to check the available trace information.
- Setting up e.g. binary traces for all channels of a primary rate line might also cause performance issues, which means that binary traces might drop trace information in order to not cause timing problems with the FAX protocol.
- On the other hand, if the trace levels are too low, you do not have enough information to find out the reason for the problems.
When setting up traces, check the available disk space and increase the number and the size of Trace Files depending on the available disk space using registry values listed below. Good values (if they fit into the available disk space) are:
KCS TCOSS provides different types of Traces described below: Enabling and configuring these traces is described in the manuals and in a separate Solution: 00000048 “Tracing FAX communication problems”
- The TCOSS Trace
provides general information and trace lines for LS1 Link connection- or latency problems. But, because of the fact that Link errors will also interrupt the FAX transmission, it is recommended to enable this trace by setting the registry value HKLM\SOFTWARE\TOPCALL\TCOSS\TraceLevel=0x1083 hex
- The ISDN Trace
doesn’t provide trace information for the FAX protocol itself. But it contains ISDN CallerID and CalledID and can be therefore used to detect calls from/to specific problem numbers within the trace files.
- The Modem Trace
tracks the entire T.30 Fax protocol. If possible, it should be enabled for checking line quality problems. If the amount of trace information is an issue (too much traffic, too many channels), it is better to turn off the Modem trace rather than turning off the binary trace.
- The Binary Trace
should be enabled for checking line quality problems. It should at least be enabled for specific originators (see configuration details below). It is also recommended to increase the number of Binary Trace Files within the registry (default value: 50):
By default it is recommended to use a the Trace file size of 50000 kB for the binary traces as explained in the section above. But if you want to take binary Traces for long FAX transmissions (> 10 pages) it is necessary to increase also the maximum File size for the binary trace files within the registry value: HKLM\SOFTWARE\TOPCALL\TCOSS\BinaryTrace\MaxTraceFileSize>=100000 (50 page documents)
- FAX Line quality trace:
This trace provides less information than the modem trace, but it can be activated for many channels.
You will see specific trace lines if there are failure responses (FTT, PPR) or retransmits initiated in the T.30 Fax protocol.
The FAX Line quality trace is enabled by setting additionally the flag ‘0x80 hex’ for the TraceFlags of a specific Node, so in combination with ISDN, binary and modem traces you have to set the registry value
- BER Test (Bit Error Rate Test):
A BER Test requires an additional ISDN Testing device and a specific setup in the number conversion table of the FAX channel configuration. See Solution 00000048 “Tracing FAX communication problems” and 00000047 “How to interpret BER (bit error rate) test results?” for details. In many cases BER tests cannot be used because:
- The connection from the testing device to the customer is not entirely digital end-to-end.
- The customer uses methods for echo cancellation.
- The call route contains also VoIP components (gateway), which convert the signals.
You can easily recognize whether the BER test works or not. If the BER Test is setup correctly in the FAX channel configuration, you will hear the confirmation tone when calling the DDI extension configured for the BER test. If you then enable the BER Test on your ISDN Testing device and you don’t receive any echo packets (Rx counter does not increase), the BER test doesn’t work in this environment.
Step 6: Try to reproduce the problems with a well known and working counter party
The best approach is to check if the problems also occur when ‘sending to / receiving from’ a well-known and working environment (e.g. the Line server in your office):
- Enable maximum Traces (including ISDN, Modem and Binary traces) on both sides (in your office and on customer side). If the traces are enabled on both sides you can compare the FAX signal on the sending and receiving side. By doing so you can detect, if any components in the call route/communication path (e.g. VoIP Gateways) change the signal (signal level, timing, sequence of signals,…)
- Use complex documents for testing. The chance to detect problems is much higher if the amount of FAX data transferred per page is large. We recommend using at least 5 page documents and the page data should be the CCITT Test image (or an image with equal complexity) on each page. Documents containing complex data (many black/white changes) on one page are better than documents containing many pages with simple content.
Testing with such documents will also help in detecting reference clock issues, which are not directly on the ISDN connection between LS1 and PABX/PSTN (such issues can be see within the TCOSS Trace), but anywhere in the call route in-between, e.g. on a Voice gateway.
- By default use ECM for sending the test documents.
- Do several tests to get a good statistical summary:
One successful test does not verify sufficiently that the line is really working reliably.
- Do parallel tests using several lines. In some cases (e.g. if VoIP gateways are involved) the problems start if several channels/lines are active at the same time.
If you use the feature “Locking of send commands to equal numbers” (Configuration line 11 of the FAX channel = 1), use different receiver numbers for testing.
- It is not sufficient to check whether the test documents are transferred correctly (no error reported in the Outbox). You have to check the traces to find out, whether the transmission was error free. Details on checking the traces are explained in steps 7 and 8.
- Use different times during a day for testing:
In many cases transmission/reception errors are reported during peak hours. These are not necessarily the peak hours of the customer, if the counter party is located in a different time zone.
- If you do not have a proper environment in your office, you can contact BC technical support in Vienna. They can provide you with FAX numbers for their test environments, which can be used for testing. And they will help you with matching the traces captured in their environment with the traces taken at the customer site.
- If you are not sure whether your FAX channel configuration could cause the problems, check the situation also with the standard configuration (especially regarding country specific settings). This is especially important for analogous lines (TC26 line interface).
- If you encounter a problem and you pass over the traces to the next level of support, please verify the following things:
Step 6.1: Adopt trace settings if the problem is not reproducible with the well-known counter party:
If you cannot reproduce the problem in the previous step, it is necessary to adapt the trace settings:
- For the outbound direction, setup the binary and modem traces only for specific channels, reroute the troublesome send orders to the specified tracer channels using appropriate rr99 routing entries.
- In the inbound direction you can setup binary traces for specific originator numbers (ISDN caller IDs) or recipient numbers (DDIs). This is done by using specific conversion rules in the number conversion table of the FAX channel configuration. See chapter “6.3.5 Binary Trace for Specific Originators” within the TCOSS Configuration manual (tcoss_cnf_78814_Ed02.pdf, page 231)
When using this option you must turn off the unconditional binary Traces, that means Configuration Line 242, positions 5 and 6 must be set to ‘00 00’, but still the flag (0x4) for enabling the binary traces within the registry value “TraceFlags\Node_xxx” must be enabled
- If the inbound problems cannot be related to specific senders and the problems can be seen on all channels, it is better to turn off the Modem Trace and to turn on the “FAX Line quality trace” instead. This is done with the flag ‘0x80 hex’ in the registry value “TraceFlags\Node_xxx”, so you should configure there a value of 0x85 hex (with binary Trace, without ISDN Trace) or 0x95 hex (with binary and ISDN Trace).
Step 7: Analyzing binary traces
Binary Traces are stored with the filenames TCOSS_xxx.btr, xxx is an increasing number. You can convert them using the BTrc.exe tool described in Step 1 (Btrc.exe must be called providing the filename of the .btr file as command line parameter). After the conversion you will get a TCOSS_xxx.asc and TCOSS_xxx.wav file in the same directory, xxx is again the same number as for the original .btr file.
Every time TCOSS creates or finishes a binary trace file you will find the following trace entry within the normal TCOSS Trace files, you can easily search for such trace entries using the search term “btr”:
- BTR Open File C:\Tcoss\Trace\TCOSS_xxx.btr for Channel yy: (yy)
- BTR Closes File C:\Tcoss\Trace\TCOSS_xxx.btr for Channel yy: (yy)
The TCOSS_xxx.asc file
can be checked with a normal text editor.
- In the first line of the asc file you will see the TCOSS channel, which was used to capture the BTR file.
- In each line of the .ASC file you will find a relative time stamp (starts with 0 when the file is created). This time stamps are helpful when matching T.30 commands reported in the ASC file with the time axis shown in the Wave Editor.
- After the time stamp you see Rx or Tx indicating whether the command/signal was received (Rx) or Sent (Tx)
- Filtering out T.30 FAX protocol:
If you enter the search string “t30” in your text editor you will get the entire T.30 FAX protocol communication, similar to the modem trace. A list of T.30 commands can be found within the chapter “10.2.2 Index of Used HDLC Abbreviations Used in Recommendation T.30” of the TCOSS ISDN Manual
Especially interesting are the T.30 abbreviations: “PPR”, “FTT”, “RTN” and “RTP”.
These T.30 signals indicate Errors in the transmitted Fax data.
The receiver side recognizes the problem and responds with one of these failure responses:
“FTT – failure to train”: The fax “training” (= learn phase) was not received correctly. The transmission speed must be reduced and the training must be repeated. There is a maximum of 3 retries.
“PPR – partial page request”: This is a response code when using Error correction mode (ECM). The checksum of the received ECM data was not correct. The PPR signal requests, that these data are retransmitted.
“RTN – retrain negative” and “RTP – retrain positive”: The received FAX page contained too many bad pixel lines. With RTP an additional training is done (reduced speed), with RTN the Fax transmission is aborted.
- Detecting signal interruptions:
It is also possible that the FAX communication fails caused by short interruptions of the signal. You can find such cases when searching for “NOCARRIER”. The trace entry “NOCARRIER” does not necessarily indicate that there was a FAX problem, but when it occurs while sending/receiving data, it will cause problems.
- Detecting dropped BTR elements:
If you enable the binary trace on many channels of the same line server (e.g. on all channels of a PRI line) you might face the effect that binary trace elements are dropped. Dropping the trace information is done if the binary trace information cannot be written “in time”. By dropping the trace information, TCOSS ensures that the FAX protocol does not fail due to timeouts. You will hear such dropped signals also in the Wave file, but there you will not know whether the interruptions are caused by dropped BTR elements or really by dropouts of the line signal. The ASC file on the other hand clearly marks dropped elements with the trace
xx btr element(s) dropped, (yyy ..zzz)
Simply search for “dropped” within the ASC file when you hear signal interruptions in the Wave. If you do not find the “dropped” trace lines, the interruptions in the wave file are real signal interruptions.
The TCOSS_xxx.wav file
should be checked with a visual Wave Editor like Audacity mentioned in step 1.
If you use an ordinary media player (like Windows media player) to listen to that audio signal, it is much more difficult to detect problems:
- The “Left” channel “L” (upper part) will show the received signals,
the “Right” channel “R” (lower part) will show the transmitted signals (from the Line server’s point of view)
- You can see the signal strength (y-axis) and compare it for good and bad cases.
- If you want to listen to one specific signal (e.g. only the received signal) drag the slider on the left side of the signal view to either “L” or “R”
- Except for V.34, the FAX protocol is a half duplex protocol, i.e. at any time either sender or receiver has a signal, the other party is silent.
- A normal FAX communication is a request/response protocol. For example – after a transmitted FAX page you should immediately see the page confirmation from the receiver side. Long delays indicate a problem (e.g. VoIP gateways with latency issues on the network will delay the FAX protocol)
- Visible peaks or short interruptions indicate line quality problems.
- If you have binary traces from the same call (taken at the customer site and taken in your office), you can load both wave files into the same instance of Audacity. By doing so you can compare the signal levels on sender and receiver side and you can detect, whether other components (VoIP gateways) alter the signal in any way:
Examples for Binary traces (Wave output):
In the example shown below you see a successful 5 page fax transmission (Page 1 is short, pages 2..4 are complex)
Track 1, L-Signal (upper part): Receiver side, received signal
Track 1, R-Signal (lower part): Receiver side, sent signal
Track 2, L-Signal (upper part): Sender side, received signal
Track 2, R-Signal (lower part): Sender side, sent signal
The next example shows a failed transmission (XO Error = 3 learn attempts unsuccessful). In this case the problems were caused by a faulty VoIP Gateway, which was placed somewhere in the communication path. The FAX signal was modified and completely destroyed by the VoIP Gateway: The meaning of the tracks is the same as for example 1.
Matching the sender side trace and the receiver side trace was only possible by using specific test cycles (in a test cycle with a total amount of 3 messages, the 2nd binary trace taken on the sender side was compared with the 2nd binary trace taken on the receiver side).
The next example shows a situation with a short (70 msec) signal interruption marked out dark grey. The receiver side (LS1) interprets the interrupt as “End of FAX data” and expects a FAX command (MPS or EOP) to be received after the page data. As this is not received it sends a RNR (Receiver not ready) signal and the communication is aborted:
- Summary of search terms for the ASC file: t30, ppr, ftt, rtn, rtp, NOCARRIER, dropped
Step 8: Analyzing TCOSS Trace files:
TCOSS Trace files are stored in ‘C:\TCOSS\Trace’ with the Filename TCOSSxx.trc (xx = numeric). Depending on the amount of activated Traces you can search for different search terms. If you are interested in getting a combined list of different trace strings, it is recommended to use a text editor in which you can enter search terms with regular expressions, for example the following search term will list you all binary trace lines, modem traces and messages which were closed with an error:
When using regular expressions you must also “escape” some special characters, for example the brackets, which is done with a leading backslash character.
For example, the following regular expression will give you all ISDN Trace related trace lines for channel 10:
<< .*\(10\)|\(10\) >>
Take care that you have to enable a checkbox “Regular Expression” in the search window, if you want to use this functionality.
Details on how to use regular expressions can be found on the internet, e.g. under: http://en.wikipedia.org/wiki/Regular_expressions
The following sections gives you a list of search strings which can be used to search for problems in the TCOSS traces, assuming that the general ‘TCOSS\TraceLevel’ is set to ‘0x1083 hex’.
- “ id:”
Please note the leading blank. You will find those trace entries in the general TCOSS Trace which also have an Event log entry, for example Link errors to the line servers. After the ID: you will see the Event-ID.
- “err no.=” or “err no.=[1-9]”
This will give you TAMIO trace entries, which are written when a FAX send order is completed. A value of 0 indicates a successful send order, values from 1-9 indicate an error Xx. The 2nd search method requires regular expressions. See also solution ‘00000050 - TAMIO Fax Error Codes’.
- “:MOD” or “xx:MOD”
This will filter out the modem trace, if you specify additionally a channel number (xx), you will get only the modem traces for this channel. Of course only if modem traces are enabled.
Indicates that a binary Trace file was opened or closed.
- “PPR” or “FTT” or “RTN” or “RTP”
These search strings will detect T.30 Fax protocol failure responses. You should not find any PPR’s within your TCOSS trace when you do the tests with the 5 page long test documents with your customer. You can search for those strings, when either Modem trace or FAX line quality trace is enabled.
- “slip” or “clock”
Use these search terms when checking for ISDN reference clock issues. Take care that you will only see clock issues which are detected directly on the ISDN line where LS1 is connected to. Trace entries like
BackGroundThread Slip Error Counter=0 0 0 0
indicate no problem. Reference clock problems will destroy the fax signal and are also common reasons for FAX problems.
- “=ERROR 2,”
Trace entries like .211=ERROR 2,27,410603FF indicate, that FAX data were received with CRC errors. Such trace entries, if shown rather often, indicate quality problems with the line.
- “FaxSend” “FaxReceive” “RxInfo”
These are search terms, which can be used if the FAX Line Quality trace is enabled, they will show you if PPRs are received or if for example a received FAX page contains bad pixel lines (value after Info=) or Fax reception was interrupted due to a loss of signal (Carrier lost):
RxInfo: Channel 12 received 227 lines, Info=0 0 9600 EQ=0 -7dBm Carrier lost
RxInfo: Channel 12 received 2316 lines, Info=1 1 9600 EQ=0 -7dBm
FaxSend PPR message has been received
FaxSend FTT message has been received
These are trace lines (part of the modem trace) indicating the eye quality, which is a value that indicates how good the received signal is. You will get it only for inbound messages. Interesting is the first value after “peye=”. A value of 0 is good, the higher the value is, the worse the received signal is:
N3/T7D .000=31:MOD rx_tcs *peye=0(83), res=1
N3/T81 .000=33:MOD rx_tcs *peye=6(440), res=1
These trace entries indicate a performance problem on the TCOSS system. The FAX channel TUM (user module) had to wait for the TAM (application module) to respond. In most cases it indicates a performance problem with the hard disk or with the dedicated LAN in a Tandem system. TAWIS delays can cause FAX problems if the delay is too long.
WARNING: TAWIS delay 4422 ms on TAM channel 30, cmd type 1
Step 9: Final conclusions:
In most cases FAX reception or FAX sending problems are caused by bad lines or other components located anywhere in the communication path of the message (VoIP Gateways, WAN connections,…). Very rarely such problems are caused by configuration problems of the TCOSS Fax channels (a configuration issue normally influences all send orders not only specific recipients).
To verify whether the problem is located in the environment of your customer or in the environment of their counter parties (senders/recipients) you can run tests from your customer with a well-known, well-working counter party (e.g. the Line server in your office). Use multi-page (>= 5 pages), complex documents, send with Error correction mode (ECM) and check, if the traces contain PPRs (partial page requests). Check both directions (sending and receiving); they might take different call routes. Use also different times (during a day) for testing and also test parallel send orders.
If the conclusion of those tests is, that the problem is in the environment of your customer, you can help him to detect and solve the problems using the questions listed in Step 4. PABX logs and Bit Error rate tests are also good tools to detect line quality problems.
Additionally, line quality problems at customer site can be caused by:
- Damaged hardware (LS1).
- Faulty cabling/connectors or cables exceed the max. allowed length.
- Missing termination resistors (ISDN).
If the conclusion of these tests is that the problem lies in the environment of their counter party, you cannot solve the problem. In the outbound direction you can try to improve the situation by reducing the transmission speed for those specific recipients causing problems. Other modulations (V.29, V.27) might also help. Error correction mode (ECM) should not be turned off. Reducing the transmission speed can be done using rr99 routing entries like shown below. The examples change the transmission speed for the FAX number 1234567.
Check different speed settings to find out the best result:
F:1234567~,F:E361234567~, use V.29 9600 Bd or
F:1234567~,F:E261234567~, use V.29 7200 Bd or
F:1234567~,F:E161234567~, use V.27 4800 Bd
It might also help to test with different Higher Layer/Lower Layer capability settings (Configuration line 251 of the ISDN FAX channel) if the problems are caused by Voice Gateways doing Voice compression, but it will only change the situation if the gateway really checks the ISDN HLC before applying the voice compression methods (unlikely).
In some cases it might help to improve the situation by doing some specific fine-tuning in the T.30 Fax protocol, for example by changing different timers. But this cannot be done in the field, it requires investigation and configuration- or code changes in R&D.
- Article “Tracing FAX communication problems”
- Article “TAMIO Fax Error Codes”
- Article “How to interpret BER (bit error rate) test results?”
- Is the problem situation really part of the trace
- Are the provided traces consistent, that means:
Are the provided binary traces really those files being referenced in the TCOSS trace?
- When exactly did you have the problem (time stamp)?
- On which channel?
- Sender and/or recipient numbers involved in this call.
- Error codes reported
- If you did several tests using different configurations, mark out clearly the used configuration for a specific trace.
- Include all used configurations/registry settings and other relevant information about the installation (operating system, virtual of physical…)
Keywords: T.30, Audacity, regular Expressions, Journals, XO, XQ, XT, XN, XS, clock synchronization, Trunk, Gateway, Echo cancellation, Voice compression methods, Voice activity detection, Binary Trace, Fax Line quality trace, Modem Trace, BER-Test, ECM (Error correction mode), CCITT, ITU,