Troubleshooting eComStation - A Guide to the Troubled

No one likes exceptions, but they do happen. The kernel defines two types of exception reports

• Process Exception Reports
• Kernel IPE Exception Reports

Some applications define private exception handlers which replace the default Process Exception report with an application specific report. Examples of application specific exception reports include

• GCC LIBC Exception Reports.
• eWorkPlace Exception Reports.
• Innotek Font Library Exceptions.
• Exceptq Exception Reports.

The format of a process (i.e. ring3) exception report is

01-> 02-25-1999  10:58:35  SYS3175  PID 00b2  TID 0001  Slot 0068
02-> E:\CLASSES\LABS\LAB26\BEDBUG.EXE
03-> c0000005
04-> 1bf94e24
05-> P1=00000001  P2=00000000  P3=XXXXXXXX  P4=XXXXXXXX
06-> EAX=00000000  EBX=00060210  ECX=0002881c  EDX=00060210
07-> ESI=00000001  EDI=00000002
08-> DS=0053  DSACC=d0f3  DSLIM=1fffffff
09-> ES=0053  ESACC=d0f3  ESLIM=1fffffff
10-> FS=150b  FSACC=00f3  FSLIM=00000030
11-> GS=0000  GSACC=****  GSLIM=********
12-> CS:EIP=005b:1bf94e24  CSACC=d0df  CSLIM=1fffffff
13-> SS:ESP=0053:000287ec  SSACC=d0f3  SSLIM=1fffffff
14-> EBP=000287f8  FLG=00012206

15-> DOSCALL1.DLL 0002:00004e24

The format of a Kernel IPE (i.e. ring0) exception report will be

1-> <IPE specific Message>

2->  THE SYSTEM DETECTED AN INTERNAL PROCESSING
     ERROR AT LOCATION ##xxxx:yyyyyyyy - aaaa:bbbb

3->  lllll , ffff
4->  038600d1
5-> INTERNAL REVISION 6 . 307  DATE: 92/03/01

or

1->  TRAP 0002        ERRCD= 0000  ERACC= ****  ERLIM= ********
      EAX= 7d240a58  EBX= ff202fdc  ECX= 00064423  EDX= 00003624
      ESI= fff3272c  EDI= 7d240004  EBP= 00004a44  FLG= 00003202
2->   CS:EIP= 0160 : fff702a6  CSACC= c09d  CSLIM= ffffffff
      SS:ESP= 0030 : 00004a38  SSACC= 1097  SSLIM= 00003fff
3->   DS= 0158  DSACC= c0f3  DSLIM= ffffffff  CR0= fffffffb
4->   ES= 0158  ESACC= c0f3  ESLIM= ffffffff  CR2= 1a060014
      FS= 0000  FSACC= ****  FSLIM= ********
      GS= 0000  GSACC= ****  GSLIM= ********
     THE SYSTEM DETECTED AN INTERNAL PROCESSING
     ERROR AT LOCATION ##0160:fff6453f - 000d:a53f

     60000 , 9084
     038600d1
     INTERNAL REVISION 6 . 307  DATE: 92/03/01

Created when GCC LIBC (i.e. libc063.dll) receives an exception report. The format of GCC LIBC report will be

Killed by SIGILL
pid=0x0061 ppid=0x0024 tid=0x0001 slot=0x009b pri=0x0200 mc=0x0001 N:\MOZILLA-TEST\SEAMONKEY\SEAMONKEY.EXE
LIBC063 ffffffff:ffffffff
cs:eip=005b:20d15cc1 ss:esp=0053:0011eaf8 ebp=0011eb14 ds=0053 es=0053 fs=150b gs=0000 efl=00212206
eax=20d42348 ebx=0011eb50 ecx=20da1618 edx=0011eb50 edi=0000000c esi=20d15cc0
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

• Exception reports are written to the standard error output (i.e. the command line).
• Popuplog.os2 is suppressed.
• Easy to miss report if the application is a GUI application.
• To save output, redirect output to a file.
• Use the standard command line redirection syntax (1>error.log 2>&1) or use Rich Walsh's Run!.

Created when xfldr.dll (i.e eWorkplace or xWorkplace) or some DLL it loads generates an exception.

• The exception reports are written to xwptrap.log or ewptrap.log.
• The exception handler is based on exceptq exception handler code so the report format is similar.
• The report file is written to the %TMP% directory or root of boot drive.

Generated when the Innotek Font library (i.e. ft2lib.dll) generates an exception.

• The report format is similar to the standard process exception report.
• The exception reports are written to ft2excpt.log.
• The report file is written to root of boot drive.
• It usually requires a reboot to restore normal operation.

Generated by applications to use the exceptq exception handler.

• The exceptq handler is based on code orginally developed by IBM and released as Open Source.
• The exception reports are written to files named pppp_t.trp where pppp is the process id and t is the thread ordinal.
• The exception reports are written to the working directory.
• Exceptq is distributed as a shared DLL.
• Exceptq uses standard .sym files to convert addresses to symbolic names.
• Fm/2 supports exceptq exception reporting.
• Some versions of the Mozilla apps support exceptq exception reporting.

Tracing is extremely valuable when you need to analyze application activity over time. There are a number of general purpose tracing tools.

• OS/2 Trace Facility
• OS2TRACE
• PMSPY

• Provides API level tracing.
• Ships with eCS/OS2.
• Implemented by TRACE.EXE and TRACEFMT.EXE.
• Limited output formatting
• Some documented trace points appear to no long operate.
• Two trace points are known to trap the kernel.
• Trace Setup Reference is a cookbook setup guide.
• Trace Control Tool provides a convenient wrapper around the trace command syntax. The tool uses configuration files to provide repeatable trace setups.

• Provides API level tracing.
• Somewhat instrusive because it patches the binaries.
• Excellent output formatting.
• Has a GUI setup wrapper.
• Available from Hobbes. or from Dave's home page.

• Provides PM window message tracing
• Has good filtering options.
• Output typically requires a programmer to interpret.
• Download from OS/2 Site

Logging in typically defined as recording unexpected events. This is a fuzzy defintion because the distinction between log data and trace data if often not black and white.

There are a large number of logging tools. The list includes

• System Error Log
• Mozilla Log Facility
• Innotek Font Library Log Facility
• eWorkplace Unexpected Condition Log
• GCC LIBC logging
• TCP/IP Syslog Facility

• The log entries are essentially a mini-process dumps with application specific data.
• The log data is often of limited usefulness because the captured data is not well documented.
• The logging facility is implemented by kernel APIs.
• Log record formatting is implemented by SYSLOGPM.

• The Mozilla logging facility is comprehensive and configurable.
• It generally requires developer assistance to setup and interpret
• The documentation at Mozilla Logging provides a good overview of what can be done.

• The log facility records detailed font processing activity.
• The log data is written to files named ft2lib_x.log, where x is the library instance.
• The files are written to root of boot drive.
• Logging is controlled by HKEY_LOCAL_MACHINE\Software\InnoTek\InnoTek Font Engine\Logging registry key.
• The logs are recommended for finding broken font files.

• This log facility records unexpected conditions detected by eWorkPlace/XWorkplace.
• The log files are named ewplog.log or xwplog.log.
• The files are written to the %TMP% directory or the root of boot drive.

• Records unexpected condtions by libc (i.e. libc063.dll).
• The log files have names of the form libc_pppp.log, where pppp is the process id.
• The files are written to working directory.

• The syslog facility implements a standard TCP/IP service on well-known UDP port 514.
• IBM includes a syslogd implementation in \tcpip\bin.
• The IBM syslogd is configured with the syslog.cnf files stored in the %ETC% directory.
• A third party implementation known as syslogd v3a, available from hobbes, is more feature complete than the IBM implementation.
• Syslogd v3a is configured with the syslog.conf file stored in the %ETC% directory.

The process dump facility generates dump files in response to otherwise unhandled process exceptions or on demand.

• The facility is configured by DUMPPROCESS statment or PDUMPUSR command.
• The dump files are written to the configured directory.
• The dump files are formatted by Process Dump Formatter (pmdf).
• The Process Dump Facility is documented in x:\OS2\SYSTEM\RAS\PROCDUMP.DOC.
• The Process Dump Reference is a somewhat dated cookbook setup guide.
• The Process Dump Control script implements a convenient wrapper to setup and capture process dumps.

The trap dump facility generates dump files in response to otherwise unhandled kernel exceptions or on demand.

• The facility is configured by the TRAPDUMP and PDUMPSYS commands.
• System dump files are written to named volume and destroy all content.
• System dump files are formatted by the Process Dump Formatter (pmdf).
• The Trap Dump Facility is documented in x:\OS2\SYSTEM\RAS\PROCDUMP.DOC
• Trap Dump Reference is a somewhat dated cookbook setup guide.
• Dump Trap Screen extracts the trap screen data from a dump file. The utility is designed for systems that do not have a trap dump volume, but that do have a diskette drive.

• Pmdf formats process dump files, system dump filess and system log facility dump files.
• Pmdf is a generic tool which is configured to match the system that generated the dump file.
• Pmdf is configured with a combination of executable files (i.e. df_ref.exe), symbol (.sym) files and structure definition (.sdf) files.
• The configuration files are built for a specific version of an executable or kernel. However, some files will work for multiple versions of a given excutable.
• For 14.104a kernels, use 14.105 df_ret.exe.
• The pmdfvers.lst file defines the directories that contain the configuration files for a specific system.
• Pmdfvers.lst entries are indexed by the kernel internal version id which is similar to the bldlevel string.
• Kernel Debugger Reference is a cheatsheet reference to the kernel debugger and the Process Dump Formatter.

The system analysis tools analyze and report data that is defined within the kernel

• Theseus implements comprehensive tools for memory and process analysis.
• Pstat implmenents a simple listing of loaded processes and DLLs.
• Psfiles lists all open files.
• Pssem lists all semaphores.
• Top displays process status in various formats.
• Cadh includes a process process status listing amoung its features.

Debuggers allow the operation of the kernel or an application to be controlled and examined in very fine detail.

• Kernel debugger
• OpenWatcom debuggers (wdw and wd).
• VAC debuggers (icsdebug and idebug).
• ICAT debugger

Disassemblers are useful for understanding programs that do not have available source code.

• ida
• Theseus
• pmdf

• Everyone should be aware of the existance of "The OS/2 Debugging Handbook, #0.7a (sg244640.inf)." It's included in the OS/2 Warp Toolkit on your eCS CD.
• OS/2 Diagnostic Tools is the home for many of the tools and documents discussed here.
• Join the developer community.