PBS Professional 11.2 Release Notes      21 December 2011

Copyright (c) 2003-2011 Altair Engineering, Inc.  All rights reserved.

PBS(TM), PBS Works(TM), PBS GridWorks(R), PBS Professional(R), PBS 
Analytics(TM), PBS Catalyst(TM), e-Compute(TM), and e-Render(TM) are 
trademarks of Altair Engineering, Inc. and are protected under 
U.S. and international laws and treaties. All other marks are the 
property of their respective owners. 

ALTAIR ENGINEERING INC. Proprietary and Confidential. Contains Trade 
Secret Information. Not for use or disclosure outside ALTAIR and its 
licensed clients. Information contained herein shall not be 
decompiled, disassembled, duplicated or disclosed in whole or in part 
for any purpose. Usage of the software is only as explicitly permitted 
in the end user software license agreement. 

Copyright notice does not imply publication. 

For the most recent information, log in to the PBS Professional
website, 
	 www.pbsworks.com 
with your site ID and password and follow the
"Client Login" link, then the "Download Software" link.
-------------------------------------------------------------------------
Contents

	Contact Information
	Licensing Information
	Supported Platforms
	Upgrading
	Special Notes (Security)
	Special Notes (Licensing)
	Special Notes (Installation)
	Special Notes (General)
	Special Notes (External Interface Changes)
	Special Notes (Documentation)
	Special Notes (Hooks)
	Special Notes (HPC Basic Profile)
	Special Notes (Windows)
	Special Notes (SGI)
	Special Notes (Cray)
	Special Notes (MPI)
	Special Notes (AIX)
	Special Notes (Linux)
	New Features
	Modifications and Bug Fixes
	Known Bugs, Errors, and Warnings
	Third Party Legal Notices

-------------------------------------------------------------------------
Contact Information

        Altair Engineering, Inc.
        1820 E. Big Beaver Road
        Troy, MI 48083-2031 USA

	www.pbsworks.com

	PBS Sales:        
        pbssales@altair.com
        248.614.2400	

	PBS Support:
	pbssupport@altair.com 
	248.614.2425

-------------------------------------------------------------------------
Licensing Information

Terms of use for this software are available online at 
      http://www.pbspro.com/UserArea/agreement.html, 
and are also included in the PBS Professional Administrator's Guide
and the PBS Professional User's Guide.

PBS uses a new Altair license server for licensing.  The Altair
license server must be installed and configured before PBS is
installed and configured for permanent use.  See the PBS Professional
Installation & Upgrade Guide and the Altair License Management System
Installation and Operations Guide for additional information.

-------------------------------------------------------------------------
Supported Platforms 

For the following platforms, you can download all PBS components:

        HP-UX 11.23 and later on Itanium

        IBM AIX 5.3 with TL9 or later on POWER architectures
	IBM AIX 6.x and 7.1 on POWER architectures

        Red Hat Enterprise Linux 4 (AS, ES, WS) 
		    on x86, x86_64, and ia64
        Red Hat Enterprise Linux 5 
		    on x86, x86_64, and ia64
	Red Hat	Enterprise Linux 6 on x86 and x86_64

        CentOS 5.4, 5.5, 5.6, and 6.0 on x86 and x86_64

        SGI Altix with SGI ProPack 5, 6, and 7 on ia64
        SGI ICE/XE with SGI ProPack 5 and 6 on x86_64
	SGI ICE/XE/UV with SGI Performance Suite 1 on x86_64

        Solaris 10 on SPARC and x86_64

        SuSE SLES 9, 10, 11 on x86, x86_64, and ia64

	Windows 7 on x86 and x86_64
	Windows XP Professional, SP1 and later, on x86 and x86_64
	Windows Server 2003 on x86 and x86_64
	Windows Server 2008 on x86 and x86_64
	Windows Server 2008 R2 on x86_64
	Windows Vista on x86 and x86_64


	Cray Linux Environment (CLE) 2.2 on XT4 and XT5
	Cray Linux Environment (CLE) 3.0 on XT6
	Cray Linux Environment (CLE) 3.1 on XT5, XT6, XE5, and XE6
	Cray Linux Environment (CLE) 4.0 on XE6

For the following platforms, you can download only the license server:

        OS X 10.5/10.6 on x86 and x86_64



MPI
PBS runs jobs with all known implementations of MPI.  PBS is integrated
more tightly with some versions, providing support for resource tracking
and job cleanup after a failure.  These are listed here:

    MPICH 1.2.5, 1.2.6, and 1.2.7 on Linux
    MPICH2 1.0.3, 1.0.5, 1.0.7 on Linux
    MPICH-GM (mpich-1.2.6.14b) on Linux
    IBM POE on AIX 5.x and 6.x, including HPS support
    Intel MPI 2.0.022, 3, and 4 on Linux
    HP MPI 1.08.03 on HP-UX 11 on Itanium 2
    HP MPI 2.0.0 on Linux 
    Platform MPI 8.0 on Linux
    LAM/MPI 6.5.9, 7.0.6, 7.1.1 on Linux 
    SGI MPI (MPT) on Linux on SGI platforms, including over InfiniBand
    MVAPICH 1.2.7 and 2.0 on Linux



Support for CSA:
Support for CSA is deprecated.  The following platforms supported CSA:
    ProPack version  OS version  Architecture libcsa version libjob version
    -----------------------------------------------------------------------
    PP6		     SLES10	 x86_64	      libcsa.so.3    libjob.so.2
    PP6		     SLES10	 ia64	      libcsa.so.3    libjob.so.2
    PP6		     SLES11	 ia64	      libcsa.so.4    libjob.so.2
    PP6		     SLES11	 x86_64	      libcsa.so.4    libjob.so.2
    PP7		     SLES11	 ia64	      libcsa.so.4    libjob.so.2
    (none)	     SLES10	 x86_64	      libcsa.so.1    libjob.so
    (none)	     SLES11	 x86_64	      libcsa.so.1    libjob.so
Any system running PBS and Performance Suite will not support CSA.


Support for User Space mode on InfiniBand switches:
    Platform:             AIX 5.3 on Power6
    Parallel Environment: 4.3.2.5
    libnrt:               2.4.5.5
    Switches:		  Qlogic InfiniBand SilverStorm 9120
    Connection to switch: Compute nodes connect to each Qlogic switch 
			  with 2 networks per HCA adapter.  There are 
			  4 links per switch and a total of 8 links per 
			  node.

Windows Domains:
    Active Directory Service domains are supported.
    Windows NT domains are not supported. 
    Mixed mode is not supported.

    Mixing Windows systems with non-Windows systems in one PBS
    complex is not supported.  Peer scheduling between Windows and non-Windows
    complexes is not supported.

    For more specific Windows platform information, see 
    Special Notes (Windows).  

Licensing:
    For a list of supported platforms for the license manager, see the
    license manager release notes.  These are available from the PBS
    Works Documentation Download Area, at
    https://secure.altair.com/UserArea/docs.php


-------------------------------------------------------------------------
Upgrading

1. PBS 11.0 and later uses a new Altair license server, which must be installed and
   configured before PBS is upgraded.  See the PBS Professional
   Installation & Upgrade Guide and the Altair License Management System
   Installation and Operations Guide.

2. Upgrading a Cray to PBS 11.0 and later requires a migration upgrade.  See the
   PBS Professional Installation & Upgrade Guide.

3. No rolling upgrades:
   You should upgrade all MoMs and the server/scheduler at the same
   time; an older MoM cannot interoperate with a newer server.  For
   example, an 11.2 server cannot send jobs to a 10.4 or even an 11.1
   MoM; the older MoMs will reject the jobs.

4. To choose between overlay and migration when upgrading 
   to PBS Professional 11.0 and later:

   If you are running PBS version 5.3 or later, an overlay upgrade
   is recommended.

   If you are running a PBS version older than 5.3, a migration upgrade
   is recommended.

   PBS on Windows can be upgraded via migration only.


-------------------------------------------------------------------------
Special Notes (Licensing)

PBS uses a new Altair license server, which MUST be installed and
configured before PBS is installed and configured.  You CANNOT use a
FLEX server with PBS 11.0 or later. Follow the instructions in the Altair
License Management System Installation and Operations Guide.

Chapter 3, Licensing, in the PBS Professional Installation & Upgrade
Guide describes how to configure PBS Professional to use the license
server.

If you find that the PBS server host becomes unresponsive a short time
after startup, the server may be trying to contact the wrong license
server.  See the Troubleshooting chapter in the PBS Professional
Administrator's Guide.

The pbs_license_file_location server attribute is deprecated.  The
installation script unsets this attribute during an upgrade.  This is
intentional.  Use instead the pbs_license_info server attribute.

The minimum number of PBS Works licenses that can be checked out is
one (1).  If the pbs_license_min server attribute is set to zero or
unset, PBS automatically sets the number to one.

For version 11.2, the default value for the pbs_license_linger_time
server attribute is set to 1 year (31536000 seconds). The previous
default value was 3600 seconds.  This means that any unused licenses
are checked back in to the license server after a year, instead of
one hour.  This value can be overridden by setting the value of
pbs_license_linger_time via qmgr.

-------------------------------------------------------------------------
Special Notes (Installation)

Under UNIX/Linux, we recommend that the PBS administrator creates a
new data service account called "pbsdata" before installing PBS.  This
account must be created correctly.  See the PBS Professional
Installation & Upgrade Guide for details.

-------------------------------------------------------------------------
Special Notes (General)

0. Do not perform partial removal of custom resources.  It is 
   important to remove custom resources correctly.  See the 
   instructions in the PBS Professional Administrator's Guide.

1. Dependencies on Linux shared object libraries:

   Altair distributes separate PBS packages for different supported
   Linux distributions. Customers running on SLES and RHEL
   distributions should take care to use the correct package for their
   systems. For example, in a complex made up of RHEL 4 and RHEL 5
   hosts, the RHEL4 package should be used on the RHEL 4 hosts, and
   the RHEL5 packages should be used on the RHEL 5 hosts.

   If PBS Professional 10.2 or later is deployed onto Linux
   distributions other than the supported SLES and RHEL distributions,
   there may be runtime problems with dependencies on shared object
   libraries, typically libssl.so or libcrypto.so.

   Customers who encounter such problems should first determine which
   supported distribution is most similar to that installed on their
   systems. For example, openSUSE 10.x is similar to SLES 10, and
   CentOs distributions are similar to RHEL. The customer should then
   try the PBS package that is supported for the similar system.

   If the runtime dependencies are still not resolved, there are two
   other approaches that may be tried. Before trying either approach,
   first run ldd on pbs_mom to get name and version number of
   libraries that pbs requires. pbs_mom needs to be extracted from the
   PBS rpm by using the Linux rpm2cpio and cpio commands.

   Here is an example using an installed SLES 10 x86_64 pbs_mom:

     shell> ldd /opt/pbs/default/sbin/pbs_mom
     libdl.so.2 => /lib64/libdl.so.2 (0x00002b23b0a59000)
     libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002b23b0b5d000)
     libssl.so.0.9.8 => /usr/lib64/libssl.so.0.9.8 (0x00002b23b0c96000)
     libc.so.6 => /lib64/libc.so.6 (0x00002b23b0ddd000)
     libcrypto.so.0.9.8 => /usr/lib64/libcrypto.so.0.9.8 (0x00002b23b100d000)
     /lib64/ld-linux-x86-64.so.2 (0x00002b23b093c000)

   The Linux incompatibilities are generally associated with
   libcrypto.so, which is required by libssl.so.

   Having identified the requisite version of libcrypto.so and
   libssl.so, there are two options:

   (1) Locate and install an OpenSSL compatibility library that will
       satisfy the dependency. In the example above, the library needs to
       satisfy the dependency on libcrypto.so.0.9.8.

   (2) Download and install a version of OpenSSL that corresponds to the
       dependency determined above (e.g., libcrypto.so.0.9.8).


2. Starting with version 10.4, PBS requires more disk space than prior
   versions.  Package size of PBS with Python is almost double that of
   these older versions.  For example, under SLES, the size is now
   81mb instead of 49mb, and under RHEL, the size is now 42mb instead
   of 21mb.


3. The default umask behavior changed in 10.4 due to the fix for
   SPID 4487.  Under 10.2 and earlier versions of PBS, if
   -Wsandox=PRIVATE was specified, the job's execution directory was
   created with permissions drwxr-xr-x .  With this fix, the default
   job umask is applied, so unless a different umask is specified for
   the job's execution directory, it has permission drwx------.  To
   allow read access by Group or Other, it is necessary to specify the
   appropriate umask when submitting the job.  See the PBS Professional
   User's Guide.


4. Do not use different versions of PBS within the same complex.
   All machines using the same PBS server should run the same version
   of PBS.  Do not mix major or minor versions.  For example, do not
   run 11.2 and 10.2 in the same complex.


5. Do not use qmgr to set or change the vnode sharing attribute, or a
   vnode's ncpus, vmem or mem.  Instead, create a new vnode definition
   file and use "pbs_mom -s insert" to add the file.  Then HUP the
   MOM.  There are requirements for the vnode definition file.  See
   the PBS Professional Administrator's Guide, section 3.5, "How to
   Configure MOMs and Vnodes".


6. All printable characters are allowed in a resource value, except
   for comma, space, colon, quotes, and ampersand, but are subject to
   some restrictions such as those imposed by select and place.  Also,
   if these resource values are used as pathnames, they are subject to
   the restrictions of the operating system.

 
7. Issue 5909 (Bug 5375): pbs_mom emits error "unable to get my domain 
   name"

   Starting with release 7.0, pbs_mom may emit the message

	 "pbs_mom: pbs_mom, Warning: unable to get my domain name"

   when PBS is started.  The mere presence of this warning on a system
   that has been upgraded from a previous release of PBS should not
   cause concern: the same check was performed in previous releases;
   the only change is that the warning was suppressed.

   The actual cause of the warning may be
   -  pbs_mom is unable to find the fully-qualified domain name of the
      host on which it is executing
   -  although the host's name (i.e. the value of uname -n) is fully
      qualified, the host's native name service has been configured to
      prefer other name service backends over DNS, and the preferred
      service uses an unqualified name for the host.  For example, systems
      using a name service switch configuration file, nsswitch.conf, may 
      have a line of the form
           hosts: files dns
      in conjunction with a hosts file entry in which the host's name is
      not fully qualified.

   To address the problem, administrators should ensure that the
   result of converting the host's name (obtained via gethostname())
   to an address (using gethostbyname()) and converting that address
   to a name (using gethostbyaddr()) results in the original host
   name.

   See the pbs_hostn(8B) manual page for more information.


8. SPID 15770: Change in qdel behavior
   
   The -Wdelay option to qdel is silently ignored.  The delay between
   the SIGTERM and SIGKILL signals is determined by each queue's
   kill_delay attribute.  The default value for this attribute is changed
   to 10 seconds.


9. X11 forwarding limitation with Sun ssh 1.1

   A working X11 forwarding configuration depends on the underlying
   ssh version. Sun sshd version Sun_SSH_1.1 does not work
   correctly. Updating to version Sun_SSH_1.1.2 provides an sshd with
   working X11 Forwarding. We recommend that customers use version
   1.1.2 or newer. Sun sshd differs from openssh (as installed on
   BSD/linux systems) in important ways, so we do not recommend that
   customers replace Sun_SSH with openssh.

10. Enabling X11 forwarding

    In order to enable X11 forwarding, add the path to the xauth binary
    to each MoM's pbs_environment file.  For example:
       echo $PATH
       /bin:/user/bin

       -bash-3.2$ which xauth
       /usr/bin/X11/xauth

    The entry in the pbs_environment file should be the following:
       PATH=/bin:/usr/bin:/usr/bin/X11


-------------------------------------------------------------------------
Special Notes (External Interface Changes)

All external interface changes took place in previous releases.

Changes
-------

0.  New -T option to qstat command shows estimated start time.

1.  Server cannot be killed with SIGINT.

2.  The pbs-report command will be deprecated.  It will be moved to the 
    unsupported directory in a future release.

3.  New server and queue resource usage limits are incompatible 
    with old server and queue resource usage limits.  See 
    "Managing Resource Usage" in the PBS Professional Administrator's 
    Guide.

4.  Enabling job history changes the behavior of dependent jobs. If a
    job j1 depends on a finished job j2 for which PBS is maintaining
    history then j1 will go into the held state. If job j1 depends on a
    finished job j3 that has been purged from the historical records
    then j1 will be rejected just as in previous versions of PBS where
    the job was no longer in the system.

5.  8.0: Change to pbsnodes output: "Host =" replaced with "Mom =" 

6.  The resource arch is only used inside of a select statement.

7.  pbs_hostid command no longer used.  Use "almutil -hostid"..

8.  When a job is requeued, a new timeout message may be printed.

9.  When a reservation is confirmed, the message printed to the 
    scheduler log is different from previous versions.

10. MOM now exits with an exit value of 1 if an error is detected while
    reading the configuration file.  In previous versions of PBS, MOM
    did not exit.

11. Some MOM error messages no longer have a leading WARNING flag.

12. Behavior of qdel changed.  See Special Notes (General).

13. Behavior of qsub, qstat, qselect, pbs_mom expanded.  

14. Change in pbsnodes output: on hyper-threaded Linux systems, pbsnodes 
    now reports the number of processors listed in /proc/cpuinfo.

15. Change to log levels: see the PBS Professional Administrator's Guide
    for the table titled "PBS Events and Log Event Classes"

16. Ability to submit a job with "-l <resource name>", i.e. without
    giving a value for the resource, will not work in future releases. 
    This behavior was not supported, and will no longer work in future
    releases.

17. (11.0) A message in the server log file indicating that a vnode 
    name is already defined by another MOM is no longer logged.

18. Consumable resources that do not appear in the "resources" line in
    the sched_config file will no longer appear in a job's exec_vnode
    or reservation's resv_nodes. In previous releases they were
    included, but not used.  Most notable examples are mpiprocs and
    ompthreads.

19. 11.2: Support in PBS for CSA is deprecated.

20. 11.2: New -X option to qsub.

21. 11.2: New accounting log field called "project=<project name>"
    added to S, E, and R records.


Files that no longer exist (11.0)
--------------------------
 PBS_HOME/server_priv/nodes
 PBS_HOME/server_priv/resvs/ and all *.RB files under this directory
 PBS_HOME/server_priv/serverdb


Deprecated scheduler configuration parameters 
---------------------------------------------
 assign_ssinodes 
 cpus_per_ssinode 
 key
 load_balancing_rr 
 mem_per_ssinode 
 preempt_checkpoint 
 preempt_fairshare 
 preempt_requeue 
 preempt_starving 
 preempt_suspend 
 sort_by 
 sort_priority option to job_sort_key
 strict_fifo 
 sync_time


Obsolete and deprecated resources
---------------------------------
 The -l nodes=nodespec form is replaced by the -l select= and -l place= 
    statements.
 The -l resc=rescspec form is replaced by the -l select= statement.
 Properties are replaced by boolean resources.
 The resource host is only used inside of a select statement.
 The resource arch is only used inside of a select statement.
 The nodect resource is obsolete.   The ncpus resource should be used 
    instead. Sites which currently have default values or limits based 
    on nodect should change them to be based on ncpus.
 The neednodes resource is obsolete.
 The ssinodes resource is obsolete.
 The ppn resource is deprecated.
 The nodes resource is no longer used.
 The mpp* resources are deprecated, meaning the following are deprecated:
   mppwidth
   mppdepth
   mppnppn
   mppmem
   mpparch
   mpphost
   mpplabels
   mppnodes


Server and Vnode deprecations
-----------------------------
 The time-shared node type is no longer used.
 The :ts suffix is obsolete.
 The cluster node type is no longer used.
 The lictype vnode attribute is deprecated.
 The license vnode attribute is deprecated.
 The server attribute node_pack is deprecated.
 The server attribute pbs_license_file_location is deprecated, 
   and replaced by pbs_license_info.


Command deprecations
--------------------
 The -a option to the qselect command is deprecated.
 The -Wdelay=nnnn option to the qdel command is deprecated.
 The -c option to the pbsnodes command is deprecated.
 The pbs_hostid command is no longer available.
 The -a option to the pbs_sched command is deprecated.

API deprecations
----------------
 pbsrescquery
 pbs_rescquery
 avail
 totpool
 usepool
 configrm


Altix: no longer used
---------------------
Requests for the following values will return a PBSE_RMUNKNOWN error:
 cpuset_small_mem
 cpuset_small_ncpus
 memreserved
 max_shared_nodes
 nodersrcs
 shared_cpusets
 small_job_spec


New server attributes
---------------------
   reserve_retry_init
   reserve_retry_cutoff
   max_queued
   max_queued_res.RES
   max_run
   max_run_res.RES
   max_run_soft
   max_run_soft_res.RES
   job_history_enable
   job_history_duration
   max_concurrent_provision
   backfill_depth (10.4)
   est_start_time_freq (10.4)
   pbs_license_info (11.0)

New vnode attributes
--------------------
   hpcbp_enable
   hpcbp_user_name
   hpcbp_webservice_address
   hpcbp_stage_protocol
   provision_enable
   current_aoe
   in_multivnode_host (11.0)

New reservation attributes
--------------------------
   reserve_retry

New queue attributes
--------------------
   max_queued
   max_queued_res.RES
   max_run
   max_run_res.RES
   max_run_soft
   max_run_soft_res.RES   

New job attributes
------------------
   Exit_status
   Stageout_status
   Submit_arguments
   estimated
   placement includes "vscatter" (11.0)
   placement includes "exclhost" (11.0)
   forward_x11_cookie (11.2)
   forward_x11_port (11.2)
   project (11.2)

New job states
--------------
   F Finished
   M Moved
   
New vnode states
----------------
   provisioning
   wait-provisioning
   sharing attribute includes "default_exclhost", "force_exclhost" (11.0)
   resv_exclusive (11.0)


New MOM configuration variables
-------------------------------
   cpuset_error_action (10.4)
   $aix_largepagemode
   vnodedef_additive (11.0)

New resources
-------------
   start_time (10.4)
   exec_vnode (10.4)
   vntype (11.0)
   nchunk (11.0)
   PBScrayhost (Cray only) (11.0)
   PBScraynid (Cray only) (11.0)
   PBScrayorder (Cray only) (11.0)
   PBScraylabel_<label> (Cray only) (11.0)
   PBScrayseg (Cray only) (11.0)
   naccelerators (11.2)
   accelerator_memory (11.2)
   accelerator (11.2)
   accelerator_model (11.2)

New scheduler attributes
------------------------
   do_not_span_psets (11.0)

-------------------------------------------------------------------------
Special Notes (Documentation)

1. The 10.2 and later release of PBS Professional includes a new book
   called the PBS Professional Reference Guide.  This book
   contains reference material for PBS Professional, including
   commands, configuration parameters, and attributes.

2. The title of the PBS Professional External Reference Specification 
   has been changed to "PBS Professional Programmer's Guide".

3. Some manual pages that were reproduced in the PBS Professional 
   Administrator's Guide have been removed from the guide.  They 
   can be found in the PBS Professional Reference Guide,
   and as online manual pages.

4. The pre-9.2 PBS Professional Administrator's Guide has been split
   into two books: the PBS Professional Installation and Upgrade Guide
   and the PBS Professional Administrator's Guide.  The PBS
   Professional Installation and Upgrade Guide contains the
   instructions for pre-installation planning, and for installation
   and upgrading.


-------------------------------------------------------------------------
Special Notes (Hooks)

Administrators who are writing provisioning hooks need to be aware
that the Python pbs.event().reject() function fails to log the error
message when used from a provisioning hook.  This function behaves
correctly for all other hooks supported by PBS Professional.


-------------------------------------------------------------------------
Special Notes (HPC Basic Profile)

Supported Platforms for HPC Basic Profile: 
1.  The PBS server is supported on RHEL4(x86_64) and SLES 10 (x86_64)
2.  The HPCBP MOM is supported on RHEL4(x86_64) and SLES 10 (x86_64)
3.  Client commands are supported on Linux on x86 and x86_64

Supported commands and features:
    Some commands and features are not supported with HPC Basic Profile.
    See the PBS Professional Administrator's Guide.

Job submission:
    Job submission for HPC Basic Profile requires specific syntax.
    See the PBS Professional User's Guide.

Configuration:
    Using HPC Basic Profile requires specific configuration of PBS.
    See the PBS Professional Administrator's Guide.

-------------------------------------------------------------------------
Special Notes (Windows)

0. PBS path information no longer saved in AUTOEXEC.BAT

   Any value for PATH saved in AUTOEXEC.BAT may be lost after
   installation of PBS.  If there is any path information that needs
   to be saved, AUTOEXEC.BAT must be edited by hand after the
   installation of PBS.  PBS path information is no longer saved in
   AUTOEXEC.BAT.

1. Change in requirements for submitting jobs under Windows

   Each batch script (.bat) in a job script must have "call" in front
   of it.  For example, MyJob.bat, which calls Step1.bat and Step2.bat
   should look like this:
   #PBS -l walltime=1:00:00
   call E:\Step1.bat
   call E:\Step2.bat


2. If the full path name of a PBS command contains spaces, the
   path name must be quoted. If an argument to a PBS option contains
   spaces, it must be quoted. Examples:

   "\Program Files\PBS Pro\exec\bin\qstat" -Bf
   qsub -W stagein="the_input@hostA:c:\Documents and Settings\pbstest\my_input"


3. Specify only Windows DNS address for Windows Host

   For any privileged action requested of the server (e.g. qmgr
   requests), PBS allows it if the requesting user is an admin-type of
   user (i.e. member of "Domain Admins" or "Administrators), and the
   requesting host is the local host.  PBS compares what it initially
   thinks is the local host name against the requesting host name.

   The server may return "Unauthorized Request" for privileged actions
   requested by a legitimate user.  The problem arises from having the
   local host configured with two or more DNS addresses resolving to
   different names for the local host name.  In particular, a site may
   have set up a UNIX-driven DNS as the root domain, and a
   Windows-driven DNS in a subdomain to service domains in an Active
   Directory environment. Specifying both addresses in the local
   host's list of DNS addresses may give rise to this problem.  So it
   is important that in this case, you must specify only the Windows
   DNS address for the Windows host.


4. Requirement for using preemption via group soft limits:

   All jobs must be submitted using the -W group_list option 
   to qsub.  This ensures that the egroup job attribute is set 
   properly, which is required for this type of preemption to work.
   If the -W group_list option to qsub is not used, the egroup
   job attribute may not be set properly, and the number of jobs
   counted as being over or not over their soft limit may be inaccurate.

   For example if userA belongs to pbsgrpA then the egroup will be
   pbsgrpA for the following job submission: 
      qsub -W group_list=pbsgrpA job.script


5. Windows MPICH 1.2.5

   To run PBS jobs that make use of Windows MPICH 1.2.5, you need to
   specify the "-localroot" argument when invoking mpirun to get
   better performance, or to get around the error "failed to
   communicate with the barrier command". Here is a sample PBS job
   script:

    C:\DOCUME~1\pbstest>type job.scr 
    echo begin 
    type %PBS_NODEFILE% 
    "C:\Program Files\MPICH\mpd\bin\mpirun" -localroot -np 2 -machinefile 
    %PBS_NODEFILE% \winnt\temp\netpipe -reps 3 
    echo done


6. Windows platform restrictions: 

    Any platform used for the following should be a Server-type
    of Windows OS:
    1. shared PBS_HOME in a server failover configuration
    2. location for network-shared user home directories 

7.  Windows domains

    PBS Professional is supported on Windows XP Pro and Windows 2003
    Server if the domain controller server is configured native. While
    PBS Professional supports Active Directory Service domains, it
    does not support Windows NT domains. Running PBS in an environment
    where the domain controllers are configured in mixed mode is not
    supported.

8.  Installation on 64-bit platform has different path from 32-bit

    PBS Installation on a 64 bit Windows XP host (x64): PBS Installer
    will prompt for the destination path with "Drive:\Program Files
    (x86)\PBS Pro".  This drive is the name of the partition/drive
    where Windows has already been installed. During installation,
    system-related files are copied to the respective location in the
    relative path Windows\SysWOW64.

9.  Jobs requeued in slow file system

    The PBS server might mark the execution node as down during file
    stage operation.  For a system on a slow network where file
    copying takes a long time, MOMs may be marked down when they are
    up, due to the copy delay.  A MOM that is busy waiting for file
    copy to finish may be marked down by the server.  As a result,
    jobs on this MOM are requeued. To avoid this, set
    node_fail_requeue to zero.


10. Memory reporting and enforcement

    Physical memory usage can be reported.  Memory limits can be
    enforced.  Virtual memory (vmem) is not reported.


11. Installation Location

    The PBS installer requires and thus installs the Microsoft 2005
    redistributable pack.  Please refer to Microsoft documentation for
    further details on this package.


12. xcopy bug: do not use xcopy inside job scripts

    The xcopy command sometimes expects input from the user. Because
    of this, it must be assigned an input handle. Since we do not
    create the job process with an input handle assigned, XCOPY will
    fail/behave abnormally if used inside a PBS job script. This is a
    bug in XCOPY for which there is no known patch.
    Customers are therefore advised not to use the XCOPY command inside
    job scripts and use COPY/ROBOCOPY/PBS_RCP command instead.


-------------------------------------------------------------------------
Special Notes (SGI)

0. cpusets: the behavior of pbs_mom has changed with respect to actions
   taken when a CPU set setup failure occurs.  See the documentation on the
   cpuset_error_action MoM configuration variable.

1. UV: PBS does not automatically generate vnode placement set
   information.

2. Altix: cpuset_small_mem, cpuset_small_ncpus, max_shared_nodes,
   nodersrcs, shared_cpusets and small_job_spec are no longer used.
   Requests for those values will return a PBSE_RMUNKNOWN error.

3. There is a known issue with SLES9 where ProPack 4 can crash.  This
   is caused by SuSE bug 85380.  It is strongly recommended that you
   upgrade to the latest version of SLES9, e.g. SLES9 SP4.  If you 
   install SP4, make sure that you point to SGI for YOU updates.  

4. Issue 5353 (bug 12069) and issue 5354 (bug 12070) changed the
   behavior of PBS on ICE systems with respect to PBS-generated vnode
   definitions: previously PBS automatically generated the placement
   information if running with CPU set support on execution nodes.
   This was changed to not generate placement automatically, instead
   providing a script the customer could call to do so.

   Customers upgrading from 9.2 to 9.2patch or later will encounter
   the fact that an existing use of pbs_mom.cpuset on an ICE system
   will have generated placement information but the new one will not.

   Startup time may take minutes.

5. When using SGI UV machines, make sure that the MTU size is consistent
   on all systems in the PBS complex.

6. SGI's MPT comes with either ProPack or Performance Suite.
   MPT may depend on array services to do work across
   multiple physical hosts and a bug in the Linux kernel prior to
   version 2.6.32.13-0.4.1 may prevent that from succeeding.



-------------------------------------------------------------------------
Special Notes (Cray)

1. PBS is more tightly integrated with Cray:  
   a. PBS job submitters can use the PBS select and place language.
   b. The mpp* resources are deprecated.
   c. PBS supplies several Cray-specific resources.
   See the PBS Professional Administrator's Guide and the PBS 
   Professional User's Guide. 

2. Upgrading a Cray to PBS 11.0 and later requires a migration upgrade.  See the
   PBS Professional Installation & Upgrade Guide.

3. The Cray XT5h system is comprised of both XT5 and X2 compute nodes.
   A typical qsub request that does not specify the node architecture
   will cause the job to run on XT5 nodes.  For example,

      qsub -lmppwidth=2 -lmppdepth=3 jobscript.sh

   If the jobs should run on X2 nodes, the request must be:

      qsub -lmppwidth=2 -lmppdepth=3 -lmpparch=X2 jobscript.sh

   Running jobs on a mixture of XT5 and X2 nodes is not supported.


4. For Cray XT only: The number of licenses to allocate to a job is
   based on whichever is greater: ncpus or (mppwidth*mppdepth).
   The value of mppdepth defaults to 1, if not specified.


-------------------------------------------------------------------------
Special Notes (MPI)

1. The default for starting wrapped mpids is rsh.  If you are wrapping
   HP MPI with the PBS pbs_mpihp, and you want to use something other
   than rsh, be sure to set PBS_RSHCOMMAND in PBS_HOME/pbs_environment.  
   For example:

   PBS_RSHCOMMAND=ssh

2. MPD rings
   As has been observed with mpich2-1.0.3, when starting MPD daemons,
   only one MPD per user on an execution host can be run; multiple MPD
   rings can be created per user as long as each participating
   execution host is in only one ring. Otherwise, jobs with
   intersecting execution hosts on their MPD rings can interfere with
   one another.

   Suggested workaround:

   Given that only 1 MPD per user on an execution host can be run,
   do one of the following:

   1. Set 1 job per user to run on a PBS node by setting:

        # qmgr -c "set node <node_name> max_user_run=1"

   2. Force users to submit jobs to run on nodes exclusively using the excl
      keyword in -l place as in:

        % qsub -l select=2:ncpus=1:mpiprocs=2 -l place=free:excl job.scr

      In addition, to set this default job-wide resource on the server, 
      the admin can set:

        # qmgr -c "set server resources_default.place=free:excl"
 
   Note: Workaround 2 prevents other users from using the same node if it has
   more than 1 resources_available.ncpus.

3. If you will use suspend/resume with Platform MPI, make sure that 
   MPI_BUNDLE_MPIDS is not set.

-------------------------------------------------------------------------
Special Notes (AIX)


1.  Use of Large Page Mode
    AIX 5.2 and AIX 5.3

    An IBM bug exists that prevented PBS from enabling Large Page Mode
    for AIX 5.0 through 5.3.  The bug reference is:

    IY80185: SETPCRED() TRUNCATES POINTERS TO 32 BITS

    IBM has now fixed this bug and provides a Fix Pack for AIX 5.2
    and 5.3.  The bug does not exist in AIX 6.0 and higher.  See:

    http://www-01.ibm.com/support/docview.wss?uid=isg1IY80185

    Before installing/running PBS on AIX 5.2 or 5.3, a Fix Pack must
    be installed.  For AIX 5.2 use AIX APAR IY80185.  For AIX 5.3 use
    AIX APAR IY90555.  The program oslevel will tell you the version
    of AIX that is installed.

       instfix  -ik IY80185
    will tell you if you have the patch for AIX 5.2.

       instfix -ik IY90555
    will tell you if you have the patch for AIX 5.3.

    Contact IBM support for more information on installing Fix Packs.

2.  The default PBS_HOME directory for AIX was changed from 
    /usr/spool/PBS to /usr/local/spool/PBS.

3.  InfiniBand state window never shows NRT_WIN_RUNNING

    In AIX 7.1, the InfiniBand state can be misreported.  The 
    state window never shows NRT_WIN_RUNNING, and this can lead
    a user to believe that their job is not in US (user) mode.
    This is IBM bug 166907.

-------------------------------------------------------------------------
Special Notes (Linux)

Altair distributes separate PBS packages for different supported Linux
distributions. Customers running on SLES and RHEL distributions should
take care to use the correct package for their systems. For example,
in a complex made up of RHEL 4 and RHEL 5 hosts, the RHEL4 package
should be used on the RHEL 4 hosts, and the RHEL5 package should be
used on the RHEL 5 hosts.

If PBS Professional is deployed onto Linux distributions other than the
supported SLES and RHEL distributions, there may be runtime problems with
dependencies on shared object libraries, typically libssl.so or libcrypto.so.

Customers who encounter such problems should first determine which
supported distribution is most similar to that installed on their
systems. For example, Fedora Core 6 is similar to RHEL 5. The customer
should then try the PBS package that is supported for the similar
system.

If the runtime dependencies are still not resolved, there are two other
approaches that may be tried. Before trying either approach, first run ldd on
pbs_mom to get name and version number of libraries that pbs requires. pbs_mom
needs to be extracted from the PBS rpm by using the Linux rpm2cpio and cpio
commands.


Here is an example using an installed SLES 10 x86_64 pbs_mom:

shell> ldd /opt/pbs/default/sbin/pbs_mom
libdl.so.2 => /lib64/libdl.so.2 (0x00002b23b0a59000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002b23b0b5d000)
libssl.so.0.9.8 => /usr/lib64/libssl.so.0.9.8 (0x00002b23b0c96000)
libc.so.6 => /lib64/libc.so.6 (0x00002b23b0ddd000)
libcrypto.so.0.9.8 => /usr/lib64/libcrypto.so.0.9.8 (0x00002b23b100d000)
/lib64/ld-linux-x86-64.so.2 (0x00002b23b093c000)

The Linux incompatibilities are generally associated with
libcrypto.so, which is required by libssl.so.

Having identified the requisite version of libcrypto.so and libssl.so,
there are two options:

(1) Locate and install an OpenSSL compatibility library that will
satisfy the dependency. In the example above, the library needs to
satisfy the dependency on libcrypto.so.0.9.8.

(2) Download and install a version of OpenSSL that corresponds to the
dependency determined above (e.g., libcrypto.so.0.9.8).



-------------------------------------------------------------------------

New Features

See also chapter 1 of the PBS Professional Administrator's Guide.
Note that Bugzilla numbers are no longer the primary bug-tracking
tool; instead Issue numbers are used for this.  Issues are not listed
by OS.

New Features in Release 11.2:

Issue   BZ     Title
-----   -----  -----------------------------------------------------------
4222	8404   Support for X11 forwarding via qsub. 
4697 	9823   Projects-based Limits.
4968 	10858  Support Intel MPI 3.0/4.0
17643 	       oom_adj awareness in PBSPro, so mom cannot be killed but her children can be
18153 	       Support Cray Basil 1.2
187520 	       Cray GPU Support
197525 	       Support SGI ProPack 7 on Altix "classic"
214100	       Supply customers with a supported way to view the job scripts of submitted jobs.
215868	       Support AIX 7.1
224693	       Support CentOS 6
235282	       Update basil.h to support basil_support


New Features in Release 11.1:

Issue   BZ     Title
-----   -----  -----------------------------------------------------------
16591	       Support CentOS
17760	       Deprecate pbsnodes -c option
197487	       Support CentOS (installation)
205513	       Support Interlagos (AMD Chip) on Cray
210663	       Support RHEL 6 on x86 and x86_64
210664	       Support RHEL 6 on ICE/XE/UV
210665	       Support SGI Performance Suite on RHEL and SLES
211710	       Server should not wait for single node jobs to start 

New Features in Release 11.0:

Issue   BZ     Title
-----   -----  -----------------------------------------------------------
16224	       Introduce "personal mode" configuration for hostname resolution for PPBS (phase 1)
182208	       Introduce "personal mode" configuration for acct running PBS on Windows under PPBS
20203	       Introduce "personal mode" configuration for acct running PBS on Windows under PPBS (phase 1)
175173	       Eliminate the need for having a Windows Domain User Service Account for 
	       Windows Domain environments - Phase 2
183670	       Installer changes for configurable service account (for RFE 182208)
5363	       PBS Professional Server/MOM Scalability
189890	       Package PostgreSql with PBS Pro
195254	       Generic OS Provisioning (phase 2)
184394	       Expose Vnodes Information in Hooks
184788	       Enhance PBS's notion of vnode scattering and excl
16543	       Improving integration with Cray platforms
187329	       Support LM-X for PBS Professional


New Features in Release 10.4:

Issue   BZ     Title
-----   -----  -----------------------------------------------------------
2819	4822   Query for job scheduling order
2525	4146   Ensure the top N most deserving jobs are not delayed by backfilling
4962	10844  Common cross-platform job execution environment (aka Python Everywhere)
15527	       Support SGI Ultraviolet Hardware configuration
182830	       log resource usage when job is requeued


New Features in Release 10.2:   

Issue   BZ     Title
-----   -----  -----------------------------------------------------------
4046	7898   Support site-specific allocation management
4783	10290  Support SLES 11 on x86, x86_64, and ia64
5155	11392  Generic OS Provisioning
5293	11850  Support User Space mode on IBM Infiniband switches without LoadLeveler
17731	       Checkpoint by walltime interval
18677	       Scheduler fails to restart on alarm call & deprecate pbs_sched -a


New Features in Release 10.1:   

Issue   BZ     Title
-----   -----  -----------------------------------------------------------
1646   	2424  	Extend client commands (e.g., qstat, qdel) to track jobs qmoved to 
		other servers by default 
1866 	2807 	Add "per named user" resource limits (like max_user_res)
2626 	4387 	Add "per named group" resource limits (for ncpus)
3007 	5109 	Limit specific users/groups on number of jobs/resources 
		(server & queue level)
3163 	5671 	create "max_group_res_queueable" attribute
3614 	6822 	Add ability to retrieve information about finished jobs
4054 	7911 	Make reservations more fault tolerant
4241	8449	Ability to specify command to run for interactive jobs (qsub -I)
4462 	9172 	Finer Granularity Control on Users/Groups
4526 	9349 	Metascheduling to Windows HPC Server 2008 via OGSA HPC Basic Profile
4753	10088	add command arguments to qsub
4998 	10957 	Add ability for sites to give certain users higher limits than 
		the general limits.
5118 	11315 	Support Windows Server 2008 on AMD64 and EM64T (x86_64)
18022 		Allow epilogue to requeue job for restart from a checkpoint file

New Features in Release 10.0:   

Issue   BZ     Title
-----   -----  -----------------------------------------------------------
1879	2835   Support Windows XP Professional and 2003 Server on AMD64 and EM64T
2245	3599   Site-specific hook for additional checks at submit time
2583	4288   Site-specific hook to qsub to modify args/input parameters
2799	4791   Ensure "Large" jobs get started (without "starving")
3657	6950   Site-specific hook for pbs_rsub
4464	9176   Install PBS into version defined sub-directories
5074	11201  Support ProPack 6 on SGI Altix Itanium, XE, and ICE platforms



-------------------------------------------------------------------------
Modifications and Bug Fixes for Release 11.2:

Issue   BZ      Title
-------------------------------------------------------------------------
2181 	3427 	Epilogue in Windows has issues with passed arguments
3250 	5847 	Node erroneously marked "down" due to heavy memory use and paging
3927 	7625 	Behavior Change in Server Startup When Node is Not Available
19647 		Don't run epilogue multiple times
175010 		server remembers vnodedef sharing setting after vnodedef removed
175793 		Unsupported API's in pbs_ifl
181287 		Problem with reservation confirmation when the difference between 
		reservation start time & submission time is very less
187303 		Stageout is moving System/Hidden Files on Vista 64 bit in Personal Mode
191080 		Bug in currently unused code in resource.c
192052 		Deleted custom resource should not show up elsewhere in PBS
193146		qsub request gets intermittenly rejected due to alarm call .
193147 		hook causing submit rejection
207690 		can't tell if a moved job finished, even with job history enabled on all servers
210066 		qmgr -c "help" doesn't display the version number.
210808 		readme file reports PBS Pro 10.x versus 11.00
213731 		Stageout is creating a copy of system file inside the same folder
213781 		function remtree returns error when path does not exist
215727 		resources assigned twice for job requiring provisioning
216866 		INSTALL should preserve server FQDN
219563 		job staging directory not getting deleted even after successful stagein and stageout
220361 		Failed provision needs to set node comment
220989 		Windows pbs_send_job writes "Log opened" whenever server sends job to mom
222125 		Reservation inside an exclusive reservation gets confirmed
222856 		PBS Init script in an infinite loop, will not allow machine to boot.
223458 		qterm options -f, -F did not stop the primary and secondary servers in 
		a failover setup
223743 		Node order of multi-vnode setup is not maintained upon restart
224178 		Performance of mem resource usage calculation could be better
224348 		PBS Pro 11.1 INSTALL script does not include CentOS 5.6
224611 		pbs_server crash when creating node, likely because of hostname resolution failure
225092 		Declaring indirect resources in a vnode definition does not achieve expected behavior
225189 		Checkpoint restart of PBS job fails because 
		PBS confirms the new ALPS reservation with checkpointed job's old PAGG
225296 		Anomalous accounting log with provisioning
225336 		RUNJOB event is not honored
225408 		CPR: unable to restart a checkpointed job in PBS 11.1 GA
225839 		PBS Installation fails on specifying non-default location and non-root user.
225929 		Inappropriate memory handling
225933 		Heap overflow problem
226196 		Core dump in pbs_sched on error 15139
227020 		qsub -l select accepting " invalid resource" and wrong syntax, from 11.1 onwards
227106 		Overlay upgrade fails to start when upgrading from 11.1.0
228161 		UV mom_get_sample() doesn't find enough processes
228456 		qmgr seg-faults when set with a wrong syntax
228644 		pbs_sched hangs in a deadlock when holidays file doesn't have the YEAR line
229490 		Default for pbs_license_min needs to be larger, or increase pbs_license_linger time.
229587 		Failover: secondary pbs_server fails to launch pbs_sched if /etc/pbs.conf 
		is not present
229624 		Installation does not save dataservice acct when installing to non-default PBS_EXEC
229744 		job's staging and execution directory removed when staging fails
230047 		Prolog.c has CRLF, LF line terminators
230158 		Job's requesting boolean=false may never run on sites with placement sets
230363 		Restarting PBS server shows substate of finished job (JOB_SUBSTATE_FINISHED) 
		as moved job (JOB_SUBSTATE_MOVED)
230364 		The job substate is not updated with JOB_SUBSTATE_FINISHED (92) on finish 
		executing the moved array job.
230929 		qmgr core dump with "p q @default"
231046 		Adding UNIX man pages to builds
231396 		INSTALL script will fail on crayxt
232244 		Queue attributes that are unset gets set after a PBS restart
232489 		Networking errors may leave qruns hanging waiting for the scheduler's reply
232953 		Normal user gains admin privilege if Server host name is localhost
233441 		INSTALL script needs updating for newer CentOS releases
234941 		Undefined resource for PBS 11.1
235183 		Cray migration upgrade: Job rejected in mpp translation after qmove
235728 		segmentation fault core dump from server's children, server still running
236205 		When pbs_rsub -Wqmove fails, the message returned from the Server is 
		incorrectly formated
236635 		PBS MOM generated 46GB log file leading to PBS crash


Modifications and Bug Fixes for Release 11.1:

Issue   BZ      Title
-------------------------------------------------------------------------
4746 	10045 	array job log message shows problem during job purge
4930 	10764 	Increase length limit on usernames and groupnames
14681 		reinitialize user rlimits on SIGHUP
16452 		pbs_probe not complaining when it should about file permissions
16777 		terminate script throws req_reject error though script has executed successfully
17323 		When a job is preempted, it should be "terminated" and 
		NOT "killed" before being restarted
18067 		packages/linux/wrap does not check for rpmbuild failure
18869 		improve handing of failed hook setup
177790 		Failure to recover dependent jobs on PBS server restart in peered environment
180675 		specifying users who are except to log into moms their process are being killed
184408 		pbs_rsub -W qmove option times out in only 10 seconds
191559 		Windows Installer - failure when double-quote is used in password
193610 		Job running into dedicated time
199153 		qstat -Q includes finished jobs
201353 		MPI integration on Fluent IB...
205398 		Stageout directory fail on windows
207045 		mom message for $action script can be confusing
207264 		function pointers are called without checking them for NULL
208690 		pbs_py_spawn() does not work with multi-node jobs
209056 		rm.3B man page missing from packaging
209240		job_sort_formula overrides fairshare, or not?
209729 		exec_vnode remains set for the job after the job launch 
		fails because pbs_server's fork failed
209940 		FGC limit of a resource can be overallocated upon being first requested
210234 		scheduler crash when backfilling in the presence of 
		job in a reservation that was qrun -H on nodes outside the reservation
210253 		setting single_signon_password enable from "f" to "t" complains about finished jobs
210401 		server crash on SRB project test server
210502 		topjobs can be delayed due to limits
210856 		In failover, both servers are active with postgres running on both.
210938 		qmgr -c "help" not updated with the correct copyrights text
211052 		Boolean custom resource fails if not host-level
211108 		Buffer overflow with standing reservations
211118 		Scheduler will reference the global fairshare tree when it shouldn't.
211508 		PBS qmgr gives sytax error for user name in FGC limit
211544 		pbs_server will fail on connecting to LMX license server if 
		pbs_license_min forces to check out all the tokens from license server.
211753 		hook infrastructure should treat resources as case-insensitive
211936 		pbs_server unnecessary updates to DB make pbsnodes command very slow
212021 		Automatically added "vntype" to the sched_config resources line for Cray installs
212024 		top jobs + fairshare: top jobs incorrectly punished from cycle to cycle.
212099 		windows code error: getLocalGroups() not calling NetUserGetLocalGroups().
212101 		qsub -koe is problematic when root can not write
212116 		Complex fairshare tree's may be incorrectly scheduled after 
		jobs are run due to double counting
212141 		PBS v11 won't install on Japanese localized Windows machine
212315 		extra vnode showed up in exec_vnode
212509 		readonly resources cannot be written to from a hook
212594 		qmgr operations on nodes are slower than in v10.4
212719 		Submission scripts with multibyte characters are not supported
212925 		uninitalized variable can crash server when deleting natural vnode
213009 		Permissions on job stderr/stdout when job is running 
		are not set according to -Wumask=
213340 		issue with inserting vnodes on HP-UX 11 (Itanium)
213441 		Not able to fetch licenses on x14-32-rhel6 due to bad TZ env value to PBS
213749 		"no server database exists" after changing hostname
214244 		pbs_probe doesn't detect ownership problems on $PBS_HOME/datastore.
214322 		Some server attributes are not preserved across upgrade
214700 		pbs_mom parsing of vnodedef files doesn't recognize space
215828 		Job submission checks can be circumvented in certain circumstances
216035 		Server doesn't report that queue doesn't exist
216182 		PBS does not report and error if PBS_DATA_SERVICE_USER edited manually
216212 		pbs_server crash in hooks with long dependency list
216246 		server fine grained limits are not retained upon restart
216919 		Dynamic resource assumed infinite if script fails
217067 		Should be root/admin to set acl_roots
217436 		Unable to start database service if database account does not have home directory
217533 		Scheduler may not correctly calculate resources_available value on a queue
217641 		PBS Windows installer still using "GridWorks"
217657 		Subjob's do not inherit the reserve_ID attribute
217822 		unsetting a FGC limit on a particular resource unsets the whole attribute
217998 		qstat -w argument seems excessively wide compared to 10.4.0 qstat -w
218228 		UNSUPPORTED: pbs_dtj does not pass -n to remote nodes
218667 		warning in scheduler file fifo.c
219486 		0x8000401B (The client is not allowed to launch this server) Error on PBS startup
219578 		Restart of PBS Server seg faults following job_save 
		failures due to out of disk space issue
220020 		Memory leak in pbs_server when provisioning
220246 		qmgr command causes pbs_server to crash
220710 		When a job terminates because of walltime violation, 
		termination action timeout is not respected
220769 		Possible server crash due to unchecked sprintf call
220829 		To update EDD of 5363 due to Issue 220818
221896 		Uninitialized variable prov_vnode_list in find_prov_vnode_list() 
		makes pbs_server to segfault 


Modifications and Bug Fixes for Release 11.0:

Issue           Title
-------------------------------------------------------------------------
20417 		Mom rejects job with incorrect walltime request
20746 		Have MS deal better with current stream not getting response
180321 		job arrays do not work on AIXIB
182299 		Placement sets are not sorted when asking placement per-job
187276 		mom cleaning job data directory after failed job staging
188358 		Suspected bug in simulate code
188834 		PBS should use job_create(3) independent of CSA
190154 		qstat shows duplicate job identifiers in its output
193147 		PBS_ibwins hook causing submit rejection
193230 		set sched / print sched command in qmgr are inconsistent
193632 		PBS changing USERPROFILE environment variable
195285 		Current sister Mom error message is not decriptive when she can not join the job
196052 		Call to the task manager initialization (tm_init) crashes (seen with MPI integration)
196803 		Job in status "T" and stack
196851 		pbs server not responding to commands
197414 		add libcsa.so.4 to the list of acceptable choices for SLES 11, ProPack 6
197683 		Only do case sensitive matching of paths on Linux and Unix
198013 		Poor Scheduling performance
198115 		PBS Scheduler goes into infinite loop and jobs stay in 'Q' state
198551 		Zero length file created by symlink attack
200774 		resources.available.ncpus limit on the queue not working with PBSPro 10.4
202431 		Job exists in two places when job history is enabled
202462 		pbs_server restart clears 'exec_host' attribute info for Finished jobs
202465 		Restarting pbs_server everyday with job_history_duration set to 24:00:00 
		causes jobs to remain in server forever
202617 		preemption failing if preemtped job takes >30s to go away
203516 		NASA is experiencing slow cycles (Upwards towards 40m) phase 2
203912 		prologue gets run as user on AIX
204674 		Scheduler assigns mpiprocs to a node causing node to become job-exclusive
204914 		PBS breaks syslogd message length limits causing pbs_server crash
204917 		Peer Scheduling cannot be used with Backfill and Estimated Start Time
205201 		Server Segmentation Fault when max_concurrent_provision is >5
205433 		Allow space characters in path names of qsub's -o and -e options
205451 		PBS_RSHD hosts.equiv file expecting newline, otherwise rcp fails
205722 		suspending qstat can hang the server
206664 		aix infiniband code does not check for port state
206783 		Stageout directory fail on windows (magic 65536 bytes) 



Modifications and Bug Fixes for Release 10.4:

Issue   BZ      Title
-------------------------------------------------------------------------
2339   	3805  	qmgr print server prints reservation queues
3709 	7080 	Scheduler may fail to run large jobs in multi-resv environment
4487 	9245 	permissions on tmpdir differ between Moms
20989 		Job deleted due to cpuset failure creation should be requeued
175772 		job_sort_key: "preempt_priority HIGH" doesn't sort as expected
176947 		Use of uninitialized variable in pbs_rdel
177918 		problems with limits.c
179551 		missing quotes in pbs_mpihp causes job run failures
179652 		node sorting not working within a reservation
180474 		File staging is not happening with stageout having directory as 
		destination path
180672 		job walltime is wrong when suspended job is deleted
181383 		pbsnodes <unknown-node> command returns 'NULL' as output
181986 		jobs rejected instead of queued in route queue when execution queue's
		max_queued hit
182350 		Memory leak in parse_at_list()
183262 		PBSPro 10.1 server crash
183584 		server can reject MOM RESET message
183649 		qmoved job array with 2 subjobs has subjobs incorrectly marked as 
		finished (X)
184068 		Walltime killed message
184912 		PBS no longer reports if job killed because memory limit exceeded in stderr
185336 		Emerald RC3 mpplables ignored w/ mppdepth
185365 		MOMs not reconnecting properly to server after nodes restarted
186694 		Cray 753938 scheduler crash
186785 		Security Hole in PBSPro Mom /var/spool/PBS/spool
187149 		Job messages repeat when job deleted from routine queue
187306 		Compilation error when PBS source not configured with python (BZ 14319)
190584 		HP MPI wrapper does not support mpirun syntax
191132		Submit a job with space in master file name is not working in windows 
		2008 64 bit machine
191875 		Scheduler looping and not scheduling jobs 


-------------------------------------------------------------------------
Modifications and Bug Fixes for Release 10.2:

Issue   BZ      Title
-------------------------------------------------------------------------
2226   	3542    log file entries for SCP events have trailing "^M"s
4078 	7992 	scheduler crashes when placement resource value appears multiple times
4472 	9196 	reservation job gets "resource unavailable" - clean up in progress
5142 	11370 	rcp supplied with PBS does not accept "." in user names
5250 	11658 	Log message for missed occurrences should be improved
5324 	11974 	Alps reservation still exists for a PBS job after the job is gone
5327 	11979 	pbs_mom times out ending qsub but ALPS is not aware
14717 		qrun/qmoved jobs shows Not running in qstat for state R
16155 		PBS init script fails to touch "/var/lock/subsys/pbs" on RHEL systems.
16540 		Old node comment returns after server restart
17905 		Invalid chars in vnodedef file can prevent server from starting
18677 		scheduler fails to restart on alarm call
18890 		qselect -s option is ignored for finished subjobs
19138 		Dependant jobs do not run after failover occurs
19189 		If we set limit on one resource , it is replacing the previous 
		limit set on different resources.
19592 		PBS_CONF_FILE may be getting set to muliple entries by PBS
19627 		AUTOEXEC.BAT PATH may be deleted
19646 		Scheduler receive buffer filling and dropping UDP packets
19905 		Server is not recognizing the state of reservation as Degraded 
		when a node is made as stale.
20230 		max_queuable will look at finished jobs if job history is turned on
20303 		Can't shutdown or restart secondary PBS Server when in a HA configuration.
20366 		No scheduler allocations when primary login node is unavailable on Cray XT
20417 		Mom rejects job with incorrect walltime request
20479 		blocking job never returns if there is a dependancy problem
20511 		Order of set_job and prologue execution different between batch 
		and interactive-batch jobs
20561 		Unnecessary install directory /opt/pbs/... on Cray boot node
20721 		Hook throws exception when multiple managers/operators set w/ same hostname
21373 		When staging out files from Job dir, the subdirectories structure is not maintained.
174607 		pbs_rsub -H option ignores all but first host in comma-separated list
175641 		POE error when running jobs on IBM IB.
175644 		Increase string length limit of PBS directives
175769 		Core dump from tm_request() in mom_comm.c
176420 		Hook for queuejob clobbers PBS_O_WORKDIR
176751 		Server doesn't allow unsetting of Resource_List['nodes']
176918 		poe crashing when different version lib used than what pbs was compiled with
177457 		Under certain circumstances PBS accepts jobs asking for undefined resources:
177787 		Mom kills stagein's when $restrict_user is set.
177911 		Server core dump with long list of chunks
178452 		Large page mode fails using PBS10.1 for AIX5.3
178718 		bad cookie in TM request reveals the correct cookie in MOM log
179048 		ALPS reservation not deleted when PBS deletes a job (Cray 747486)
179067 		AIX MOM crashes on P6 with IB
180438 		Jobs submission fails for "stage-in of files with spaces" only when 
		jobs are submitted to a Linux instance of PBS.
180551 		mom core dump when terminate script encounters zero tasks 


Modifications and Bug Fixes for Release 10.1:

Issue   BZ      Title
------------------------------------------------------------------------------------
1554  	2316  	preempting job fails to run when lower prio job can't be preempted
2122 	3242 	undefined external in pbs_submit API
3011 	5114 	long host name gives pbs_mom, pbs_sched, and qmgr indigestion
3443    6341    Can add same group multiple times to acl_groups
3701 	7054 	default PATH given to user jobs may not be sufficient
4159 	8200 	ImpersonateLoggedOnUser error in preemption test
4209 	8371 	warnings in rpp.c and popen.c
4262 	8541 	change error message that appears when there is no usage database
4398 	9016 	pbs_rmget usage incorrect
4501 	9294 	Job run ... on hosts should be on "vnodes"
4714 	9883 	When computer restarted, node shows down.
4822 	10428 	qselect -s X is not legal even though it is documented.
4865 	10549 	make over limit enforcement the same action as qdel
5023 	11039 	jobs won't run if pbs_license_max set to > 2147483647
5034 	11087 	Server leaks memory during resource assignment and reservation submission
5203 	11517 	Dispatch of jobs in reservation on multi-node system appears to have 
		incorrect placement
5261 	11677 	no server logging that showed missed occurence
5305 	11895 	mppwidth on nons-Cray floods scheduler log
5311 	11922 	pbs-report gives non-useful error message if run by non-root
5361 	12139 	PBS jobs not correctly passing signals
14629 		Non-cpusetted 9.2 pbs_mom reports incorrect number of physical CPUs on Altix
14762 		job with -lselect=0 stalls the scheduling cycle
15350 		Misspelled argument in Mom config file can result in Mom crashing executing the script
15382 		pbs-report, account field incorrectly set to date
15437 		job state mismatch cpuset cleanup problem/cpuset collision
15438 		cpuset cleanup problem
15464 		Scheduler simulation fails if a vnode associated to a reservation is unavailable
15497 		Race condition in end of job processing can cause Mom to not be told 
		to delete job/cpuset
15974 		Minor problem with qstat and long comments
16025 		scheduler needs to be more trusting
16389 		Installer does not recognize Altix systems correctly for RHEL and SLES
16836 		-W sandbox=private doesn't work with job arrays
16838 		typo in init script causes limits not to be set back to original values
16850 		ulimit manipulation in init.d/pbs should not be SGI specific
16933 		Reservation with associated "place" not recovered after server restart
16940 		import a base64 python hook as test crashes pbs server daemon
16976 		pbs server segfaults while submitting a job with -V option and a queuejob 
		hook is enabled
17456 		PBS mom detects only half of the physical CPUs
17580 		PBS' use of xcopy in Windows XP fails in German version
17635 		Hooks mangling incoming environment variables that contain commas
17822 		PBS Server internal error with hooks, caused by two Job_Name attributes 
		being sent to server.
17859 		Error 15007 on Windows - ( errno 10038 )
17906 		server crashes if invalid vnode spec in qrun -H
17999 		PBS_NODEFILE removed after qhold initiate checkpoint-abort of a running job
18019 		negative ncpus request or select with no ncpus (and no default) causes sched crash
18086 		Server segfaults when executing certain hooks
18254 		mom setting incorrect pcpus value on windows system.
18387 		server Crash on Windows when job is submitted
18400 		pbs_mom seg-faults with numerous job start errors
18419 		server can send jobs to nodes that are rejecting requests from sister nodes
18735 		shared and exclusive job in reservation assigned to same blade
18818 		Hooks crashing server
18825 		Server crash when hook enabled
18853 		Non-rerunnable jobs that cannot be started are deleted rather retried
18877 		PBS hook causing server to crash.
18975 		Queued jobs with exec host 


Modifications and Bug Fixes for Release 10.0:

Issue	BZ	Title
---------------------------------------------------------------------------------------
3048	5244	Slowness while installing PBS (optimization issue)
3541	6638	kill_delay queue attribute has no effect on qdel
3586	6738	after a password change, user's password doesn't work on node
4789    10317	File stage-in fails for read-only files
4865	10549	make over limit enforcement the same action as qdel
4905	10631	job script with "for /f" cmd construct does not work with qsub
5003	10993	cycle harvesting job does not change state (U) when rerun
5188	11458	The job directory should be deleted if there is a stage-in failure 
5195	11485	make tmpdir handling consistent between UNIX and Windows
5208	11542	Backfill job delayed
5243    11645   standing reservation state should change to DJ (Deleting Jobs) between occurrences
5245	11650   skipping first occurrence of a standing reservation leads to out of 
		sequence processing of server's work tasks
5246	11651   job submitted to a confirmed reservation does not run if reservation 
		overlaps another reservation
5247	11652   job did not run after submitting a confirmed standing reservation
5255	11665   pbs_command hang up, after the server restart after standing reservation ends
5260	11676   processing deletion of running jobs at end of occurrence may corrupt memory
5265	11683   check if occurrence duration exceeds reservation duration is erroneous
5295    11860	For Cray XT, license jobs based on mppwidth*mppdepth
5305	11895	mppwidth on nons-Cray floods scheduler log		
5313   	11925  	patch to handle requests for MPP resources with advance reservations
5318	11943	Scheduler takes very long to consider job with malformed select statement 
5325	11975	backfilling with strict ordering is not honoring walltime requests
5329 	11986 	Scheduler does not always honor backfill and will schedule other jobs instead
5333 	11991 	PBS Reservations are considered endless, problems with back to back reservations
5336 	12011 	Jobs are held in "H" status after several ALPS errors.
5339	12014	patch to handle requests for MPP resources with advance reservations
5340	12015	Scheduler does not always honor backfill and will schedule other jobs instead
5341	12016	PBS Reservations are considered endless, problems with back to back reservations
5342	12017	Jobs are held in "H" status after several ALPS errors
5346	12029	High priority jobs unable to run due to backfill problem
5355	12075   Advance reservations are not being respected
14492 		pbs_probe complained about missing /usr/local/pbs/tcltk/include,
14516 		mom segfaults while dereferencing a null pointer in a unusual path of job clean-up
14518 		mom segfaults while dereferencing a null pointer in a unusual path of job clean-up
14546 		Advance reservation can be oversubscribed
14556 		Jobs in reservations hang
14562	        Scheduler may oversubscribe resources in the presence of a running 
		standing reservation 
14595 		Server Segfault on Restart with -t hot option
14635 		Random mom segfaults
14636 		user qdel another users job
14656 		Rare security problem on Linux platform [9.2.2 GA version]
14659 		root owned processes when ulimit is too low
14671 		Rare security problem on Linux platform [8.0.7 GA version]
14674 		root owned processes when ulimit is too low
14683 		root owned processes when ulimit is too low [9.2.3 version]
14699 		Found PBS_MOM issue with PBS latest build on windows 2003.
14709 		Scheduler may oversubscribe resources in the presence of a running 
		standing reservation
14710 		Minor: Uninitialized variable in pbs_ical.c
14726 		Overlay and Migration upgrade are not working with the latest build
14771 		Advance reservations are not being respected
14798 		more missing curly braces in mpp.c
14835 		server leaks memory during regular job and array submission
14844 		Order job is not reflecting in the job in UI.
14845 		Job delete runs immediately and prevents user for interacting with the UI for a while
14858 		need to merge changes made to 8.1.5c to the PBSPro_9_2_branch
14903 		Mom went down in file staging load test
14904 		Mom went down twice in file staging load test
14950 		PBSvnodedefs file being created for SGI ICE without cpusets
14951 		PBSvnodedefs file being created for SGI ICE without cpusets
14991 		mom segfault upon qdel of a running job
14992 		nodal string arrary resource value is being ignored
15007 		sched segfaulting, stime on Q jobs
15187 		Scheduler consuming all memory
15238 		backfill job delayed 
15420 		backfill job delayed
15443 		Scheduler consuming all memory
15579 		Scheduler 15016 error message
15596 		B record does not contain number of standing reservations and 
		number of standing reservation occure





-------------------------------------------------------------------------

Known Bugs, Errors, and Warnings

This section lists known deficiencies and warnings as well as unresolved
problem reports filed by customers.

Note that Bugzilla numbers are no longer the primary bug-tracking
tool; instead Issue numbers are used for this.  Issues are not listed
by OS.

SPID	BZ	Summary
------------------------------------------------------------------------------------
2565 	4227 	multi-task jobs don't work using tm interface on Windows
4088 	8030 	Server accepts connection before testing "connection table full"
4324 	8712 	scheduler ignores node/queue association when node_group_key set
4344 	8779 	cycle harvesting on some Linux (FedCore 5,SLES10) not working
4346 	8785 	pbs_rsub not computing start time from an end time and duration
4510 	9315 	pbsrun_wrap/unwrap should know if installed in shared location
4640 	9651 	locate function in server does not work for subjobs
4706 	9858 	pcpus shows on all vnodes in output of pbsnodes/qmgr
		  The pbsnodes command displays pcpus in the output for all 
		  vnodes. This is cosmetic only and doesn't affect functionality.
4782 	10281 	prologue run as user in interactive job
4801 	10355 	pbsnodes -v produces results when used without -a
4869 	10558 	Memory leak in licensing code in Windows server
4901 	10625 	Incorrect MAIL FROM syntax used when PBS sends mail
4908 	10639 	Losing acl_roots setting after doing qterm -t quick/net start
4909 	10644 	tracejob usage statement doesn't show that we need jobid
4946 	10802 	Scheduler runs multi-cpu job when it would exceed max_load
4970 	10863 	Mom doesn't notice a line removal from her config file when HUP-ed.
4997 	10946 	HP MPI and preemptions: some jobs fail to resume
5020 	11035 	req_reject error for jobs from MOM while running 10,000 jobs load 
		test with formula and eligible_tim
5030 	11076 	cp behavior differs from rcp/scp when sandbox=PRIVATE
		  Workaround:
		  When using sandbox=PRIVATE, use an absolute path in
		  the "remote_path" portion, where the format is
		  local_path@hostname:remote_path
5047 	11124 	restart script fails because PBS_NODEFILE is not defined
		  The file containing the list of nodes allocated to a
		  job (the path given by the environment variable
		  PBS_NODEFILE) is not available when the action script
		  for "restart" is run.  
		  Workaround:
		  If the "checkpoint" action script saves a private copy
		  of PBS_NODEFILE, this can be used by the "restart" script
		  in place of the one provided by PBS.
5070 	11186 	Unable to use no_multinode_jobs on multi-vnoded system
5076 	11204 	pbs_rcp unable to stage-in/stage-out the large file ( > 4 GiB )
5089 	11244 	SYSLOG levels do not match documentation
5140 	11364 	Reservation starts and runs after required resources made unavailable
5194 	11477 	illegal qsub syntax "-l mem=200mb:ncpus=2" not rejected
5209 	11543 	server stops responding after deleting job array with 1024 subjobs
5217 	11566 	kerberized PBS not pulling the jobs from peer servers
5225 	11589 	Using tm interface within a job results in an ALPS reservation that 
		doesn't get deleted
5292 	11849 	parsing of quoted lines in sched_config should tolerate line breaks
5326 	11976 	Problem with importing environment into job (qsub -V) when a shell 
		function is defined
5337 	12012 	advance reservations don't work well unless "ncpus" is removed from 
		the scheduler's resource list
14643 		FREQ=HOURLY in combination with PBS_TZID=Asia/Calcutta causes Bad 
		time specification(s) error
		  Workaround:
		  On later Linux updates, the system's zoneinfo files
		  may have some countries represented under a different
		  name than previous releases. In this case,
		  Asia/Calcutta was replaced by Asia/Kolkata. It is
		  best to use Linux's tzselect utility to determine
		  what to set PBS_TZID to.
14816 		ICE 9.2.3 Slow start PBS server with a lot of down nodes
		  Workaround: all nodes have to be up.
16011 		user vs Admin permission problems on Windows
16127 		Address warnings from VS 2008 related to secure functions
16938 		mom directories not created if PBS_START_MOM=0 first time PBS started 
		after install
		  Workaround: Rerun /usr/pbs/default/etc/pbs_habitat
17564 		Scheduler does not pay attention to pbs_license_max
17942 		Tunable Formula is overridden by fairshare
		  Workaround: use one or the other, not both
17964 		Standing reservations are not adjusting to Daylight Savings Time
		  This release note impacts only those sites that meet the following
		  criteria:
		  1) PBS Professional is installed on a non-Linux operating system.
		  2) OS timezone information files that have not been updated after 2008.
		  3) The complex where PBS Professional is installed is in a
		  geographical location that underwent a change in the timezone rules,
		  e.g., a change in the start and end time of daylight savings time.
		  The computation of standing reservation occurrences start times may be
		  scheduled at a different time than expected. The difference should
		  reflect the discrepancy between the old and new timezone rule for the
		  given location.
18147 		Job comment incorrect for finished jobs
18585 		Post Installation Fails on Windows
		  Workaround: Creating and enabling a 'workq' as well as setting 
		  scheduling to 'true' will resolve this issue as these are primary 
		  post installation steps.
18854 		qmgr does not throw error when it can't write to $PBS_HOME
18863 		incorrect error message for unauthorized users	
19442 		comma's in default_qsub_arguments value are lost when 
		qmgr -c 'p s' is redirected back to qmgr
		  Workaround: create a wrapper for qsub that adds the default values
20605 		Typos in source code comments
21058 		Job fails to run: could not chdir to %USERPROFILE%\My Documents\PBS Pro
		  Workaround:
		  Assign user an explicit home directory so that MOM doesn't 
		  use local home directory
21176 		Job_Name incorrectly listed as python string in pbs_job_attributes man page
175540 		mom timeouts with lots of -v vars and interactive jobs
176691 		Language lawyer issue with 10.1 sources
176935 		Server restart modifies node comment
176936 		Macro use causes recalculation
176943 		mom sometimes requires restart to read mom_priv/config file properly
177354 		rerunnable also misspelled as a hooks attribute
177714 		Changes from SPID 17731 not propagated to qselect command
178296 		scheduler doesn't warn admins (even via log) when dedicated time file 
		has errors
179122 		PBS Server crash on P5 with HPS and P6 with IB
179313 		tracejob command not up to date with filtering levels
180671 		Implement better check to confirm primary and secondary aren't both active
181840 		Job array in state 'B' can have misleading comment
181841 		tk.tcl not being found when running xpbsmon on SLES 11
		  Workaround:
		  Under SLES11, you'll need to manually update the following 2 files:
		  /opt/pbs/default/tcltk/lib/tk8.3/listbox.tcl: 
		  comment out lines 182-184 to look like:
		  #bind Listbox <MouseWheel> {
		  #    %W yview scroll [expr {- (%D / 120) * 4}] units
		  #}
		  /opt/pbs/default/tcltk/lib/tk8.3/text.tcl: 
		  comment out lines 464-466 to look like:
		  #bind Text <MouseWheel> {
		  #    %W yview scroll [expr {- (%D / 120) * 4}] units
		  #}
181889 		printjob fails with 'malloc error' when used in prologue script.
182458 		reject(errmsg) in provisioning hook does not print errmsg to server log
182540 		Interactive jobs terminating prematurely on Cray
184664 		pbs_rsub -lhost=<unknown host> -R <start time> -E <end time> does not 
		throw error
185376 		pbs_server should not send email containing bare "LF"
185580 		strict_ordering doesn't keep jobs from running on cpus freed due to 
		preepmtion
185847 		All jobs in queue accrue eligible time even when there are more than 
		the limit
185854 		Licenses do not match running jobs
186870 		qsub silent abort on '#' character
187236 		qsub: illegal -W value (":")
187393 		Stagein is deleting the folder in which the files are being staged in 
		PBS on 64 bit Vista
190589 		Overlay upgrade causes breaks wrapped MPI executables functionality
190646 		PBSPro node grouping feature and queue=<queue> problem
191079 		$kbd_idle & HM macros not cooperating
		  Workaround:
		  1. Open up HM
		  2. Go to Preferences
		  3. Go to Keyboard Settings
		  4. Go to Shortcut key and select Shift
		  5. Go to the Function keys and re-assign to keys that work.
193136 		Insufficient log message format
194878 		If server_dyn_res script never returns, scheduler will hang
		  Workaround:
		  If a server_dyn_res script has a possibility of
		  hanging, admins should take extra care to guard
		  their scripts with alarms.
196917 		Cycle harvesting is not working on windows 7 system
199027 		peer scheduled job schedselect does not reflect what used to run job
199651 		Using qsub -v and -V together
200017 		cosmetic error in qstat -f output when estimated.* attributes are present
200790 		Job submission being refused because of PBS_ibwins hook
		  Workaround:
		  Disable PBS_ibwins hook, by moving away PBS_ibwins.*
		  files from "$PBS_HOME/server_priv/hooks".
200803 		pbs_idled build attempted even though no header files present
202852 		error messages must go to standard error
203500 		Warning compiling with -Wall
203904 		Unevaluated Variable "Failed to run: Job %s has finished (15139)" 
		log message
205435 		Race condition with qsub command line and #PBS directives in job script.
		  Workaround: 
		  Use command-line options to qsub
205525 		PBS released alps reservation before running the batch epilogue 
206420 		placement sets smaller than one host + exclhost result in jobs not 
		being assigned full hosts
		being assigned full hosts
		  The exclhost feature should not be used with
		  placement sets smaller than a host.  If the two are
		  used together it may lead to jobs requesting
		  exclhost sharing hosts.
206488 		Restarting pbs_server causes duplicate accounting records for reservations
206538 		Inconsistent behavior of "qsub--" on UNIX and Linux platforms
206823 		Error in calls to disrcs()
208133 		An attribute/resource silently fails to get set in a modifyjob hook 
		when job is already running.
208804 		Permission of /tmp on MOM
211357 		unsupported warning message does not appears when installing opensuse 11.x
213008 		default log level is not actually used 
213451 		Windows accounting logs missing resources_used.ncpus in E record
218347 		Support comma character in file names in stagin directives
218412 		Estimated start time stays set when job is held
218444 		Need to include Python setuptools
219238 		Job final report on resource usage from qstat does not match 
		accounting record.
219839 		duplicate BLOCK host / port message
220001 		module(s) we're missing in pbs_python causing the import 
		hashlib, no _md5 erro
		  Workarounds (There are two):
		  1) We're missing the hashlib.py* files and _hashlib.so files.
		  Copy these from another python installation:
		     # cp /usr/lib64/python2.6/hashlib* /opt/pbs/default/lib/python/python2.5/
		     # cp /usr/lib64/python2.6/lib-dynload/_hashlib.so /opt/pbs/default/lib/python/python2.5/shared/
		  You have to do something similar for optparse and textwrap.  There may not be binaries.
		     # cp /usr/lib64/python2.6/optparse.* /opt/pbs/default/lib/python/python2.5/
		     # cp /usr/lib64/python2.6/textwrap.* /opt/pbs/default/lib/python/python2.5/
		  2) Ugly, but works in a hook:
		     import sys
		     sys.path = ['', '/usr/lib/python26.zip', '/usr/lib64/python2.6', 
		     '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', 
		     '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', 
		     '/usr/lib64/python2.6/site-packages', '/usr/local/lib64/python2.6/site-packages', 
		     '/usr/local/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages', 
		     '/opt/pbs/default/python/lib/python25.zip', '/opt/pbs/default/python/lib/python2.5', 
		     '/opt/pbs/default/python/lib/python2.5/plat-linux2', '/opt/pbs/default/python/lib/python2.5/lib-tk', 
		     '/opt/pbs/default/python/lib/python2.5/lib-dynload', '/opt/pbs/default/python/lib/python2.5/site-packages']
		     import pbs
		     import hashlib
		  The paths on your system may be different (depending on the system python version 
		  and PBS install location), so here is what we did. We went into the 
		  system Python interpreter and did:
		     Import sys
		     print sys.path
		  And did the same in /opt/pbs/default/python/bin/python. The sys.path = line is then 
		  just [output_from_system_python, output_frompbs_python].  
220009 		exec_host and exec_vnode are cleared when an array subjob enters 
		the X state
220159 		Possible logic error in match_string_array()
220665 		accounting value for cput very high
223016 		When "node_group_enable" is True, "queue" node property is not honored.
		  Workaround:
		  Use "resources_available" to route jobs to nodes.
223365 		Job array subjobs should have job_state=H(eld) after 
		failing/requeueing 20 times.
225095 		node sorting based on an indirect resource not working as expected
		  Workaround: 
		  use qmgr to set indirect resources
225932 		Inappropriate memory handling followup
228242 		Windows Installer not checking for NTFS Dependency, and PBS Fails to start
229718 		hook throwing time out exception
231220 		Serious scaling issues in pbs_server, continued
231266 		Running jobs in reservations may cause top jobs to get start 
		times in the past
231347 		Slow scheduling cycle and qmove/qalter might cause jobs running 
		on nodes with incompatible resources
231781 		'-V' or '-v' option in default_qsub_arguments not working in PBS 11.x
232709 		qsig appears to signal more than just a session leader
234258 		tracejob results not in chronological order
234466 		Takes way too long to add many nodes to database
		  Workaround:
		  db_tool delivered in fall of 2011; contact Support for more info
234776 		incorrect placement set metadata when an indirect resource is 
		defined on a node in the placement set
235290 		naccelerator value is represented in size
235326 		bug with -k oe qsub option?
235779 		Scheduling error occurs: Could not find node solution in 
		run_update_resresv()
236010 		pbs_licensing_dir needs to be removed after failed installation, 
		before new installation
236011 		Attributes w/ unknown values in hook should be set to "None", not ""
236206 		Possible use of uninitialized pointer


----------------------------------------------------------------------------------
Third Party Legal Notices

The following copyrights apply:

/*
**  avltree - AVL index routines by Gregory Tseytin.
**
**
**    Copyright (c) 2000 Gregory Tseytin <tseyting@acm.org>
**      All rights reserved.
**
**    Redistribution and use in source and binary forms, with or without
**    modification, are permitted provided that the following conditions
**    are met:
**    1. Redistributions of source code must retain the above copyright
**       notice, this list of conditions and the following disclaimer as
**       the first lines of this file unmodified.
**    2. Redistributions in binary form must reproduce the above copyright
**       notice, this list of conditions and the following disclaimer in the
**       documentation and/or other materials provided with the distribution.
**
**    THIS SOFTWARE IS PROVIDED BY Gregory Tseytin ``AS IS'' AND ANY EXPRESS OR
**    IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
**    OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
**    IN NO EVENT SHALL Gregory Tseytin BE LIABLE FOR ANY DIRECT, INDIRECT,
**    INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
**    NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
**    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
**    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
**    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
**    THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
**
**
*/

/*
 * Copyright (c) 1988, 1993
 *      The Regents of the University of California.  All rights reserved.
 *
 * This code is derived from software written by Ken Arnold and
 * published in UNIX Review, Vol. 6, No. 8.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 * 3. All advertising materials mentioning features or use of this software
 *    must display the following acknowledgement:
 *      This product includes software developed by the University of
 *      California, Berkeley and its contributors.
 * 4. Neither the name of the University nor the names of its contributors
 *    may be used to endorse or promote products derived from this software
 *    without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
 * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
 * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
 * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
 * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 * SUCH DAMAGE.
 *
 * $FreeBSD: src/lib/libc/gen/popen.c,v 1.14 2000/01/27 23:06:19 jasone Exp $
 */


/*
* Copyright (c) 2003-2010 Cray Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* - Neither the name Cray Inc. nor the names of its contributors may be
* used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/


License for libical:
-------------------
The contents of this file are subject to the Mozilla Public License
Version 1.1 (the "License"); you may not use this file except in
compliance with the License. You may obtain a copy of the License at
http://www.mozilla.org/MPL/

Software distributed under the License is distributed on an "AS
IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or
implied. See the License for the specific language governing rights
and limitations under the License.

     The Original Code is libical. 

     The Initial Developer of the Original Code is Eric Busboom

     All Rights Reserved. 

     Contributor(s): See individual source files.

-------------------------------------------------------------------------