PBS Professional 10.4 Release Notes 1 June 2010 Copyright (c) 2003-2010 Altair Engineering, Inc. All rights reserved. PBS(TM), PBS Works(TM), PBS GridWorks(R), PBS Professional(R), PBS Analytics(TM), PBS Catalyst(TM), e-Compute(TM), and e-Render(TM) are trademarks of Altair Engineering, Inc. and are protected under U.S. and international laws and treaties. All other marks are the property of their respective owners. ALTAIR ENGINEERING INC. Proprietary and Confidential. Contains Trade Secret Information. Not for use or disclosure outside ALTAIR and its licensed clients. Information contained herein shall not be decompiled, disassembled, duplicated or disclosed in whole or in part for any purpose. Usage of the software is only as explicitly permitted in the end user software license agreement. Copyright notice does not imply publication. For the most recent information, log in to the PBS Professional website, www.pbsgridworks.com with your site ID and password and follow the "Client Login" link, then the "Download Software" link. ------------------------------------------------------------------------- Contents Contact Information Licensing Information Supported Platforms Upgrading Special Notes (Licensing) Special Notes (General) Special Notes (External Interface Changes) Special Notes (Documentation) Special Notes (Hooks) Special Notes (HPC Basic Profile) Special Notes (Windows) Special Notes (SGI) Special Notes (Cray) Special Notes (Solaris) Special Notes (MPI) Special Notes (AIX) Special Notes (Linux) New Features Modifications and Bug Fixes Known Bugs, Errors, and Warnings Third Party Legal Notices ------------------------------------------------------------------------- Contact Information Altair Engineering, Inc. 1820 E. Big Beaver Road Troy, MI 48083-2031 USA www.pbsgridworks.com PBS Sales: pbssales@altair.com 248.614.2400 PBS Support: pbssupport@altair.com 248.614.2425 ------------------------------------------------------------------------- Licensing Information Terms of use for this software are available online at http://www.pbspro.com/UserArea/agreement.html, and are also included in the PBS Professional Administrator's Guide and the PBS Professional User's Guide. PBS uses FLEXlm for licensing. The FLEXlm license server must be installed and configured before PBS is installed and configured for permanent use. See the PBS Professional Installation & Upgrade Guide and the Altair License Management System Installation Guide for additional information. ------------------------------------------------------------------------- Supported Platforms For these platforms, you can download a FLEXlm license server and/or all PBS components. HP-UX 11.22 and later on Itanium2 IBM AIX 5.3 with v5.3.7.0 or later libpthreads library (bos.rte.libpthreads) on POWER architectures IBM AIX 6.x on POWER architectures Red Hat Enterprise Linux 4 and 5 (AS, ES, WS) on x86, x86_64, and ia64 SGI Altix with SGI ProPack 5 and 6 on ia64 and x86_64 Sun Solaris 10 on SPARC and x86_64 SuSE SLES 9 on x86, x86_64, and ia64 Windows XP Professional, SP1 and later, on x86 and x86_64 Windows Server 2003 on x86 andx86_64 Windows Server 2008 on x86 and x86_64 Windows Vista on x86 and x86_64 Unicos/lc 2.0 on Cray XT3 and XT4 Unicos/lc 2.1 on Cray XT4, XT5, XT5h, and XT6 For these platforms, you can download all PBS components. SGI Altix ICE/XE with SGI ProPack 5 and 6 on x86_64 SGI UV with SGI ProPack 7 and SLES 11 on x86_64 SuSE SLES 10, 11 on x86, x86_64, and ia64 MPI PBS runs jobs with all known implementations of MPI. PBS is integrated more tightly with some versions, providing support for resource tracking and job cleanup after a failure. These are listed here: MPICH 1.2.5, 1.2.6 on Linux 2.4 on x86, AMD64, EM64T, Itanium2 MPICH 1.2.5, 1.2.6 on Linux 2.6 on x86, AMD64, EM64T MPICH 1.2.7 on x86 Linux MPICH2 1.0.3, 1.0.5, 1.0.7 on Linux MPICH-GM (mpich-1.2.6.14b) on Linux IBM POE on AIX 5.x and 6.x, including HPS support Intel MPI 2.0.022 on Linux HP MPI 1.08.03 on HP-UX 11 on Itanium 2 HP MPI 2.0.0 on Linux 2.4 & 2.6 on x86, AMD64, EM64T, Itanium2 LAM/MPI 6.5.9, 7.0.6, 7.1.1 on Linux 2.4 and 2.6 on x86, AMD64, EM64T and Itanium2 SGI MPI (MPT) on Linux on Altix/x86_64 and XE SGI MPI (MPT) on Linux on Altix/Itanium2 SGI MPI (MPT) over Infiniband MVAPICH 1.2.7 and 2.0 on Linux Support for CSA: ProPack version OS version Architecture libcsa version libjob version ----------------------------------------------------------------------- PP5 SLES10 ia64 libcsa.so.1 libjob.so PP5 SLES10 x86_64 libcsa.so.2 libjob.so.2 PP6 SLES10 x86_64 libcsa.so.3 libjob.so.2 PP6 SLES10 ia64 libcsa.so.3 libjob.so.2 PP6 SLES11 ia64 libcsa.so.4 libjob.so.2 Support for User Space mode on InfiniBand switches: Platform: AIX 5.3 on Power6 Parallel Environment: 4.3.2.5 libnrt: 2.4.5.5 Switches: Qlogic InfiniBand SilverStorm 9120 Connection to switch: Compute nodes connect to each Qlogic switch with 2 networks per HCA adapter. There are 4 links per switch and a total of 8 links per node. Windows Domains: Active Directory Service domains are supported. Windows NT domains are not supported. Mixed mode is not supported. Mixing Windows systems with non-Windows systems in one PBS complex is not supported. Peer scheduling between Windows and non-Windows complexes is not supported. For more specific Windows platform information, see Special Notes (Windows). ------------------------------------------------------------------------- Upgrading 1. PBS uses a FLEXlm license server, which must be installed and configured before PBS is upgraded. See the PBS Professional Installation & Upgrade Guide and the Altair License Management System Installation Guide. 2. It is suggested that you upgrade your FLEXlm license server to the latest version. See the Altair License Management System Installation Guide. 3. To choose between overlay and migration when upgrading to PBS Professional 10.4: If you are running PBS version 5.3 or later, an overlay upgrade is recommended. If you are running a PBS version older than 5.3, a migration upgrade is recommended. PBS on Windows can be upgraded via migration only. ------------------------------------------------------------------------- Special Notes (Licensing) PBS uses a FLEXlm license server, which must be installed and configured before PBS is installed and configured. Follow the instructions in the Altair License Management System Version 9.0 Installation Guide. Chapter 5, Licensing, in the PBS Professional Installation & Upgrade Guide describes how to configure PBS Professional to use the license server. Issue 4695 (Bug 9813): If you are using the server-list redundant license server configuration, note that only the first server will hand out licenses. Make sure that that server has enough licenses for all your virtual CPUs. ------------------------------------------------------------------------- Special Notes (General) 0. Do not perform partial removal of custom resources. It is important to remove custom resources correctly. See the instructions in the PBS Professional Administrator's Guide. 1. Dependencies on Linux shared object libraries: Altair distributes separate PBS packages for different supported Linux distributions. Customers running on SLES and RHEL distributions should take care to use the correct package for their systems. For example, in a complex made up of RHEL 4 and RHEL 5 hosts, the RHEL4 package should be used on the RHEL 4 hosts, and the RHEL5 packages should be used on the RHEL 5 hosts. If PBS Professional 10.2 or later is deployed onto Linux distributions other than the supported SLES and RHEL distributions, there may be runtime problems with dependencies on shared object libraries, typically libssl.so or libcrypto.so. Customers who encounter such problems should first determine which supported distribution is most similar to that installed on their systems. For example, openSUSE 10.x is similar to SLES 10, and CentOs distributions are similar to RHEL. The customer should then try the PBS package that is supported for the similar system. If the runtime dependencies are still not resolved, there are two other approaches that may be tried. Before trying either approach, first run ldd on pbs_mom to get name and version number of libraries that pbs requires. pbs_mom needs to be extracted from the PBS rpm by using the Linux rpm2cpio and cpio commands. Here is an example using an installed SLES 10 x86_64 pbs_mom: shell> ldd /opt/pbs/default/sbin/pbs_mom libdl.so.2 => /lib64/libdl.so.2 (0x00002b23b0a59000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002b23b0b5d000) libssl.so.0.9.8 => /usr/lib64/libssl.so.0.9.8 (0x00002b23b0c96000) libc.so.6 => /lib64/libc.so.6 (0x00002b23b0ddd000) libcrypto.so.0.9.8 => /usr/lib64/libcrypto.so.0.9.8 (0x00002b23b100d000) /lib64/ld-linux-x86-64.so.2 (0x00002b23b093c000) The Linux incompatibilities are generally associated with libcrypto.so, which is required by libssl.so. Having identified the requisite version of libcrypto.so and libssl.so, there are two options: (1) Locate and install an OpenSSL compatibility library that will satisfy the dependency. In the example above, the library needs to satisfy the dependency on libcrypto.so.0.9.8. (2) Download and install a version of OpenSSL that corresponds to the dependency determined above (e.g., libcrypto.so.0.9.8). 2. Version 10.4 requires more disk space than previous versions. Package size of PBS with Python is almost double that of previous versions. For example, under SLES, the size is now 81mb instead of 49mb, and under RHEL, the size is now 42mb instead of 21mb. 3. The default umask behavior has changed in 10.4 due to the fix for SPID 4487. Under 10.2 and earlier versions of PBS, if -Wsandox=PRIVATE was specified, the job's execution directory was created with permissions drwxr-xr-w. With this fix, the default job umask is applied, so unless a different umask is specified for the job's execution directory, it has permission drwx------. To allow read access by Group or Other, it is necessary to specify the appropriate umask when submitting the job. See the PBS Professional User's Guide. 4. Do not use different versions of PBS within the same complex. All machines using the same PBS server should run the same version of PBS. Do not mix major or minor versions. For example, do not run 10.4 and 10.2 in the same complex. 5. Do not use qmgr to set or change the vnode sharing attribute, or a vnode's ncpus, vmem or mem. Instead, create a new vnode definition file and use "pbs_mom -s insert" to add the file. Then restart the MOM. See the PBS Professional Administrator's Guide, section 2.1, "The qmgr Command". 6. All printable characters are allowed in a resource value, except for comma, space, colon, quotes, and ampersand, but are subject to some restrictions such as those imposed by select and place. Also, if these resource values are used as pathnames, they are subject to the restrictions of the operating system. 7. Issue 5909 (Bug 5375): pbs_mom emits error "unable to get my domain name" Starting with release 7.0, pbs_mom may emit the message "pbs_mom: pbs_mom, Warning: unable to get my domain name" when PBS is started. The mere presence of this warning on a system that has been upgraded from a previous release of PBS should not cause concern: the same check was performed in previous releases; the only change is that the warning was suppressed. The actual cause of the warning may be - pbs_mom is unable to find the fully-qualified domain name of the host on which it is executing - although the host's name (i.e. the value of uname -n) is fully qualified, the host's native name service has been configured to prefer other name service backends over DNS, and the preferred service uses an unqualified name for the host. For example, systems using a name service switch configuration file, nsswitch.conf, may have a line of the form hosts: files dns in conjunction with a hosts file entry in which the host's name is not fully qualified. To address the problem, administrators should ensure that the result of converting the host's name (obtained via gethostname()) to an address (using gethostbyname()) and converting that address to a name (using gethostbyaddr()) results in the original host name. See the pbs_hostn(8B) manual page for more information. 8. SPID 15770: Change in qdel behavior The -Wdelay option to qdel is silently ignored. The delay between the SIGTERM and SIGKILL signals is determined by each queue's kill_delay attribute. The default value for this attribute is changed to 10 seconds. ------------------------------------------------------------------------- Special Notes (External Interface Changes) Changes ------- 0. New -T option to qstat command shows estimated start time. 1. Server cannot be killed with SIGINT. 2. The pbs-report command will be deprecated. It will be moved to the unsupported directory in a future release. 3. New server and queue resource usage limits are incompatible with old server and queue resource usage limits. See "Managing Resource Usage" in the PBS Professional Administrator's Guide. 4. Enabling job history changes the behavior of dependent jobs. If a job j1 depends on a finished job j2 for which PBS is maintaining history than j1 will go into the held state. If job j1 depends on a finished job j3 that has been purged from the historical records than j1 will be rejected just as in previous versions of PBS where the job was no longer in the system. 5. 8.0: Change to pbsnodes output: "Host =" replaced with "Mom =" 6. The resource arch is only used inside of a select statement. 7. pbs_hostid command no longer used. Use lmhostid. 8. When a job is requeued, a new timeout message may be printed. 9. When a reservation is confirmed, the message printed to the scheduler log is different from previous versions. 10. MOM now exits with an exit value of 1 if an error is detected while reading the configuration file. In previous versions of PBS, MOM did not exit. 11. Some MOM error messages no longer have a leading WARNING flag. 12. Behavior of qdel changed. See Special Notes (General). 13. Behavior of qsub, qstat, qselect, pbs_mom expanded. 14. Change in pbsnodes output: on hyper-threaded Linux systems, pbsnodes now reports the number of processors listed in /proc/cpuinfo. 15. Change to log levels: see the PBS Professional Administrator's Guide for the table titled "PBS Events and Log Event Classes" 16. Ability to submit a job with "-l ", i.e. without giving a value for the resource, will not work in future releases. This behavior was not supported, and will no longer work in future releases. Deprecated scheduler configuration parameters --------------------------------------------- assign_ssinodes cpus_per_ssinode key load_balancing_rr mem_per_ssinode preempt_checkpoint preempt_fairshare preempt_requeue preempt_starving preempt_suspend sort_by sort_priority option to job_sort_key strict_fifo sync_time Obsolete and deprecated resources --------------------------------- The -l nodes=nodespec form is replaced by the -l select= and -l place= statements. The -l resc=rescspec form is replaced by the -l select= statement. Properties are replaced by boolean resources. The resource host is only used inside of a select statement. The resource arch is only used inside of a select statement. The nodect resource is obsolete. The ncpus resource should be used instead. Sites which currently have default values or limits based on nodect should change them to be based on ncpus. The neednodes resource is obsolete. The ssinodes resource is obsolete. The ppn resource is deprecated. The nodes resource is no longer used. Server and Vnode deprecations ----------------------------- The time-shared node type is no longer used. The :ts suffix is obsolete. The cluster node type is no longer used. The lictype vnode attribute is deprecated. The license vnode attribute is deprecated. The server attribute node_pack is deprecated. Command deprecations -------------------- The -a option to the qselect command is deprecated. The -Wdelay=nnnn option to the qdel command is deprecated. The -c option to the pbsnodes command is deprecated. The pbs_hostid command is no longer available. The -a option to the pbs_sched command is deprecated. API deprecations ---------------- pbsrescquery pbs_rescquery avail totpool usepool Altix: no longer used --------------------- Requests for the following values will return a PBSE_RMUNKNOWN error: cpuset_small_mem cpuset_small_ncpus memreserved max_shared_nodes nodersrcs shared_cpusets small_job_spec New server attributes --------------------- reserve_retry_init reserve_retry_cutoff max_queued max_queued_res.RES max_run max_run_res.RES max_run_soft max_run_soft_res.RES job_history_enable job_history_duration max_concurrent_provision backfill_depth (10.4) est_start_time_freq (10.4) New vnode attributes -------------------- hpcbp_enable hpcbp_user_name hpcbp_webservice_address hpcbp_stage_protocol provision_enable current_aoe New reservation attributes -------------------------- reserve_retry New queue attributes -------------------- max_queued max_queued_res.RES max_run max_run_res.RES max_run_soft max_run_soft_res.RES New job attributes ------------------ Exit_status Stageout_status Submit_arguments estimated New job states -------------- F Finished M Moved New vnode states ---------------- provisioning wait-provisioning New MOM configuration variables ------------------------------- cpuset_error_action (10.4) $aix_largepagemode New resources ------------- start_time (10.4) exec_vnode (10.4) ------------------------------------------------------------------------- Special Notes (Documentation) 1. The 10.2 and later release of PBS Professional includes a new book called the PBS Professional Reference Guide. This book contains reference material for PBS Professional, including commands, configuration parameters, and attributes. 2. The title of the PBS Professional External Reference Specification has been changed to "PBS Professional Programmer's Guide". 3. Some manual pages that were reproduced in the PBS Professional Administrator's Guide have been removed from the guide. They can be found in the PBS Professional Reference Guide, and as online manual pages. 4. The pre-9.2 PBS Professional Administrator's Guide has been split into two books: the PBS Professional Installation and Upgrade Guide and the PBS Professional Administrator's Guide. The PBS Professional Installation and Upgrade Guide contains the instructions for pre-installation planning, and for installation and upgrading. ------------------------------------------------------------------------- Special Notes (Hooks) Administrators who are writing provisioning hooks need to be aware that the Python pbs.event().reject() function fails to log the error message when used from a provisioning hook. This function behaves correctly for all other hooks supported by PBS Professional. ------------------------------------------------------------------------- Special Notes (HPC Basic Profile) Supported Platforms for HPC Basic Profile: 1. The PBS server is supported on RHEL4(x86_64) and SLES 10 (x86_64) 2. The HPCBP MOM is supported on RHEL4(x86_64) and SLES 10 (x86_64) 3. Client commands are supported on Linux on x86 and x86_64 Supported commands and features: Some commands and features are not supported with HPC Basic Profile. See the PBS Professional Administrator's Guide. Job submission: Job submission for HPC Basic Profile requires specific syntax. See the PBS Professional User's Guide. Configuration: Using HPC Basic Profile requires specific configuration of PBS. See the PBS Professional Administrator's Guide. ------------------------------------------------------------------------- Special Notes (Windows) 0. PBS path information no longer saved in AUTOEXEC.BAT Any value for PATH saved in AUTOEXEC.BAT may be lost after installation of PBS. If there is any path information that needs to be saved, AUTOEXEC.BAT must be edited by hand after the installation of PBS. PBS path information is no longer saved in AUTOEXEC.BAT. 1. Change in requirements for submitting jobs under Windows Each batch script (.bat) in a job script must have "call" in front of it. For example, MyJob.bat, which calls Step1.bat and Step2.bat should look like this: #PBS -l walltime=1:00:00 call E:\Step1.bat call E:\Step2.bat 2. If the full path name of a PBS command contains spaces, the path name must be quoted. If an argument to a PBS option contains spaces, it must be quoted. Examples: "\Program Files\PBS Pro\exec\bin\qstat" -Bf qsub -W stagein="the_input@hostA:c:\Documents and Settings\pbstest\my_input" 3. Specify only Windows DNS address for Windows Host For any privileged action requested of the server (e.g. qmgr requests), PBS allows it if the requesting user is an admin-type of user (i.e. member of "Domain Admins" or "Administrators), and the requesting host is the local host. PBS compares what it initially thinks is the local host name against the requesting host name. The server may return "Unauthorized Request" for privileged actions requested by a legitimate user. The problem arises from having the local host configured with two or more DNS addresses resolving to different names for the local host name. In particular, a site may have set up a UNIX-driven DNS as the root domain, and a Windows-driven DNS in a subdomain to service domains in an Active Directory environment. Specifying both addresses in the local host's list of DNS addresses may give rise to this problem. So it is important that in this case, you must specify only the Windows DNS address for the Windows host. 4. Requirement for using preemption via group soft limits: All jobs must be submitted using the -W group_list option to qsub. This ensures that the egroup job attribute is set properly, which is required for this type of preemption to work. If the -W group_list option to qsub is not used, the egroup job attribute may not be set properly, and the number of jobs counted as being over or not over their soft limit may be inaccurate. For example if userA belongs to pbsgrpA then the egroup will be pbsgrpA for the following job submission: qsub -W group_list=pbsgrpA job.script 5. Windows MPICH 1.2.5 To run PBS jobs that make use of Windows MPICH 1.2.5, you need to specify the "-localroot" argument when invoking mpirun to get better performance, or to get around the error "failed to communicate with the barrier command". Here is a sample PBS job script: C:\DOCUME~1\pbstest>type job.scr echo begin type %PBS_NODEFILE% "\Program Files\MPICH\mpd\bin\mpirun" -localroot -np 2 -machinefile %PBS_NODEFILE% \winnt\temp\netpipe -reps 3 echo done 6. Windows platform restrictions: Any platform used for the following should be a Server-type of Windows OS: 1. shared PBS_HOME in a server failover configuration 2. location for network-shared user home directories 7. Windows domains PBS Professional is supported on Windows XP Pro and Windows 2003 Server if the domain controller server is configured native. While PBS Professional supports Active Directory Service domains, it does not support Windows NT domains. Running PBS in an environment where the domain controllers are configured in mixed mode is not supported. 8. Installation on 64-bit platform has different path from 32-bit PBS Installation on a 64 bit Windows XP host (x64): PBS Installer will prompt for the destination path with "Drive:\Program Files (x86)\PBS Pro". This drive is the name of the partition/drive where Windows has already been installed. During installation, system-related files are copied to the respective location in the relative path Windows\SysWOW64. 9. Jobs requeued in slow file system The PBS server might mark the execution node as down during file stage operation. For a system on a slow network where file copying takes a long time, MOMs may be marked down when they are up, due to the copy delay. A MOM that is busy waiting for file copy to finish may be marked down by the server. As a result, jobs on this MOM are requeued. To avoid this, set node_fail_requeue to zero. 10. Memory reporting and enforcement Physical memory usage can be reported. Memory limits can be enforced. Virtual memory (vmem) is not reported. ------------------------------------------------------------------------- Special Notes (SGI) 0. cpusets: the behavior of pbs_mom has changed with respect to actions taken when a CPU set setup failure occurs. See the documentation on the cpuset_error_action MoM configuration variable. 1. UV: PBS does not support generating vnode placement set information. 2. Altix: cpuset_small_mem, cpuset_small_ncpus, max_shared_nodes, nodersrcs, shared_cpusets and small_job_spec are no longer used. Requests for those values will return a PBSE_RMUNKNOWN error. 3. There is a known issue with SLES9 where ProPack 4 can crash. This is caused by SuSE bug 85380. It is strongly recommended that you upgrade to the latest version of SLES9, e.g. SLES9 SP4. If you install SP4, make sure that you point to SGI for YOU updates. 4. Issue 5353 (bug 12069) and issue 5354 (bug 12070) changed the behavior of PBS on ICE systems with respect to PBS-generated vnode definitions: previously PBS automatically generated the placement information if running with CPU set support on execution nodes. This was changed to not generate placement automatically, instead providing a script the customer could call to do so. Customers upgrading from 9.2 to 9.2patch or later will encounter the fact that an existing use of pbs_mom.cpuset on an ICE system will have generated placement information but the new one will not. Startup will take a long time. ------------------------------------------------------------------------- Special Notes (Cray) 1. The Cray XT5h system is comprised of both XT5 and X2 compute nodes. A typical qsub request that does not specify the node architecture will cause the job to run on XT5 nodes. For example, qsub -lmppwidth=2 -lmppdepth=3 jobscript.sh If the jobs should run on X2 nodes, the request must be: qsub -lmppwidth=2 -lmppdepth=3 -lmpparch=X2 jobscript.sh Running jobs on a mixture of XT5 and X2 nodes is not supported. 2. For Cray XT only: The number of licenses to allocate to a job is based on whichever is greater: ncpus or (mppwidth*mppdepth). The value of mppdepth defaults to 1, if not specified. Cray-specific Modifications and Bug Fixes for Release 10.4: Issue BZ Title ---------------------------------------------------------------------- 186694 Cray 753938 scheduler crash Cray-specific Modifications and Bug Fixes for Release 10.2: Issue BZ Title ---------------------------------------------------------------------- 5324 11974 Alps reservation still exists for a PBS job after the job is gone 5327 11979 pbs_mom times out ending qsub but ALPS is not aware Cray-specific Modifications and Bug Fixes for Release 10.1: Issue BZ Title ---------------------------------------------------------------------- 15868 PBS/Alps does not respect labels ------------------------------------------------------------------------- Special Notes (Solaris) In order to run the FLEXlm license server on Solaris 10 or later, you must take extra steps which are not described in the documentation. Contact Altair's Support services. ------------------------------------------------------------------------- Special Notes (MPI) 1. The default for starting wrapped mpids is rsh. If you are wrapping HP MPI with the PBS pbs_mpihp, and you want to use something other than rsh, be sure to set PBS_RSHCOMMAND in PBS_HOME/pbs_environment. For example: PBS_RSHCOMMAND=ssh 2. MPD rings As has been observed with mpich2-1.0.3, when starting MPD daemons, only one MPD per user on an execution host can be run; multiple MPD rings can be created per user as long as each participating execution host is in only one ring. Otherwise, jobs with intersecting execution hosts on their MPD rings can interfere with one another. Suggested workaround: Given that only 1 MPD per user on an execution host can be run, do one of the following: 1. Set 1 job per user to run on a PBS node by setting: # qmgr -c "set node max_user_run=1" 2. Force users to submit jobs to run on nodes exclusively using the excl keyword in -l place as in: % qsub -l select=2:ncpus=1:mpiprocs=2 -l place=free:excl job.scr In addition, to set this default job-wide resource on the server, the admin can set: # qmgr -c "set server resources_default.place=free:excl" Note: Workaround 2 prevents other users from using the same node if it has more than 1 resources_available.ncpus. ------------------------------------------------------------------------- Special Notes (AIX) 1. Use of Large Page Mode AIX 5.2 and AIX 5.3 An IBM bug exists that prevented PBS from enabling Large Page Mode for AIX 5.0 through 5.3. The bug reference is: IY80185: SETPCRED() TRUNCATES POINTERS TO 32 BITS IBM has now fixed this bug and provides a Fix Pack for AIX 5.2 and 5.3. The bug does not exist in AIX 6.0 and higher. See: http://www-01.ibm.com/support/docview.wss?uid=isg1IY80185 Before installing/running PBS on AIX 5.2 or 5.3, a Fix Pack must be installed. For AIX 5.2 use AIX APAR IY80185. For AIX 5.3 use AIX APAR IY90555. The program oslevel will tell you the version of AIX that is installed. instfix -ik IY80185 will tell you if you have the patch for AIX 5.2. instfix -ik IY90555 will tell you if you have the patch for AIX 5.3. Contact IBM support for more information on installing Fix Packs. 2. The default PBS_HOME directory for AIX was changed from /usr/spool/PBS to /usr/local/spool/PBS. ------------------------------------------------------------------------- Special Notes (Linux) Altair distributes separate PBS packages for different supported Linux distributions. Customers running on SLES and RHEL distributions should take care to use the correct package for their systems. For example, in a complex made up of RHEL 4 and RHEL 5 hosts, the RHEL4 package should be used on the RHEL 4 hosts, and the RHEL5 packages should be used on the RHEL 5 hosts. If PBS Professional is deployed onto Linux distributions other than the supported SLES and RHEL distributions, there may be runtime problems with dependencies on shared object libraries, typically libssl.so or libcrypto.so. Customers who encounter such problems should first determine which supported distribution is most similar to that installed on their systems. For example, openSUSE 10.x is similar to SLES 10, and CentOs distributions are similar to RHEL. The customer should then try the PBS package that is supported for the similar system. If the runtime dependencies are still not resolved, there are two other approaches that may be tried. Before trying either approach, first run ldd on pbs_mom to get name and version number of libraries that pbs requires. pbs_mom needs to be extracted from the PBS rpm by using the Linux rpm2cpio and cpio commands. Here is an example using an installed SLES 10 x86_64 pbs_mom: shell> ldd /opt/pbs/default/sbin/pbs_mom libdl.so.2 => /lib64/libdl.so.2 (0x00002b23b0a59000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002b23b0b5d000) libssl.so.0.9.8 => /usr/lib64/libssl.so.0.9.8 (0x00002b23b0c96000) libc.so.6 => /lib64/libc.so.6 (0x00002b23b0ddd000) libcrypto.so.0.9.8 => /usr/lib64/libcrypto.so.0.9.8 (0x00002b23b100d000) /lib64/ld-linux-x86-64.so.2 (0x00002b23b093c000) The Linux incompatibilities are generally associated with libcrypto.so, which is required by libssl.so. Having identified the requisite version of libcrypto.so and libssl.so, there are two options: (1) Locate and install an OpenSSL compatibility library that will satisfy the dependency. In the example above, the library needs to satisfy the dependency on libcrypto.so.0.9.8. (2) Download and install a version of OpenSSL that corresponds to the dependency determined above (e.g., libcrypto.so.0.9.8). ------------------------------------------------------------------------- New Features See also chapter 1 of the PBS Professional Administrator's Guide. Note that Bugzilla numbers are no longer the primary bug-tracking tool; instead Issue numbers are used for this. Issues are not listed by OS. New Features in Release 10.4: Issue BZ Title ----- ----- ----------------------------------------------------------- 2819 4822 Query for job scheduling order 2525 4146 Ensure the top N most deserving jobs are not delayed by backfilling 4962 10844 Common cross-platform job execution environment (aka Python Everywhere) 15527 Support SGI Ultraviolet Hardware configuration 182830 log resource usage when job is requeued New Features in Release 10.2: Issue BZ Title ----- ----- ----------------------------------------------------------- 4046 7898 Support site-specific allocation management 4783 10290 Support SLES 11 on x86, x86_64, and ia64 5155 11392 Generic OS Provisioning 5293 11850 Support User Space mode on IBM Infiniband switches without LoadLeveler 17731 Checkpoint by walltime interval 18677 Scheduler fails to restart on alarm call & deprecate pbs_sched -a New Features in Release 10.1: Issue BZ Title ----- ----- ----------------------------------------------------------- 1646 2424 Extend client commands (e.g., qstat, qdel) to track jobs qmoved to other servers by default 1866 2807 Add "per named user" resource limits (like max_user_res) 2626 4387 Add "per named group" resource limits (for ncpus) 3007 5109 Limit specific users/groups on number of jobs/resources (server & queue level) 3163 5671 create "max_group_res_queueable" attribute 3614 6822 Add ability to retrieve information about finished jobs 4054 7911 Make reservations more fault tolerant 4241 8449 Ability to specify command to run for interactive jobs (qsub -I) 4462 9172 Finer Granularity Control on Users/Groups 4526 9349 Metascheduling to Windows HPC Server 2008 via OGSA HPC Basic Profile 4753 10088 add command arguments to qsub 4998 10957 Add ability for sites to give certain users higher limits than the general limits. 5118 11315 Support Windows Server 2008 on AMD64 and EM64T (x86_64) 18022 Allow epilogue to requeue job for restart from a checkpoint file New Features in Release 10.0: Issue BZ Title ----- ----- ----------------------------------------------------------- 1879 2835 Support Windows XP Professional and 2003 Server on AMD64 and EM64T 2245 3599 Site-specific hook for additional checks at submit time 2583 4288 Site-specific hook to qsub to modify args/input parameters 2799 4791 Ensure "Large" jobs get started (without "starving") 3657 6950 Site-specific hook for pbs_rsub 4464 9176 Install PBS into version defined sub-directories 5074 11201 Support ProPack 6 on SGI Altix Itanium, XE, and ICE platforms New Features in Release 9.2: ID Summary ----- ----------------------------------------------------------------- 2337 New eligible wait time for scheduling (and starving) 3682 make Linux install package (rpm) relocatable 4899 Ability to create standing reservations 5157 Support Per-job Staging and Execution Directories 8614 Wild cards in file staging commands 9971 Supporting ProPack 5 on SGI Altix ICE/XE clusters 10258 Support IBM AIX 6 10614 Restrict user-level access to a job's requested resources 10648 Add parenthesis, unary -/+, and division operations to tunable formula 11040 IBM AIX 6.1 on Power 4, 5, and 6 architectures New Features in Release 9.1: ID Summary ---- ----------------------------------------------------------------- 6829 Use tunable formula to compute job priorities 9053 add logging to pbs_mom -s operations 9138 Support SLES 10 SP1 on x86, x86_64, & ia64 New Features in Release 9.0: ID Summary ---- ----------------------------------------------------------------- 3786 Ability to get version info for all executables (and files) 3812 Implement Units/Token-based License System 4915 MPI Integration: IBM POE on HPS (US mode) 5107 Support a single user belonging to two fairshare groups 5219 MPI Integration: MVAPICH for InfiniBand on Linux on x86, x86_64, IA64 5533 pbsadmin must not be a member of "Domain Admins" 6088 Allow additional characters in resource value in nodes file 6109 Be able to sort nodes by available (as in unused) resources 7071 Ability to change limits in RPP. 7569 Allocate "unset" resources as if the value was set to 0 8139 difficult to have back-slash as a character in a resource value 8288 Support RHEL 5 AS, ES, WS, on x86, x86_64, ia64, and PowerPC 8655 MOM Redesign for Large Altix Systems 9090 Mom Support for Large Page Mode on AIX 9282 Scheduler should emit a debug message if it's primetime or not ------------------------------------------------------------------------- Modifications and Bug Fixes for Release 10.4: Issue BZ Title ------------------------------------------------------------------------- 2339 3805 qmgr print server prints reservation queues 3709 7080 Scheduler may fail to run large jobs in multi-resv environment 4487 9245 permissions on tmpdir differ between Moms 20989 Job deleted due to cpuset failure creation should be requeued 175772 job_sort_key: "preempt_priority HIGH" doesn't sort as expected 176947 Use of uninitialized variable in pbs_rdel 177918 problems with limits.c 179551 missing quotes in pbs_mpihp causes job run failures 179652 node sorting not working within a reservation 180474 File staging is not happening with stageout having directory as destination path 180672 job walltime is wrong when suspended job is deleted 181383 pbsnodes command returns 'NULL' as output 181986 jobs rejected instead of queued in route queue when execution queue's max_queued hit 182350 Memory leak in parse_at_list() 183262 PBSPro 10.1 server crash 183584 server can reject MOM RESET message 183649 qmoved job array with 2 subjobs has subjobs incorrectly marked as finished (X) 184068 Walltime killed message 184912 PBS no longer reports if job killed because memory limit exceeded in stderr 185336 Emerald RC3 mpplables ignored w/ mppdepth 185365 MOMs not reconnecting properly to server after nodes restarted 186694 Cray 753938 scheduler crash 186785 Security Hole in PBSPro Mom /var/spool/PBS/spool 187149 Job messages repeat when job deleted from routine queue 187306 Compilation error when PBS source not configured with python (BZ 14319) 190584 HP MPI wrapper does not support mpirun syntax 191132 Submit a job with space in master file name is not working in windows 2008 64 bit machine 191875 Scheduler looping and not scheduling jobs ------------------------------------------------------------------------- Modifications and Bug Fixes for Release 10.2: Issue BZ Title ------------------------------------------------------------------------- 2226 3542 log file entries for SCP events have trailing "^M"s 4078 7992 scheduler crashes when placement resource value appears multiple times 4472 9196 reservation job gets "resource unavailable" - clean up in progress 5142 11370 rcp supplied with PBS does not accept "." in user names 5250 11658 Log message for missed occurrences should be improved 5324 11974 Alps reservation still exists for a PBS job after the job is gone 5327 11979 pbs_mom times out ending qsub but ALPS is not aware 14717 qrun/qmoved jobs shows Not running in qstat for state R 16155 PBS init script fails to touch "/var/lock/subsys/pbs" on RHEL systems. 16540 Old node comment returns after server restart 17905 Invalid chars in vnodedef file can prevent server from starting 18677 scheduler fails to restart on alarm call 18890 qselect -s option is ignored for finished subjobs 19138 Dependant jobs do not run after failover occurs 19189 If we set limit on one resource , it is replacing the previous limit set on different resources. 19592 PBS_CONF_FILE may be getting set to muliple entries by PBS 19627 AUTOEXEC.BAT PATH may be deleted 19646 Scheduler receive buffer filling and dropping UDP packets 19905 Server is not recognizing the state of reservation as Degraded when a node is made as stale. 20230 max_queuable will look at finished jobs if job history is turned on 20303 Can't shutdown or restart secondary PBS Server when in a HA configuration. 20366 No scheduler allocations when primary login node is unavailable on Cray XT 20417 Mom rejects job with incorrect walltime request 20479 blocking job never returns if there is a dependancy problem 20511 Order of set_job and prologue execution different between batch and interactive-batch jobs 20561 Unnecessary install directory /opt/pbs/... on Cray boot node 20721 Hook throws exception when multiple managers/operators set w/ same hostname 21373 When staging out files from Job dir, the subdirectories structure is not maintained. 174607 pbs_rsub -H option ignores all but first host in comma-separated list 175641 POE error when running jobs on IBM IB. 175644 Increase string length limit of PBS directives 175769 Core dump from tm_request() in mom_comm.c 176420 Hook for queuejob clobbers PBS_O_WORKDIR 176751 Server doesn't allow unsetting of Resource_List['nodes'] 176918 poe crashing when different version lib used than what pbs was compiled with 177457 Under certain circumstances PBS accepts jobs asking for undefined resources: 177787 Mom kills stagein's when $restrict_user is set. 177911 Server core dump with long list of chunks 178452 Large page mode fails using PBS10.1 for AIX5.3 178718 bad cookie in TM request reveals the correct cookie in MOM log 179048 ALPS reservation not deleted when PBS deletes a job (Cray 747486) 179067 AIX MOM crashes on P6 with IB 180438 Jobs submission fails for "stage-in of files with spaces" only when jobs are submitted to a Linux instance of PBS. 180551 mom core dump when terminate script encounters zero tasks Modifications and Bug Fixes for Release 10.1: Issue BZ Title ------------------------------------------------------------------------------------ 1554 2316 preempting job fails to run when lower prio job can't be preempted 2122 3242 undefined external in pbs_submit API 3011 5114 long host name gives pbs_mom, pbs_sched, and qmgr indigestion 3443 6341 Can add same group multiple times to acl_groups 3701 7054 default PATH given to user jobs may not be sufficient 4159 8200 ImpersonateLoggedOnUser error in preemption test 4209 8371 warnings in rpp.c and popen.c 4262 8541 change error message that appears when there is no usage database 4398 9016 pbs_rmget usage incorrect 4501 9294 Job run ... on hosts should be on "vnodes" 4714 9883 When computer restarted, node shows down. 4822 10428 qselect -s X is not legal even though it is documented. 4865 10549 make over limit enforcement the same action as qdel 5023 11039 jobs won't run if pbs_license_max set to > 2147483647 5034 11087 Server leaks memory during resource assignment and reservation submission 5203 11517 Dispatch of jobs in reservation on multi-node system appears to have incorrect placement 5261 11677 no server logging that showed missed occurence 5305 11895 mppwidth on nons-Cray floods scheduler log 5311 11922 pbs-report gives non-useful error message if run by non-root 5361 12139 PBS jobs not correctly passing signals 14629 Non-cpusetted 9.2 pbs_mom reports incorrect number of physical CPUs on Altix 14762 job with -lselect=0 stalls the scheduling cycle 15350 Misspelled argument in Mom config file can result in Mom crashing executing the script 15382 pbs-report, account field incorrectly set to date 15437 job state mismatch cpuset cleanup problem/cpuset collision 15438 cpuset cleanup problem 15464 Scheduler simulation fails if a vnode associated to a reservation is unavailable 15497 Race condition in end of job processing can cause Mom to not be told to delete job/cpuset 15974 Minor problem with qstat and long comments 16025 scheduler needs to be more trusting 16389 Installer does not recognize Altix systems correctly for RHEL and SLES 16836 -W sandbox=private doesn't work with job arrays 16838 typo in init script causes limits not to be set back to original values 16850 ulimit manipulation in init.d/pbs should not be SGI specific 16933 Reservation with associated "place" not recovered after server restart 16940 import a base64 python hook as test crashes pbs server daemon 16976 pbs server segfaults while submitting a job with -V option and a queuejob hook is enabled 17456 PBS mom detects only half of the physical CPUs 17580 PBS' use of xcopy in Windows XP fails in German version 17635 Hooks mangling incoming environment variables that contain commas 17822 PBS Server internal error with hooks, caused by two Job_Name attributes being sent to server. 17859 Error 15007 on Windows - ( errno 10038 ) 17906 server crashes if invalid vnode spec in qrun -H 17999 PBS_NODEFILE removed after qhold initiate checkpoint-abort of a running job 18019 negative ncpus request or select with no ncpus (and no default) causes sched crash 18086 Server segfaults when executing certain hooks 18254 mom setting incorrect pcpus value on windows system. 18387 server Crash on Windows when job is submitted 18400 pbs_mom seg-faults with numerous job start errors 18419 server can send jobs to nodes that are rejecting requests from sister nodes 18735 shared and exclusive job in reservation assigned to same blade 18818 Hooks crashing server 18825 Server crash when hook enabled 18853 Non-rerunnable jobs that cannot be started are deleted rather retried 18877 PBS hook causing server to crash. 18975 Queued jobs with exec host Modifications and Bug Fixes for Release 10.0: Issue BZ Title --------------------------------------------------------------------------------------- 3048 5244 Slowness while installing PBS (optimization issue) 3541 6638 kill_delay queue attribute has no effect on qdel 3586 6738 after a password change, user's password doesn't work on node 4789 10317 File stage-in fails for read-only files 4865 10549 make over limit enforcement the same action as qdel 4905 10631 job script with "for /f" cmd construct does not work with qsub 5003 10993 cycle harvesting job does not change state (U) when rerun 5188 11458 The job directory should be deleted if there is a stage-in failure 5195 11485 make tmpdir handling consistent between UNIX and Windows 5208 11542 Backfill job delayed 5243 11645 standing reservation state should change to DJ (Deleting Jobs) between occurrences 5245 11650 skipping first occurrence of a standing reservation leads to out of sequence processing of server's work tasks 5246 11651 job submitted to a confirmed reservation does not run if reservation overlaps another reservation 5247 11652 job did not run after submitting a confirmed standing reservation 5255 11665 pbs_command hang up, after the server restart after standing reservation ends 5260 11676 processing deletion of running jobs at end of occurrence may corrupt memory 5265 11683 check if occurrence duration exceeds reservation duration is erroneous 5295 11860 For Cray XT, license jobs based on mppwidth*mppdepth 5305 11895 mppwidth on nons-Cray floods scheduler log 5313 11925 patch to handle requests for MPP resources with advance reservations 5318 11943 Scheduler takes very long to consider job with malformed select statement 5325 11975 backfilling with strict ordering is not honoring walltime requests 5329 11986 Scheduler does not always honor backfill and will schedule other jobs instead 5333 11991 PBS Reservations are considered endless, problems with back to back reservations 5336 12011 Jobs are held in "H" status after several ALPS errors. 5339 12014 patch to handle requests for MPP resources with advance reservations 5340 12015 Scheduler does not always honor backfill and will schedule other jobs instead 5341 12016 PBS Reservations are considered endless, problems with back to back reservations 5342 12017 Jobs are held in "H" status after several ALPS errors 5346 12029 High priority jobs unable to run due to backfill problem 5355 12075 Advance reservations are not being respected 14492 pbs_probe complained about missing /usr/local/pbs/tcltk/include, 14516 mom segfaults while dereferencing a null pointer in a unusual path of job clean-up 14518 mom segfaults while dereferencing a null pointer in a unusual path of job clean-up 14546 Advance reservation can be oversubscribed 14556 Jobs in reservations hang 14562 Scheduler may oversubscribe resources in the presence of a running standing reservation 14595 Server Segfault on Restart with -t hot option 14635 Random mom segfaults 14636 user qdel another users job 14656 Rare security problem on Linux platform [9.2.2 GA version] 14659 root owned processes when ulimit is too low 14671 Rare security problem on Linux platform [8.0.7 GA version] 14674 root owned processes when ulimit is too low 14683 root owned processes when ulimit is too low [9.2.3 version] 14699 Found PBS_MOM issue with PBS latest build on windows 2003. 14709 Scheduler may oversubscribe resources in the presence of a running standing reservation 14710 Minor: Uninitialized variable in pbs_ical.c 14726 Overlay and Migration upgrade are not working with the latest build 14771 Advance reservations are not being respected 14798 more missing curly braces in mpp.c 14835 server leaks memory during regular job and array submission 14844 Order job is not reflecting in the job in UI. 14845 Job delete runs immediately and prevents user for interacting with the UI for a while 14858 need to merge changes made to 8.1.5c to the PBSPro_9_2_branch 14903 Mom went down in file staging load test 14904 Mom went down twice in file staging load test 14950 PBSvnodedefs file being created for SGI ICE without cpusets 14951 PBSvnodedefs file being created for SGI ICE without cpusets 14991 mom segfault upon qdel of a running job 14992 nodal string arrary resource value is being ignored 15007 sched segfaulting, stime on Q jobs 15187 Scheduler consuming all memory 15238 backfill job delayed 15420 backfill job delayed 15443 Scheduler consuming all memory 15579 Scheduler 15016 error message 15596 B record does not contain number of standing reservations and number of standing reservation occure Modifications and Bug Fixes for Release 9.2: ID OS Summary ---- ---- --------------------------------------------------------------- 3505 All tracejob can print wrong log entries operating on >1 job 5148 All Rapid qsub/qstat causes Server to be unresponsive (15007) 6398 All scheduler prints same message if job is over queue or server resource limit 6777 All Scheduler thinks it's primetime when it ended an hour ago 7370 All potential buffer overflow - TMPDIR handling 8393 All Scheduler doesn't do a case insensitive comparison with hostnames 8656 All file descriptor leak if malloc fails 8796 All qdel -Wsuppress_email not accepted on command line 8801 All Confusing error message with incorrect qdel default argument 10489 All multiple sister failures results in multiple obits and delete job 10548 All Clean up all job processes after qdel or walltime enforcement 10574 All Unable to delete a running multinoded job after server restart. 10659 All Scheduler can mis-order chunks in the exec_vnode on sites with multi-noded machines 10790 All on server recovery, job in unknown substate could crash server 10806 All SR# 62744 localhost not processed correctly for qsub -o or -e options 10908 All Jobs preempted by requeue do not release licenses in time 10997 All scheduler still schedules jobs even if license server is down 11002 All scheduler can over subscribe queue resources if resources_assigned is unset 11015 All server_dyn_res ignores command arguments 11143 All Preemption by requeue cause all licenses in use but no jobs run 8369 Linu mom logs warning for multi-core cpu 10848 Linu INSTALL script mkdir /tmp/pbsroot$$ creates a race condition 10849 Linu INSTALL script denial of service via a symlink attack 11195 Linu Server on large ICE system sending out vnode-host-map repeatedly 9110 Wind duplicate environment entry with qsub -V 9913 Wind install of PBS on a standalone host without pbsadmin fails 11093 Wind accounting_logs show alt_id with homedir info with spaces. 10656 Alti SR 61983 Mom can crash when sister takes too long to free cpusets 10847 Alti Qhold in prologue causes server to crash 10699 AIX wrapped poe does not support non-HPS systems 10815 othe Inserting a vnodedefs file adds cpu0 to resources_available.ncpus 11003 othe job_sort_key not commented out in sched_config 11105 othe Init script could make a mistake when trying to find active server during shutdown Modifications and Bug Fixes for Release 9.1: ID OS Summary ---- ---- --------------------------------------------------------------- 5221 All acl_groups not checked when set on a route queue 6337 All Permissions of /etc/pbs.conf are set by umask of installing user 6340 All Deleting a node does not unassociate it from a queue 6580 All qstat does not detect all conflicts in options 7005 All preemption via checkpoint can be incorrectly restarted in a single cycle 9281 All server_dyn_res is not white-space robust 9296 All Disallow illegal values for vnode sharing attribute 9771 All email should be sent out when FLEX gives out expiring notice 9783 All multi-complex PBS sharing 1 FLEX server can over-license jobs 9853 All scheduler crash when reservation is rejected due to not enough licenses 9896 All scheduler's internal exec_vnode can be cut short if exec_vnode has more chunks than vnodes on system 9931 All Checkout of GlobalGridWorks* and GWProfessional not showing 9985 All race condition with SIGCHLD and reading from dynamic script 10046 All Limit of 5 peer queues in Scheduler 10200 All Modify failover primary takeover from secondary procedure 10419 All Server crash when a subjob is force deleted. 10496 All pbs_remsh script doesn't exit after printing usage 9195 Linu Reservation job not started (depending on walltime & other resvs) 10029 Linu Unable to execute PBS HP MPI jobs: can't find PBS_NODEFILE 9942 Alti checkpointed job does not restart after both server and mom shutdown 10311 Alti Call SGI shared memory reporting interfaces less often 6077 AIX AIX DLPAR partitions ncpus detected incorrectly 10038 AIX HPS incorrectly setup when multiple adapters have the same netid 10207 AIX environment space is too small for large HPS machine 10280 AIX wrapped poe does not preserve command line args correctly 10352 AIX IBM hps library modifies Mom's umask Modifications and Bug Fixes for Release 9.0: ID OS Summary ---- ---- --------------------------------------------------------------- 3971 All qstat output fields may be too small for display of possible values 7852 All job freed twice if queue is missing when server restarts 7956 All Oversubscribed nodes / resources_available.ncpus is negative 8447 All Server should do case insensitive host name comparisons 8583 All scheduler crashes when queues have node_group_key and backfilling/preemption 8597 All mom hangs when job with large number of attached tasks exits 8681 All Job wide mem limit not enforced for inherited mem limit 8688 All pbsnodes usage information differs from man page 8689 All pbsnodes -[od] operates on natural vnode rather than host 8691 All pbsnodes -[docr] operates on vnodes 8721 All make server resistant to corruption in tracking file 8722 All double free in server of UPDATE3 message information 8725 All Scheduler thinks a job can never run, even though it could fit statically 8772 All server can reference through freed structure on node_down_requeue 8811 All on Server recovery, node resources not reallocated to jobs 8820 All reply_hello23() job list may be prematurely truncated 8895 All qsub -o option gives short host name in path 8921 All Server license about to expire comment retained on restart 8958 All Starving job incorrectly delayed by backfill jobs 8966 All job (including CPU set) cleanup not done upon RPP failure in im_request() 8980 All Cannot increase walltime limit for multi-node job 9080 All Scheduler crash with job arrays and backfill 9150 All "mem" limit may result in PBSE_BADATVAL on 32 bit system 9160 All place=pack ignored on complexes without non-multivnoded systems 9230 All Node offline state removed when job run on node 9232 All large number of vnodes overflows qmgr limits 9265 All qstat of subjob range of non-existent array can crash server 9503 All Job substate JOB_SUBSTATE_TERM not processed on recovery 9549 All Server not sending vnodemap to Mom when she updates vnodes 9559 All Server and Mom cycling send output back for rerun job 9689 All CR in the accounting logs 9747 All Job wanting mem in bytes will not be broken up across vnodes (and will not run) 9762 All Malformed select error kill pbs_server 3965 Linu Resource tracking for multi-threaded codes using NPTL 6359 Linu Linux on IA64 mom rpm has wrong dependencies 8300 Linu Cannot create vnodes with qmgr "c n mom=mom" syntax 8784 Linu Jobs sent to low priority nodes first. 8902 Linu Adding/Deleting nodes via qmgr takes long time 8433 Wind windows installer not doing case sensitive checking of passwords 8528 Wind email notification on windows isn't flexible and causes mail not to be sent 7771 Alti Unable to delete cpuset at end of job 8606 Alti Jobs fail at start due to cpuset anomalies 9071 Alti memory placement problems with CPU sets and suspend/resume 9114 Alti Vnode set to free if allocated excl but no cpus allocated. 9116 Alti Large node file incomplete after server restart (in failover configuration) 9339 Alti preemption when requesting hosts won't work on multi-vnoded systems 8815 Sola Server crash in catch_child 8744 othe pbs_init.d can cause errors if default grep is nonstandard 8890 othe pbs_init.d script should prefer PBS_MOM_HOME over PBS_HOME 9064 othe pbs_init.d should remove preexisting PBSvnodedefs if not using pbs_mom.cpuset 9767 othe missing man pages in mom-only package ------------------------------------------------------------------------- Known Bugs, Errors, and Warnings This section lists known deficiencies and warnings as well as unresolved problem reports filed by customers. Note that Bugzilla numbers are no longer the primary bug-tracking tool; instead Issue numbers are used for this. Issues are not listed by OS. 1870 2814 Scheduler might miscalculate inner fairshare-tree nodes 1944 2949 PBS windows can't do overlay upgrade 2085 3184 Preemption via requeue could fail (and does under windows) It is possible that the Scheduler will attempt to run a high priority job before a preempted job has been successfully requeued, causing the high priority job to be delayed until the next scheduling cycle. 2181 3427 Epilogue in Windows has issues with passed arguments 2281 3675 Invalid log message says starving jobs make jobs in reservations not run 2290 3708 Server receives an obit storm (many many obits) 2303 3735 License installation prompt (?) gets replaced by a letter during PBSPro installation. 2565 4227 multi-task jobs don't work using tm interface 2740 4659 generic checkpoint may cause held jobs due to pbs_passwd failure If the mom_priv\config file has $restart_transmogrify set to true, and if a task is checkpointed, restarted and checkpointed again, then the session ID passed to the second checkpoint action will be from the new session ID. This will cause pbs_password to fail and the released job to stay in the held state 2921 4972 Server first locks up and then marks all nodes down Workaround: Do NOT manually mark nodes down via qmgr or pbsnodes. Instead, qrerun the jobs, then mark the node offline. 3011 5114 long host name gives pbs_mom, pbs_sched, and qmgr indigestion 3138 5619 ambiguous word size 3142 5626 several versions of addclient() don't add all addresses for given host 3250 5847 Node erroneously marked "down" due to heavy memory use and paging 3351 6071 Scheduler can't handle consecutive holidays which cross year boundary 3360 6099 after a crash, jobs on sisterhood are not cleaned up 3361 6101 MOM blocks on CSA call Workaround: Turn off node_fail_requeue 3362 6102 Round robin to nodes resetting pointer at top of hour 3377 6140 Resources_used differ from MOM and Accounting Logs The accounting information provided to the epilogue script does not match the FINAL accounting information for the job. The cputime given to the epilogue in argument number 7 is less than what is shown by tracejob as the total cputime for the job. 3381 6144 ProPack 4 mom doesn't properly handle change in CPUs after SIGHUP On Altix systems using pbs_mom.cpuset on ProPack 4 or newer, changing the value of cpuset_create_flags in $PBS_HOME/mom_priv/config may result in CPU set creation failures when subsequent jobs are started. To avoid this problem, change the value of cpuset_create_flags and restart (or send a SIGHUP to) pbs_mom.cpuset only when it is idle. 3458 6369 chkconfig inconsistencies 3460 6371 Scheduler doesn't write helpful log if no node is defined 3462 6373 qstat showed walltime after job requeued due to node_fail_requeue 3506 6509 Scheduler not correctly ordering fairshare 3525 6589 PBS startup doesn't address dependencies on remote file systems 3558 6674 server's accounting record end time - start time != job's walltime 3620 6842 Server node file not updated when resource_available.X unset 3648 6908 MOM goes down while trying to run a job 3661 6968 2 duplicate start entries found for a job in accounting log 3662 6973 Scheduler may incorrectly conclude that resources are unavailable 3669 6985 Scheduler doesn't immediately determine a job is not able to be run 3750 7262 error log after calling conn_qsub should not use errno 3768 7303 Resources allocated from multiple vnodes when not required 3816 7388 PBS windows is slow. 3829 7405 Scheduler crash when freeing memory 3990 7763 RPP fails when starting job with 256 nodes 4007 7788 mpi child process not killed when job qdel-ed 4065 7951 no newline in resourcedef file makes server unable to run jobs Workaround: Try not to edit PBS_HOME/server_priv/resourcedef file with "notepad" application but if you do, be sure to explicitly add a newline character at the end of the file. Notepad is notorious for not explicitly adding that newline at the end of the file if it wasn't explicitly typed in. 4071 7966 Unexpected behavior trying to qrun a job requesting 0 chunks. 4088 8030 Server accepts connection before testing "connection table full" 4125 8134 Secondary Server can take over even if Primary running 4162 8206 smp_cluster_dist does not work with node grouping 4165 8213 coding problem in resource_resv.c 4182 8276 cycle harvesting: after qrun, node does not go back to busy 4185 8292 Have RPM not strip pbs binaries 4199 8338 manual qruns during a scheduling cycle are deferred until later 4218 8392 -W stagein directories not deleted or overwritten 4220 8398 Failed generic checkpoint leaves job state indeterminate When a generic checkpoint fails, MOM will not reset the job state correctly. The job will continue to run but any further attempts to checkpoint the job will give the error: "Request invalid for state" and the MOM log will have the message "req_holdjob failed: Checkpoint active". There is no workaround. 4235 8435 /etc/init.d/pbs on solaris gives message "grep: illegal option -q" When starting PBS with the init.d script, a usage message from grep may be output: machine.company.com# /etc/init.d/pbs start Starting PBS grep: illegal option -- q Usage: grep -hblcnsviw pattern file . . . PBS mom PBS sched ... The init.d/pbs script is also called by the INSTALL program. The -q flag is a standard flag for grep but SunOS puts the standard CLIs in /usr/xpg4/bin, not in the default PATH. The version of grep in /usr/bin does not accept the -q flag. If the init.d/pbs script is run where PATH does not have /usr/xpg4/bin before /usr/bin, this message will be output. It does not cause an error and can be ignored. 4254 8498 windows does not show individual task cputime The log messages that show the individual task cputime always report zero under windows. For example: 10/12/2006 17:41:56 M task 00000001 cput= 0:00:00 ... 10/12/2006 17:41:56 M vm9-2 cput= 0:01:12 mem=0kb The one task had cputime of 01:12 but it showed zero in the first message. 4257 8509 Better error when qsub rejects selectspec with old-style resource request in script The following error message: qsub: "-lresource=" cannot be used with "select" or "place", resource is: mem is printed when a job's resource request combines old-style -l or -l nodes with a select or place statement. This can occur when a job is submitted with an old-style resource request in the job script and a select or place statement in the command line. 4261 8532 pbs_rsub -Wqmove doesn't allow default -I option to be changed 4266 8551 Strange interaction between xcopy,move and PBS. Workaround: Do the xcopy as the last line of the batch script. 4304 8662 security patch for 8.0 causes qmsg failure Workaround: This problem applies to BSD and the like including Mac OS (and perhaps Tru64). On those systems, when a file is created, the group ownership of the file is set by the kernel to that of the parent directory regardless of the effective gid of the creating program. The workaround is to add commands to the prologue to find the job's output/error files and do a chgrp command to that of the user (argv[3] to the prologue). Since the prologue has the job's stdout/stderr as the prologue's stdout/stderr, they could write a small C program that looked up the gid associated with the group name, and use the chown(2) call. 4324 8712 scheduler ignores node/queue association when node_group_key set If node_group_key is set at the server level, exclusive node association with queues will be ignored. The workaround is to set node_group_key on those queues to the same value as on the server. 4344 8779 cycle harvesting on some Linux (FedCore 5,SLES10) not working 4346 8785 pbs_rsub not computing start time from an end time and duration 4371 8889 scheduler does not honor -l place=pack when backfilling Workaround: request one chunk with N CPUs, instead of N chunks, each with 1 CPU and place=pack. 4376 8903 INSTALL succeeded on hpux1 with lacking disk space 4377 8904 qrun appears to hang for several seconds 4378 8918 pbs_mpihp not recognizing "-TCP" option. Workaround: Contact Support. 4385 8940 rpm returns error "scriptlet failed" 1. Comment out: #rpm -U --nodeps $rpm Instead, use: rpm -U --nodeps -nopost $rpm 2. Run the postinstall script by hand to finish the installation. 4386 8952 possible bug in make_schedselect() - needs investigation 4407 9027 error not returned when queue type changed and jobs in queue 4452 9147 When mother superior comes up after being down, multi-node jobs are always requeued 4511 9316 pbsrun.intelmpi may not be passing "--rsh" or "-r" options. pbsrun.intelmpi mishandles how -r and --rsh are passed along to the Intel mpirun command. If one always wants mpirun to use ssh instead of rsh, one should edit $INTEL_MPI_HOME/bin64/mpirun.actual (after the pbsrun_wrap has been performed), and change: other_mpdboot_opt="" to: other_mpdboot_opt="--rsh=ssh" After this change the -r or --rsh options must not be specified to the pbsrun.intelmpi command in the job script. 4553 9420 A starving job preempts in one cycle and is preempted in the next - repeat 4571 9461 Can not submit two exclusive reservations into the system 4596 9507 pbsrun MPI jobs run under mpich2-1.0.5 does not finish. Workaround: Edit PBS_EXEC/lib/MPI/pbsrun.mpich2.init so that: replace line: options_to_pbs_attach="-j $PBS_JOBID -s -P" with: options_to_pbs_attach="-j $PBS_JOBID" 4639 9642 Scheduler doesn't correctly backfill around reservations 4640 9651 locate function in server does not work for subjobs 4650 9676 Scheduler doesn't correctly take server dynamic resources into account for jobs in reservations 4655 9688 pbs init.d script not reporting correct pid for secondary server 4659 9708 RPP fails when packets are sent from an unexpected interface 4666 9746 Job in exit state, Mom sending obits 4691 9803 peer scheduling doesn't set peer'd job comment 4706 9858 pcpus shows on all vnodes in output of pbsnodes/qmgr The pbsnodes command displays pcpus in the output for all vnodes. This is cosmetic only and doesn't affect functionality. 4742 10020 Job in routing queue pointing to itself will be routed forever 4782 10281 prologue run as user in interactive job 4801 10355 pbsnodes -v produces results when used without -a 4820 10426 qsub -V doesn't construct environment variables pointing to func PBS' qsub -V does not pass correctly to a running job the environment variables pointing to "functions". For instance, once can define a function vvv in bash as follows: $ cat bash-func-and-pbs vvv () { echo Hello ; } export -f vvv and submitting a job with qsub -V would get syntax error upon encountering the passed vvv environment: bash: vvv: line 2: syntax error: unexpected end of file bash: error importing function definition for `vvv' The workaround is to have the file holding the function definition be read (sourced) in advance inside the job, or in .bashrc or .login. 4848 10511 PBS init script omits possible ypbind dependency 4869 10558 Memory leak in licensing code in Windows server 4900 10624 preemption via requeue can cause node oversubscription 4901 10625 Incorrect MAIL FROM syntax used when PBS sends mail 4908 10639 Losing acl_roots setting after doing qterm -t quick/net start 4909 10644 tracejob usage statement doesn't show that we need jobid 4929 10763 server does not close open file descriptors opened by parent 4946 10802 Scheduler runs multi-cpu job when it would exceed max_load 4958 10837 rpms need dependency to coreutils 4961 10843 PBS erroneously allocates reserved resources from confirmed reservations 4970 10863 Mom doesn't notice a line removal from her config file when HUP-ed. 4997 10946 HP MPI and preemptions: some jobs fail to resume 5020 11035 req_reject error for jobs from MOM while running 10,000 jobs load test with formula and eligible_tim 5030 11076 cp behavior differs from rcp/scp when sandbox=PRIVATE Workaround: When using sandbox=PRIVATE, use an absolute path in the "remote_path" portion, where the format is local_path@hostname:remote_path 5047 11124 restart script fails because PBS_NODEFILE is not defined The file containing the list of nodes allocated to a job (the path given by the environment variable PBS_NODEFILE) is not available when the action script for "restart" is run. Workaround: If the "checkpoint" action script saves a private copy of PBS_NODEFILE, this can be used by the "restart" script in place of the one provided by PBS. 5070 11186 Unable to use no_multinode_jobs on multi-vnoded system 5076 11204 pbs_rcp unable to stage-in/stage-out the large file ( > 4 GiB ) 5089 11244 SYSLOG levels do not match documentation 5140 11364 Reservation starts and runs after required resources made unavailable 5194 11477 illegal qsub syntax "-l mem=200mb:ncpus=2" not rejected 5209 11543 server stops responding after deleting job array with 1024 subjobs 5217 11566 kerberized PBS not pulling the jobs from peer servers 5225 11589 Using tm interface within a job results in an ALPS reservation that doesn't get deleted 5292 11849 parsing of quoted lines in sched_config should tolerate line breaks 5326 11976 Problem with importing environment into job (qsub -V) when a shell function is defined 5337 12012 advance reservations don't work well unless "ncpus" is removed from the scheduler's resource list 14643 FREQ=HOURLY in combination with PBS_TZID=Asia/Calcutta causes Bad time specification(s) error Workaround: On later Linux updates, the system's zoneinfo files may have some countries represented under a different name than previous releases. In this case, Asia/Calcutta was replaced by Asia/Kolkata. It is best to use Linux's tzselect utility to determine what to set PBS_TZID to. 14816 ICE 9.2.3 Slow start PBS server with a lot of down nodes Workaround: all nodes have to be up. 15646 discover of vnode very slow Two workarounds : 1. Stop the server (qterm -t quick -s) Edit the PBS_HOME/server_priv/nodes file, append " sharing=force_excl" without the quotes, but with the space before 'sharing' to each node entry in the file (not to the first two lines of course). Start the server (/etc/init.d/pbs start) or 2. Set the server attribute "default_qsub_arguments to "-lplace=excl", then each job will request exclusive access to the nodes allocated (using qmgr) 15772 Unrecognized PBS_TZID environment variable Submitting a standing reservation with a PBS_TZID environment variable set to a timezone location that can not be understood by PBS Professional, may result in the following error message being displayed: "Unrecognized PBS_TZID environment variable". The environment variable should be set to a timezone location known to PBS Professional. The appropriate zone location of the client can be obtained from the machine on which the PBS server is installed: 1) On Linux platforms: it can be identified using either, the tzselect command (if available), or from the underlying operating system's zone.tab timezone location file, which may be found under /usr/share/zoneinfo/zone.tab 2) On all other platforms: it can be identified from the list of libical supported zoneinfo locations available under $PBS_EXEC/lib/ical/zoneinfo/zones.tab 15955 Code Cleanup and improvement to correctly handle unchecked memory allocations 16011 user vs Admin permission problems on Windows 16127 Address warnings from VS 2008 related to secure functions 16150 acl_resv_users does not a@host Workaround: While documentation says that acl_resv_users may have an @host string in the syntax, admins should only set allowed users by username. 16452 pbs_probe not complaining when it should about file permissions 16938 mom directories not created if PBS_START_MOM=0 first time PBS started after install Workaround: Rerun /usr/pbs/default/etc/pbs_habitat 17564 Scheduler does not pay attention to pbs_license_max 17942 Tunable Formula is overridden by fairshare Workaround: use one or the other, not both 17964 Standing reservations are not adjusting to Daylight Savings Time This release note impacts only those sites that meet the following criteria: 1) PBS Professional is installed on a non-Linux operating system. 2) OS timezone information files that have not been updated after 2008. 3) The complex where PBS Professional is installed is in a geographical location that underwent a change in the timezone rules, e.g., a change in the start and end time of daylight savings time. The computation of standing reservation occurrences start times may be scheduled at a different time than expected. The difference should reflect the discrepancy between the old and new timezone rule for the given location. 18147 Job comment incorrect for finished jobs 18585 Post Installation Fails on Windows Workaround: Creating and enabling a 'workq' as well as setting scheduling to 'true' will resolve this issue as these are primary post installation steps. 18741 MOM segfault 18854 qmgr does not throw error when it can't write to $PBS_HOME 18863 incorrect error message for unauthorized users 19118 Python hooks machinery does not automatically export new scheduler/server limit attributes 19442 Server qsub defaults Workaround: create a wrapper for qsub that adds the default values 19580 Job fails to start due to communication failure with sister MOM 20605 Typos in source code comments 21058 Job fails to run: could not chdir to %USERPROFILE%\My Documents\PBS Pro Workaround: Assign user an explicit home directory so that MOM doesn't use local home directory 21176 Job_Name incorrectly listed as python string in pbs_job_attributes man page 175540 mom timeouts with lots of -v vars and interactive jobs 176691 Language lawyer issue with 10.1 sources 176935 Server restart modifies node comment 176936 Macro use causes recalculation 176943 mom sometimes requires restart to read mom_priv/config file properly 177354 rerunnable also misspelled as a hooks attribute 177714 Changes from SPID 17731 not propagated to qselect command 177790 Failure to recover dependent jobs on PBS server restart in peered environment 178296 scheduler doesn't warn admins (even via log) when dedicated time file has errors 179122 PBS Server crash on P5 with HPS and P6 with IB 179313 tracejob command not up to date with filtering levels 179680 qrun -H command is not provisioning the job requesting for an existing aoe 180321 qdel of job arrays take a long time on AIX HPS and InfiniBand enabled AIX systems cannot use array jobs. There is no workaround. If an array job is submitted by mist\ ake, it should be put on hold (qhold) before deleting it. 180439 several obsolete interfaces appear in 180671 Implement better check to confirm primary and secondary aren't both active 180675 specifying users who are except to log into moms their process are being killed 180913 windows build script does not check compile status 181287 Problem with reservation confirmation when the difference between reservation start time & submission time is very less 181840 Job array in state 'B' can have misleading comment 181841 tk.tcl not being found when running xpbsmon on SLES 11 Workaround: Under SLES11, you'll need to manually update the following 2 files: /opt/pbs/default/tcltk/lib/tk8.3/listbox.tcl: comment out lines 182-184 to look like: #bind Listbox { # %W yview scroll [expr {- (%D / 120) * 4}] units #} /opt/pbs/default/tcltk/lib/tk8.3/text.tcl: comment out lines 464-466 to look like: #bind Text { # %W yview scroll [expr {- (%D / 120) * 4}] units #} 181856 behavior of max_queued_res 181889 printjob fails with 'malloc error' when used in prologue script. 182299 Placement sets are not sorted when asking placement per-job 182458 reject(errmsg) in provisioning hook does not print errmsg to server log 182691 jobs getting blocked when dedicated time in use long before ded time hits 184408 pbs_rsub -W qmove option times out in only 10 seconds 184664 pbs_rsub -lhost= -R -E does not throw error 185121 installation script won't install server on RHEL 5.4 Workaround: edit install script 185376 pbs_server should not send email containing bare "LF" 185580 strict_ordering doesn't keep jobs from running on cpus freed due to preepmtion 185854 Licenses do not match running jobs 186870 qsub silent abort on '#' character 187236 qsub: illegal -W value (":") 187276 mom cleaning job data directory after failed job staging PBS may sometimes remove the job output directory whether or not staging out of files was successful. This happens when stageout starts and then fails. This can happen when there is insufficient space at the stageout destination. It is recommended to make sure that there is sufficient destination space for files to be staged out. 187393 Stagein is deleting the folder in which the files are being staged in PPBS on 64 bit Vista 188358 Suspected bug in simulate code 188834 PBS should use job_create(3) independent of CSA Workaround: Install CSA 190154 qstat shows duplicate job identifiers in its output 190589 Overlay upgrade causes breaks wrapped MPI executables functionality 190646 PBSPro node grouping feature and queue= problem 190939 Incorrect message in tracejob "MPP: No matching MPP host found." 191079 $kbd_idle & HM macros not cooperating Workaround: 1. Open up HM 2. Go to Preferences 3. Go to Keyboard Settings 4. Go to Shortcut key and select Shift 5. Go to the Function keys and re-assign to keys that work. 191080 Bug in currently unused code in resource.c ---------------------------------------------------------------------------------- Third Party Legal Notices The following copyrights apply: /* ** avltree - AVL index routines by Gregory Tseytin. ** ** ** Copyright (c) 2000 Gregory Tseytin ** All rights reserved. ** ** Redistribution and use in source and binary forms, with or without ** modification, are permitted provided that the following conditions ** are met: ** 1. Redistributions of source code must retain the above copyright ** notice, this list of conditions and the following disclaimer as ** the first lines of this file unmodified. ** 2. Redistributions in binary form must reproduce the above copyright ** notice, this list of conditions and the following disclaimer in the ** documentation and/or other materials provided with the distribution. ** ** THIS SOFTWARE IS PROVIDED BY Gregory Tseytin ``AS IS'' AND ANY EXPRESS OR ** IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES ** OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. ** IN NO EVENT SHALL Gregory Tseytin BE LIABLE FOR ANY DIRECT, INDIRECT, ** INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT ** NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, ** DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY ** THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT ** (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF ** THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ** ** */ /* * Copyright (c) 1988, 1993 * The Regents of the University of California. All rights reserved. * * This code is derived from software written by Ken Arnold and * published in UNIX Review, Vol. 6, No. 8. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD: src/lib/libc/gen/popen.c,v 1.14 2000/01/27 23:06:19 jasone Exp $ */ -------------------------------------------------------------------------