MPE/iX Large Files Overview

by Craig Fairchild
Commercial Systems Division

Introduction

This document describes the changes to the MPE/iX operating system as a result of the initial addition of Large File functionality. In this document, Large Files are defined as any file greater than 4GB - 64KB (4,294,901,760 bytes) which was the maximum file size prior to release 6.5. For the sake of brevity, this value is referred to somewhat inaccurately as "4GB" for the rest of this document.

Overview

The addition of Large File support to MPE/iX increases the maximum file size supported by MPE/iX from the former limit of 4GB to 128GB for release 6.5. In the future, additional increases in supported file size can be expected as customer needs demand.

In this first release of Large File support, two types of files are supported for file sizes beyond 4GB. These are ordinary, fixed length record files, and a new type of KSAM file, KSAM64. A phased approach to Large File support has been taken to get Large File functionality to market in the quickest possible time frame. Several customer surveys have indicated that these two file types are the most needed for Large File support.

Large File support of KSAM files requires the addition of yet another KSAM file type. Due to the internal structure of CM KSAM and KSAM XL files, these file types cannot support file sizes beyond 4GB. The implication of this was that in order to meet our customer demand of Large File KSAM support, a new KSAM file type needed to be defined. This new KSAM file type is the KSAM64 file, or file type 7. KSAM64 files have an improved, generalized internal structure that allows them to support file sizes from one record to the maximum object size of 128GB. KSAM64 files are externally identical to KSAMXL files (and support all CM KSAM intrinsic calls as well).

Large File support has been added to MPE/iX as seamlessly as possible. Large File support has been implemented within the OS so that Large Files are available on all HP 3000 models. Large Files are created by simply specifying a file limit which is sufficiently large. This can be done interactively through the BUILD or FILE commands, or programmatically through the FOPEN and HPFOPEN intrinsics. Opening an existing Large File is as transparent as opening any other file. All system intrinsics will work on Large Files in the exact same manner as any normal sized file. As a matter of fact, if your program accesses files through MPE/iX intrinsics, it is highly likely that there will be no need to make any changes at all to operate on Large Files.

A large part of the ease of Large File use lies in the fact that access to files in MPE/iX is done through a record-oriented interface. All interfaces to the OS still deal with signed 32-bit integers to indicate record numbers. This means that there are really two factors that serve to limit maximum file sizes, the maximum number of records in a file, and the maximum object size supported on MPE/iX. The following table shows how the record size of an ordinary fixed-length record file can affect the maximum size of file.

Record Size	Maximum # of Records	Maximum Object Size	Maximum Resulting File Size
1 byte	2,147,483,647	128GB	2GB
40 bytes	2,147,483,647	128GB	80GB
60 bytes	2,147,483,647	128GB	120GB
80bytes	1,717,986,918	128GB	128GB

About this Document

This document is intended to describe all the external functions, features, and changes that have been introduced with Large File support. The document itself is broken into four sections:

Large File Compatibility Issues. This section covers the small number of issues that could arise when accessing Large Files.
Changes to the Command Interpreter. This section describes the enhancements that have been added to several CI commands to support Large Files and KSAM64 files.
New and Enhanced Intrinsics. This section reveals the upgrades to several existing intrinsics for Large File support, as well as introducing a new set of intrinsics to aid in user-mapped file access.
Miscellaneous. This section discusses the impacts of Large Files on various subsystems and utilities, and covers other miscellaneous Large File topics.

Summary of Large File Compatibility Issues

This section covers the compatibility issues that users need to be aware of when dealing with Large Files. In most cases applications and users are unaware that the file that they are operating on is a Large File.

Even though Large Files are highly transparent, there are some rare situations that can produce results that are either incompatible or in error when dealing with Large Files instead of normally-sized files. The following sections highlight and describe the issues that can affect access to Large Files.

User-mapped Access to Files

MPE/iX currently supports two types of user-mapped access to files; short-mapped access and long-mapped access. Short-mapped access provides a 32-bit pointer for a user application to directly access a file. There are several limitations to short-mapped access. For example, a file must be 4MB or less and the total of all short-mapped access files for a process must be less than 6MB. Long-mapped access provides a 64-bit pointer to access a file of any size, up to the previous maximum file size of 4GB.

Problem Description

Due to the way that existing MPE/iX compilers emit instructions to access memory, problems can result when accessing Large Files with a Large File pointer. Long pointers on PA-RISC hardware are defined to have two separate parts; a space ID, and an offset within that space. This also corresponds to the way virtual memory is defined in PA-RISC -- a 64-bit address broken down into the high 32-bits defining the space of the address, and the lower 32-bits defining the offset within that space.

When building instructions to access memory, the space ID must first be loaded into a space register. The offset portion of the address is then loaded into a general register and the space register/general register pair is used to de-reference virtual memory. When adding to the pointer, the general register is incremented and then used in conjunction with the space register that was previously loaded. When the maximum file size supported on the system was 4GB (the size of each space), all valid file offsets were found within a single space, and there was no need to change the space ID.

The advent of Large Files makes it possible for a single file to cross multiple space ID boundaries. This presents a problem for existing code which uses long-mapped access. When the general register holding the offset is incremented to the point that it overflows, logically it should reference the first word in the next space. However, because the space ID is held in a separate space register, the overflow of the general register is lost. The net effect is that access loops back to the beginning of the current space rather than moving on to the next space as intended.

Compatibility Issue

In order to prevent applications from inadvertently stumbling across this data access problem, normal long-mapped access to Large Files has been made illegal. Any attempt to open a Large File for long-mapped access (HPFOPEN option 21) results in an error. In order to open a Large File for user-mapped access, a new large-mapped option must be passed (HPFOPEN option 87). This new large-mapped option works on both Large Files as well as normal-sized files. In programs that wish to continue to access Large Files through user-mapped access, all instances of HPFOPEN calls for long-mapped access (option 21) need to be changed to large-mapped access (option 87). The pointer that is returned is still a 64-bit pointer which is equivalent in every way to the pointer type returned by the long-mapped option.

By specifying this new large-mapped access option, the option assumes that the caller is indicating their awareness of cross space ID problem and ensures that the program logic correctly handles any cross space ID buffer.

Workaround

The workaround to this problem is actually fairly simple. Any access that would normally span a space ID boundary needs to be broken into two separate accesses, one for each side of the space ID boundary. This can be done within the calling application directly, or by calling the new intrinsics described in section 3 of this document. In order to open a Large File for user-mapped access, the new HOP_OPTION_LARGE_MAPPED (87) option must be passed instead of the long-mapped option.

FLABELINFO

Two current items of the FLABELINFO intrinsic are unable to return information on some Large Files. These items are described in the following table:

Item #	Description	New Item #	Description
28	This item returns the size of the file limit of a file in bytes. It returns the value in an unsigned 32-bit integer. For any Large File, an error is returned if this item is called since the true file limit could not be expressed in a 32-bit value.	62	This item returns the number of bytes in the file limit, but instead returns the value in a signed 64-bit integer. This item can be used for all files regardless of whether or not they are Large Files.
58	This item was recently added to MPE/iX to return the number of sectors of disk space currently allocated to a file. This value is returned in a 32-bit signed integer. While this is adequate for all files supported as of 6.5, future release may support file sizes which will exceed the capacity of this item (512 GB).	63	This item forms a replacement for item 58. It also returns the number of sectors allocated to a file, but returns this information in a signed 64-bit integer. Because of this increased size, this item can reliably be used to get this disk space information for any sized file.

32-bit Integer Overflow

There are a number of FFILEINFO, FGETINFO and FLABELINFO items that return information on the maximum file size of a file, or the current size of a file. While each of these items (with the exception noted above) works without change on Large Files, it is possible that programs that perform arithmetic on the values returned from these functions could experience integer overflow traps when dealing with very large files. An example of such an operation is multiplying the number of records in a file by the record size to get an estimation of the total size of a file. With normal-sized files, the result of this multiplication is guaranteed to never overflow an unsigned 32-bit integer. However, with Large Files, the result of this multiplication could be a value up to 128 GB, which could be much larger than a value that can be represented in 32-bits. In such cases, the result of the multiplication would need to be assigned to a 64-bit integer.

Intrinsic	Item # or Parameter	Item Name	Description
FFILEINFO	4	FF_REC_SIZE	A 16-bit signed integer which returns the record size associated with the specified file
FFILEINFO	9	FF_RECORD_PTR	A 32-bit signed integer which returns the record number of the current logical data pointer associated with the specified file number
FFILEINFO	10	FF_EOF_PTR	A 32-bit signed integer which holds the current EOF record number of the specified file
FFILEINFO	11	FF_FILE_LIMIT	A 32-bit signed integer that returns the record number of the file limit of the file (the maximum size that the file may grow to)
FFILEINFO	14	FF_BLK_SIZE	A 16-bit signed integer that returns the block size associated with the specified file
FFILEINFO	67	FF_HPE_REC_SIZE	A 32-bit signed integer that returns the record size associated with the given file
FFILEINFO	68	FF_HPE_BLK_SIZE	A 32-bit signed integer that returns the record size associated with the given file
FLABELINFO	12	FLI_FILE_LIMIT	A 32-bit signed integer that returns the maximum number of records in the specified file
FLABELINFO	19	FLI_RECORD_END_OF_FILE	A 32-bit signed integer that returns the current data pointer record number associated with the given file number
FGETINFO	N/A	lrecptr	A 32-bit signed integer that returns the current data pointer record number associated with the given file number
FGETINFO	N/A	eof	A 32-bit signed integer that returns the current EOF record number
FGETINFO	N/A	filelimit	A 32-bit signed integer that returns the maximum number of records in the specified file

If any of the items listed above are used in mathematical computations, especially when a record number item is multiplied by a record or block size, care must be taken to determine whether or not it is possible to overflow a 32-bit integer.

AIF:OS Impact

The Architected Interfaces for operating system information (AIF:OS) are a set of privileged routines that provide a supported means of accessing and altering internal system data structures. Because of the need to enhance these data structures to provide Large File support, some corresponding changes to a few AIF:OS routines are required.

Each of the affected AIF:OS items follows a similar form. A 32-bit value is currently being returned, which cannot express the capacity of a Large File. In each case a new item number has been defined which works with a 64-bit value and can be used safely for either large or normal sized files.

The following table describes each of the AIF:OS items that return errors for Large Files, as well as the new items that have been added to supplant the older items:

AIF Function	Item #	New Item #	Description
AIFFILELGET	4033	4101	This item returns the byte offset of the current data pointer associated with the file number that was passed to `AIFFILELGET`. Item 4033 returns a 32-bit unsigned integer and results in an error when called against a Large File. The 4101 item returns a 64-bit integer and works for all file sizes.
`AIFFILELPUT`	4033	4101	This item allows a caller to set the data pointer associated with a file number to a new byte offset within a file. Item 4033 allows a caller to pass in a 32-bit unsigned integer to specify the new data pointer value. Given its size, this item can never set the data pointer past the first 4GB of a Large File. The new 4101 item number allows the caller to specify a 64-bit integer to indicate the new data pointer value. The new 4101 item number works for all files, providing the value specified is within the file limit of the specified file.
AIFFILEGGET	5017	5101	These items return the byte offset of the End Of File pointer for a given file. Item 5017 returns a 32-bit unsigned integer and generates an error when called against a Large File. Item 5101 returns a 64-bit integer which can be safely be returned for both large and normal sized files.
AIFFILEGPUT	5017	5101	These items allow the EOF of a file to be set to a specified byte offset. Item 5017 allows the caller to specify a 32-bit unsigned integer for the new EOF offset. Due to the size of this value, it is not possible to use this item to set an EOF beyond 4GB in any file. Item 5101 allows the caller to specify a 64-bit integer which can be used to set the EOF to any offset in any file, providing that the EOF offset is within the file limit of the specified file.
AIFFILEGGET	5018	5102	These items return the byte offset of the file limit of a file, or the maximum size that the file can possibly grow to. Item 5018 returns this value in a 32-bit unsigned integer and returns an error if called against a Large File. Item 5102 returns this information in a 64-bit integer and can be safely used with all file sizes.
AIFFILEGGET	5048	5103	These items return the number of sectors of disk space that currently are allocated to a given file. Item number 5048 returns this value in a signed 32-bit integer and returns an error for a Large File, if that Large File has more than 512GB of disk space allocated to it. Item 5103 returns this sector count in a 64-bit integer and reliably works on files of any size.

OPTIONS Enhancement

As a side effect of adding support for the KSAM64 file type, a new file type value, 7, has been allocated for KSAM64 files. The foptions structure is a 16-bit value made up of various bit fields. Several intrinsics accept or return foptions including FOPEN, FFILEINFO, FLABELINFO, and FGETINFO. The definitions of the foptions are:

0:1: Reserved
1:1: Record format extension bit
2:3: File Type
5:1: File Equations disallowed
6:1: Labeled tape
7:1: Carriage control
8:2: Record format
10:3: Designator type
13:1: Binary/ASCII
14:2: File domain

The addition of KSAM64 support in no way alters the format of the foptions structure. Rather, a previously undefined value in the file type field, 7, is now defined. The list of defined file types which can be returned in the foptions is:

0 Ordinary file type
1 CM KSAM
2 RIO
3 KSAMXL
4 CIR
5 SPOOL
6 MSG
7 KSAM64

In general, the definition of this previously undefined value will not cause any compatibility issues for programs. The only time that issues could arise is if code performed some type of validity check on the file type field of the foptions. In the case that user code is checking and rejecting a value of 7 in the file type field, the workaround would be to eliminate this check, or add 7 to the list of valid values.

Command Interpreter Changes

Although the impact of Large Files is negligible on most CI commands, there are several minor enhancements to a few commands. All CI commands work on Large Files in the exact same manner as they do for normal sized files. RENAME still renames Large Files, PURGE still purges them, and so on. The commands documented in this section have been enhanced to support some Large File features.

BUILD Command Changes

With the first release of Large File support, only two file types are being supported for Large File sizes; ordinary, fixed length record files, and KSAM files. Because of inherent limitations in the internal format of KSAMXL and CM KSAM files, it is necessary to introduce a new type of KSAM file, KSAM64 files, in order to go beyond a 4GB sized KSAM file. The internal format of the KSAM64 index area is structured so that file sizes from 1 record all the way to 1TB can be easily supported.While unfortunately, this means that MPE/iX supports three different varieties of KSAM files (CM KSAM, KSAMXL and KSAM64), the good news is that all three of these file types are functionally equivalent and can be used interchangeably by programs. The only external difference between KSAMXL and KSAM64 files is that a different file type must be used to create them. Other than this, all other KSAM options are specified identically.

The only change to the BUILD command is the addition of the KSAM64 file type keyword. This keyword can be included in the BUILD command as a replacement for the KSAMXL keyword. KSAM64 files may be of any file size, but are the only KSAM file type that can be larger than 4GB. The new syntax for the BUILD command is:

     BUILD filereference                                          [F] 
        [;REC=[[recsize][,[blockfactor][,[U][,BINARY]]]]] 
                                         [V][,ASCII ] 
                                         [B] 
        [{;CCTL  }] 
         {;NOCCTL} 
        [;TEMP] 
        [;DEV=[[dsdevice]#][device]] 
        [;CODE=filecode] 
        [;DISC=[numrec][,[numextents][,[initialloc]]]]         [{;RIO  }] 
         {;NORIO}                                                                        {;STD   } 
         {;MSG   } 
        [{;CIR   }] 
         {;KSAMXL} 
         {;SPOOL }
         {;KSAM64}                                                              [;ULABEL=numlabels]                                                             [;KEY={^filereference2}] 
              {keyinfo        }                                                         [;FIRSTREC=recnum] 
        [{;REUSE  }] 
         {;NOREUSE} 
 
        [;LANG={langid  }] 
               {langname}                                                               [{;DEFBLK}] 
         {;OPTMBLK}

In order to create a Large File, all that is required is to specify a file limit of a sufficiently large number. For example, to create a ordinary, fixed length record file with 80 byte records that had a maximum capacity of about 8GB, a file limit of 100,000,000 records would need to be specified. For example, the following command could be used:

BUILD TESTFILE;REC=-80,,F,ASCII;DISC=100000000

Specifying a file limit that is greater than 4GB for any type of file other than ordinary fixed length record files and KSAM64 files will result in an error.

FILE Command Changes

The FILE and BUILD command syntaxes always mirror each other. It follows, then, that the new KSAM64 file type keyword is also being added to the FILE command, in the exact same way as the BUILD command. The new FILE syntax is included below:

                           =*formaldesignator 
                           =$NULL 
                           =$NEWPASS 
     FILE formaldesignator =$OLDPASS 
                           =$STDIN 
                           =$STDINX 
                           =$STDLIST 
                           =filereference[:nodespec] 
                                          [,filedomain]                                [;DEV=[[envname]#][device][,outpri][,numcopies] 
      [;VTERM] 
      [;ENV=envfile[:nodespec]] 
      [;option] 
      [;access] 
      [;disposition] 
      [{;DEFBLK}] 
       {;OPTMBLK}                                                               option              Any valid option for the FILE command.                      SYNTAX FOR OPTION 
                                    {F}      ;REC=[recsize][,[blockfactor][,{U}[,{BINARY}]]] 
                                    {V}  {ASCII } 
     ;DEN=[density] 
     ;DISC=[numrec][,numextents][,initialloc] 
     ;CODE=[filecode]      [ {;RIO  } ] 
       {;NORIO} 
       {;STD} 
       {;KSAMXL} 
       {;SPOOL}
       {;KSAM64}      [ {;MSG} ] 
       {;CIR}      [;ULABEL=numlabels]      [;KEY={^filereference2}] 
           {keyinfo}      [;FIRSTREC=recnum]      [ {;REUSE}   ] 
       {;NOREUSE}      [;LANG= { langid  } ] 
             { langname}

LISTF[ILE | TEMP] Command Changes

The LISTF, LISTFILE, and LISTFTEMP commands all can be used to display information about Large Files. All of the various LISTF and LISTFILE formats continue to work with normal sized files as well as Large Files. File type information about KSAM64 files is displayed in the LISTF family of commands as a lowercase 'k'.

In order to display information about Large Files, two new file format options have been introduced. The LISTF, LISTFILE, and LISTFTEMP commands now support options 10 (SUMMARYWIDE) and 11 (DISCWIDE). File format option 10 is the Large File equivalent of file option "1", containing all the same information, but in a new format that allows for the expression of larger values. Similarly, the format option 11 is the Large File equivalent of file option "2". Again, the same basic information is displayed, but the format has been changed to allow for a greater range of values to be displayed.

The syntax of the LISTF, LISTFILE, and LISTFTEMP commands has not been changed with the addition of the new 10 and 11 format options. The same syntax can be used with the new values of 10 and 11 as the format. The output of the new formats varies depending on whether or not MPE/iX syntax names are being displayed or HFS syntax names are being displayed.

The format for each of the MPE/iX and HFS syntax varieties of the new formats 10 and 11 can be seen from the examples shown below:

:listf @,10                                                                     ACCOUNT=  SYS         GROUP=  EXAMPLE Name     Access FCode RecSiz Type            EOF    File Limit                  --------  ERWS  ----- ------ ----- ------------- ------------- BIGFILE                 1024 FA                0    1023456789 
ICE       E     NMPRG    256 FB              832           832 
XKSM64                    80 FAk               1          1023
XKSMXL                    80 FAK               1          1023 
XRAND      RW             80 FA              157           157 
XRAND2      W            252 VA              396            49 
YRAND           NMOBJ    256 FB               22          4000

:listfile ./@,10 
PATH= /SYS/EXAMPLE/

Access FCode RecSiz Type            EOF    File Limit Name 
 ERWS  ----- ------ ----- ------------- ------------- ------------ 
                1024 FA                0    1023456789 BIGFILE 
 E      NMPRG    256 FB              832           832 ICE 
                  80 FAk               1          1023 XKSM64 
                  80 FAK               1          1023 XKSMXL 
  RW              80 FA              157           157 XRAND 
   W             252 VA              396            49 XRAND2 
        NMOBJ    256 FB               22          4000 YRAND

:listf @,11 
ACCOUNT=  SYS         GROUP=  EXAMPLE

Name     Access FCode RecSiz Type            EOF    File Limit Disk Usage  Exts 
--------  ERWS  ----- ------ ----- ------------- ------------- --------KB ----- BIGFILE                 1024 FA                0    1023456789          0     0 
ICE         E   NMPRG    256 FB              832           832        208     1 
XKSM64                    80 FAk               1          1023         64     1
XKSMXL                    80 FAK               1          1023         52     1
XRAND      RW             80 FA              157           157         16     1 
XRAND2      W            252 VA              396            49         16     1 
YRAND           NMOBJ    256 FB               22          4000          8     1

:listfile ./@,11 
 PATH= /SYS/EXAMPLE/ Access FCode RecSiz Type            EOF    File Limit Disk Usage  Exts Name 
  ERWS  ----- ------ ----- ------------- ------------- --------KB ----- -------- 
                1024 FA               0    1023456789          0     0 BIGFILE 
  E      NMPRG   256 FB             832           832        208     1 ICE 
                  80 FAk              1          1023         64     1 XKSM64 
                  80 FAK              1          1023         52     1 XKSMXL
  RW              80 FA             157           157         16     1 XRAND 
   W             252 VA             396            49         16     1 XRAND2 
        NMOBJ    256 FB              22          4000          8     1 YRAND

The display for the format options 10 and 11 is the same for each of the LISTF, LISTFILE, and LISTFTEMP commands. One of the unique additions of these two new options is the "ERWS" column. This column indicates whether or not the file is currently opened, and if so, for what type of access. An "E" in the E column indicates that the file is opened for Exclusive access, the "R" in the R column indicates that the file is opened for Read access, the "W" in the W column indicates that the file is opened for Write access, and the "S" in the S column indicates that the file is being Stored by the STORE or TurboSTORE utility. The column may have one or more of these characters specified, or none, depending on how the file is currently accessed.

Intrinsics Changes

As part of the addition of Large File support to MPE/iX several intrinsics changes have been introduced. This encompasses both the modification of existing intrinsics, as well as the introduction of a new set of intrinsics. This chapter describes all of the intrinsics changes that have been implemented as a part of Large File support.

Existing Intrinsics

Relatively few changes have been necessary to the existing file system intrinsics in order to implement Large Files. The majority of these changes are in support of adding 64-bit return values for file byte offsets.

HPFOPEN Changes

As described in the section "User-Mapped Access to Files", when performing user-mapped access to Large Files, care must be taken to not attempt to dereference a buffer that crosses a SID boundary. In order to ensure that programs are aware of this caveat, traditional attempts to open a Large File for user-mapped access will fail. In order to open a Large File for user-mapped access, a new HOP_OPTION_LARGE_MAPPED option (item number 87) must be used, as described in the following table:

Item Name	Item Number	Item Size	Description
`HOP_OPTION_LARGE_MAPPED`	87	@64	This item returns a 64-bit pointer value which points to the start of the file data of the specified file. This item works for any sized file, but is the only user-mapped access option that works for Large Files (files whose file limit is greater than 4GB). Note that not all file types allow user-mapped access. If this item is requested against a file type that does not support user-mapped access, the `HPFOPEN` request will result in a failure.

FLABELINFO Changes

Three new items will be supported by the FLABELINFO intrinsic. These new options are described in the table below:

ItemName	Item Number	Item Size	Description
`FLI_FILE_LIMIT_BYTES_64`	62	8 bytes	This item returns a 64-bit integer which indicates the maximum file size of the specified file in bytes. This is the 64-bit equivalent of `FLABELINFO` item 28.
`FLI_COMPUTED_SECTORS_64`	63	8 bytes	This item returns a 64-bit integer which holds the number of 256 byte sectors of disk space that are currently allocated to the specified file. This item is the 64-bit equivalent to the `FLABELINFO` item 58.
`FLI_FILE_LIMIT_IS_LARGE`	64	4 bytes	This item returns a 32-bit integer which can be used as a simple test to see if the file limit of a file is greater than 4GB. This item will return two values:0 = The file limit is less than 4GB1 = The file limit is greater than 4GB

FFILEINFO Changes

Two new items have been added to the FFILEINFO intrinsic. These items are described in the table below:

Item Name	Item Number	Item Size	Description
`FF_FILE_LIMIT_BYTES`	109	8 bytes	This item returns a 64-bit integer indicating the maximum size of the file, or the file limit, in bytes.
`FF_FILE_LIMIT_IS_LARGE`	110	4 bytes	This item returns a 32-bit integer which can be used to easily tell if the specified file has a file limit greater than 4GB. This item returns two values:0 = The file limit is less than or equal to 4GB 1 = The file limit is greater than 4GB.

New Intrinsics

The following new intrinsics are being added to aid in the support of user-mapped file access. These intrinsics can be used for pointer manipulation and data movement on any user-mapped file; short-mapped, long-mapped, or large-mapped. They are especially recommended for use with the large-mapped HPFOPEN option.

HPFADDTOPOINTER

NM callable only.This routine can be used to perform arithmetic on a 64-bit pointer value. Byte offsets can be added to or subtracted from a pointer by specifying either a positive or negative offset value.

Syntax

@64 I64 @64 I32

HPFADDTOPOINTER( base_ptr, offset, return_ptr, status );

Parameters

base_ptr

64-bit pointer by reference (required)
The base_ptr can be a 64-bit pointer to an object of any type.

offset

64-bit signed integer by reference (required)
The offset can be any positive or negative value. Specifying a positive value moves the return_ptr forward from the previous base_ptr, while a negative value moves the return_ptr backward from the base_ptr.

return_ptr

64-bit pointer by reference (required)
The return_ptr is an output parameter that has the new pointer value returned to it. It can be a 64-bit pointer to an object of any type.

status

32-bit signed integer by reference (optional)
Returns the status of the HPFADDTOPOINTER call. If no errors or warnings are encountered, status returns 32 bits of zero. If errors or warnings are encountered, status is interpreted as two 16-bit fields. Bits (0:16) comprise status.info. A negative value indicates an error condition, and a positive value indicates a warning condition. Bits (16:16) comprise status.subsys. The value represents the subsystem that set the status information.




	NOTE: No attempt is made to verify that the pointer value returned is a legitimate pointer to a valid object. Any invalid pointers are detected and generate errors when the pointers are dereferenced.

HPFMOVEDATA

NM callable only.

This routine can be used to efficiently move data from a source buffer to a target buffer.

Syntax

               I64        @64       @64       I32 
HPFMOVEDATA ( count, source_ptr, target_ptr, status )

Parameters

count

64-bit signed integer by reference (required)
The count parameter allows the caller to specify the number of bytes to move from the source buffer to the target buffer.

source_ptr

64-bit pointer by value (required)
The source_ptr can be a 64-bit pointer to any valid object that the calling process has access to. The buffer may be in the caller's stack, heap, or obtained by opening a file with user-mapped access.

target_ptr

64-bit pointer by value (required)
The target_ptr can be a 64-bit pointer to any valid object that the calling process has access to. The buffer may be in the caller's stack, heap, or obtained by opening a file with user-mapped access.

status

32-bit signed integer by reference (optional)
Returns the status of the HPFMOVEDATA call. If no errors or warnings are encountered, status returns 32 bits of zero. If errors or warnings are encountered, status is interpreted as two 16-bit fields. Bits (0:16) comprise status.info. A negative value indicates an error condition, and a positive value indicates a warning condition. Bits (16:16) comprise status.subsys. The value represents the subsystem that set the status information.




	NOTE: When calling the `HPFMOVEDATA` intrinsic, it is important to ensure that the source and target buffers are not overlapping. The results of a `HPFMOVEDATA` call when source and target buffers are overlapping are undefined. If source and target buffers are overlapping, the `HPFMOVEDATALTOR` or `HPFMOVEDATARTOL` intrinsics should be used.

HPFMOVEDATALTOR

NM callable only.

This routine can be used to efficiently move data from a source buffer to a target buffer. If the source and target buffers were viewed horizontally, like a line of text, the data movement is performed by starting at leftmost position of the source buffer (to the leftmost position of the target buffer) and proceeding to the rightmost.

Syntax

                    I64       @64        @64      I32 
HPFMOVEDATALTOR ( count, source_ptr, target_ptr, status )

Parameters

count

64-bit signed integer by reference (required)
The count parameter allows the caller to specify the number of bytes to move from the source buffer to the target buffer.

source_ptr

target_ptr

status

32-bit signed integer by reference (optional)
Returns the status of the HPFMOVEDATALTOR call. If no errors or warnings are encountered, status returns 32 bits of zero. If errors or warnings are encountered, status is interpreted as two 16-bit fields. Bits (0:16) comprise status.info. A negative value indicates an error condition, and a positive value indicates a warning condition. Bits (16:16) comprise status.subsys. The value represents the subsystem that set the status information.




	NOTE: This intrinsic is especially useful when the source and target buffers are overlapping. The `HPFMOVEDATALTOR` intrinsic is typically used when the target buffer's address is to the left (smaller) of the source buffer's address. Moving the data from the left to the right ensures that the data in the source buffer is copied to the target buffer before it is overwritten itself.

HPFMOVEDATARTOL

NM callable only.

This routine can be used to efficiently move data from a source buffer to a target buffer. If the source and target buffers were viewed horizontally, like a line of text, the data movement is performed by starting at the rightmost position of the source buffer (to the rightmost position of the target buffer) and proceeding to the leftmost.

Syntax

                   I64       @64        @64       I32 
HPFMOVEDATARTOL ( count, source_ptr, target_ptr, status )

Parameters

count

64-bit signed integer by reference (required)
The count parameter allows the caller to specify the number of bytes to move from the source buffer to the target buffer.

source_ptr

target_ptr

status

32-bit signed integer by reference (optional)
Returns the status of the HPFMOVEDATALTOR call. If no errors or warnings are encountered, status returns 32 bits of zero. If errors or warnings are encountered, status is interpreted as two 16-bit fields. Bits (0:16) comprise status.info. A negative value indicates an error condition, and a positive value indicates a warning condition. Bits (16:16) comprise status.subsys. The value represents the subsystem that set the status information.




	NOTE: This intrinsic is especially useful when the source and target buffers are overlapping. The `HPFMOVEDATARTOL` intrinsic is typically used when the target buffer's address is to the right (larger) of the source buffer's address. Moving the data from the right to the left ensures that the data in the source buffer is copied to the target buffer before it is overwritten itself.

HPFFILLDATA

NM callable only.

This routine can be used to efficiently initialize a buffer with a specified character value.

Syntax

                I64       @64       CV       I32 
HPFFILLDATA ( count, buffer_ptr, fill_char, status )

Parameters

count: 64-bit signed integer by reference (required)
A positive count of the number of bytes in the buffer indicated by the buffer_ptr parameter that should be initialized.
buffer_ptr: 64-bit pointer by value (required)
A pointer to the buffer that should be initialized. The buffer_ptr may point to any valid object in your stack, heap, or a file that has been opened with user-mapped access.
fill_char: Character value by value (required)
The character value that should be used to initialize the specified buffer. Any value in the range of 0 through 255 can be specified, including all printable and non-printable ASCII characters.
status: 32-bit signed integer by reference (optional)
Returns the status of the HPFFILLDATA call. If no errors or warnings are encountered, status returns 32 bits of zero. If errors or warnings are encountered, status is interpreted as two 16-bit fields. Bits (0:16) comprise status.info. A negative value indicates an error condition, and a positive value indicates a warning condition. Bits (16:16) comprise status.subsys. The value represents the subsystem that set the status information.

Miscellaneous

This section covers several issues that are related to Large Files that may be of interest to the reader.

STORE/RESTORE Impact

The STORE and RESTORE utilities have been modified to handle Large Files. This enhancement was accomplished in such a fashion that STORE tapes with Large Files are completely compatible with previous versions of the operating system. A STORE tape created on release 6.5 can be taken to any other MPE/iX system and be successfully restored. On systems prior to 6.5, any Large File on the tape is not recognized as a valid file and will not be restored onto the system. Although Large Files are skipped over in this scheme, all other files will be able to be restored.

Any attempt to store a Large File using the ;TRANSPORT STORE option results in an error at store time.

In addition to the STORE/RESTORE and TurboSTORE products all supporting Large Files, most, if not all, 3rd party backup products have also been enhanced to support Large Files. If you rely on 3rd party backup tools, please refer to your vendor for information regarding the proper updates and procedures for using these tools on release 6.5.

Backdating Procedures

If for some reason it is necessary to "backdate" your system from the 6.5 release to a prior MPE/iX release, very little extra work is required due to Large Files. Before attempting the backdate operation, the 6.5 system must be shut down completely with the CNTRL-A SHUTDOWN command. You must also wait for the final system shutdown message to appear before resetting the system and beginning the UPDATE command on the earlier MPE/iX release.

Any Large Files that are in the directory structure of the system are unusable once the system is backdated to a release earlier than 6.5. The Large Files can be purged from the older system via the :PURGELINK command. Any other attempt to open or manipulate a Large File results in an error. Once the system is again updated to release 6.5 or beyond, the Large Files are once again accessible.

Utility and Subsystem Support

Nearly all utilities and subsystems support Large Files without change. This runs the spectrum from FCOPY to SORT. The only word of caution is that not all utilities and subsystems support file sizes that are less than 4GB. This includes such things as EDITOR or HPEDIT. The general rule of thumb is if it doesn't work on a 3GB file, it won't work on a Large File.

One exception of note is the DSCOPY subsystem. DSCOPY has not been enhanced to be able to transfer Large Files, or the new KSAM64 files. The FTP utility has been enhanced to do network transfers of these files, and is the preferred utility for network file transfer.

Impact on IMAGE Database

The introduction of Large Files has had no impact on the IMAGE/SQL product. Large database support is still accomplished by use of Jumbo Datasets. Therefore there are no compatibility issues to be concerned with in regards to changes in your IMAGE/SQL databases.

HP SRC

Version A.01.29 of HP SRC does not support KSAM64 files. If an attempt is made to check in a KSAM64 file, error 387 will be reported.

JAVA

Java programs will be able to access large MPE/iX files by using Java Native Interface (JNI) to call MPE/iX intrinsics, but will not be able to access Large Files via the java.io APIs. The java.io APIs treat all files as bytestream files and access these files through the POSIX C language APIs. The first release of Large Files for MPE/iX does not include support for bytestream files larger than 2GB in size; therefore, there can be no Large Files that are also bytestream files in this release.

HPSEARCH/iX

HPSearch/iX has been changed to provide for searching of Large Files. Additionally, it has been enhanced to search KSAM64 and KSAMXL files.

HPGLANCEPLUS/iX

HPGlancePlus/iX has been changed to display file information about Large Files. Also, KSAM64 files are included in the File Display on the Process Screen and will show a type of KSM64.

HPBROWSE/iX

HPBrowse/iX supports viewing Large Files and KSAM64 Files.

HPEDIT/iX

HPEdit/iX does not support files of 2,000,000 records or more, and so, does not support Large Files.

COBOL/iX

COBOL II/iX supports Large Files. No changes are necessary to existing COBOL II/iX programs to read, write, and update Large Files. KSAM64 files are supported by the Indexed I-O module. When you open a file with indexed organization, that file is permitted to be an existing KSAM64 file. When COBOL II/iX creates a file with indexed organization, by default it will create a KSAM XL file. However, you may override the default by issuing a FILE command specifying KSAM64 as the file type.

COBOL II/CM

COBOL II/CM does not support KSAM64 files. An attempt to open or create a KSAM64 file with a CM COBOL program will result in file status 39 and, typically, a program abort with COBOL error 648.

C/iX

C/iX does not support Large Files. You must issue your own intrinsic calls if you want to access Large Files from C. Attempting to open a Large File using the C/iX Library function fopen() will fail and set errno to ESYSERR (50).

MPE/iX Large Files Overview

Technical documentation

» Table of Contents

» Index

Introduction

Overview

About this Document

Summary of Large File Compatibility Issues

User-mapped Access to Files

Problem Description

Compatibility Issue

Workaround

FLABELINFO

32-bit Integer Overflow

AIF:OS Impact

OPTIONS Enhancement

Command Interpreter Changes

BUILD Command Changes

FILE Command Changes

LISTF[ILE | TEMP] Command Changes

Intrinsics Changes

Existing Intrinsics

HPFOPEN Changes

FLABELINFO Changes

FFILEINFO Changes

New Intrinsics

HPFADDTOPOINTER

HPFMOVEDATA

HPFMOVEDATALTOR

HPFMOVEDATARTOL

HPFFILLDATA

Miscellaneous

STORE/RESTORE Impact

Backdating Procedures

Utility and Subsystem Support

Impact on IMAGE Database

HP SRC

JAVA

HPSEARCH/iX

HPGLANCEPLUS/iX

HPBROWSE/iX

HPEDIT/iX

COBOL/iX

COBOL II/CM

C/iX