Large Size Datasets in TurboIMAGE/iX

Introduction

This enhancement in TurboIMAGE/iX removes the current 80 GB limit on the size of a dataset so as to allow users to have really large datasets. Now the only limiting factor is the number of entries in a dataset, which can be up to 2**31 -1.As the solution involves changes in the internal data structures of IMAGE, a new utility DBBIGSET provides an easy way for existing databases to migrate to the new structures. Once migrated, the structural details will be transparent to the users. User interface changes have been made to DBSCHEMA, DBUTIL and DBINFO. Old and new format databases can co-exist in the same system.

Dataset Size Limitations

The IMAGE database management system has been around for more than two decades.When HP first introduced the product, the maximum size of a dataset was limited to the MPE file size. However, when users felt the need for much larger dataset sizes, the 'JUMBO' feature was introduced. With this feature, a user can create a dataset with a size greater than the MPE file size of 4GB. Even though the dataset physically spans across multiple MPE files called chunk control files and chunkset files, the JUMBO feature allows them to be seen as a single logical file. Even with this feature, users could only define a dataset up to 40 GB in size, because of the limit of block numbers, whereas the JUMBO feature per se doesn't have this restriction. Earlier, IMAGE used to transfer data in blocks, which necessitated addressing of a record through a block number and an offset within the block. The maximum number of blocks allowed to be addressed in IMAGE was 2**23 -1, and the maximum block size was 5120 bytes. The total file size with these, turns out to be 42,949,667,840 bytes (approximately 40 GB). A quick solution has been provided in IMAGE version C.07.10 released with MPE/iX version 6.0, wherein the limit of 40 GB has been extended to 80 GB. This was done by increasing the maximum number of blocks to 2**24 -1. The current enhancement lifts the 80 GB restriction by changing the way that records are addressed internally.

Entryname vs Entrynumber formats

To understand the solution in detail, a brief discussion of the terms entryname and entrynumber is worthwhile. As discussed above, in IMAGE, records (entries) were referenced through a 24 bit block number and an 8 bit offset (slot number) within the block. This format is referred to as 'entryname' format. For example, the 101st record in a dataset, with a blocking factor of 8, will be referenced as hex c05, represented as the 24-bit blocknumber of 12 (hex c) and a slot within the block of 5. Now we can use a full 32-bit number to refer a record number. We call this 'entrynumber' format. So, the 101st record in a dataset will be represented as just 101, irrespective of the blocking factor used.All existing databases prior to this release use entryname format. With this solution, a database can be created either in the entrynumber format or the entryname format. However, if large datasets greater than 80 GB are to be created, it can be done only if the database is created in entrynumber format.

Migration Utility

Existing databases can be moved to support large datasets through the migration utility provided with the solution. The migration utility is DBBIGSET.PUB.SYS, which has PM capability and has to be executed from the group where the database exists. Migration is a one-time activity, and it requires exclusive access to the database. Since the internal structures are being changed, a full backup of the database must be taken before starting the migration. Alternatively, if sufficient resources are available one could make a copy of the database, do the migration on the copy, verify the integrity and copy the migrated database back to the original. No data integrity checking of the database is performed during this process. During migration, the RL'CONDITION word in root file label #0 will be set to 'LS'. If for any reason the migration can't be completed, appropriate error messages will be displayed, and the user must revert back to the old database environment from the previous backup. The syntax for running migration utility is

	NOTE: It should be kept in mind that, migration involves changes to internal structures of the database, and a backup must be taken before starting the migration.

Entrynumber format vs. Jumbos vs. MPE Large Files

It has to be noted that this enhancement per se will not make use of the large file feature of MPE release 6.5. Creating a database in entrynumber format or migrating a database to entrynumber format just allows the datasets to grow beyond 80 GB (which is the current limit). There is no harm in keeping the database in entrynumber or entryname format, if the dataset size is less than 80 GB. Also, there is no impact if some databases are kept in entrynumber format and others in entryname format in the same system. However, as of version C.09.02 of IMAGE, the only mechanism to have a dataset size greater than 4 GB, is to use 'Jumbo' feature. This will be the case, till IMAGE makes use of the large file feature of MPE.

Conclusion

In summary, this release