UniVerse Files

1
UniVerse Files
Neil Morris, UniVerse Technical Support

2
Credits and Acknowledgements
Presenter
• Neil Morris, UniVerse Technical Support
Developers
• Joel Yates, UniVerse Technical Support
• Neil Morris, UniVerse Technical Support
©2015 Rocket Software, Inc. All Rights Reserved.

3
Abstract
 This session will take you through the File Types available for use with
UniVerse. The UniVerse File Header structure will be examined. A more
detailed review of UniVerse dynamic hashed, static hashed, and btree file
header structures is included. And finally a review of the group and record
structures associated with dynamic and static hashed files. An overall
understanding of the UniVerse file types and structures can aid in file
configuration and tuning.

4
Agenda
UniVerse files general
• Directories, SQL, Multilevel, Distributed, Btree, Dynamic and Static
hashed
UniVerse file header
• Magic, Type, Modulus, Separation, and MKDBtimestamp
BTree/Index files
Dynamic and static hashed files
UniVerse file structures
• Group structure and Record structure

5
Naming Conventions - LONGNAMES
 By default UniVerse limits file names to 14 characters.
>CREATE.FILE FILE.WITH.A.LONG.FILENAME DYNAMIC
WARNING: An operating system file will be created with a truncated
name.
Creating file "FILE.WITH000" as Type 30.
Creating file "D_FILE.WITH000" as Type 3, Modulo 1, Separation 2.
Added "@ID", the default record for RetrieVe, to "D_FILE.WITH000".
>CT VOC FILE.WITH.A.LONG.FILENAME
FILE.WITH.A.LONG.FILENAME
0001 F
0002 FILE.WITH000
0003 D_FILE.WITH000
>

6
Naming Conventions - LONGNAMES
Changing the default behavior.
 Execute 'LONGNAMES ON' in the account to enable the
feature in that account.
 Execute 'LONGNAMES ON NEWACC' in the main uv account
to enable the feature by default for any new universe accounts
 Use 'uv.load -longnames' when installing or upgrading
UniVerse to enable this feature by default.

7
Naming Conventions – Special Characters
Character substitutions will occur when characters with special
meanings at the operating system level are used in a filename
or in the key to an item in a Type 1 or 19 file.
For example on UNIX:
>CREATE.FILE MY/FILE DYNAMIC
Creating file "MY?FILE" as Type 30.
Creating file "D_MY?FILE" as Type 3, Modulo 1, Separation 2.
Added "@ID", the default record for RetrieVe, to "D_MY?FILE".

8
On Windows:
>CREATE.FILE MY|FILE DYNAMIC
Creating file "MY%VFILE" as Type 30.
Creating file "D_MY%VFILE" as Type 3, Modulo 1, Separation 2.
Added "@ID", the default record for RetrieVe, to "D_MY%VFILE".
Refer to 'Naming files' in System Description manual for a
complete list.

9
Windows special files.
Windows allows certain names to be used to
directly access a device or feature. Attempting to
use these names for UniVerse file names or
record keys for Type 1 and 19 files will result in
errors.

10
Windows special files.
>CREATE.FILE AUX.FILE 18 1 1
An operating system file already exists with the name
"AUX.FILE".
>COPYI FROM VOC TO BP AUX.PORT
Failed to write record "AUX.PORT" to file "BP"!
0 records copied.
Windows treats 'AUX' as a reference to COM1 and restricts
these operations.

11
Naming Conventions – WINDOWS Files
CON: Send data to console
PRN: Send data to LPT1 (line printer)
AUX: Send data to COM1
NUL: Disregard data (same as /dev/null on UNIX)
LST: Send data to LPT1 (line printer)
LPT1-LPT4: Send data to parallel port
COM1-COM4: Send data to serial port

12
Directories
Type 1: Defined by the presence of '.Type1' file in the
directory
Type 19: Defined by the absence of '.Type1' file in the
directory
On Windows this is a hidden file, use 'dir /ah' to view

13
Type 1 File Record Keys
Key is broken into 14 character pieces and items
stored using sub-directory structure.
A key length divisible by 14 will use '?' as the file
name.
The key length is restricted to a maximum length of 41
characters on both Windows and UNIX.

14
>CREATE.FILE TEST 1
>COPY FROM VOC TO TEST LIST,FOURTEEN.XXXXX
LIST,FOURTEEN.XXXXXZ
>SH -c "find TEST -print"
TEST
TEST/.Type1
TEST/FOURTEEN.XXXXX
TEST/FOURTEEN.XXXXX/?
TEST/FOURTEEN.XXXXX/Z
>©2015 Rocket Software, Inc. All Rights Reserved.

15
>COPY FROM VOC TO TYPE1
LIST,12345678901234567890123456789012345678901234567890
File name "12345678901234567890123456789012345678901234567890" too long.
Truncated to "12345678901234567890123456789012345678901".
1 record copied.
>SH -c "find TYPE1 -print"
TYPE1
TYPE1/.Type1
TYPE1/12345678901234
TYPE1/12345678901234/56789012345678
TYPE1/12345678901234/56789012345678/9012345678901

16
Converting Type 1 Files
Use RESIZE to convert a Type 1 file to Type 19 and vice versa.
>RESIZE TEST 19
>SH -c "find TEST -print"
TEST
TEST/FOURTEEN.XXXXX
TEST/FOURTEEN.XXXXXZ

17
Converting Type 1 Files
WARNING: Items not conforming to the Type 1
format will be lost. Verify before resizing.
>SELECT TEST
3 record(s) selected to SELECT list #0.
>>SELECT TEST
2 record(s) selected to SELECT list #0.
"FILENAMETHATEXCEEDSFOURTEENBYTES" not found.
>>

18
Type 1 Files on Windows
On Windows Type 1 files do not use the sub-
directory structure.
The primary difference between a Type 1 and
Type 19 file is the 41 character record key limit
on the Type 1 file.

19
Type 19 Files
 On both UNIX and Windows a Type 19 file is simply a directory at the
operating system level.
 The maximum record key size is limited by the operating system
maximum file name length.
0001 OPEN 'TEST' TO F.FILE ELSE STOP 'CANNOT OPEN TEST'
0002 KEY1 = STR('X','255');KEY2 = STR('X','256')
0003 WRITE 'TEST' ON F.FILE,KEY1
0004 WRITE 'TEST' ON F.FILE,KEY2
0005 END
>RUN BP X
Program "X": Line 4, Fwrite failed at DBwrite for file TEST.
Program "X": Line 4, WRITE failure.

20
Type 1 and 19 files: NFS and SHARES
Caution regarding using basic WRITE.
Line terminator on UNIX is new-line, char(10)
Line terminator on Windows is carriage-return &
new line, char(13) : char(10)

21
Type 1 and 19 Files: NFS and SHARES
WRITE operation will fail on UNIX to Windows share
(cifs) with ALLOWNFS. Instead use sequential
processing (WRITEBLK, WRITESEQ).
On UNIX a WRITE to a Type 1 or 19 file is done using
a temporary file to avoid loosing an existing item in the
event the write fails. This is accomplished using link()
and unlink() calls which are not supported on cifs
mounts.

22
Machine Class: Big & Little endian
Refers to the manner in which the hardware
manages binary (ie., integer) data.
For example integer 1 in hexidecimal
Big endian : 01000000
Little endian: 00000001

23
Machine Class: Big & Little endian
Big endian (swapped)
• Windows
• Linux
• Sun Intel
Little endian (not swapped)
• AIX
• HPUX
• Sun Sparc

24
Machine Class: Converting
UniVerse files contain a combination of binary
and string data
File must be converted when moved between
different machine classes
The fnuxi or FORMAT.CONV command is used
to perform the conversion

25
Machine Class
>FORMAT.CONV -m1 TEST
>DUMP TEST
00000000: 0C01EFAC 03000000 00000000 17000000
00000010: 04000000 00000000 00000000 00000000
>FORMAT.CONV -m0 TEST
>DUMP TEST
00000000: ACEF010C 00000003 00000000 00000017
00000010: 00000004 00000000 00000000 00000000
DUMP is a basic program which uses READBLK to process the file in 'raw'
mode.

26
SQL Tables and Views
 Must be used within a SQL Schema
 Protected by GRANTs
 Tracked in uvhome/sql/catalog database
SQL Tables
 Use standard static and dynamic hashed files.
 Will contain a SICA block with table properties
 Use LIST.SICA to display SICA contents
SQL Views
 Virtual view of parts of an SQL Table
 Can limit what parts of the table can be viewed or updated

27
Multilevel Files
 Organizational method for managing files
 Subfiles can be any type of file including directories (1 & 19)
 Relies on application to maintain the association by accessing
the appropriate subfile.
 Multilevel files originated in the PICK environment
Example:
CUSTOMERS
US
CANADA
MEXICO

28
Multilevel Files
Creating:
CREATE.FILE CUSTOMERS,US 18 3791 4
CREATE.FILE DATA CUSTOMERS,CANADA DYNAMIC
Access:
LIST CUSTOMERS,CANADA NAME ZIP
OPEN “CUSTOMERS,US” TO F.FILE ELSE …

29
Multilevel Files
>CT VOC CUSTOMERS
CUSTOMERS
0001 F
0002 CUSTOMERS
0003 D_CUSTOMERS
0004 M
0005
0006
0007 CANADAýUS
0008 CANADAýUS

30
Multilevel Files
>SH -c "ls -l CUSTOMERS"
total 4
drwxr-xr-x 2 root root 96 Sep 1
10:55 CANADA
-rw-r--r-- 1 root root 2048 Sep 1
10:55 US
>

31
Distributed Files
 Logical grouping of files treated as single universe file
 Parts can be static hashed or dynamic hashed
 Part file can be a member of more than one distributed file
 Partitioning algorithm based on @ID contents
 All items within a given part must adhere to the algorithm for
each distributed file of which the part is a member
 Distributed file must be defined local for UVNet access, parts
can be remote
 Cataloged in &PARTFILES&

32
Distributed Files
Advantages
Parts can be accessed independent of the distributed
file. For example INVOICES comprised of multiple
parts where each part represents a year.
Can reduce the amount of time a file has to be offline
for maintenance. For example BUILD.INDEX,
RESIZE, fixtool, etc..
Distributed file is transparent to the application.

33
Distributed Files
Disadvantages
Increased demands for file units.
Performance impact when opened local to
subroutines and functions called frequently.
Note: BSCAN only works against individual
parts.

34
Distributed Files
Universe commands
 DEFINE.DF
 VERIFY.DF
 REBUILD.DF
 LIST.DF
Example:
DEFINE.DF INVOICES ADDING INVOICE.2005 1 INTERNAL "FIELD(@ID,'*',1)"

35
UniVerse Files
BTree
Static hashed
Dynamic hashed
These files are comprised of a series of buffers
Buffers are also called frames or groups
In a BTree the term used is leaf

36
UniVerse Files BTree
Header buffer
Index buffer
Data buffer
Oversize item buffer
Free buffer
The initial buffer of a BTree is often referred to as the
'root' since all access begins at that point.

37
UniVerse Files Hashed (Dynamic/Static)
 Header
 Primary buffer (1 per group)
 Overflow buffer to expand group beyond single
primary buffer
 Oversized item buffer to store large items
 Free buffer, space available to be reused

39
UniVerse File Header Buffer
 BTree, Static hashed, and Dynamic hashed
files all begin with a header buffer
 The header buffer is used to store information
about the structure of the particular file

40
The format of the header buffer is the same for all files
Some information is common across different types of
files
Other information in the header will vary based on the
type of file
 A dynamic file header will contain merge and split
factors
These locations would be unused in a static hashed
file

41
The minimum amount of information that must be
present in a file header is:
MAGIC
TYPE
MODULUS
SEPARATION
MKDBTimestamp

42
The size of the header buffer depends on the separation
of the file.
Odd separation the header is 1024 bytes
Even separation the header is separation x 512
Only the first 1024 bytes of the header are used
The file header is covered in more detail later.

43
UniVerse File MAGIC
32-bit integer value that identifies the file as a UniVerse
file.
First 16 bits is always hexadecimal 'ACEF'
Next byte is style code which indicates if the file is
32-bit (01) or 64-bit (02)
Next byte is revision used to identify the file structure

44
UniVerse File MAGIC
Example: ACEF010C
ACEF - universe file
01 – 32-bit file
0C - created at or after release 9.5.1C

45
UniVerse File MAGIC
Style code
 The style code defines the type of integer values used to
navigate the file
 A style '01' file uses signed 32-bit integer pointers
 A style '02' file uses signed 64-bit integer pointers
 Style is not directly related to the physical size of the file
 However a style '01' (32-bit) can not exceed 2GB
 The maximum value for a signed 32-bit integer is 2GB
 A style '02' (64-bit) has no specific limit other than what would
be imposed by the operating system

46
UniVerse File Type
UniVerse file types include:
1 Directory with special handling for key size
2-18 Static hashed files. Type controls record hashing
19 Directory
25 BTree
27 Distributed file
30 Dynamic hashed file
41 SQL View

47
UniVerse File Modulus
 Identifies the number of primary groups
within the file
 Each group in a hashed file can be
updated independent of other groups in the
file

48
UniVerse File Separation
 The separation determines the size of the
buffers within the file
 The separation is expressed in terms of
512 byte blocks
 A separation of 4 would equate to a buffer
size of 2048.

49
UniVerse File MKDBTimestamp
The UTC timestamp when the file is created
 After a resize, the MKDBTimestamp is replaced
by the time of the resize operation
The original timestamp is also saved and stored
during the resize operation
The original value is used should a file have
been rotated out during the resize operation

50
UniVerse File MKDBTimestamp
There are numerous tools on the internet to
allow you to quickly convert a UTC
timestamp to a readable time and date. For
example:
http://www.epochconverter.com/epoch/unix-hex-timestamp.php

52
BTree/Index Files
Used for UniVerse Index files.
 File Type 25
 Separation 16 (buffer size 8192)
 Items filed in sort order.
 Sort order may be numeric or text
 All access begins at the same point 'root
leaf'

53
BTree/Index Files
Buffers or leafs that makeup a BTree file.
Header
Index
Data
Oversized item
Free

54
Btree/Index Files
 The style of an index is based on the style of
the data file when the index is created
 An index created on a 32-bit file will be 32-bit
 An index created on a 64-bit file will be 64-bit
 If a file is resized between 32-bit and 64-bit, the
style of the index will not change
 An index would need to be deleted and
recreated to have it match the new style

55
Btree/Index Header
Required header information
Magic
Type (25)
Modulus (1 though not used)
Separation 16 (8K)
Status flags (covered later)
Optional
Free chain pointer

56
BTree/Index Files
Index buffer
Used to divide the BTree into subsets
based on sort order.
Index leaf points to other index or data
leafs
May reference up to 384 index or data
buffers

57
BTree/Index Files
Data buffer
May contain up to 128 items.

58
BTree/Index Files
Oversized item buffer
 Used to store data for any item over
approximately 7k
 Additional buffers may be linked as needed
to store the item.

59
Btree/Index Files
Free chain pointer
As buffers become free they become part of the free
chain
Existing free chain pointer is written as forward pointer
in free buffer
The free buffer address is written to the file header as
free chain pointer
The same logic applies for static and dynamic hashed
files

60
BTree/Index Files
 The Btree structure can greatly reduce the number of read
operations required to locate a specific index value in the
index.
 Each index read reduces potential items by a factor of 384.

61
BTree/Index Files
 The record id’s within an index item are sorted and accessed
sequentially.
 This is done to optimize certain Retrieve operations selecting
multiple indexed fields
 However, poor performance can occur when updating an
indexed value containing a large volume of keys.
 A very large number of read/write operations may be required
to insert a new record id into an index item containing
thousands of keys
 Caution should be used when creating an index where a large
number of keys go to a single indexed value

62
Btree/Index Files
 All index files have an INDEX.MAP file
 The INDEX MAP file contains information regarding the
specific indexes
 Data contained within the INDEX.MAP includes but is not
limited to:
• index file (ie., INDEX.000)
• index name
• index type (A, C, D, or I)
• (S)ingle or (M)ultivalued
• Flag indicating if index need to be built
• Flag indicating if null values are included
• Flag indicating if index is disabled

63
BTree/Index Commands
blook
CREATE.INDEX
BUILD.INDEX
LIST.INDEX
DISABLE.INDEX
ENABLE.INDEX
UPDATE.INDEX
SET.INDEX
DELETE.INDEX
ENCRYPT.INDEX
REENCRYPT.INDEX
DECRYPT.INDEX

64
Indexes and Retrieve
NO.INDEX keyword instructs Retrieve to not use an
available index
REQUIRE.INDEX keyword instructs Retrieve to only
execute the command if an index is available
EXPLAIN keyword will provide data on whether
Retrieve is using an index
Note, when using multiple indexes in a select that
returns a significant portion of the file, not using the
index may perform better.

65
Indexes and Retrieve
 The use of an index is based on comparing the dictionary
reference to data in the INDEX.MAP
 Field name is NOT a factor in determining if an index will be
used
 Field, I-Type code, Correlative must match
 Single/Multivalue indicator must match
 Justification (L & T or R) must match
 The ‘In DICT’ column of LIST.INDEX will indicate if the
dictionary item matches the data within the INDEX.MAP

66
Indexes and BASIC
INDICES is a function which can return information on
the indexes on the file
SELECTINDEX is a statement which can return a
select list of items from an index
BSCAN is a function which allows traversing an index
in ascending or descending order
BSCAN allows partial values as the beginning point of
a search

67
Concurrent BUILD.INDEX
The indexing properties of a file are determined at the
time it is opened
A process which has the file open would not be aware
of new indexes created after the open
BUILD.INDEX CONCURRENT should only be used
after all users have reopened the file when a new
index has been created

68
Dynamic and Static Hashed Files

69
The buffers which comprise these files are the same
The difference is where these buffers are located
The static hashed file uses a single file to store
information
For a dynamic file:
• The header and primary buffers (one per group) are
in the DATA.30 file
• All other buffers are in the OVER.30 file
The OVER.30 header only holds the free chain pointer

70
A simple static file might be represented as:
Header
Primary
Primary
Primary
Primary
Overflow
Overflow
Special
Overflow
Oversized Item
Overflow

71
A dynamic file would look as follows:
DATA.30 OVER.30
Header
Primary
Primary
Primary
Primary
Header
Overflow
Overflow
Special
Overflow
Oversized Item
Overflow

72
The modulus of a static hashed file is constant
The modulus is loaded when the file is opened
The modulus of a dynamic hashed file can change
while the file is open
The modulus of an open dynamic file is stored in the
T30FILE shared memory structure
All users having the dynamic file open use the values
stored in the shared memory structure

73
The “smat –d” command displays the T30FILE contents
 Access to the T30FILE structure is protected by the
T30FILE semaphore
On a system with a large number of open dynamic files,
contention for access to the T30FILE structure may
exist
If possible, files that do not significantly change size
should remain static hashed

74
The buffers that make up static and dynamic files
include
 Header
 Primary buffers
 Group overflow buffers
 Oversized record buffers
 Free buffers
 Special purpose buffers

75
Special purpose buffers
 SICA: SQL Tables and views. Triggers.
Note - this buffer is encrypted
 Part file: Partitioning algorithm information for
each distributed file of which the part file is a
member

76
Dynamic and Static Hashed File Type
Record hashing is achieved by performing a
calculation on the record key
The file type (static) or hashing algorithm
(dynamic) determines the calculation used on the
key
The calculation returns a value which determines
the primary group location for the record

77
The objective when choosing a file type or
hashing algorithm is to equally distribute the
records among the available primary groups
UniVerse provides tools to assist with generating
an even distribution
• HASH.AID, FILE.STAT, HASH.HELP, etc.

78
 UniVerse provides17 different file types or
hashing algorithms for static hashed files
The different file types are based on the
characteristics of the record key
Two file types or hashing algorithms are available
for dynamic files

79
Static Hashed File Type
2 keys end with numbers.
3 keys end mainly with numbers.
4 keys end with letters.
5 keys end with full range of ASCII characters.
6 keys begin with numbers.
7 keys begin mainly with numbers.
8 keys begin with letters.
9 keys begin with full range of ASCII characters.

80
10 keys are numbers.
11 keys are mainly numbers.
12 keys are letters.
13 keys are full range of ASCII characters.
14 entire keys are numbers.
15 entire keys are mainly numbers.
16 entire keys are letters.
17 entire keys are full range of ASCII characters.
18 entire keys are of arbitrary form.

81
Dynamic Hashed File Type
Dynamic files have two available file types:
20 GENERAL
21 SEQ.NUM

82
Choosing the best file type for a static hashed file may
be challenging
Determining the target values for modulus and
separation initially is a good approach
Tools such as HASH.AID can be used to assist in
selecting the most appropriate file type

83
Dynamic and Static Hashed Separation
 Separation determines the size of the buffers that
make up the file
 For a dynamic file the separation is documented as
GROUP.SIZE
 Group size 1 is a separation of 4 where group size 2 is
a separation of 8
 The RESIZE command can be used to select a
different separation for a dynamic file

84
>CREATE.FILE TEST DYNAMIC
>GROUP.STAT TEST
Bytes Records File= TEST Modulo= 1 Sep= 4
>RESIZE TEST 30 1 16
>GROUP.STAT TEST
Bytes Records File= TEST Modulo= 1 Sep= 16

85
 Choosing a separation involves balancing cpu and disk
 The fewer records in a group, the less cpu required to
retrieve an item
The more records contained within a buffer, the fewer
I/O operation required to process the file

86
Given a file contains items with an average record size of
150 bytes:
• Separation of 1 can fit 3 items per buffer

87
With 3 items per buffer, the target item will be found on
average in 1.5 compares
With 6 items per buffer, the target item is found on
average in 3 compares
With 13 items per group, the target item on average is
found in 7.5 compares

88
Using previous example and a 1,000,000 record file:
• A separation of 1 or 2 file results in a 170mb file
• A separation of 4 file would be 157mb file
Choose a separation that balances disk utilization with
the fewest items stored per group
Note - large record sizes may impact the choice of
separation

89
Dynamic and Static Hashed Modulus
Choosing the separation value will simplify the
choice of modulus
Divide total records by the estimated number of
records per group to get modulus
For a dynamic file this method can factor into the
choice for minimum modulus

90
Dynamic and Static Hashed Modulus
For static hashed files, the choice of modulus
should take into account expected growth
If the file is expected to grow, choosing a larger
modulus would be appropriate
How much larger would depend on expected
growth and how often files are examined and
scheduled for maintenance

91
Dynamic Hashed Files
 Routine file maintenance can be minimized with the
use of dynamic files
A dynamic file modulus can grow and shrink based on
the file contents
Using dynamic files with ALLOWNFS from multiple
systems is NOT recommended

92
In addition to the header information already mentioned, a
dynamic file header contains:
 Current modulus
 Base modulus
 Next group to split
 Data bytes (filesp)
 Large record size
 Hashing algorithm (GENERAL/SEQ.NUM)
 Split factor
 Merge factor

93
Dynamic Hashed Files Modulus
The modulus value in a dynamic file is referred to
as the ‘minimum modulus’
By default, the minimum modulus is set to 1
The ‘current modulus’ is used to determine the
actual number of groups in the file

94
Dynamic Hashed Files Modulus
The current modulus is loaded to the T30FILE
memory structure when the file is first opened
The current modulus stored in T30FILE will then
be used for all I/O being done to the file.

95
Dynamic Hashed Files Base Modulus
Base modulus is used as part of the hashing
algorithm
Base modulus is equal to the largest power of
two of the current modulus
For example, a current modulus of 216 would
translate to a base modulus of 128
A current modulus of 260 would translate to a
base modulus of 256

96
Dynamic Hashed Files Base Modulus
Decimal Hex Binary
Curr Mod: 216 D8 0000 0000 1101 1000
Base Mod: 128 80 0000 0000 1000 0000
Curr Mod: 260 104 0000 0001 0000 0100
Base Mod: 256 100 0000 0001 0000 0000

97
Dynamic Hashed Files Next Split
The next split pointer determines the next group that will
be split
Next split = (Current Modulus) – Base Modulus + 1
Group split will be discussed in more detail later

98
Dynamic Hashed Files filesp
 Filesp (or data bytes) is the byte count of the item data in the
file
 This value excludes data stored in oversized item buffers
 Filesp, current modulus, and separation are used to determine
file load.
 File load determines if a split or merge operation is necessary

99
File load calculation
• Filesp / (current modulus * separation * 512) * 100
• Percentage of data in file compared to physical size
Given the following dynamic file values
• filesp: 31636
• current mod: 20
• separation: 4
File load = (31636 / 40960) * 100 = 77%

100
Filesp and fixtool
fixtool may report a filesp error when scanning a file that
is being actively updated
The filesp value is maintained in the T30FILE memory
structure.
Minor changes to the file will change the filesp value but
the information is not immediately flushed to disk.

101
Dynamic Hashed Files Large Record
 Large record size specifies the number of bytes at which a record is
considered a “large record”
 Large records are stored in the OVER.30 file
 By default large record size is 80% (less record header) of the buffer size
 A 32-bit file with separation of 4 will have large record size set to 1628
 (2048 – 12) * .8 = 1628

102
Dynamic Hashed Files Hashing Algorithm
Dynamic files have two possible hashing algorithms
GENERAL which is type 20
SEQ.NUM which is type 21
These file types are only valid with dynamic files

103
Dynamic Hashed Files Split Factor
Split factor is expressed as a percentage, default 80
After an update the file load is calculated using the
formula discussed previously
If the calculated load is greater than the split factor, a
split operation will occur

104
Dynamic Hashed Files Merge Factor
 Merge factor is also expressed as a percentage with a default
of 50
 If the calculated file load is less than this value a merge
operation will occur
 Note - the merge operation will only occur if the current
modulus is larger than the minimum modulus
 The file will never decrease below minimum modulus

105
Dynamic Hashed Files T30FILE
 T30FILE is a configurable shared memory structure, which is
scanned when opening a dynamic file
 If the file is found, the usage count is incremented by 1; otherwise, a
new entry is added to the table
 The usage counter is decrement by 1 when the file is closed
 The table entry is cleared when usage count hits 0
 The table size is based on the T30FILE uvconfig parameter

106
Dynamic Hashed Files T30FILE
The dynamic file header is updated periodically from the
shared memory table
The memory structure gets flushed to the header on the
following operations:
 On a split operation
 On a merge operation
 When the file is closed by a UniVerse process

107
Dynamic Hashed Files SELECT
 An active select counter is maintained in the T30FILE
memory structure
Split and merge operations are blocked when the file is
being selected
This prevents potentially skipping records which were
merged or processing split records multiple times

108
Dynamic Hashed Files ALLOWNFS
Closing a dynamic file flushes information from memory
to the file header
Accessing dynamic files from multiple UniVerse
systems via ALLOWNFS may cause corruption
If UVNet is not being used, out of date information from
the remote system can be flushed to the local file
header
This can occur even when the remote access is simply
to read the file.

109
Dynamic Hashed Files ALLOWNFS
An example of file corruption
User A on Sys1 opens file local to Sys1
User B on Sys2 opens the same file via ALLOWNFS
User A updates the file causing split
User A closes the file
User B closes the file flushing memory from Sys2 which
wipes out header changes made by User A

110
SUSPEND.FILES and Dynamic Files
 The SUSPEND.FILES command can be used to pause database
updates
 This can be used when making a backup copy of the database
 Running fixtool on the file copy may report an error related to filesp
 Minor changes to the file may change filesp but not cause a split or
merge
 In this situation, the updated filesp value would not be flushed to disk
 The fixtool error in this case can be ignored

111
Dynamic and Static Hashed Files Header Info
Additional header info
• NLS
• Indexes
• RESIZE
• Automatic Data Encryption
• Transaction Logging
• SQL Tables
• FILE.USAGE
• uvbackup
• UV Replication
• AUDIT

112
Dynamic and Static Hashed File Header NLS
When using National Language Support (NLS).
Map name - Map to use when reading and writing to the
file
Map checksum - Checksum used to ensure the map
has not been altered
Sort map name - Map to use when sorting file
Sort map checksum - Checksum used to ensure the
sort map has not been altered

113
Dynamic and Static Hashed File Indexes
Status flags
• Indexes are enabled or disabled
• If updates were applied while indexes were disabled
Pathname for indexing directory
• Maximum of 256 bytes for revision 12 (x0C)
• Maximum of 128 bytes for older revisions

114
Dynamic and Static Hashed File RESIZE
Status flag to indicate the file is being resized
concurrent
Old MKDBTimestamp. Used to validate file when
file resized while rotated out
Current resize path which defines the temporary
file being used to resize the file

115
Dynamic and Static Hashed File Header Encryption
A 32-bit value which exists for an encrypted file
• Revision (8bits)
• Encryption key (24bits)
The current revision level is x01
The key is used to locate the file in &ENCINFO&
The &ENCINFO& file contains information on the
encryption status of the file

116
Dynamic and Static Hashed File Header UVTL
Contains the record id to the UV.TRANS record
If UVTL Checkpoint mode is enabled
• Inconsistent bit. Indicating the checkpoint process found
structural problems with file
• Last UVTL log where file was processed by checkpoint
deamon
• Last UVTL log containing a warmstart transaction for this file

117
Dynamic and Static Hashed File Header SQL
Schema
Column count
Column map
SICA Pointer (discussed previously)

118
Dynamic and Static Hashed File Header Statistics
 UniVerse file usage statistics stored in the file header
• FILE.USAGE
• FILE.USAGE.CLEAR
• FILE.USAGE.OFF
 Seventeen 32-bit integer counters recording statistics related to
file I/O operations
 UTC Timestamp of last FILE.USAGE.CLEAR operation

119
Dynamic and Static Hashed File Header uvbackup
The values maintained by uvbackup are maintained by
certain other commands such as COUNT
• UTC Timestamp when uvbackup or count occurred
• Flag indicating type of access
• Count of the number of items in the file

120
Dynamic and Static Hashed File Header UVDR
UniVerse Data Replication stores information in the file
header
Subscriber files are marked with a read only flag
Flag (1=Published, 2=Subscribed)
Key to either UV_UDRPUB or UV_UDRSUB file item
Note - this is NOT related to the new U2 Replication
functionality

121
Dynamic and Static Hashed File Header Misc
Audit log flag - Indicates the file is used by the recently
added Audit Logging feature

123
UniVerse Files Primary Buffer
The primary buffer is the starting point for all groups
The primary buffer for group 1 follows immediately
after the header
For a dynamic file, the primary buffers are only
located in the DATA.30 file

124
The location of primary buffer is determined based on
the physical offset from the beginning of the file
The hashing algorithm determines which primary
group the key hashes
The RECORD command can be used to return the
group in which a record hashes
The primary buffer address can be calculated using
the primary group number and the file separation

125
A record hashing to group 100 in a separation 4 file
will be in the primary group starting at address
• 206848 decimal
• 32800 hexadecimal
IF MOD(Separation,2)
THEN HdrSz = 1024
ELSE HdrSz = Separation * 512
Addr = ((Group - 1) * Separation * 512) + HdrSz

126
UniVerse Files Primary and Overflow Buffers
The structure of primary and overflow groups are the
same
Overflow groups extend a primary group which is full
A buffer is a collection of items/records
All space within a buffer (separation * 512) is
accounted for by UniVerse

127
UniVerse Files: Item/Record Types
There are two type of items:
Normal
• Hashed: len(item) < (separation * 512)
• Dynamic: len(item) < Large record size
Oversized

128
UniVerse Files Item/Record Types
The illustrations that follow will use:
TEST32: 32-bit hashed file, separation of 4
TEST64: 64-bit dynamic file, separation of 8

129
UniVerse Files Normal Item
The int in a 32-bit file is 4 bytes and 8 in a 64-bit file
The following information exists for an item
• (int) Forward pointer
• (int) Blink
• (int) Flags
An existing item has the following
• Key
• Segment mark
• Record data

130
UniVerse Files: Normal Item
Item 1 group 1 TEST32
00000800 : 00000838 00000038 00000803 56455249 | ...8...8....VERI
00000810 : 46592E53 514CFF56 FE42502E 4F2F5645 | FY.SQL V?BP.O/VE
Item 1 group 1 TEST64
0000000000001000 : 0000000000001028 0000000000000028 | .......(.......(
0000000000001010 : 00000C0300000000 4639FF44FE39FEFE | ........F9 D?9??
0000000000001020 : FE313554FE530002 0000000000001068 | ?15T?S.........h

131
UniVerse Files: Normal Item Forward Pointer
Forward pointer indicates the position of the next item
in the group
If the forward pointer points outside the bounds of the
current buffer it must be buffer aligned
That is, it must point to the beginning of another
buffer
A forward pointer of zero indicates the last item in the
group.

132
UniVerse Files Normal Item Blink
The blink or backward link value is similar to a
checksum
It is a bitxor of the size of the current item and a
previous address
It is used by UniVerse to validate that the data
and pointers are not corrupted

133
For the first item in a buffer, the previous address
is the beginning address for the prior group buffer
For other items in the group, the previous
address is a pointer back the prior item within that
buffer

134
 Record size is calculated by subtracting the
current address from the forward pointer
 If the forward pointer is zero or points to another
buffer
• The size is calculated by subtracting the current
address from the end of the current buffer

135
00000800 : 00000838 00000038 00000803 56455249 | ...8...8....VERI
00000810 : 46592E53 514CFF56 FE42502E 4F2F5645 | FY.SQL V?BP.O/VE
Size = x838 – x800 (x38 or decimal 56)
Previous address = x38 ^ x38 (zero, indicating primary buffer)
00000838 : 0000085C 00000824 00000C03 44495658 | ......$....DIVX
00000848 : FF56FE72 61646978 FE45FE47 56FE4449 | V?radix?E?GV?DI
Size = x85c – x838 (x24 or decimal 36)
Previous address = x24 ^ x824 (x800 or decimal 2048)

136
UniVerse Files Flag Field
The flag field is used to define certain properties of the
item
 The size of the flag field is either 32-bit or 64-bit
depending on the file style
 However, only 16 bits are actually used

137
Big endian (treating as 16-bit int)
Bit Description Bit Description
1Uvbackup weekly 9Oversized item
2Uvbackup daily 10Oversized item buffer
3Fwd to OVER.30 11Record padded
4Blink to DATA.30 12New style padding
5Group process by resize 13Part file buffer
6not used 14Sica buffer
7Records deleted from group 15Free item
8not used 16Not used

138
The bit values on a little endian machine will be
reversed
More detailed slides on the flag field are included at the
end of the presentation

139
UniVerse Files: Normal Item
For a normal item, the record key will immediately
follow the flag field
The record key will be terminated by a segment mark
(xFF)
The record data follows the segment mark

140
UniVerse Files: Large Items
In a static hashed file, a large item is any item whose
size exceeds the file separation
In a dynamic file, a large item is any item whose size
exceeds the LARGE.RECORD.SIZE parameter

141
Large items are handled differently between static and
dynamic hashed files
For static hashed file:
• Data from the end of the record is removed and written to
oversized buffers
• This process continues until the remainder of the record fits into
the current buffer

142
In a dynamic file all of the record data is stored in
oversized buffers in OVER.30
The primary buffer contains:
• Record key
• Pointer to oversized buffer address
• A count of the number of oversized buffers used

143
When the ‘large item’ bit is set in the flag field, the flag
field will be followed by two integer values:
• The first is the address of the first oversized buffer
• The second is buffer count for the item
This differs from a normal item, where the item key
immediately follows the flags field
The size of the integer values are based on whether the
file is 32-bit or 64-bit

144
00000400 : 000005A0 000001A0 00000903 00004200 | ..............B.
00000410 : 0000001F 58FF4C69 6E65206E 756D6265 | ....X Line numbe
Address for first buffer
Number of buffers needed to store item
In a dynamic file the address is always pointing to a
location in the OVER.30 file.

145
The oversized buffer contains forward pointer, blink,
and flags fields in the same format as already
discussed
The size for calculating or verifying the blink is always
equal to the buffer size based on separation
00004200 : 00004000 00000600 00000A00 37FE4C69 | ..@.........7?Li
00004210 : 6E65206E 756D6265 72203238 FE4C696E | ne number 28?Lin

146
Summary
What we covered today
The different types of files available for use in
UniVerse
The UniVerse file header structure
Dynamic hashed, static hashed, and BTree files
The group and record structures associated with the
dynamic and static hashed files

147
Disclaimer
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED
IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.
IN ADDITION, THIS INFORMATION IS BASED ON ROCKET SOFTWARE’S CURRENT PRODUCT PLANS AND STRATEGY,
WHICH ARE SUBJECT TO CHANGE BY ROCKET SOFTWAREWITHOUT NOTICE.
ROCKET SOFTWARE SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR
OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
• CREATING ANY WARRANTY OR REPRESENTATION FROM ROCKET SOFTWARE(OR ITS AFFILIATES OR ITS OR
THEIR SUPPLIERS AND/OR LICENSORS); OR
• ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF
ROCKET SOFTWARE.

148
Trademarks and Acknowledgements
The trademarks and service marks identified in the following list are the exclusive properties of Rocket Software,
Inc. and its subsidiaries (collectively, “Rocket Software”). These marks are registered with the U.S. Patent and
Trademark Office, and may be registered or pending registration in other countries. Not all trademarks owned by
Rocket Software are listed. The absence of a mark from this page neither constitutes a waiver of any intellectual
property rights that Rocket Software has established in its marks nor means that Rocket Software is not owner of
any such marks.
Aldon, CorVu, Dynamic Connect, D3, FlashConnect, Pick, mvBase, MvEnterprise, NetCure,
Rocket, SystemBuilder, U2, U2 Web Development Environment, UniData, UniVerse, and
wIntegrate
Other company, product, and service names mentioned herein may be trademarks or service marks of
others.

UniVerse Files

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a UniVerse Files

Similar a UniVerse Files (20)

Último

Último (20)

UniVerse Files