HP MAP File Transfer Access and Management
File Transfer, Access, and Management, or FT AM, is an OSI Standard that defines the framework upon which layer seven file transfer services can be built for accessing and managing files across open systems.
by Steven W. Manweiler
THE BASIC DISTRIBUTED file system capability for OSI communication is defined by the file service standard called File Transfer, Access, and Maoage-menl (FTAM) FTAM defines facilities for Ihe transfer of files between open systems and provides a framework for accessing and managing files across open systems.
While FTAM provides functionality similar to conventional lile transfer services, FTAM is not strictly a file transfer service. It also provides the ability to perform functions such as reading and writing parts of remote files, and obtaining or modifying attribute information associated with a remote file. FTAM should not be thought of only as a file transfer service, but rather as a framework on which services such as file transfer can be built. In fact, a service such as a distributed file system could be built using a suitable FTAM implementation. However, the main strength of FTAM is that it is a system independent file service. Any two vendors with FTAM implementations can communicate with each other using FTAM regardless of the types of file systems supported by those vendors.
FTAM is defined in the International Standards Organization [ISO) document ISO 8571,1 This document defines the file service, the file protocol, and a common model of a file system. However, it does not define a user interface. Various committees are currently working on defining interfaces for FTAM, or have already done so. One example of an existing programmatic interface for FTAM was defined by the North American MAP/TOP [Manufacturing Aulomation Protocol/Technical Office Protocol) Users Croup.2
Another standards organization involved with FTAM is the U.S.A. National Institute of Standards and Technology [NIST)."' Four times a year NIST hosts a workshop in which vendors meet to develop implementation agreements for existing OSI standards.
This article describes the basic concepts of FTAM. the functionality of FTAM io HP MAP 3.0, and some aspects of HP's design and development of FTAM.
Overview of FTAM
The FTAM standard (ISO 8571) defines FTAM in three parts: ihe virtual file store [VFS), the sendees, and the protocol. The virtual file store defines how to handle different files structures, the services define the functionality that can be requested by the user, and the protocol defines the order in which the services can be used.
Virtual File Store
The virtual filestore, or VFS, is the most important aspect of FTAM. Because of the wide variety of file systems and file structures available on different systems, a common representation of a file system is needed to facilitate the transfer and access of files between dissimilar systems. The VFS is this common [virtual) file system. It defines a set of actions [operations) that can be performed on virtual files, a sel of attributes associated with virtual files, and the structure (model) of virtual files. Each FTAM implementation is responsible for mapping the features of the VFS onto the mechanisms provided by its file system. VFS File Actions. The actions that can be performed on VFS files are similar to those provided by most file systems. Users can create and delete a file, open and close a file, read or change the attributes of a file, read or write all or parts of a file, and erase or move to various locations in a file.
VFS File Attributes. A set of attributes is defined for each file in ihe VFS. These attributes are properties that, aside from the data in the file, completely describe the file, A file has basic attributes such as its name, its type, and the actions that maybe applied to that file, along wilh attributes that specif accounting information, history, user access restrictions, security restrictions (e.g.. passwords), and concurrent access restrictions.
The set of attributes supported by the VFS is large enough to have a logical mapping to any file system. In fact, most real file systems support a much smaller set of attributes than that defined by the VFS. This set of attributes contains mandatory and optional attributes. All FTAM implementations must support the set of mandatory attributes, and may choose to support an optional attribute if, for example, lhat attribute has a natural mapping onto that implementa-tioo's file system.
An example of a mandatory attribute is the permitted actions attribute. FTAM defines a set of actions that can be performed on an existing file. These actions include read, insert, replace, extend, erase, read attributes, change attributes, and delete file. The permitted actions attribute for a file defines a subset of these actions that can be applied to that file. Any FTAM user is capable of performing this subset of actions on the file. FT AM defines an optional attribute called access control that is used when a finer granularity of access restriction is needed for a file. For example, it allows the creator of a file to designate the users that can access the file, and by setting the permitted actions for each user, designate how each user can access the file. If this attribute is not used, any user can access the file.
Another attribute of the virtual file store worth mentioning is its file locking mechanism, which is called concurrency control. When accessing a file, an F'TAM user can specify a separate lock for each file access action. For example, a user cart specify a shared lock for the read action and exclusive locks for the write actions such as insert, replace, extend, and erase.
Virtual File Store File Model
The most important feature of the virtual file store is its file model. This file model specifies the file data types, the file structure, the file access operations, and the encoding of file data for transfer.
The VFS uses a very general hierarchical model to describe the structure of all files (future addenda to ISO 8571 may include the definitions of other file models such as a network model or relational data bases). In this model, a virtual file is an ordered tree of any depth. Associated with each node in the tree is ail access point in the file called a file access data unit (FADUJ. These access points may or may not have user data associated with them. The user data is structured into groups called data units. Fig, 1 shows this file model.
it is possible to access all or part of the information stored in the tree. When accessing a specific node in the tree, all data contained in the subtree of a node is available to the user. Thus, to read a complete file, the user can either traverse the entire tree (preorder traversal is used), or the user can read all data associated with the root node of the tree in one operation.
Abstract Syntax Notation One (ASN.l) is used for specifying the types of information in a file and how this information is encoded for transfer. ASN. 1 provides a common way to specify the format of the data that is to be transferred by FTAM. See the article on page 11 for more about ASN.l.
Most of the file structures of existing file systems are more restricted than that specified by the VFS file model. For this reason the VFS defines the nation of a document type. A document type defines practical file types by imposing restrictions on:
■ The structure of the file
■ The types of the data in the file and the amount of structuring information present in the file
■ The actions that may be performed and when they may be performed
■ The mechanism of encoding data for transfer.
Fig. 2 shows the structure of a document type that is called FTAM-1 in the FTAM standard. FTAM-1 models simple text files such as an HP-UX text file, which is simply a sequence of bytes without any structure. This type of file consists of one FADU that contains one data unit. The data unit contains the contents of the entire file. The FTAM-1 document type restricts the file's contents to text (i.e., the file may not contain integers or floating-point numbers). The document type definition also places restrictions on how files may be transferred and accessed. For example, data in FTAM-1 files can only be written at the end of the file, and it is not possible to read parts of an FTAM-1 file, that is, the entire file must be read,
Fig, 3 shows another document type called FTAM-2, This document type models record-oriented files. In this model, each record is represented by a FADU and each FADU contains one data unit that represents the contents of the record. In addition, a root node without an associated data unit is included in the structure. This gives the user the ability to address the entire file in one operation. This is useful for situations in which the user wishes to read the entire file. If there were no root node, the user would have to read the file one record at a time in separate operations. Again, the contents of data units for Files of this document type are restricted to text.
The FTAM-2 document type definition also places restrictions on how various operations can he applied to files. For example, the erase operation can only be applied to the root FADU (i.e., only the whole file may be erased). Furthermore, new records can only be added to the end of the file and existing records cannot be changed or updated (i.e., only read access is allowed on existing records).
Fig. 1. General FTAM file model.
s- File Access Dala Unit /
Root Node Data Unit
Each FTAM implementation supports a certain set of document types. Developers will usually choose to support document types that closely model the file types supported by their operating systems plus document types that model files of other vendors for interoperability purposes. For example, the document types FTAM-1 and FTAM-3 map naturally onto HP-UX file systems, while the document type FTAM-2 maps naturally onto record-oriented file systems. FTAM-3 is identical to FTAM-1 except that FTAM-3 files contain only binary data, HP's MAP 3.0 FTAM 800 supports all three of these document types.
Services and the Protocol
FTAM defines a set of services that can be requested by the user, The services available in FTAM allow users to establish, release, and abort connections, create and delete files, open and close files, read and change the attributes of a file, read, write, and erase a file, and move to different locations in a file. The order in u:hich these services can be osed is also defined and represents the FTAM protocol. The protocol is strictly connection-oriented and follows the requestor/server model [see Fig. 4). In this model, a requestor (initiator) makes requests to a specific server (responder). The responder can handle requests simultaneously from more than one initiator. In FTAM. the initiator is the user and the responder acts as the eniity manipulating the [possibly remote) virtual file store.
During protocol exchanges, the initiator and responder build a series of nested stages or regimes. There are four such regimes that can be constructed during the course of an FTAM dialogue. These regimes are:
The FTAM regime (a connection is made between initiator and responder)
The file selection regime (a specific file is selected or created)
The file open regime (the selected file is opened) The data transfer regime (data is transferred between the initiator and responder). FTAM Regime. The FTAM regime is entered w'hen an initiator establishes a connection with a responder. When establishing a connection, the initiator and responder negotiale what can be done on the connection. This negotiation is necessary because implementations are not required to support all of the FTAM functionality. This negotiation takes place in two steps. First, the initiator sends a connect request that contains information concerning what operations the initiator wishes to perform on the connection. Next, the responder issues a connect response back to the initiator indicating which of the requested operations the responder supports.
The initiator may terminate the FTAM regime (i.e., close the connection) using the FTAM terminate request. A connection may be terminated at any time using the FTAM abort service. This is an unorderly method of terminating a connection. For example, if a connection is aborted during a data transfer, the file being accessed may be left in an undefined state.
File Selection Regime. Once the FTAM regime has been established, the next step is to select or create a file. All operations in subsequent regimes are applied implicitly to this file. When the initiator is finished with this file, a new file can be specified by exiting and reentering the file selection regime. Once the file selection regime is established, the initiator can request file management operations without entering any other regimes. For instance, the initiator may read the attributes or modify the attributes of ihe currently selected file. The file selection regime can be terminated by ihe initiator by deselecting or deleting the file using the FTAM deselect or FTAM delete service. The FTAM protocol then reenters the FTAM regime. File Open Regime. Before any data transfer can take place, the selected file must first be opened. This is the purpose Of ibe file open regime. When a file is being opened, the initiator specifies the data transfer operations to be performed on the file. Once the file open regime is established, two data access operations may be performed prior to the establishment of the data transfer regime. The initiator may either erase all or part of the file or move to a specific location in the file. When all data access and data transfer operations are completed for the currently opened Tile, the file open regime can be terminated by the initiator using
the FT AM file close service. The FT AM protocol then reenters the file selection regimeData Transfer Regime. The innermost regime of the FTAM protocol is the data transfer regime. In this regime, all or part of a file is transferred unidirectionally between the initiator and responder. There are three steps involved in this regime. First, the initiator specifies the direction of the data transfer (i.e., reading or writing) and the parts of the file to be transferred. Next, the specified data is transferred. Finally, an orderly end-of-data transfer is performed. This entire process may be performed more than once without having to close the file. Once the data transfer regime is completed, the FTAM protocol reenters the file open regime,
FTAM defines an optional error recovery protocol that facilitates recovery from errors such as a broken connection or the failure and restart of a remote host.
HP MAP 3.0 and FTAM
HP's MAP 3.0 FT AM/800 product is completely compliant with the FTAM standard ISO 8571 and the NIST Phase II Implemention Agreements.3 HP's virtual file store supports a complete implementation of three document types: FTAM-i (unstructured text files), FTAM-3 (unstructured binary files), and FTAM-2 (record-oriented text files). The VFS also supports the full functionality of concurrency control, the majority of the access control functionality, and many other attributes. The protocol implementation is also relatively complete. All major components of the protocol are implemented with the exception of restart and recovery.
HP MAP 3.0 FT AM/800 consists of three different processes: a user process, a service provider process, and a responder process [see Fig. 5], This implementation follows
Network
Fig. 4. FTAM requestor/server model the HP MAP 3.0 upper layer architecture described on page 11.
User Process
The user process contains the user's application program along with the MAP FTAM library. The MAP FTAM library-is a set of C functions callable by a user's C program. The functions in the library provide the MAP FTAM application program interface for user applications. The library validates parameters passed in calls to its functions and translates these calls into requests to the service provider process. The service provider process and the user process communicate using HP-UX domain sockets.
The MAP FTAM interface provides two types of calls: low-level calls and high-level calls. Low-level calls perform one FTAM service request while high-level calls perform a sequence of FTAM service requests. An example of a low-level call is tt_create(), which creates an FTAM file, and an example of a high-level call is ftJcopyQ. which performs all of the FTAM service requests necessary to copy a filé from ooe location to another.
To establish a connection to a remote responder using
Machine A
User Process
Machine A
User Process
HP-UX System Cells
Interprocess Communication
OSI Protocol Stack
Machine B
HP-UX System Cells
Interprocess Communication
OSI Protocol Stack
Machine B
- HP-UX System Cells
|
OS! | |
|
Interprocess | |
|
Communication | |
|
ost | |
|
Protocol | |
|
ULA = Upper Layer Architecture Fig. 5. FTAM implementation for HP MAP 3.0 the MAP FTAM interface, the user must specify the address of that responder. The MAP FTAM interface supports two forms of addressing: directory distinguished names and presentation addresses. If the user supplies a directory distinguished name, the MAP FTAM library issues a request to the X.500 directory service, which resolves the name, and returns the appropriate presentation address. The MAP FTAM library can then issue the connect request to the lower layers of the OSI stack. If the user provides a presentation address, the X.500 service is not used. See the article on page 15 for more information about X.500. Each call has Iwo modes of operation: synchronous and asynchronous. In synchronous mode, the request completes entirely before control is returned back to the caller. In asynchronous mode, the call returns once the request is sent to the initiator protocol machine. Another call is provided so the user can check the status of outstanding requests and receive the results of requests when they complete. Service Provider Process The service provider process, which is forked by ihe user process, contains three modules: the MAP interface provider. the initiator protocol machine, and the VFS module. The function of the MAP interface provider is to translate MAP FTAM requests into requests to the initiator protocol machine. For low-level MAP requests, this translation is trivial since the low-level MAP FTAM requests are modeled after (he initiator protocol machine services. However, the high-level MAP FTAM requests like ft.tcopy must be issued in a series of calls lofhe initiator protocol machine. The initiator protocol machine accepts FTAM service requests from the MAP interface provider and validates the requests to ensure thai they are made in valid protocol sequence and that the parameters are compliant with ISO 8571 and the NIST Phase II implemention Agreements. AfteT validation, one of two actions takes place. If the request involves a file on a remote system. Ihe request is encoded using ASN.1 and sent to the responder operating on the remote system. If the request involves a local file, a request is issued to the VFS module, which accesses the file from the local file store. Note that requests for local files could have been handled in the same manner as requests for remote files. This different method of local file access was added as a performance enhancement. Responder Process The responder process, which runs as a daemon (server) process, contains the responder protocol machine and the VFS modules. It waits for connect requests from an initiator and when one is received, it forks the child process that handles all operations on lhal connection. When the connection is terminated, the child process exits. The VFS modules are a library of functions that are linked w'ith both the service provider process and the responder process. The implementation is straightforward with a few exceptions. The exceptions occur in the areas of concurrency control and attribute support. Since HP-UX does not provide a file locking mechanism sufficient to handle FTAM's concurrency control features, shared memory was used to implement concurrency control. Because FTAM requires the support of attributes that do not map naturally to HP-UX file systems, shadow files are used to store various FTAM file attributes. One shadow file exists for each file accessible with FTAM. A shadow file resides in the same directory as the file it describes and makes use of a "_" prefix on the filename to make it transparent on ao HP-UX Is (list contents of a directory] command. File Copy Scenarios Three different scenarios exist when copying a file using FTAM. The first and most common scenario occurs when a file is copied from a remote node to the local node. In this scenario, one user process, one service provider process, and one responder process exist. The VFS in the service provider process handles all access to the local HP-UX file system while the VFS in the responder process on the remote node handles all access to ihe remote HP-UX file system. The data is transferred between the service provider process and the responder process using the lower layers of the OSI protocol stack. The second scenario occurs when a file is copied from one location in the local file store to another location in that same local file store. In Ihis case, there is one user process and one service provider process, but no responder process. The VFS in the service provider process handles all access to the local file system. The third and final scenario is the situation in which a file is copied from one remote file store to another (or the same] remote file store, In this case, there is one user process, one service provider, and two responder processes. The responder on the source node transfers the contents of the source file to the service provider process. The service provider process receives the data from the responder process on the source node and sends the dala to ihe responder process on the destination node. The responder process on the destinalion node receives ihe data from the service provider process and writes the data to the destination file. A more efficient method oF transferring the data would be to have the two remote responder processes communicate directly with each other. This method oF communication is not possible with the FTAM protocol. FTAM Design and Development The development of the FTAM product involved project teams from HP's Colorado Networks Division (CND) in Fort Collins. Colorado and Information Networks Division (IND) in Cupertino, California, There was also coordination with HP's Roseville Networks Division (RND) in Roseville, California, where the HP OSI Express card was developed. The FTAM initiator and responder protocol machines and the VFS module were developed at IND at the same time ihe MAP FTAM interface was being developed at CND, Since some of the interface code resides in the same physical process as the protocol machines, a great deal of communication was necessarv to ensure the compatibility oF the module interfaces. The need for close communication between geographically separate divisions, coupled with the large code size of FTAM [75 KNCSS), created a complex engineering problem that forced the project teams to follow several "best practices" and to create a Few oF iheir own, MAP FTAM Interface Development Because the design effort and FTAM expertise were distributed over two organizations, the design of the MAP FTAM interface for FTAM was difficult. At the time, the protocol machines and the NTS modules were already in the coding phase. Working from the MAP FTAM interface specification and the external specifications of the FTAM protocol machine modules, the interface designers used structured design methods for the initial design. Structure charts were used to document the high-level design, and proved to be a valuable tool for communicating the design to IND FTAM engineers at a high-level design review. This review and the tools helped to uncover many design issues that would have gone unnoticed until the integration and testing phase, which might have jeopardized the timely release of the product. The next step wras the low-level design. In this step, pseudocode for the interface modules was developed that contained the major logic in the design. Again, a design review was held to review this pseudocode and more defects w'ere uncovered. Next, the code was developed using a technique called sliced implementation. In this approach, complete vertical slices of functionality are implemented in several stages. This approach worked well with the MAP FTAM interface since this interface contains about thirty calls. These calls were grouped into four disjoint sets and implemented in four slices. The first slice contained the connection management related calls, the second slice contained the lowr-level file management calls (e.g., create, open, close], the third slice contained the low-level data transfer calls, and the last slice contained the high-level calls. The development and module testing of each slice were performed in an efficient, pipeline fashion. When the developers finished a slice, the code was handed to another engineer who was in charge of module testing each slice. While this slice was being module tested, the developers worked on Ihe next slice. Module Testing The testing of the interface modules also posed an interesting problem for several reasons. First, as the slices were completed, they could not he integrated with the protocol and VFS modules since those modules were not finished with iheir early testing phases. Second, the interface modules reside in two differenl processes (user process and service provider process), and when the initial slices wrere being completed the utility routines for such functions as interprocess communication (messaging) and managing connections with remote machines (connection management) were not finished. Finally, the amount of code in the MAP FTAM interface modules alone is about 2G KNCSS and their interfaces with other modules are numerous and complex. Given these problems, we had to develop a special method of testing these modules. Writing C code to simulate the nonexistent modules would have been too time-consuming because each stub routine would have been too general, or numerous sets of stubs would have been written generating numerous executable files. Also, because of the amount and complexity of the parameters at each module interface, validation of results would have been difficult. To aid in this testing effort, a general module testing tool was developed. This tool facilitates the rapid development of module tests by alleviating some of the problems mentioned above. Testing using this tool involves two steps. First, the tester describes the relevant interfaces between the module being tested and the nonexistent modules. Given this interface description, the tool generates an executable file that contains the code of the module to be tested, the code for an interpreter for executing module test scripts, and code for transferring control between the module interfaces and the interpreter. The second step is to wrrite test scripts that are executed by the interpreter mentioned above. These test scripts are written in a language designed specifically for module testing. The language is similar to C but is scenario-oriented and contains facilities that simplify parameter validation. Because the language is interpreted, errors in test scripts can be quickly changed and rerun without having to recompile and relink. Using this language, the tester can specify the order of events at the interface of the module being tested, These events include calls to routines in the module and calls from the module to routines in nonexistent modules. Each occurrence of a call to a nonexistent routine is handled separately in the scenario. This simplifies the task of writing stubs yet a!lowrs a great deal of flexibility since the script code for each call can be tailored for that specific instance. By forcing the tester to specify the exact scenario for a test, a more rigid test environment is created where spurious calls by the module or calls made out of order cannot go undetected. At strategic points in a test script, the tester can place statements to validale various parameters and olher data values. These statements are assertions or print statements. The printing facility provided by the tool automatically formats the output according to the type of value printed. It can even prinl an entire recursive data structure. While a script is running, the tool prints out information about the events that have occurred. When any failure occurs. the tool reports the failure and exits. The exact location of the failure in the scenario is known immediately by the tester. This information helps in the debugging process. If the information printed hy the tool is insufficient to solve the problem, the script can be reexecuted in any conventional debugger such as xdtj. Product Test Suite The test suite for the Integrated FTAM product was designed and developed as if it were an actual producl. This means that in the early stages of design, much consideration was given to creating a simple but flexible test suite that had all the characteristics of an HP product such as usability and sup portability. Typically, there are multiple test suites for a product. For example, a producl may have a functional test suite, a reliability test suite, a sLress test suite, a performance test suile, and so on. Instead of having completely independent test suites for each type of testing, the FTAM test suite encompasses all types of tests. These suites were developed under a common umbrella, sharing a common architecture, interface and output format. Among the advantages gained by using this philosophy are that a large amount of common code can be shared across tests and the learning curve for supporting and using the tests is reduced considerably. The FTAM test suite features one uniform command line interface for all of the different types of tests. The interface contains many options that give the user a great deal of flexibility in controlling the degree of testing. Options exist for specifying the test or set of lest cases to run, the number of iterations for each test, how long to run the tests, the number of connections to use during testing, the names of the nodes (machines) to be used during testing, and so on. For example, the user can specify that a set of reliability tests run for 72 hours using a given set of nodes running under a certain level of stress. There is also a single configuration file that can be used to specify the values of various parameters needed during testing. The test suite can be run manually or automatically using an automated test tool such as the internal HP tool known as the scaffold." The output printed by the test suite contains only simple passed or failed messages. This allows a novice user to determine the results of a test easily. If a test fails, the parameter information of the call that failed is printed to aid in solving the problem. The output printed by the test suite is uniform (and deterministic) across all tests. Each test in the suite is written in a uniform manner so that all tests have some similar characteristics such as the same architecture, interface, and naming conventions. Once the design of one test is understood, the design of all tests for the product is also understood, A large number of library routines exist that were designed to be as test case independent as possible. These two factors greatly simplify the addition, modification, and deletion of tests or test cases. The time spent designing and developing the test suite has been worth the cost. The test suite has proven to be very usable and all members of the FTAM team have become expert users of the suite within a short time. Tests have been added and modified by different team members with very few problems. Other project teams have already adopted the architecture and methodology of this test suite in the test suites for their products. Finally, a point worth mentioning is that the amount of code for the test suite is actually smaller than that of the product. This is a noteworthy achievement given the complex nature of the MAP FTAM interface. Performance Once the initial product was integrated, performance tests were run to determine FTAM's file transfer throughput rate. We discovered that FTAM could only transfer a file at a rate of 3 kbytes/s on an HP 9000 Series 815 computer. For this reason, a performance task force was organized consisting of engineers representing all areas of the OSI architecture. This task force identified major improvement areas by taking performance numbers at different layers and then analyzing execution traces to determine the most significant areas for performance improvement. The areas selected included the addition of the VFS to the service provider process in the initiator, modification of transport configuration parameters, and reduction in the use of heap memory. With these changes, FTAM's throughput increased from its initial value to SO kbytes/s on two HP 9000 Series 815 machines, and to 225 kbytes/s on two HP 9000 Series 835 machines. Acknowledgments Many individuals contributed to the design and development of the FTAM product. The engineers and managers involved with FTAM were Suhas Badve, Rajiv Batra. Atul Bhatnagar, Alan Burke, )eff Conrad, Dave Hendricks, Lisa Kozlowski-Owen, Paul Melmon, John Smith, and Don Tiller. References 1. Information Processing Systems - Open Systems interconnection - File Trans/er, Access and Management, ISO 8571 [Parts 1-4], International Organization tor Standardisation, 1908. 2. Manufacturing Automation Protocol Specification, Version 3.0, General Motors Corporation. 1988. Stable ImpJementntion Agreements for Open Systems Interconnection Protooois, N1ST Special Publication 5H0-162, National Institute of Standards and Technology, December, 1E138. 4. C. D. Fuget rind B J, Scott, "Tools for Automating Software Test Packagii Execution," Hewieft-Pcckard Journal Vol. 37. no. | |
Post a comment