Controlling the File Transfer
The current Secure Shell protocol and current clients do not transfer
any information about the files to be transferred, only the file
contents as a byte stream. This is sufficient for Unix-type files if the
sender and receiver use the same CCS.
SSH Tectia Server for IBM z/OS needs more information: which transfer format to use,
what code sets are involved, and what the file characteristics
are. Some of this information may be transported in the protocol in
the future versions of SSH Tectia Client. In SSH Tectia Server for IBM z/OS, the information is
either encoded in the filename (Advice String), read from a
file transfer profile, or in case of file transfer client from
environment variables.
Advice String
Since the SFTP server decodes the name for the dataset, the filename
in the request can be used to convey other information. The following
filename convention is used: any filename that starts with
/ftadv: has the format
/ftadv:advstr/realfilename
where advstr is the advice string and realfilename is
the dataset name or a filename to be further processed by the server.
The advice string is a sequence of name=value pairs delimited by commas.
Some names have a one-letter abbreviation for easier usability.
Note: File transfer client applications support only advice string
options with abbreviation, that is, the parameters X, F, D, C, E, A, J, I,
T, O, R, B, L and M.
The file transfer server supports all names and abbreviations.
The advice string names and abbreviations are as follows:
-
TRANSFER_MODE=transfer_mode, X=transfer_mode The transfer mode specifies whether codeset and line delimiter conversions
are performed. Possible values are:
X=bin: Codeset and line delimiter conversions are not performed.
X=text: Codeset and line delimiter conversions are performed.
Default: none
Note: If TRANSFER_MODE is not given but both TRANSFER_CODESET
and TRANSFER_FILE_CODESET or TRANSFER_TRANSLATE_TABLE are present
conversions are performed.
-
TRANSFER_FORMAT=transfer_format, F=transfer_format The byte stream consists of the bytes that are transferred as payload in
the SFTP protocol packets. The byte stream has one of the following
formats: stream, line, or record. All three
formats may have data consisting of text, non-text data, or a mixture of
these. When creating an MVS dataset, a record that is longer than the
maximum or fixed record length for the dataset will be truncated. When
writing to datasets with fixed record lengths, short records will be
filled with binary zeroes if you use the record transfer format and with
blanks if you use the line transfer format.
F=stream: The stream transfer format contains the data bytes of
the dataset but no structural information. If a dataset with a fixed record
length is transferred with the stream format and recreated with the same
record length, the record structure will be preserved. Variable length
records will not be recreated properly if transferred with the stream format.
F=line: The line transfer format is record-based. It uses
delimiter characters to mark the end of a record. The delimiter character
may be a Carriage Return or a Newline. When writing to or reading from
datasets with ASA control characters, a Form Feed is also treated as a
delimiter. The table below shows the values of these characters in EBCDIC
and ASCII. Data sent to SSH Tectia Server for IBM z/OS in the line transfer format must be
in EBCDIC or must be converted to EBCDIC during the transfer.
Delimiter EBCDIC ASCII
Name Dec Oct Hex Name Dec Oct Hex
\r Carriage Return CR 13 015 0x0D CR 13 015 0x0D
\n Newline NL 21 025 0x15 LF 10 012 0x0A
\f Form Feed FF 12 014 0x0C FF 12 014 0x0C
Note that ASCII does not have a NL character, instead LF is used to
delimit lines.
Avoid conversions that transform an ASCII Line Feed (LF/10/012/0x0A)
into an EBCDIC Line Feed (LF/37/045/0x25) or an EBCDIC Newline
(NL/21/025/0x15) into an ASCII Next Line (NEL/133/0205/0x85).
Be aware that sending a double delimiter, e.g. \r\n
or \n\r, to SSH Tectia Server for IBM z/OS will result in two
records. The transfer-line-delimiter and
file-line-delimiter advice string attributes can be used to
cause the SSH Tectia Server for IBM z/OS server or client program to convert between the
line delimiter conventions.
SSH Tectia Server for IBM z/OS sends \n as the Server Newline Convention
in the server initialization SFTP protocol message.
When transferring line format data to and from MVS files with ASA line
printer control characters, SSH Tectia Server for IBM z/OS will convert between the
control characters and line delimiter characters, as described in the
IBM book z/OS C/C++ Programming Guide, SC09-4765-03, Chapter
8.
Datasets transferred in the line transfer format and recreated on a
mainframe will not necessarily be identical.
F=record: The record transfer format is record-based. Each record
is preceded by a length field consisting of a 4-byte big-endian binary
integer, which indicates the number of data bytes in the record. Note that
the format is not the same as the record descriptor word in datasets with
RECFM=V or VB.
A dataset that is transferred with the record transfer format can be
recreated as any dataset type.
Default: line.
-
TRANSFER_FILE_CODESET=codeset, D=codeset The data in the dataset has the specified codeset. codeset
is the codeset name that is known to the iconv function of the
system performing the conversion. The available codesets can be listed by
invoking the iconv command at a USS prompt with the -l
option:
> iconv -l
Default: none
-
TRANSFER_CODESET=codeset, C=codeset During the transfer the data has the specified codeset. codeset
is the codeset name that is known to the iconv function
of the system performing the conversion. The available codesets can be
listed by invoking the iconv command at a USS prompt with the
-l option:
> iconv -l
Default: none
-
TRANSFER_TRANSLATE_TABLE=translate_table, E=translate_table translate_table is the name of the table that specifies the codeset
conversion. If set, this attribute overrides the transfer codeset and file
codeset attributes. The table is always applied in the normal
direction, that is, the first character array is used for incoming
(from the line to the dataset) data and the second array for outgoing
data. If the opposite translation is needed, e.g. the dataset contains
ASCII and should be transferred as EBCDIC, the users (or their system
programmer) can prepare a table dataset with the character arrays in
reversed order (e.g. with the system utility CONVXLAT or by editing an
existing translate dataset).
-
TRANSFER_TRANSLATE_DSN_TEMPLATES=dsn_templates, A=dsn_templates dsn_templates specifies the search templates for the translate
table. Write '%T' to show the point where the translate table name (see above)
is to be inserted. Delimit the templates with a plus character. The data set name
templates must not contain slashes, instead they must be preceded by
two or three underscores. See Section Dataset Access.
The first translate table dataset that is found is used to perform the code
conversion.
Note! The translate table must translate line delimiters into EBCDIC NL
characters. See Section Transfer Format.
Default: none
-
TRANSFER_FILE_LINE_DELIMITER=line_delimiter, J=line_delimiter The transfer file line delimiter specifies the newline convention used
in the file. Possible values are:
J=mvs: The line delimiter used in the file is NL (\n, 0x0a).
J=unix: The line delimiter used in the file is NL (\n, 0x0a).
J=dos: The line delimiter used in the file is LFNL (\r\n, 0x0d0a).
J=mac: The line delimiter used in the file is LF (\r, 0x0d).
Default: none
-
TRANSFER_LINE_DELIMITER=line_delimiter, I=line_delimiter The transfer line delimiter specifies the newline convention used during
the file transfer. Possible values are:
I=mvs: The line delimiter during the transfer is NL (\n, 0x0a).
I=unix: The line delimiter during the transfer is NL (\n, 0x0a).
I=dos: The line delimiter during the transfer is LFNL (\r\n, 0x0d0a).
I=mac: The line delimiter during the transfer is LF (\r, 0x0d).
Default: none
-
TYPE=file_type, T=file_type The file type specifies the type of the dataset when the dataset is created.
T=hfs: The type of the created dataset is HFS.
T=po: The type of the created dataset is PDS.
T=poe: The type of the created dataset is PDSE.
T=vsam: The type of the created dataset is VSAM.
T=ps: The type of the created dataset is PS.
Default: po, if dataset name includes member, otherwise ps
-
RECFM=file_org, O=file_org file_org specifies the dataset organization. Possible values are all
valid combinations of the following letters:
F Fixed
V Variable
U Undefined
B Blocked
S Spanned or standard
M Machine line printer codes
A ASA line printer codes
Default: vb
-
LRECL=file_reclen, R=file_reclen Maximum record length or fixed record length.
Default: 4096, for VSAM, 80, if dataset organization is F or FB, otherwise 1024
-
BLKSIZE=file_blocklen, B=file_blocklen Maximum block size.
Default: none
-
SIZE=file_size, L=file_size Size estimate in bytes for dataset allocation.
Default: 1000000
-
DIRECTORY_SIZE=directory_size, M=directory_size Number of 256-byte records in the directory.
Default: 10
-
KEYOFF=key_offset Specifies the key offset. The position of the first byte of the key in
records of the specified VSAM dataset.
Default: none
-
KEYLEN=key_length Specifies the length in bytes of the keys used in the dataset.
Default: none
-
SPACE_UNIT=space_unit Unit of space allocation for a dataset.
Possible values are:
SPACE_UNIT=blks: Allocation unit is blocks.
SPACE_UNIT=cyls: Allocation unit is cylinders.
SPACE_UNIT=trks: Allocation unit is tracks.
SPACE_UNIT=avgreclen: Allocation unit is average record length.
Default: none
-
SPACE_UNIT_LENGTH=space_unit_length The size of the allocation unit if allocation unit (SPACE_UNIT) is set
to blocks (blks) or average record length (avgreclen).
Default: none
-
PRIMARY_SPACE=primary_space Primary space allocation for a dataset.
Default: none
-
SECONDARY_SPACE=secondary_space Secondary space allocation for a dataset.
Default: none
-
DATACLAS=data_class Specifies the data class of a dataset.
Default: none
-
MGMTCLAS=management_class Specifies the management class of a dataset.
Default: none
-
STORCLAS=storage_class Specifies the storage class of system managed storage.
Default: none
-
VOLUMES=volumes A plus sign (+) separated list of volumes a dataset will (or does, if
it already exists) reside on.
Default: none
-
UNIT=unit The type of device or group of devices that the dataset will (or does, if it already
exists) reside on (maximum length of 8). If an element exceeds its
maximum allowable length, it is truncated to that length.
It is also possible to specify a device address, although there is no need
for device addresses in this release. Precede a four digit address with an underscore.
Default: none
-
LIKE=like Specifies the name of a model dataset from which the RECFM,
BLKSIZE, and LRECL attributes are to be copied. The name
must be the full DSN of a cataloged dataset and must be preceded with three
underscores.
You must include the TYPE attribute when using LIKE unless
you are creating a PS data set and the model is a PS data set.
Default: none
-
REFDD=refdd Specifies the name of the JCL DD statement from which the attributes
are to be copied.
Note: The REFDD option is not available anymore in SSH Tectia Server for IBM z/OS
5.2.2 and later. REFDD is replaced by LIKE.
Default: none
-
PROFILE=file_transfer_profile, P=file_transfer_profile The file transfer profile specifies the named profile used for the
file transfer. The profile name is case-sensitive. With special
profile name P=% no profiles are used. This also prevents
profile matching based on file name.
Default: none
Example Advice Strings
Below is an example of a filename that requests the transfer of a PDS member
with the line transfer format and codeset conversion from EBCDIC to an ASCII
codeset.
/ftadv:D=IBM-1047,C=ISO8859-1,F=line/__personal.cntl/idlist
The following example requests the transfer of a dataset with the line
transfer format and codeset conversion using the translate table
USR1.SFTP.TCPXLBIN, if it exists, or TCPIP.SFTP.TCPXLBIN.
Different combinations of underscore and slash ("__", "_/",
"/_", or "//") in front of the filename indicate that the
file is an MVS dataset.
/ftadv:F=line,E=SFTP,A=___USR1.%T.TCPXLBIN+___TCPIP.%T.TCPXLBIN/__DATA1.FILE1
The example below names an HFS file to be transferred without changes.
Transfer mode is set to binary (X=bin) to avoid conversion and to
override any defaults set in the matching file transfer profiles or
environment files.
/ftadv:X=bin,T=HFS,F=stream/profcopy
The next example uses the named file transfer profile myprofile. The
advice string also sets the dataset codeset to ISO8859-15. This
value overwrites the value specified in the profile.
/ftadv:P=myprofile,d=iso8859-15/testfile
Note: Advice strings can also be used with directory names in
file transfer GUIs. With advice strings, file transfer can be controlled
in the GUI. For example in the Windows GUI, if you use the directory
/ftadv:P=WIN/____path/to/current/directory/, all file transfers
to and from the directory /path/to/current/directory use the
advice string /ftadv:P=WIN/. This means that all file transfers
use the named profile WIN (see Section File Transfer Profiles). Note also that the HFS root
directory must be given as ____ (four underscores) because of the
naming convention introduced in the product.
Note: Advice string parameters are case-insensitive, with the
exception of file transfer profile names. For example, P=win and
P=WIN point to different profiles.
This example uses the LIKE attribute when creating an MVS PS data
set with the name "userid.TEST.CPY1". The server copies
RECFM, LRECL, and BLKSIZE from the PDS
"COMP.DATA.CNTL".
/ftadv:like=___COMP.DATA.CNTL,t=PS/__TEST.CPY1
This example uses the LIKE attribute when creating a PDSE,
"COMP.CODE". The record characteristics are copied from a PDS,
"COMP.SOURCE". The PDSE is created when adding the first member,
PRG1. The new PDS is estimated to need 15 MB space.
/ftadv:like=___COMP.SOURCE,t=POE,size=15000000/____COMP.CODE(PRG1)
File Transfer Profiles
A file transfer profile is a mechanism for pre-configuring different types
of file transfers. Both mainframe clients and server use the same
profile mechanism. There are two types of profiles: named profiles and
filename-matched profiles.
Named profiles can be used with the file transfer advice string parameter
P. A named profile provides the default values for different
advice string parameters. Those default values can be overwritten with
advice string parameters.
The filename-matched profiles can be used for configuring advice string
default values when transferring files whose names match a certain
regular expression. Again, those default values can be overwritten with
advice string parameters.
File transfer profiles for SSH Tectia Server for IBM z/OS can be set in the files
/etc/ssh2/ssh_ftadv_config (globally for all users) and
$HOME/.ssh2/ssh_ftadv_config (for the specific user).
The syntax of the profile is:
%NAME|REGEXP ADVSTRING
Each profile starts with a profile name or an egrep style of
regular expression followed by one or more white spaces and the advice
string. The profile name must start with the percent (%)
character. The regular expression must not start with the %
character. If the % character needs to be the first character in
the regular expression, it must be escaped with a backslash (\).
Note that in order to get the regular expression escape character,
backslash, into a regular expression, it must be escaped by using
\\.
The profile name and regular expression must start from the beginning of
the line. There must not be any white space in front of the name or
regular expression.
An advice string is a comma-separated list of name-value pairs of type
name=value. There may be white spaces in the advice string. It
can span over multiple lines. There must be at least one white space
character in the beginning of a spanned advice string line. An advice
string must not contain the file transfer advisor marker /ftadv:/
or a filename.
Profiles with %NAME can be used with the advice string
P=NAME. Profiles with REGEXP are used only for matching
filenames. The first REGEXP that matches a filename is used.
Comments can be added to the file with a hash (#) character.
Everything on the line after # is ignored.
Example File Transfer Profiles
The following profile converts text files from Unix to MVS. ASCII is
converted to EBCDIC. This profile is only used if the advice string
contains the parameter P=UNIX.
%UNIX X=text,
F=line,
C=iso8859-1,
D=ibm-1047
The next profile converts text files from Windows to MVS. ASCII is
converted to EBCDIC. The line delimiter is converted from the Windows
style to the MVS style. This profile is only used if the advice string
contains the parameter P=WIN.
%WIN X=text,
F=line,
C=iso8859-1,
D=ibm-1047,
I=dos,
J=mvs
The following profile can be used when transferring text files from
MVS to another MVS. This profile is only used if the advice string
contains the parameter P=ZOS.
%ZOS X=text,
F=line
This profile can be used when transferring text files between the Unix
and MVS environments. All datasets created on the MVS side have fixed
blocked lines of 80 characters per line. Again, to use this profile,
P=FB80 needs to be added to the advice string.
%FB80 X=text,
F=line,
C=iso8859-1,
D=ibm-1047,
O=fb,
R=80
The following profile creates record format datasets with maximum record
length of 1024 bytes. The advice string P=REC is needed.
%REC F=record,
R=1024
The following profile can be used when transferring binary files.
%BIN X=bin,
F=stream
This next profile is used whenever a dataset name contains the string
TXT, txt, TEXT or text in it.
//.*(TXT|txt|TEXT|text).*$
X=text,
F=line,
C=iso8859-1,
D=ibm-1047
The following profile matches files that have the extension .txt,
.TXT, .c, .C, .h, .H,
.log, .LOG, .conf or .CONF. Codeset
conversion from ASCII to EBCDIC is performed.
.*\\.(txt|TXT|c|C|h|H|log|LOG|conf|CONF)$
X=text,
F=line,
C=iso8859-1,
D=ibm-1047
The following profile matches binary files with extensions
.gz, .Z, .tar and .bin.
.*\\.(gz|Z|tar|bin)$
X=bin,
F=stream
This profile matches all files and dataset names. Data is transferred
in text format and codeset conversion from ASCII to EBCDIC is
performed.
.*$ X=text,
F=line,
C=iso8859-1,
D=ibm-1047
Enabling Example File Transfer Profiles
File transfer profiles are not enabled by default. File transfer profiles
can be enabled by copying the example profile file
/etc/ssh2/ssh_ftadv_config.example to
/etc/ssh2/ssh_ftadv_config. File transfer profiles for SSH Tectia Server for IBM z/OS can
be set in the files /etc/ssh2/ssh_ftadv_config (globally for all
users) and $HOME/.ssh2/ssh_ftadv_config (for a specific user).
> cp /etc/ssh2/ssh_ftadv_config.example /etc/ssh2/ssh_ftadv_config
File Transfer Environment Variables for the Clients
In file transfer client (scp2 and sftp2), if no file
transfer profile is found, the default values for advice string parameters
are taken from the following environment variables. These are set in the
/etc/profile file (globally for all users) or
$HOME/.profile file for each user file (or another corresponding
file, depending on the users' shell):
SSH_SFT_ADVISOR_TRANSFER_MODE
SSH_SFT_ADVISOR_TRANSFER_FORMAT
SSH_SFT_ADVISOR_FILE_CODESET
SSH_SFT_ADVISOR_TRANSFER_CODESET
SSH_SFT_ADVISOR_TRANSLATE_TABLE
SSH_SFT_ADVISOR_TRANSLATE_DSN_TEMPLATES
SSH_SFT_ADVISOR_FILE_LINE_DELIMITER
SSH_SFT_ADVISOR_TRANSFER_LINE_DELIMITER
SSH_SFT_ADVISOR_FILE_TYPE
SSH_SFT_ADVISOR_FILE_ORG
SSH_SFT_ADVISOR_FILE_RECLEN
SSH_SFT_ADVISOR_FILE_BLOCKLEN
SSH_SFT_ADVISOR_FILE_SIZE
SSH_SFT_ADVISOR_DIRECTORY_SIZE
File transfer server does not use environment variables for default
advice string parameter values.
File Transfer Environment Variables for the Server
The environment variables for SSH Tectia Server for IBM z/OS file transfer can be set
in the files /etc/environment (globally for all users) and
$HOME/.ssh2/environment (for the specific user).
File transfer server uses the following environment variables:
SSH_SFTP_HOME_MVS
SSH_SFTP_STAGEFS_CACHE_SIZE_LIMIT
SSH_SFTP_STAGEFS_CACHE_ENTRY_LIFETIME
SSH_SFTP_STAGEFS_CACHE_REFRESH_INTERVAL
SSH_SFTP_DEBUG
SSH_SFTP_DEBUG_FILE
If SSH_SFTP_HOME_MVS is set to yes, the file transfer server
starts in the MVS side. The file transfer client sees USER prefix as its
starting directory. Default is no, the file transfer server
starts in the USS side.
The SSH_SFTP_STAGEFS_CACHE_SIZE_LIMIT variable specifies
staging cache size in bytes. It limits the use of system resources. The
default is 524288000 bytes (500 MB).
The SSH_SFTP_STAGEFS_CACHE_ENTRY_LIFETIME variable specifies
how many seconds one cache entry is stored in the cache. After the
lifetime has expired the entry is removed from the cache and system
resources are released. The default is 10 seconds.
The SSH_SFTP_STAGEFS_CACHE_REFRESH_INTERVAL variable specifies
how many seconds may pass until the cache is refreshed. The default is 5
seconds.
With SSH_SFTP_DEBUG, the debug level can be set for file
transfer server.
If SSH_SFTP_DEBUG_FILE is set, debug messages are stored to the
file named in the variable.
