File formats - Revision history

Boerner at 06:40, 4 March 2022

2022-03-04T06:40:44Z

@@ Line 61: / Line 61: @@
   END d
 In binary format, {{cc|BEGIN MYNAME}}, {{cc|END MYNAME}} and {{cc|DESCRIPTOR}} are stored in 50 byte character strings which '''must not''' be null terminated.
 lmt will assume block format for all files with file name suffix {{cc|.bin}} and {{cc|.blkcsv}}.

Boerner at 06:33, 4 March 2022

2022-03-04T06:33:12Z

← Older revision		Revision as of 06:33, 4 March 2022
Line 12:		Line 12:
	"*.csv" files must contain rectangular arrays, that is every line must contain the same number of comma-separated records.		"*.csv" files must contain rectangular arrays, that is every line must contain the same number of comma-separated records.

	lmt supports empty records, e.g.		lmt does not supports empty records, e.g.

	1,2,,4		1,2,,4

	~~which will~~ be filled with zero.		must be filled with some value (e.g zero).

	".csv" files may contain several commented lines at the top only where the comment character is "#".		".csv" files may contain several commented lines at the top only where the comment character is "#".

Boerner: /* ".csv" files */

2022-03-04T06:27:27Z

".csv" files

← Older revision		Revision as of 06:27, 4 March 2022
Line 9:		Line 9:

	== ".csv" files ==		== ".csv" files ==

			"*.csv" files must contain rectangular arrays, that is every line must contain the same number of comma-separated records.

			lmt supports empty records, e.g.

			1,2,,4

			which will be filled with zero.

	".csv" files may contain several commented lines at the top only where the comment character is "#".		".csv" files may contain several commented lines at the top only where the comment character is "#".

	~~The type of the file content is determined by its prospective use, that is~~
	*the data file is supposed to contain only real/float numbers which are transferred to integer if required,
	*a file containing an ordinary pedigree is supposed to contain only integer numbers,
	*a file containing a missing value indicator matrix is supposed to contain only character strings.

	== Block format ==		== Block format ==

Boerner: /* Block format */

2022-03-04T06:23:27Z

Block format

← Older revision		Revision as of 06:23, 4 March 2022
Line 59:		Line 59:
	END d		END d

	In binary format, {{cc\|BEGIN MYNAME}}, {{cc\|END MYNAME}} and {{cc\|DESCRIPTOR}} are stored in 50 byte character strings which '''must not''' be null terminated		In binary format, {{cc\|BEGIN MYNAME}}, {{cc\|END MYNAME}} and {{cc\|DESCRIPTOR}} are stored in 50 byte character strings which '''must not''' be null terminated.

			lmt will assume block format for all files with file name suffix {{cc\|.bin}} and {{cc\|.blkcsv}}.

Boerner: /* Block format */

2022-03-04T06:22:18Z

Block format

← Older revision		Revision as of 06:22, 4 March 2022
Line 58:		Line 58:
	5		5
	END d		END d

			In binary format, {{cc\|BEGIN MYNAME}}, {{cc\|END MYNAME}} and {{cc\|DESCRIPTOR}} are stored in 50 byte character strings which '''must not''' be null terminated

Boerner at 06:19, 4 March 2022

2022-03-04T06:19:19Z

← Older revision		Revision as of 06:19, 4 March 2022
Line 16:		Line 16:
	*a file containing an ordinary pedigree is supposed to contain only integer numbers,		*a file containing an ordinary pedigree is supposed to contain only integer numbers,
	*a file containing a missing value indicator matrix is supposed to contain only character strings.		*a file containing a missing value indicator matrix is supposed to contain only character strings.

			== Block format ==

			Files in block format can contain multiple records of very different data. That is, a single file can contain several different matrices, vector, and scalars and a mixture of those. Block files are important for supplying sparse matrices, large arrays(e.g. genomic relationship matrices), block-diagonal matrices (e.g. when residual variances are heterogenous) etc. Further, lmt may write intermediate output to a block file for having connected information in the same file.

			The general structure of a single block is:

			BEGIN MYNAME
			DESCRIPTOR
			DATA
			END MYNAME

			where {{cc\|MYNAME}} is a user-defined block name which must be unique, {{cc\|DESCRIPTOR}} is a comma-separated sequence of data descriptors, and {{cc\|DATA}} are the actual data.
			{{cc\|DESCRIPTOR}} contains at least 3 values:

			*type
			*kind
			*size
			*dimension

			where {{cc\|type}} is either {{cc\|int}} for integer, {{cc\|real}} for real or {{cc\|char}} for character. {{cc\|kind}} can be either {{cc\|scalar}} or {{cc\|array}}. {{cc\|size}} is the storage size in bit for a single value, for integer and real it should be 64, for character it should be the storage size of the longest character string in an array of strings. If {{cc\|kind}} is set to {{cc\|array}}, dimension is a vector of comma-separated values containing the dimensions of the array, where the length of the vector determines the number of dimensions. That is, for a one-dimensional array the vector is of length one, for two-dimensional the length is two and for three-dimensional the length is three. Currently only one, two and three dimensional arrays are supported.

			A block file in csv format containing an integer scalar, an integer vector, a real matrix, and a real scalar maybe:

			BEGIN a
			int,scalar,64
			5
			END a
			BEGIN b
			int,array,64,2
			1
			2
			END b
			BEGIN c
			real,array,64,2,2
			5,5
			5,5
			END c
			BEGIN d
			real,scalar,64
			5
			END d

Boerner at 05:52, 4 March 2022

2022-03-04T05:52:09Z

← Older revision		Revision as of 05:52, 4 March 2022
Line 4:		Line 4:
	*".blkcsv" for comma separated value ascii text files in block format		*".blkcsv" for comma separated value ascii text files in block format
	*".bin" for binary files in block format		*".bin" for binary files in block format
			*".coocsv" csv-format for storing sparse matrices in coordinate format.

	Note that this mechanism does not apply to files containing genotypes.		Note that this mechanism does not apply to files containing genotypes.

Boerner: Created page with "== File content determination via file name extensions == {{lmt}} automatically detects the format of input files by the [https://en.wikipedia.org/wiki/Filename_extension filename extension]. Supported extensions are ".csv" for ordinary [https://en.wikipedia.org/wiki/Comma-separated_values comma separated values] ascii text files ".blkcsv" for comma separated value ascii text files in block format *".bin" for binary files in block format Note that this mechanism does..."

2022-03-04T05:51:23Z

Created page with "== File content determination via file name extensions == {{lmt}} automatically detects the format of input files by the [https://en.wikipedia.org/wiki/Filename_extension filename extension]. Supported extensions are *".csv" for ordinary [https://en.wikipedia.org/wiki/Comma-separated_values comma separated values] ascii text files *".blkcsv" for comma separated value ascii text files in block format *".bin" for binary files in block format Note that this mechanism does..."

New page

== File content determination via file name extensions ==
{{lmt}} automatically detects the format of input files by the [https://en.wikipedia.org/wiki/Filename_extension filename extension]. Supported extensions are
*".csv" for ordinary [https://en.wikipedia.org/wiki/Comma-separated_values comma separated values] ascii text files
*".blkcsv" for comma separated value ascii text files in block format
*".bin" for binary files in block format

Note that this mechanism does not apply to files containing genotypes.

== ".csv" files ==

".csv" files may contain several commented lines at the top only where the comment character is "#".

The type of the file content is determined by its prospective use, that is
*the data file is supposed to contain only real/float numbers which are transferred to integer if required,
*a file containing an ordinary pedigree is supposed to contain only integer numbers,
*a file containing a missing value indicator matrix is supposed to contain only character strings.