Define TUTF-8 encoding and Tcl values.

Author: apnadkarni

doc/ByteArrObj.3

@@ -53,6 +53,9 @@ trigger proper error-handling), otherwise expect it to crash.
 .BE
 .SH DESCRIPTION
 .PP
+N.B. Refer to the \fBTcl_UniChar\fR documentation page for a description of the
+\fITUTF-8\fR encoding and related terms referenced here.
+.PP
 These routines are used to create, modify, store, transfer, and retrieve
 arbitrary binary data in Tcl values.  Specifically, data that can be
 represented as a sequence of arbitrary byte values is supported.
@@ -74,8 +77,7 @@ of \fIN\fR bytes is transformed into the corresponding sequence
 of \fIN\fR characters, where each byte value transforms to the same
 character codepoint value in the range (U+0000 - U+00FF).  Obtaining the
 string representation of a byte-array value (by calling
-\fBTcl_GetStringFromObj\fR) produces this string in Tcl's usual
-Modified UTF-8 encoding.
+\fBTcl_GetStringFromObj\fR) produces this string in TUTF-8 encoding.
 .PP
 \fBTcl_NewByteArrayObj\fR and \fBTcl_SetByteArrayObj\fR
 create a new value or overwrite an existing unshared value, respectively,

doc/CrtCommand.3

@@ -97,7 +97,7 @@ last value is NULL.
 Note that the argument strings should not be modified as they may
 point to constant strings or may be shared with other parts of the
 interpreter.
-Note also that the argument strings are encoded in normalized UTF-8 since
+Note also that the argument strings are encoded in normalized TUTF-8 since
 version 8.1 of Tcl.
 .PP
 \fIProc\fR must return an integer code that is expected to be one of

doc/DString.3

@@ -65,6 +65,9 @@ dynamic string.
 
 .SH DESCRIPTION
 .PP
+N.B. Refer to the \fBTcl_UniChar\fR documentation page for a description of the
+\fITUTF-8\fR encoding and related terms referenced here.
+.PP
 Dynamic strings provide a mechanism for building up arbitrarily long
 strings by gradually appending information.  If the dynamic string is
 short then there will be no memory allocation overhead;  as the string
@@ -144,7 +147,7 @@ an empty string.
 Since the dynamic string is reinitialized, there is no need to
 further call \fBTcl_DStringFree\fR on it and it can be reused without
 calling \fBTcl_DStringInit\fR. The caller must ensure that the dynamic
-string stored in \fIdsPtr\fR is encoded in Tcl's internal UTF-8 format.
+string stored in \fIdsPtr\fR is encoded in TUTF-8.
 .PP
 \fBTcl_DStringGetResult\fR does the opposite of \fBTcl_DStringResult\fR.
 It sets the value of \fIdsPtr\fR to the result of \fIinterp\fR and
@@ -160,7 +163,7 @@ copying the string. Since the dynamic string is reinitialized, there is no need
 to further call \fBTcl_DStringFree\fR on it and it can be reused without calling
 \fBTcl_DStringInit\fR. The returned \fBTcl_Obj\fR has a reference count of 0.
 The caller must ensure that the dynamic string stored in \fIdsPtr\fR is encoded
-in Tcl's internal UTF-8 format.
+in TUTF-8.
 
 .SH KEYWORDS
 append, dynamic string, free, result

doc/Encoding.3

@@ -78,11 +78,11 @@ Name of encoding to get token for.
 Points to storage where encoding token is to be written.
 .AP "const char" *src in
 For the \fBTcl_ExternalToUtf\fR functions, an array of bytes in the
-specified encoding that are to be converted to UTF-8.  For the
-\fBTcl_UtfToExternal\fR function, an array of
-UTF-8 characters to be converted to the specified encoding.
+specified encoding that are to be converted to TUTF-8.  For the
+\fBTcl_UtfToExternal\fR function, a TUTF-8 byte sequence
+to be converted to the specified encoding.
 .AP "const TCHAR" *tsrc in
-An array of Windows TCHAR characters to convert to UTF-8.
+An array of Windows TCHAR characters to convert to TUTF-8.
 .AP Tcl_Size srcLen in
 Length of \fIsrc\fR or \fItsrc\fR in bytes.  If the length is negative, the
 encoding-specific length of the string is used.
@@ -104,7 +104,6 @@ control the encoding profile to be used for dealing with invalid data or
 other errors in the encoding transform.
 The flag \fBTCL_ENCODING_STOPONERROR\fR has no effect,
 it only has meaning in Tcl 8.x.
-.PP
 Some flags bits may not be usable with some functions as noted in the
 function descriptions below.
 .AP Tcl_EncodingState *statePtr in/out
@@ -146,25 +145,24 @@ A path to the location of the encoding file.
 .BE
 .SH INTRODUCTION
 .PP
-These routines convert between Tcl's internal character representation,
-UTF-8, and character representations used by various operating systems or
-file systems, such as Unicode, ASCII, or Shift-JIS.  When operating on
-strings, such as such as obtaining the names of files or displaying
-characters using international fonts, the strings must be translated into
-one or possibly multiple formats that the various system calls can use.  For
-instance, on a Japanese Unix workstation, a user might obtain a filename
-represented in the EUC-JP file encoding and then translate the characters to
-the jisx0208 font encoding in order to display the filename in a Tk widget.
-The purpose of the encoding package is to help bridge the translation gap.
-UTF-8 provides an intermediate staging ground for all the various
-encodings.  In the example above, text would be translated into UTF-8 from
-whatever file encoding the operating system is using.  Then it would be
-translated from UTF-8 into whatever font encoding the display routines
-require.
-.PP
-Some basic encodings are compiled into Tcl.  Others can be defined by the
-user or dynamically loaded from encoding files in a
-platform-independent manner.
+N.B. Refer to the \fBTcl_UniChar\fR documentation page for a description of the
+\fITUTF-8\fR encoding and related terms referenced here.
+.PP
+These routines convert between TUTF-8
+and character representations using encodings such as
+standard UTF-8, UTF-16, ASCII, or Shift-JIS that might be expected by system
+interfaces or other software components. For instance, on a Japanese Unix
+workstation, a user might obtain a filename represented in the EUC-JP file
+encoding and then translate the characters to the jisx0208 font encoding in
+order to display the filename in a Tk widget. The purpose of the encoding
+package is to help bridge the translation gap. TUTF-8 provides an intermediate
+staging ground for all the various encodings. In the example above, text would
+be translated into TUTF-8 from whatever file encoding the operating system is
+using. Then it would be translated from TUTF-8 into whatever font encoding the
+display routines require.
+.PP
+Some basic encodings are compiled into Tcl. Others can be defined by the user or
+dynamically loaded from encoding files in a platform-independent manner.
 .SH DESCRIPTION
 .PP
 \fBTcl_GetEncoding\fR finds an encoding given its \fIname\fR.  The name may
@@ -202,7 +200,7 @@ on the resulting encoding token when that token will no longer be
 used.
 .PP
 \fBTcl_ExternalToUtfDString\fR converts a source buffer \fIsrc\fR from the
-specified \fIencoding\fR into UTF-8.  The converted bytes are stored in
+specified \fIencoding\fR into TUTF-8.  The converted bytes are stored in
 \fIdstPtr\fR, which is then null-terminated.  The caller should eventually
 call \fBTcl_DStringFree\fR to free any information stored in \fIdstPtr\fR.
 When converting, if any of the characters in the source buffer cannot be
@@ -231,7 +229,7 @@ The caller must call \fBTcl_DStringFree\fR to free up the \fB*dstPtr\fR resource
 irrespective of the return value from the function.
 .PP
 \fBTcl_ExternalToUtf\fR converts a source buffer \fIsrc\fR from the specified
-\fIencoding\fR into UTF-8.  Up to \fIsrcLen\fR bytes are converted from the
+\fIencoding\fR into TUTF-8.  Up to \fIsrcLen\fR bytes are converted from the
 source buffer and up to \fIdstLen\fR converted bytes are stored in \fIdst\fR.
 In all cases, \fI*srcReadPtr\fR is filled with the number of bytes that were
 successfully converted from \fIsrc\fR and \fI*dstWrotePtr\fR is filled with
@@ -259,7 +257,7 @@ The source buffer contained a character that could not be represented in
 the target encoding.
 .RE
 .LP
-\fBTcl_UtfToExternalDString\fR converts a source buffer \fIsrc\fR from UTF-8
+\fBTcl_UtfToExternalDString\fR converts a source buffer \fIsrc\fR from TUTF-8
 into the specified \fIencoding\fR.  The converted bytes are stored in
 \fIdstPtr\fR, which is then terminated with the appropriate encoding-specific
 null.  The caller should eventually call \fBTcl_DStringFree\fR to free any
@@ -269,7 +267,7 @@ encoding, a default fallback character will be used.  The return value is
 a pointer to the value stored in the DString.
 .PP
 \fBTcl_UtfToExternalDStringEx\fR is an enhanced version of
-\fBTcl_UtfToExternalDString\fR that transforms UTF-8 encoded source data to a
+\fBTcl_UtfToExternalDString\fR that transforms TUTF-8 encoded source data to a
 specified \fIencoding\fR. Except for the direction of the transform, the
 parameters and return values are identical to those of
 \fBTcl_ExternalToUtfDStringEx\fR. See
@@ -278,7 +276,7 @@ that function above for details about the same.
 Irrespective of the return code from the function, the caller must free
 resources associated with \fB*dstPtr\fR when the function returns.
 .PP
-\fBTcl_UtfToExternal\fR converts a source buffer \fIsrc\fR from UTF-8 into
+\fBTcl_UtfToExternal\fR converts a source buffer \fIsrc\fR from TUTF-8 into
 the specified \fIencoding\fR.  Up to \fIsrcLen\fR bytes are converted from
 the source buffer and up to \fIdstLen\fR converted bytes are stored in
 \fIdst\fR.  In all cases, \fI*srcReadPtr\fR is filled with the number of
@@ -322,7 +320,7 @@ exist.
 .PP
 \fBTcl_CreateEncoding\fR defines a new encoding and registers the C
 procedures that are called back to convert between the encoding and
-UTF-8.  Encodings created by \fBTcl_CreateEncoding\fR are thereafter
+TUTF-8.  Encodings created by \fBTcl_CreateEncoding\fR are thereafter
 visible in the database used by \fBTcl_GetEncoding\fR.  Just as with the
 \fBTcl_GetEncoding\fR procedure, the return value is a token that
 represents the encoding and can be used in subsequent calls to other
@@ -335,7 +333,7 @@ encoding procedures.
 .PP
 The \fItypePtr\fR argument to \fBTcl_CreateEncoding\fR contains information
 about the name of the encoding and the procedures that will be called to
-convert between this encoding and UTF-8.  It is defined as follows:
+convert between this encoding and TUTF-8.  It is defined as follows:
 .PP
 .CS
 typedef struct {
@@ -351,9 +349,9 @@ typedef struct {
 The \fIencodingName\fR provides a string name for the encoding, by
 which it can be referred in other procedures such as
 \fBTcl_GetEncoding\fR.  The \fItoUtfProc\fR refers to a callback
-procedure to invoke to convert text from this encoding into UTF-8.
+procedure to invoke to convert text from this encoding into TUTF-8.
 The \fIfromUtfProc\fR refers to a callback procedure to invoke to
-convert text from UTF-8 into this encoding.  The \fIfreeProc\fR refers
+convert text from TUTF-8 into this encoding.  The \fIfreeProc\fR refers
 to a callback procedure to invoke when this encoding is deleted.  The
 \fIfreeProc\fR field may be NULL.  The \fIclientData\fR contains an
 arbitrary one-word value passed to \fItoUtfProc\fR, \fIfromUtfProc\fR,
@@ -513,7 +511,7 @@ FF0400A200A3FF05FF03FF06FF0AFF2000A72606260525CB25CF25CE25C725C6
 .CE
 .PP
 The third line of the file is three numbers.  The first number is the
-fallback character (in base 16) to use when converting from UTF-8 to this
+fallback character (in base 16) to use when converting from TUTF-8 to this
 encoding.  The second number is a \fB1\fR if this file represents the
 encoding for a symbol font, or \fB0\fR otherwise.  The last number (in base
 10) is how many pages of data follow.

doc/Eval.3

@@ -62,13 +62,16 @@ The number of bytes in \fIscript\fR, not including any
 null terminating character.  If \-1, then all characters up to the
 first null byte are used.
 .AP "const char" *script in
-Points to first byte of script to execute (null-terminated and UTF-8).
+Points to first byte of script to execute (null-terminated TUTF-8 byte sequence).
 .AP "const char" *part in
 String forming part of a Tcl script.
 .BE
 
 .SH DESCRIPTION
 .PP
+N.B. Refer to the \fBTcl_UniChar\fR documentation page for a description of the
+\fITUTF-8\fR encoding and related terms referenced here.
+.PP
 The procedures described here are invoked to execute Tcl scripts in
 various forms.
 \fBTcl_EvalObjEx\fR is the core procedure and is used by many of the others.
@@ -113,7 +116,7 @@ elements of \fIobjv\fR, insuring that the values are valid until
 .PP
 \fBTcl_Eval\fR is similar to \fBTcl_EvalObjEx\fR except that the script to
 be executed is supplied as a string instead of a value and no compilation
-occurs.  The string should be a proper UTF-8 string as converted by
+occurs.  The string should be a proper TUTF-8 byte sequence as converted by
 \fBTcl_ExternalToUtfDString\fR or \fBTcl_ExternalToUtf\fR when it is known
 to possibly contain upper ASCII characters whose possible combinations
 might be a UTF-8 special code.  The string is parsed and executed directly

doc/Object.3

@@ -38,6 +38,9 @@ must have been the result of a previous call to \fBTcl_NewObj\fR.
 .BE
 .SH INTRODUCTION
 .PP
+N.B. Refer to the \fBTcl_UniChar\fR documentation page for a description of the
+\fITUTF-8\fR encoding and related terms referenced here.
+.PP
 This man page presents an overview of Tcl values (called \fBTcl_Obj\fRs for
 historical reasons) and how they are used.
 It also describes generic procedures for managing Tcl values.
@@ -140,9 +143,7 @@ typedef struct {
 .CE
 .PP
 The \fIbytes\fR and the \fIlength\fR members together hold
-a value's UTF-8 string representation,
-which is a \fIcounted string\fR not containing null bytes (UTF-8 null
-characters should be encoded as a two byte sequence: 192, 128.)
+a value's TUTF-8 byte sequence representation.
 \fIbytes\fR points to the first byte of the string representation.
 The \fIlength\fR member gives the number of bytes.
 The byte array must always have a null byte after the last data byte,

doc/OpenFileChnl.3

@@ -217,6 +217,9 @@ New value for the option given by \fIoptionName\fR.
 .BE
 .SH DESCRIPTION
 .PP
+N.B. Refer to the \fBTcl_UniChar\fR documentation page for a description of the
+\fITUTF-8\fR encoding and related terms referenced here.
+.PP
 The Tcl channel mechanism provides a device-independent and
 platform-independent mechanism for performing buffered input
 and output operations on a variety of file, socket, and device
@@ -415,7 +418,7 @@ corresponding calls to \fBTcl_UnregisterChannel\fR.
 .SH "TCL_READCHARS AND TCL_READ"
 .PP
 \fBTcl_ReadChars\fR consumes bytes from \fIchannel\fR, converting the bytes
-to UTF-8 based on the channel's encoding and storing the produced data in
+to TUTF-8 based on the channel's encoding and storing the produced data in
 \fIreadObjPtr\fR's string representation.  The return value of
 \fBTcl_ReadChars\fR is the number of characters, up to \fIcharsToRead\fR,
 that were stored in \fIreadObjPtr\fR.  If an error occurs while reading, the
@@ -448,14 +451,14 @@ platform-specific modes are described in the manual entry for the Tcl
 \fBfconfigure\fR command.
 .PP
 As a performance optimization, when reading from a channel with the encoding
-\fBbinary\fR, the bytes are not converted to UTF-8 as they are read.
+\fBbinary\fR, the bytes are not converted to TUTF-8 as they are read.
 Instead, they are stored in \fIreadObjPtr\fR's internal representation as a
 byte-array value.  The string representation of this value will only be
 constructed if it is needed (e.g., because of a call to
 \fBTcl_GetStringFromObj\fR).  In this way, byte-oriented data can be read
 from a channel, manipulated by calling \fBTcl_GetByteArrayFromObj\fR and
 related functions, and then written to a channel without the expense of ever
-converting to or from UTF-8.
+converting to or from TUTF-8.
 .PP
 \fBTcl_Read\fR is similar to \fBTcl_ReadChars\fR, except that it does not do
 encoding conversions, regardless of the channel's encoding.  It is deprecated
@@ -476,8 +479,8 @@ channel drivers, i.e. drivers used in the middle of a stack of
 channels, to move data from the channel below into the transformation.
 .SH "TCL_GETSOBJ AND TCL_GETS"
 .PP
-\fBTcl_GetsObj\fR consumes bytes from \fIchannel\fR, converting the bytes to
-UTF-8 based on the channel's encoding, until a full line of input has been
+\fBTcl_GetsObj\fR consumes bytes from \fIchannel\fR,
+based on the channel's encoding, until a full line of input has been
 seen.  If the channel's encoding is \fBbinary\fR, each byte read from the
 channel is treated as an individual Unicode character.  All of the
 characters of the line except for the terminating end-of-line character(s)
@@ -497,9 +500,9 @@ end-of-line character.  When -1 is returned, the \fBTcl_InputBlocked\fR
 procedure may be invoked to determine if the channel is blocked because
 of input unavailability.
 .PP
-\fBTcl_Gets\fR is the same as \fBTcl_GetsObj\fR except the resulting
-characters are appended to the dynamic string given by
-\fIlineRead\fR rather than a Tcl value.
+\fBTcl_Gets\fR works similarly to \fBTcl_GetsObj\fR except the read bytes
+are are appended as a TUTF-8 encoded byte sequence to the dynamic string
+given by \fIlineRead\fR rather than a Tcl value.
 .SH "TCL_UNGETS"
 .PP
 \fBTcl_Ungets\fR is used to add data to the input queue of a channel,
@@ -515,7 +518,7 @@ added to the input queue.  \fBTcl_Ungets\fR returns \fIinputLen\fR or
 .SH "TCL_WRITECHARS, TCL_WRITEOBJ, AND TCL_WRITE"
 .PP
 \fBTcl_WriteChars\fR accepts \fIbytesToWrite\fR bytes of character data at
-\fIcharBuf\fR.  The UTF-8 characters in the buffer are converted to the
+\fIcharBuf\fR.  The TUTF-8 encoded bytes in the buffer are converted to the
 channel's encoding and queued for output to \fIchannel\fR.  If
 \fIbytesToWrite\fR is negative, \fBTcl_WriteChars\fR expects \fIcharBuf\fR
 to be null-terminated and it outputs everything up to the null.
@@ -539,16 +542,16 @@ channel.  This is done even if the channel has no encoding.
 .PP
 \fBTcl_WriteObj\fR is similar to \fBTcl_WriteChars\fR except it
 accepts a Tcl value whose contents will be output to the channel.  The
-UTF-8 characters in \fIwriteObjPtr\fR's string representation are converted
+characters in \fIwriteObjPtr\fR's string representation are converted
 to the channel's encoding and queued for output to \fIchannel\fR.
 As a performance optimization, when writing to a channel with the encoding
-\fBbinary\fR, UTF-8 characters are not converted as they are written.
+\fBbinary\fR, characters are not converted as they are written.
 Instead, the bytes in \fIwriteObjPtr\fR's internal representation as a
 byte-array value are written to the channel.  The byte-array representation
 of the value will be constructed if it is needed.  In this way,
 byte-oriented data can be read from a channel, manipulated by calling
 \fBTcl_GetByteArrayFromObj\fR and related functions, and then written to a
-channel without the expense of ever converting to or from UTF-8.
+channel without the expense of encoding conversion.
 .PP
 \fBTcl_Write\fR is similar to \fBTcl_WriteChars\fR except that it does not do
 encoding conversions, regardless of the channel's encoding.  It is

doc/StringObj.3

@@ -85,11 +85,12 @@ Tcl_Obj *
 .SH ARGUMENTS
 .AS "const Tcl_UniChar" *appendObjPtr in/out
 .AP "const char" *bytes in
-Points to the first byte of an array of UTF-8-encoded bytes
-used to set or append to a string value.
+Points to a TUTF-8 (Tcl's internal modified UTF-8 encoding)
+byte sequence bytes used to set or append to a string value.
 This byte array may contain embedded null characters
 unless \fInumChars\fR is negative.  (Applications needing null bytes
-should represent them as the two-byte sequence \fI\e300\e200\fR, use
+should represent them as the two-byte sequence \fI0xC0 0x80\fR used
+in TUTF-8 to represent a null byte, use
 \fBTcl_ExternalToUtf\fR to convert, or \fBTcl_NewByteArrayObj\fR if
 the string is a collection of uninterpreted bytes.)
 .AP Tcl_Size length in