utf8xx.h File Reference

Basic UTF-8 manipulation routines by Jeff Bezanson. More...

#include <stdint.h>
#include <string.h>
#include <assert.h>
#include <stdarg.h>
#include <string>
#include <vector>

Include dependency graph for utf8xx.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Macros
#define	u_int32_t   uint32_t
#define	UTF8XX_MAXBYTES   4
#define	UTF8XX_MAXBYTES1   5
#define	isutf(c)   (((c)&0xC0)!=0x80)

Typedefs
typedef uint32_t	ucs4
typedef unsigned char	uchar
typedef std::vector< ucs4 >	ucs4str
typedef std::string	utf8str

Functions
int	u8_seqlen (const utf8str &s, size_t i)
size_t	u8_toucs (ucs4str &dst, const utf8str &src)
ucs4str	u8_toucs (const utf8str &src)
size_t	u8_toutf8 (utf8str &dst, const ucs4str &src)
utf8str	u8_toutf8 (const ucs4str &src)
size_t	u8_wc_len (ucs4 ch)
size_t	u8_ws_len (const ucs4str &src)
size_t	u8_wc_toutf8 (utf8str &dst, ucs4 ch)
utf8str	u8_wc_toutf8 (ucs4 ch)
size_t	u8_offset (const utf8str &s, int charnum)
size_t	u8_charnum (const utf8str &s, int offset)
ucs4	u8_nextchar (const utf8str &s, size_t *i)
ucs4	u8_nextcharn (const utf8str &s, size_t slen, size_t *i)
size_t	u8_strlen (const utf8str &s)
void	u8_inc (const utf8str &s, size_t *i)
void	u8_dec (const utf8str &s, size_t *i)

Detailed Description

Basic UTF-8 manipulation routines by Jeff Bezanson.

(from original utf8.c)

Basic UTF-8 manipulation routines
by Jeff Bezanson
placed in the public domain Fall 2005

This code is designed to provide the utilities you need to manipulate UTF-8 as an internal string encoding. These functions do not perform the error checking normally needed when handling UTF-8 data, so if you happen to be from the Unicode Consortium you will want to flay me alive. I do this because error checking can be performed at the boundaries (I/O), with these routines reserved for higher performance on data known to be valid.

Macro Definition Documentation

u_int32_t

#define u_int32_t   uint32_t

Referenced by hex_digit().

UTF8XX_MAXBYTES

#define UTF8XX_MAXBYTES   4

UTF8XX_MAXBYTES1

#define UTF8XX_MAXBYTES1   5

isutf

#define isutf

(

c

)

(((c)&0xC0)!=0x80)

is c the start of a utf8 sequence?

Referenced by hex_digit(), u8_charnum(), u8_dec(), u8_inc(), u8_nextchar(), u8_nextcharn(), and u8_offset().

Typedef Documentation

ucs4

typedef uint32_t ucs4

uchar

typedef unsigned char uchar

ucs4str

typedef std::vector<ucs4> ucs4str

typedef for UCS-4 strings

utf8str

typedef std::string utf8str

typedef for UTF-8 strings: use standard byte strings

Function Documentation

u8_seqlen()

int u8_seqlen	(	const utf8str &	s,
		size_t	i
	)

c++ifiy: returns length of utf-8 sequence beginning at position i of string s

returns length of next utf-8 sequence

References trailingBytesForUTF8.

u8_toucs() [1/2]

size_t u8_toucs	(	ucs4str &	dst,
		const utf8str &	src
	)

convert UTF-8 byte string src to UCS-4 wide character string dst, without error checking. Data is appended to dst.

Warning

only works for valid UTF-8, i.e. no 5- or 6-byte sequences

Parameters

dst	= destination UCS-4 string
src	= source UTF-8 byte string

Returns

number of characters converted

References offsetsFromUTF8, and trailingBytesForUTF8.

Referenced by hex_digit(), and u8_toucs().

Here is the caller graph for this function:

u8_toucs() [2/2]

ucs4str u8_toucs

(

const utf8str &

src

)

convience wrapper: UTF-8 string -> UCS-4 string

convert UTF-8 byte string src to a new UCS-4 string

References u8_toucs().

Here is the call graph for this function:

u8_toutf8() [1/2]

size_t u8_toutf8	(	utf8str &	dst,
		const ucs4str &	src
	)

convert UCS-4 wide character string to UTF-8 byte string.

Parameters

dst	= destination UTF-8 string
src	= source UCS-4 string

Returns

number of characters converted

Referenced by u8_toutf8().

Here is the caller graph for this function:

u8_toutf8() [2/2]

utf8str u8_toutf8

(

const ucs4str &

src

)

convenience wrapper: UCS-4 string -> UTF-8 string

References u8_toutf8().

Here is the call graph for this function:

u8_wc_len()

size_t u8_wc_len

(

ucs4

ch

)

(moo) get number of bytes required for representing a wide character ch in UTF-8. Returns 0 on error.

Referenced by u8_wc_toutf8(), and u8_ws_len().

Here is the caller graph for this function:

u8_ws_len()

size_t u8_ws_len

(

const ucs4str &

src

)

(moo) get number of bytes required for representing a wide character string ws in UTF-8

References u8_wc_len().

Here is the call graph for this function:

u8_wc_toutf8() [1/2]

size_t u8_wc_toutf8	(	utf8str &	dst,
		ucs4	ch
	)

append single UCS-4 character to a UTF-8 string

Parameters

dst	UTF-8 destination buffer
ch	UCS-4 character to convert

Returns

number of bytes written to dst (0 <= RETVAL <= UTF8XX_MAXBYTES)

Referenced by hex_digit(), u8_wc_toutf8(), unescapeCString(), unescapeJsonString(), and unescapeUtf8String().

Here is the caller graph for this function:

u8_wc_toutf8() [2/2]

utf8str u8_wc_toutf8

(

ucs4

ch

)

convience wrapper: UCS-4 char -> UTF-8 string

References u8_wc_len(), and u8_wc_toutf8().

Here is the call graph for this function:

u8_offset()

size_t u8_offset	(	const utf8str &	s,
		int	charnum
	)

(logical) character number to (physical) byte offset

References isutf.

u8_charnum()

size_t u8_charnum	(	const utf8str &	s,
		int	offset
	)

(physical) byte offset to (logical) character number

References isutf.

u8_nextchar()

ucs4 u8_nextchar	(	const utf8str &	s,
		size_t *	i
	)

read and return next logical character, updating an index variable

References isutf, and offsetsFromUTF8.

Referenced by hex_digit(), and u8_strlen().

Here is the caller graph for this function:

u8_nextcharn()

ucs4 u8_nextcharn	(	const utf8str &	s,
		size_t	slen,
		size_t *	i
	)

(moo): return next character, updating an index variable which may not exceed length slen

References isutf, and offsetsFromUTF8.

u8_strlen()

size_t u8_strlen

(

const utf8str &

s

)

count the number of characters in a UTF-8 string

References u8_nextchar().

Here is the call graph for this function:

u8_inc()

void u8_inc	(	const utf8str &	s,
		size_t *	i
	)

move to next character

References isutf.

u8_dec()

void u8_dec	(	const utf8str &	s,
		size_t *	i
	)

move to previous character

References isutf.