Unicode::LineBreak - UAX #14 Unicode Line Breaking Algorithm |
Unicode::LineBreak - UAX #14 Unicode Line Breaking Algorithm
use Unicode::LineBreak; $lb = Unicode::LineBreak->new(); $broken = $lb->break($string);
Unicode::LineBreak performs Line Breaking Algorithm described in Unicode Standard Annex #14 [UAX #14]. East_Asian_Width informative property defined by Annex #11 [UAX #11] will be concerned to determine breaking positions.
Following terms are used for convenience.
Mandatory break is obligatory line breaking behavior defined by core rules and performed regardless of surrounding characters. Arbitrary break is line breaking behavior allowed by core rules and chosen by user to perform it. Arbitrary break includes direct break and indirect break defined by [UAX #14].
Alphabetic characters are characters usually no line breaks are allowed between pairs of them, except that other characters provide break oppotunities. Ideographic characters are characters that usually allow line breaks both before and after themselves. [UAX #14] classifies most of alphabetic to AL and most of ideographic to ID (These terms are inaccurate from the point of view by grammatology). On several scripts, breaking positions are not obvious by each characters therefore heuristic based on dictionary is used.
Number of columns of a string is not always equal to the number of characters it contains: Each of characters is either wide, narrow or nonspacing; they occupy 2, 1 or 0 columns, respectively. Several characters may be both wide and narrow by the contexts they are used. Characters may have more various widths by customization.
break()
but accepts incremental inputs.
Give undef
as STRING argument to specify that input was completed.
Note:
This method gives just approximate description of line breaking behavior.
Use break()
and so on to wrap actual texts.
new and config methods accept following pairs. Some of them affect number of columns ([E]), grapheme cluster segmentation ([G]) (see also the Unicode::GCString manpage) or line breaking behavior ([L]).
"YES"
| "NO"
"YES"
.
Note: This option was introduced at release 1.011.
998
.
0
means unlimited (as of release 2012.01).
0
.
76
.
See also Urgent option and User-Defined Breaking Behaviors.
"YES"
| "NO"
"YES"
.
"EASTASIAN"
and "NONEASTASIAN"
.
Default context is "NONEASTASIAN"
.
In "EASTASIAN"
context, characters with East_Asian_Width property
ambiguous (A) are treated as ``wide'' and with Line Breaking Class AI as
ideographic (ID).
In "NONEASTASIAN"
context, characters with East_Asian_Width property
ambiguous (A) are treated as ``narrow'' and with Line Breaking Class AI as
alphabetic (AL).
[
ORD =>
PROPERTY ]
undef
undef
is specified, all tailoring assigned before will be canceled.
By default, no tailorings are available. See also Tailoring Character Properties.
"SIMPLE"
"NEWLINE"
"TRIM"
undef
"YES"
| "NO"
"NO"
.
[
ORD =>
CLASS ]
undef
undef
is specified, all tailoring assigned before will be canceled.
By default, no tailorings are available. See also Tailoring Character Properties.
"YES"
| "NO"
"YES"
.
"\n"
.
"NONBREAKURI"
"BREAKURI"
[
REGEX, SUBREF ]
undef
"UAX11"
undef
See also ColMax, ColMin and EAWidth options.
"CROAK"
"FORCE"
undef
"YES"
| "NO"
"YES"
.
Note:
This option was introduced by release 2012.001_29.
On previous releases, it was fixed to "NO"
.
``Default'' grapheme cluster defined by [UAX #29] does not include this
feature.
EA_Na
, EA_N
, EA_A
, EA_W
, EA_H
, EA_F
EA_Z
Note: This ``nonspacing'' value is extension by this module, not a part of [UAX #11].
LB_BK
, LB_CR
, LB_LF
, LB_NL
, LB_SP
, LB_OP
, LB_CL
, LB_CP
, LB_QU
, LB_GL
, LB_NS
, LB_EX
, LB_SY
, LB_IS
, LB_PR
, LB_PO
, LB_NU
, LB_AL
, LB_HL
, LB_ID
, LB_IN
, LB_HY
, LB_BA
, LB_BB
, LB_B2
, LB_CB
, LB_ZW
, LB_CM
, LB_WJ
, LB_H2
, LB_H3
, LB_JL
, LB_JV
, LB_JT
, LB_SG
, LB_AI
, LB_CJ
, LB_SA
, LB_XX
, LB_RI
Note: Property value CP was introduced by Unicode 5.2.0. Property values HL and CJ were introduced by Unicode 6.1.0. Property value RI was introduced by Unicode 6.2.0.
MANDATORY
, DIRECT
, INDIRECT
, PROHIBITED
Unicode::LineBreak::SouthEastAsian::supported
undef
is set.
N.B.: Current release supports Thai script of modern Thai language only.
UNICODE_VERSION
If you specify subroutine reference as a value of Format option, it should accept three arguments:
$MODIFIED = &subroutine(SELF, EVENT, STR);
SELF is a Unicode::LineBreak object, EVENT is a string to determine the context that subroutine was called in, and STR is a fragment of Unicode string leading or trailing breaking position.
EVENT |When Fired |Value of STR ----------------------------------------------------------------- "sot" |Beginning of text |Fragment of first line "sop" |After mandatory break|Fragment of next line "sol" |After arbitrary break|Fragment on sequel of line "" |Just before any |Complete line without trailing |breaks |SPACEs "eol" |Arbitrary break |SPACEs leading breaking position "eop" |Mandatory break |Newline and its leading SPACEs "eot" |End of text |SPACEs (and newline) at end of | |text -----------------------------------------------------------------
Subroutine should return modified text fragment or may return
undef
to express that no modification occurred.
Note that modification in the context of "sot"
, "sop"
or "sol"
may
affect decision of successive breaking positions while in the others won't.
Note: String arguments are actually sequences of grapheme clusters. See the Unicode::GCString manpage.
For example, following code folds lines removing trailing spaces:
sub fmt { if ($_[1] =~ /^eo/) { return "\n"; } return undef; } my $lb = Unicode::LineBreak->new(Format => \&fmt); $output = $lb->break($text);
When a line generated by arbitrary break is expected to be beyond measure of either CharMax, ColMax or ColMin, urgent break may be performed on successive string. If you specify subroutine reference as a value of Urgent option, it should accept two arguments:
@BROKEN = &subroutine(SELF, STR);
SELF is a Unicode::LineBreak object and STR is a Unicode string to be broken.
Subroutine should return an array of broken string STR.
Note: String argument is actually a sequence of grapheme clusters. See the Unicode::GCString manpage.
For example, following code inserts hyphen to the name of several chemical substances (such as Titin) so that it may be folded:
sub hyphenize { return map {$_ =~ s/yl$/yl-/; $_} split /(\w+?yl(?=\w))/, $_[1]; } my $lb = Unicode::LineBreak->new(Urgent => \&hyphenize); $output = $lb->break("Methionylthreonylthreonylglutaminylarginyl...");
If you specify [REGEX, SUBREF] array reference as any of Prep option, subroutine should accept two arguments:
@BROKEN = &subroutine(SELF, STR);
SELF is a Unicode::LineBreak object and STR is a Unicode string matched with REGEX.
Subroutine should return an array of broken string STR.
For example, following code will break HTTP URLs using [CMOS] rule.
my $url = qr{http://[\x21-\x7E]+}i; sub breakurl { my $self = shift; my $str = shift; return split m{(?<=[/]) (?=[^/]) | (?<=[^-.]) (?=[-~.,_?\#%=&]) | (?<=[=&]) (?=.)}x, $str; } my $lb = Unicode::LineBreak->new(Prep => [$url, \&breakurl]); $output = $lb->break($string);
Unicode::LineBreak object can behave as hash reference. Any items may be preserved throughout its life.
For example, following code will separate paragraphs with empty lines.
sub paraformat { my $self = shift; my $action = shift; my $str = shift; if ($action eq 'sot' or $action eq 'sop') { $self->{'line'} = ''; } elsif ($action eq '') { $self->{'line'} = $str; } elsif ($action eq 'eol') { return "\n"; } elsif ($action eq 'eop') { if (length $self->{'line'}) { return "\n\n"; } else { return "\n"; } } elsif ($action eq 'eot') { return "\n"; } return undef; } my $lb = Unicode::LineBreak->new(Format => \¶format); $output = $lb->break($string);
If you specify subroutine reference as a value of Sizing option, it will be called with five arguments:
$COLS = &subroutine(SELF, LEN, PRE, SPC, STR);
SELF is a Unicode::LineBreak object, LEN is size of preceding string, PRE is preceding Unicode string, SPC is additional SPACEs and STR is a Unicode string to be processed.
Subroutine should return calculated number of columns of PRE.SPC.STR
.
The number of columns may not be an integer: Unit of the number may be freely chosen, however, it should be same as those of ColMin and ColMax option.
Note: String arguments are actually sequences of grapheme clusters. See the Unicode::GCString manpage.
For example, following code processes lines with tab stops by each eight columns.
sub tabbedsizing { my ($self, $cols, $pre, $spc, $str) = @_; my $spcstr = $spc.$str; while ($spcstr->lbc == LB_SP) { my $c = $spcstr->item(0); if ($c eq "\t") { $cols += 8 - $cols % 8; } else { $cols += $c->columns; } $spcstr = $spcstr->substr(1); } $cols += $spcstr->columns; return $cols; }; my $lb = Unicode::LineBreak->new(LBClass => [ord("\t") => LB_SP], Sizing => \&tabbedsizing); $output = $lb->break($string);
Character properties may be tailored by LBClass and EAWidth options. Some constants are defined for convenience of tailoring.
By default, several hiragana, katakana and characters corresponding to kana
are treated as non-starters (NS or CJ).
When the following pair(s)
are specified for value of LBClass option,
these characters are treated as normal ideographic characters (ID).
KANA_NONSTARTERS() => LB_ID
IDEOGRAPHIC_ITERATION_MARKS() => LB_ID
N.B. Some of them are neither hiragana nor katakana.
KANA_SMALL_LETTERS() => LB_ID
KANA_PROLONGED_SOUND_MARKS() => LB_ID
Hiragana or katakana prolonged sound marks: U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK and U+FF70 HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK.
N.B. These letters are optionally treated either as non-starter or as normal ideographic. See [JIS X 4051] 6.1.1, [JLREQ] 3.1.7 or [UAX14].
N.B. U+3095, U+3096, U+30F5, U+30F6 are considered to be neither hiragana nor katakana.
MASU_MARK() => LB_ID
N.B. Although this character is not kana, it is usually regarded as abbreviation to sequence of hiragana &0x307E; &0x3059; or katakana &0x30DE; &0x30B9;, MA and SU.
N.B. This character is classified as non-starter (NS) by [UAX #14] and as the class corresponding to ID by [JIS X 4051] and [JLREQ].
By default, some punctuations are ambiguous quotation marks (QU).
BACKWARD_QUOTES() => LB_OP, FORWARD_QUOTES() => LB_CL
FORWARD_QUOTES() => LB_OP, BACKWARD_QUOTES() => LB_CL
BACKWARD_GUILLEMETS() => LB_OP, FORWARD_GUILLEMETS() => LB_CL
FORWARD_GUILLEMETS() => LB_OP, BACKWARD_GUILLEMETS() => LB_CL
Danish, Finnish, Norwegian and Swedish use 9-style or right-pointing punctuations (&0x2019; &0x201D; &0x00BB; &0x203A;) as both opening and closing quotation marks.
IDEOGRAPHIC_SPACE() => LB_BA
IDEOGRAPHIC_SPACE() => LB_ID
IDEOGRAPHIC_SPACE() => LB_SP
Some particular letters of Latin, Greek and Cyrillic scripts have ambiguous
(A) East_Asian_Width property. Thus, these characters are treated as wide
in "EASTASIAN"
context.
Specifying EAWidth => [ AMBIGUOUS_
*() => EA_N ]
,
those characters are always treated as narrow.
AMBIGUOUS_ALPHABETICS() => EA_N
AMBIGUOUS_CYRILLIC() => EA_N
AMBIGUOUS_GREEK() => EA_N
AMBIGUOUS_LATIN() => EA_N
On the other hand, despite several characters were occasionally rendered as wide characters by number of implementations for East Asian character sets, they are given narrow (Na) East_Asian_Width property just because they have fullwidth (F) compatibility characters.
Specifying EAWidth
as below, those characters are treated as ambiguous
--- wide on "EASTASIAN"
context.
QUESTIONABLE_NARROW_SIGNS() => EA_A
Built-in defaults of option parameters for new and config method can be overridden by configuration files: Unicode/LineBreak/Defaults.pm. For more details read Unicode/LineBreak/Defaults.pm.sample.
Please report bugs or buggy behaviors to developer.
CPAN Request Tracker: http://rt.cpan.org/Public/Dist/Display.html.
Consult $VERSION variable.
eawidth()
method was deprecated.
columns in the Unicode::GCString manpage may be used instead.
lbclass()
method was deprecated.
Use lbc in the Unicode::GCString manpage or lbcext in the Unicode::GCString manpage.
Character properties this module is based on are defined by Unicode Standard version 8.0.0.
This module is intended to implement UAX14-C2.
Character(s)
assigned to CB are not resolved.
Characters assigned to CJ are always resolved to NS.
More flexible tailoring mechanism is provided.
When word segmentation for South East Asian writing systems is not supported,
characters assigned to SA are resolved to AL,
except that characters that have Grapheme_Cluster_Break property value
Extend or SpacingMark be resolved to CM.
Characters assigned to SG or XX are resolved to AL.
Code points of following UCS ranges are given fixed property values even
if they have not been assigned any characers.
Ranges | UAX #14 | UAX #11 | Description ------------------------------------------------------------- U+20A0..U+20CF | PR [*1] | N [*2] | Currency symbols U+3400..U+4DBF | ID | W | CJK ideographs U+4E00..U+9FFF | ID | W | CJK ideographs U+D800..U+DFFF | AL (SG) | N | Surrogates U+E000..U+F8FF | AL (XX) | F or N (A) | Private use U+F900..U+FAFF | ID | W | CJK ideographs U+20000..U+2FFFD | ID | W | CJK ideographs U+30000..U+3FFFD | ID | W | Old hanzi U+F0000..U+FFFFD | AL (XX) | F or N (A) | Private use U+100000..U+10FFFD | AL (XX) | F or N (A) | Private use Other unassigned | AL (XX) | N | Unassigned, | | | reserved or | | | noncharacters ------------------------------------------------------------- [*1] Except U+20A7 PESETA SIGN (PO), U+20B6 LIVRE TOURNOIS SIGN (PO), U+20BB NORDIC MARK SIGN (PO) and U+20BE LARI SIGN (PO). [*2] Except U+20A9 WON SIGN (H) and U+20AC EURO SIGN (F or N (A)).Characters belonging to General Category Mn, Me, Cc, Cf, Zl or Zp are treated as nonspacing by this module.
the Text::LineFold manpage, the Text::Wrap manpage, the Unicode::GCString manpage.
Copyright (C) 2009-2013 Hatuka*nezumi - IKEDA Soji <hatuka(at)nezumi.nu>.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
Unicode::LineBreak - UAX #14 Unicode Line Breaking Algorithm |