Clusters: HarfBuzz Manual

## Clusters

In text shaping, a cluster is a sequence of characters that needs to be treated as a single, indivisible unit.

A cluster is distinct from a grapheme, which is the smallest unit of a writing system or script, because clusters are only relevant for script shaping and the layout of glyphs.

For example, a grapheme may be a letter, a number, a logogram, or a symbol. When two letters form a ligature, however, they combine into a single glyph. They are therefore part of the same cluster and are treated as a unit — even though the two original, underlying letters are separate graphemes.

During the shaping process, there are several shaping operations that may merge adjacent characters (for example, when two code points form a ligature or a conjunct form and are replaced by a single glyph) or split one character into several (for example, when decomposing a code point through the `ccmp` feature).

HarfBuzz tracks clusters independently from how these shaping operations affect the individual glyphs that comprise the output HarfBuzz returns in a buffer. Consequently, a client program using HarfBuzz can utilize the cluster information to implement features such as:

- Correctly positioning the cursor within a shaped text run, even when characters have formed ligatures, composed or decomposed, reordered, or undergone other shaping operations.

- Correctly highlighting a text selection that includes some, but not all, of the characters in a word.

- Applying text attributes (such as color or underlining) to part, but not all, of a word.

- Generating output document formats (such as PDF) with embedded text that can be fully extracted.

- Determining the mapping between input characters and output glyphs, such as which glyphs are ligatures.

- Performing line-breaking, justification, and other line-level or paragraph-level operations that must be done after shaping is complete, but which require character-level properties.

When you add text to a HarfBuzz buffer, each code point must be assigned a cluster value.

This cluster value is an arbitrary number; HarfBuzz uses it only to distinguish between clusters. Many client programs will use the index of each code point in the input text stream as the cluster value. This is for the sake of convenience; the actual value does not matter.

Client programs can choose how HarfBuzz handles clusters during shaping by setting the `cluster_level` of the buffer. HarfBuzz offers three levels of clustering support for this property:

- Level 0 is the default and reproduces the behavior of the old HarfBuzz library.

  The distinguishing feature of level 0 behavior is that, at the beginning of processing the buffer, all code points that are categorized as marks, modifier symbols, or Emoji extended pictographic modifiers, as well as the Zero Width Joiner and Zero Width Non-Joiner code points, are assigned the cluster value of the closest preceding code point from different category.

  In essence, whenever a base character is followed by a mark character or a sequence of mark characters, those marks are reassigned to the same initial cluster value as the base character. This reassignment is referred to as "merging" the affected clusters. This behavior is based on the Grapheme Cluster Boundary specification in Unicode Technical Report 29.

  Client programs can specify level 0 behavior for a buffer by setting its `cluster_level` to `HB_BUFFER_CLUSTER_LEVEL_MONOTONE_GRAPHEMES`.

- Level 1 tweaks the old behavior slightly to produce better results. Therefore, level 1 clustering is recommended for code that is not required to implement backward compatibility with the old HarfBuzz.

  Level 1 differs from level 0 by not merging the clusters of marks and other modifier code points with the preceding "base" code point's cluster. By preserving the separate cluster values of these marks and modifier code points, script shapers can perform additional operations that might lead to improved results (for example, reordering a sequence of marks).

  Client programs can specify level 1 behavior for a buffer by setting its `cluster_level` to `HB_BUFFER_CLUSTER_LEVEL_MONOTONE_CHARACTERS`.

- Level 2 differs significantly in how it treats cluster values. In level 2, HarfBuzz never merges clusters.

  This difference can be seen most clearly when HarfBuzz processes ligature substitutions and glyph decompositions. In level 0 and level 1, ligatures and glyph decomposition both involve merging clusters; in level 2, neither of these operations triggers a merge.

  Client programs can specify level 2 behavior for a buffer by setting its `cluster_level` to `HB_BUFFER_CLUSTER_LEVEL_CHARACTERS`.

As mentioned earlier, client programs using HarfBuzz often assign initial cluster values in a buffer by reusing the indices of the code points in the input text. This gives a sequence of cluster values that is monotonically increasing (for example, 0,1,2,3,4,5).

It is not required that the cluster values in a buffer be monotonically increasing. However, if the initial cluster values in a buffer are monotonic and the buffer is configured to use cluster level 0 or 1, then HarfBuzz guarantees that the final cluster values in the shaped buffer will also be monotonic. No such guarantee is made for cluster level 2.

In levels 0 and 1, HarfBuzz implements the following conceptual model for cluster values:

- If the sequence of input cluster values is monotonic, the sequence of cluster values will remain monotonic.

- Each cluster value represents a single cluster.

- Each cluster contains one or more glyphs and one or more characters.

In practice, this model offers several benefits. Assuming that the initial cluster values were monotonically increasing and distinct before shaping began, then, in the final output:

- All adjacent glyphs having the same final cluster value belong to the same cluster.

- Each character belongs to the cluster that has the highest cluster value not larger than its initial cluster value.