script: make-word-lisr.r title: List words in a string author: Peter W A Wood date: 2-Apr-2007 Version: 1.0.0
make-word-list.r lists all the unique words in a document.
make-word-list.r is used inside skimp.r - the simple keyword management program that is used in REBOL.org for many of its search indexes.
I would like to thank Sunanda without whom this script wouldn't have been started, tested, optimised or documented.
USAGE: make-word-list config content index-name /for-search ARGUMENTS: config -- changes to the default configuration (Type: object or none) (See below) content -- the string for which words are to be extracted REFINEMENTS: /for-search -- The character specified as the "not-prefix" is not removed from the front of words
The configuration object used by make-word-list function provides almost complete control over what makes a word. You only need to supply changes not all the entries in the configuration object. For example, the following configuration object will only recognise words starting with "a"
my-config: make object! [ word-start: charset [#"a"] probe make-word-list my-config "aword bword cword dad"The result would be ["aword" "ad"] If you are happy with the default settings, you can supply none instead of a parameter object.
name | format | use | default | notes |
alpha | charset | basic alphabet, both upper and lower case | charset [#"a" - #"z" #"A" - #"Z"] | |
digit | charset | basic digits | charset [#"0" - #"9"] | |
word-start | charset | characters that start a word | alpha | |
word-letter | charset | characters used in words | union alpha union digit charset ["~"] | There is no need to include "-", ".", "/" or "\" as words separated by these, such as paths or domain names are handled specially |
final-letters | charset | additional characters that may come at the end of a word | charset ["!" "?"] | |
number | charset | characters that make up the body of a number | number: union digit charset [".,"] | |
number-prefix | charset | characters that may be at the start of the a number | charset ["+-£$¢"] | |
number-postfix | charset | characters that may appear at the end of a number | charset ["+-"] | |
word-length | pair | minimum x maximum string lengths to be included | 1x40 | (3x50 means 1-letter, two-letter and 51+ letter strings will not be recognised |
not-prefix | string none | single character that will be used as a not prefix in searches | "~" | set to none if no not prefix is required |
stop-list | block of strings | words that will be ignored | ["a" "is" "the"] | |
ignore-tags | logic | ignore tags if true | false | |
index-pairs | logic | true to index-pairs; false to ignore them | true | |
hyphen | charset | characters used to hyphenate words | charset ["-_"] |
There are many different reasons why you may want to extract the words from a string. Two of the most common are:
The two uses may in your application need to behave slightly differently, especially with regard to handling the not-prefix. As an example, the default make-word-list function acts differently when given a string that contains tildes -- a leading tilde is preserved with the /for-search refinement:
>> make-word-list none "I have some ~tildes in t~~~his ~string~" == ["have" "i" "in" "some" "string~" "tildes" "t~~~his"] >> make-word-list/for-search none "I have some ~tildes in t~~~his ~string~" == ["have" "i" "in" "some" "t~~~his" "~string~" "~tildes"]