A simple Prolog tokenizer

Introduction

The purpose of this prolog script is to tokenize a sentence into a list of atoms of words without punctuations and spaces. In addition, all alphabets are converted into lower case letters.

See the predicates codes_atoms and remove below for the logic behind.

Output

Enter a query: |: Hi, I am alpha. What is your name? [hi,i,am,alpha,what,is,your,name] Enter a query: |: Here are some hints: 1. Buy some food. 2. Eat. [here,are,some,hints,1,buy,some,food,2,eat] Enter a query: |: I can handle multiple spaces too! [i,can,handle,multiple,spaces,too] Enter a query: |:

Source Code

% A simple prolog tokenizer % % remove items listed in RL from the % list OL to form the list NL % eg. % ?- remove([a,l,e],[h,e,l,l,o],L). % L = [h,o]. % remove(RL, OL, NL) :- RL = [H|T], delete(OL, H, NL1), remove(T, NL1, NL). remove([], X, X). % accept a list of codes and % convert them into a list of atoms of words % 1. skipping all punctuations such as % (.), (,), (:), (?), (!), % single quote and (#). % 2. convert all letters into lower case % 3. remove all spaces % eg. % ?- read_line_to_codes(user_input,Codes), % | codes_atoms(Codes,L). % |: Hello everbody, here is Robot1! How are you all? % Codes = [72, 101, 108, 108, 111, 32, 32, 101, 118|...], % L = [hello, everbody, here, is, robot1, how, are, you, all]. % codes_atoms(Codes, CleanAtoms) :- maplist(char_code, Ch1s, Codes), % change to list of chars maplist(downcase_atom, Ch1s, Ch2s), % convert chars to lowercase remove(['.',',',':','?','!','#','\''], Ch2s, Ch3s), % remove punct. atomic_list_concat(Ch3s, A1), % combine into atom atomic_list_concat(A2, ' ', A1), % split atom to atoms, sep.: space remove([''], A2, CleanAtoms). % remove extra spaces /* inactive mode */ go :- repeat, writeln('Enter a query: '), read_line_to_codes(user_input, Codes), codes_atoms(Codes, Atoms), % combine codes % into atoms of words writeln(Atoms), Atoms == [quit], !. % start go after this script is loaded. :- initialization(go, program).