A Paper on Computational Linguistic

[ScottKreiger]

Buffy,
The exact structure is something that I will have to get back to you on, it is slightly to moderately complex.
The reason I picked IBM PLI is because I have a couple of copies of it from when I worked as a senier IBM mainframe programmer in the Factory at Sabre, they were a gift so that I could work for free at home. And since the two languages in primary use at IBM shops are Cobal and PLI, I chose pli.
And you are right, they are a little expensive, about the price of a good used car. But it is the best compiler that I have, and it is the language that I am best at.
Scott|8-)>

[ScottKreiger]

Buffy,
is this what you meant by structure? There are only three main sections.
CALL CREATEVOCAB; /* CreateFlatVocabFile;*/
/* This procedure sets up the vocabulary network and fills in the */
/* various fields */
CALL LOADVOCABRECS;
CALL SETVOCABALPHAPTRS;
/*** Set Definition Words Pointers ***/
/*** Adding Related Words ***/
PUT SKIP EDIT('ADDING RELATED WORDS AND THEIR POINTERS')(A);
CALL GETOTHERRELATEDWORDS;
PUT SKIP EDIT('READING WORD COMMAND FILE - WORDCMDS.DAT')(A);
PUT SKIP EDIT('SETTING COMMANDS TO CORRESPONDING WORDS')(A);
PUT SKIP EDIT('SENTENCE INPUT FILLS IN SENTENCE USAGE RECS')(A);
PUT SKIP EDIT('FINISHED CREATING VOCAB NETWORK')(A);

CALL CREATEKNOWLEDGEBASE;
/* This procedure sets up the initial knowledge base using the */
/* initial sentences as input */
PUT SKIP EDIT('READING INITIAL SENTENCES INTO KNOWLEDGE BASE')(A);
CALL FINDSENTPARTS; /* FINDS SENTENCE,SUBJECT,PREDICATE,KERNAL */
CALL FINDSENTWORDS; /*set vocab word ptrs for sentence words*/
CALL SETSENTWORDS2; /*set vocab word ptrs for sentence words*/
CALL SETSUBPREDWORDPTRS;
CALL SETKERNALWORDPTRS;
CALL SETSENTIDEAINTERP;
/* CALL SETSENTGROUPPTRS; */
/*PUT SKIP EDIT('ANALYZING KNOWLEDGEBASE')(A);*/
/*CALL ExpandKnowledgeBase;*/
' ADDITIONAL DERIVED SENTENCES INPUTED')(A,A);*/
/*PUT SKIP EDIT(SentCnt,' TOTAL SENTENCES INPUTED')(A,A);*/
CALL FINDIDEARELATEDSSENTS;
CALL SETPARATHEMES; /*SWKEE*/
CALL SETPARAFLAGS;
CALL FINDRELATEDPARATHEMES;
PUT SKIP EDIT('FINISHED CREATING KNOWLEDGEBASE')(A);

CALL USERINPUT;
/* THIS PROC IS THE USER INPUT PART OF THE PROGRAM. IS IS SET TO USE */
/* A FILE FOR TESTING PURPOSES */
/*PUT SKIP EDIT('INPUT IS QUERY')(A);*/
CALL INPUTISQUERY;
/*PUT SKIP(2) EDIT('INPUT IS COMMAND')(A);*/
CALL INPUTISCOMMAND;

I am light on the user input section, but i think you get the idea from the knowledge base as to how this thing works.
Scott|8-)>

[Buffy]

Originally Posted by ScottKreiger

Buffy,
is this what you meant by structure? There are only three main sections.

This looks like the structure of your program as far as specifying the modules. The structure I was referring to though is that in your paper you have an obvious syntax for the knowledge rules as well as a "translation" (?) of the user input, that is well structured, and could be parsed with a simplistic lexical analyzer (possibly akin to your FINDSENTWORDS), and LALR-or-somesuch-style parser (possibly akin to your FINDSENTPARTS), these two pieces are what lex and yacc do, and they do what you ask for when you said you were looking for a "good syntax checker/analyzer for the user input and for the generation of the knowledge base."

You might want to dive into both a discussion of your approach to building your knowledge base and how you use it (a really facinating topic all on its own). Also, how you make sense of the "user input."

Apropos of nothing I'll mention that in terms of linguistic parsing, the most popular approach these days has to do with creating a knowledge base of existing, pre-translated text that is used to match input, rather than the older and in the past much more popular approach of trying to fully parse and analyze the text and only later looking for idioms and context within the knowledge base for disamibugation.

Infinite lookahead,
Buffy

[PuGZ]

While I didn't understand everything you said in your post, I'll re-read it over the coming days and just thought I'd let you know that as a layperson with an interest in the topic, I found this absolutely fascinating. Keep up the good work!

[ScottKreiger]

Thx, am looking forward to more critiques!
And don't knock yerself fer bein a layperson, i do home improvements myself for a living:-)
But on a more serious note, i wrote this paper for the general comp sci undergrad, i wanted to reach more than just those few who are specialists in the field. Besides, when i worked at a place where i was surrounded by 2000+ programmers, i learned that the best way to avoid creating more questions than i answered was to use simpler words, fact is, i had to go down the vocabulary of a sixth grader to make almost everything clear. The fact that you find this paper understandable and interesting combined with your admittance to being a layperson as far as comp ling goes is a testament to my success.
Scott|8-)>

[ScottKreiger]

Buffy,
where can i find yacc and lex?
But as far as looking up in the knowledgebase as a precursor to parsing user input, i don't think i agree that it is better, at least for my approach, (i am old school, after all). But who can tell? Maybe when the knowledgebase has enough sentences, although you should also realize that i also keep track of all user input and do refer to previous user input when parsing/processing input.
One of my main issues is the knowledgebase. The construction of knowledge, ie where do you start, what order do you introduce the sentences so as to be able to build upon earlier knowledge, and the fact that it is my opinion that there should be 1 - 5 million sentences in the knowledgebase has me concerned a little. The intended size of the knowledgebase is why I created so many pointers as shortcuts to move around quickly, no matter how big a programming headache this gave me! But the size has me a little concerned about the hardware this may require. I had to buy a new pc more than once to run the program over the years. 4 yrs ago the vocabulary network took 12 hrs to initialize, until i rewrote it slightly to include alpha shortcuts that brought the time down to 45 minutes. My current pc (< 6 mos old) runs the entire main prog in about 20 secs, so hopefully i will be able to expand the vocab and KB quite a bit before hitting the hardware wall again.
But i digress, what i what yacc/lex for is the expansion of the KB. I do this as a seperate program that creates the flat file that is used by the main, therefor the program layout shouldn't be a concern.
Scott|8-)>

[Buffy]

Originally Posted by ScottKreiger

where can i find yacc and lex?

"lex" ("lexical analyzer") and "yacc" ("yet another compiler compiler") are the original names of the programs built into Berkeley Unix, and are embedded in the licensed versions of this OS (see FreeBSD or OpenBSD). Good freeware versions are:

• lex: flex
• yacc: bison

Originally Posted by ScottKreiger

But as far as looking up in the knowledgebase as a precursor to parsing user input, i don't think i agree that it is better, at least for my approach, (i am old school, after all). But who can tell?

A lot of the people who have worked on this problem are old school, and its taken them a long time--and a lot of practical testing--to realize that blind sentence fragment matching provides better translation of language, due both to idiomatic meanings as well as "world knowledge" (implied understanding based on the context or related concepts to the actual language content).

Originally Posted by ScottKreiger

Maybe when the knowledgebase has enough sentences, although you should also realize that i also keep track of all user input and do refer to previous user input when parsing/processing input.

And this is where the implementation issue comes in. Matching is better than Parsing only if you have an *enormous* knowledge base at least to start with (its a point of debate as to whether these knowledge bases can be pruned over time to eliminate redundancy in some automated fashion).

It actually sounds to me like you're backing into this approach too:

Originally Posted by ScottKreiger

The construction of knowledge, ie where do you start, what order do you introduce the sentences so as to be able to build upon earlier knowledge, and the fact that it is my opinion that there should be 1 - 5 million sentences in the knowledgebase has me concerned a little.

I guess I'm still a little bit unclear as to what exactly you're putting in the knowledgebase. This quote seems to indicate you're actually storing the original input (that's the Matching approach), as opposed to the distilled "meaning" of the input.

Maybe you want to describe a bit about what the process is that you're using for both building the knowledge base and then how you use it.

Skipping whitespace,
Buffy