colorless Color Forth ver. 0.4 (updated 2/1/01) This is the informal results of a personal R&D effort to understand the 1X programming methodological style and the pragmatic application of Color Forth syntactic structures. As such, the source code (both MASM and Forth) is the definitive guide along with the working code. * Forth ASCII source text files have a .p suffix because a) I used to sell my Forth under the trade name Precise Forth. b) .p was an unused suffix. ** Don't try and run any of the 32bit Forths on 8086/88, 80186 or 80286 CPUs. They perform _NO_ CPU checks and figure you are smart enough to know better. Or anyhow, smart enough to not try that more than once.:-) I would like to thank the following folks for providing Eforth, without which this project would have been very difficult: 8086 figForth by Thomas Newman, 1981 and Joe smith, 1983 aFORTH by John Rible bFORTH by Bill Muench eForth 1.0 by Bill Muench and C. H. Ting, 1990 Eforth was just perfect. Small, unencumbered and elegant. I also wish to express my appreciation to Jeff Fox for providing the web site from which I drew the inspiration for this project, his DOS file I/O, and Chuck Moore for Forth in all its glory. UPDATE 2/1/01: There is now a 32bit Color Forth with complete source, binaries, partial documentation and working Color editor. It has all of the features of the below mentioned 4word, plus color output, single character color interpreter extensions, stack underflow error test, number BASE input and output (old version just had hex and decimal), extreme conditionals and more factoring. See my home page: http://www.modest-proposals.com/ for a link to download 4wordx.zip (x is version number). ------------- Zip contents: The other files in this directory are: Chuk4th.txt :- Chuck Moore's extracted thoughts from transcripts of his lectures at Ultratechnology.com. MlBase.txt :- rough design document that was the basis for this project. MinWord.txt :- 4word definition list ordered by necessity. GPL.txt :- GNU Public License. All of this is GPL. The following directories contain the doc, source, support and binaries of: 4thice :- First attempt at a 16bit colorless Color Forth w/reference eforth. PROK :- Minimal 32bit direct threaded subroutine 1x study. XT4 :- Reference 32bit direct threaded address eforth. 4WORD :- 32bit direct threaded subroutine colorless Color Forth ---------------------- Part 1: 4thIce and Ice Started this project after reading a Chuck Moore quote: "First, Color Forth, if you boot it, will come up with an interpreter. It will be able to read source code like this. That is all it can do. Presumably the first thing you will do with this interpreter is build up some backgrounds so you could put some text on the screen. The second thing is implement a keyboard interface so you could type something to put on the screen. The third thing you might do would be to incorporate some kind of disk access so you could get data." Looked like an interesting challenge. So I took a hacked up 16bit eforth MASM source and started cutting out code. Then adding, re-arranging and re-coding. Added IFDEFs to generate both a full old Forth kernel with new mods for purposes of debugging, as well as a 'minimal' kernel. Ended up with 4thIce (pronounced forthish) as the train wreck of old and new, and its smaller brother Ice that fits Chuck's above quote. Found out a lot about coding attachments (mine, not code addons!). Hard to remove old reliables like 'catch', 'throw', 'hex' and 'u.'. Then when it came time to chop the math functions, I found I just couldn't. Ended up leaving 'um*' and 'um/mod'. Rationalized that because I was writing this for production purposes (dream on,sucker), these functions would be required. Translation: did NOT want to recode them hi-level in terms of Add and Shift. Pragmatic note: discovered that division is required only for numeric output, but not until I was putting the boot extension source together. In a later implementation, I found a more reasonable compromise. See 4word below. Ice has a simple, stupid actually, main loop. It looks up words in the dictionary and executes them. It exits to DOS if a word is not found. Simple and demanding. No numeric input. No compiler. BEGIN TOKEN DUP WHILE FIND 0= IF BYE THEN DROP @EXECUTE REPEAT The bare kernel basic word list is: ----------------------------------------------- empty context xy r@ go handler doVAR >r init hld 0= r> fload 'number = swap rload 'eval 2! over load csp aligned dup bload #tib cell- drop bsave >in cell+ b@r+ create/hopen span rp! b!r+ hopen tmp sp! b@a+ >asciiz ucase bl b!a+ carry fmem cmove a@ hclose base fend a! hread 'expect werd c@ hwrite 'prompt 0< c! (hopen/create) 'echo 1- @ (hopen) 'tap 1+ ! @ds 'emit um/mod doLIT fs '?key um* donx overt rp0 2/ execute sp@ sp0 2* branch rp@ doUSER - ?branch forth up negate exit doVOC getxy not doLIST last gxy + bkp upo cls um+ bye np tx! or u0 cp rx? xor current ?rx and ----------------------------------------------- (114 words) But that's not a minimum kernel like Jeff and Chuck and others talk about!!!! Yet it is a first attempt at a minimal self-extending kernel. What the heck, the kernel was still only 2734 bytes and it couldn't do anything but just interpret. A really minimal kernel could be done in about 45 Forth words. But when I had figured that out was when I put 4word (see below) together and was using a slightly different construction criteria. 'Just interpret' is much more powerfull than you might think. The boot kernel, iBoot.p, takes 358 tokens (space delimited strings) to define the dictionary header definition '$,n' and another 169 tokens to define ','. From that point on, the compiler simplifies to defining 'DEFN' for creating dictionary entries from the input text and defining ']' for compiling the following token in code. All without numeric input. From there, smooth sailing for a typical Forth-in- Forth compiler. By the time the boot matched the code functionality of the original kernel, it was 372 lines comprised of about 5000 tokens. This includes numeric input, output, command line, 'abort', 'words' and 'dis' (SEE). But the real benefit from this, was to understand how the whole Forth system was put together. I won't provide detailed explanations, because it is as much a feel as it is conscious processes. You have to do it yourself. Added an ASCII text editor so that I could justify the overhead of 'word' parsing white-space CRLF text. Have always hated blocks, which force you to use a dumb block editor AND waste space. Also, blocks made it harder to share source (you try to explain to someone, not into Forth, why the 'only' format is 16 lines of 64 characters, no CR/LFs). This also enables the direct application of a full blown text editor to the development effort. The text editor is a working example, not the primary means used to develop 4thIce/Ice. NOTE: All of the Forth interpreters understand tab characters, but the example text editor does not. This feature can be added but was left out in order to keep the code simple. I also added Jeff Fox's simple DOS file I/O, after I got the boot working, to aid in further program development. The debbuger was a simple extension to 'see'. Added the capability to do nested Forth disassembly of displayed words, single step execution of the current displayed word, accompanied by a stack display. This takes 75 lines of code. -------------------------- Part Duh...er Deux: Prok32 The second effort was to hack up a 32bit direct threaded subroutine implementation that I have been using professionally for 10 years. Personal non ANS syntax, multi-tasking. Kind of my programming history in source. 32bit Precise Forth with an OK twist, or prok32, was stripped as far as I could stand it (ohhhh, my babeee!). Because it is a 32 bit implementaion, the smallest I could scrunch its kernel was 8506 bytes. But this included numeric I/O, interpreter, old style compiler, multi-tasking basics and DOS File I/O. I was going to take a break from Color Forth syntax, so just applied minimization to the kernel and tried to (poorly) apply 1x programming to the boot, editor and debugger. See the readme in the PROK subdirectory and the source for specific documentation. What I got out of that exercise, was a trimmed down kernel and a possible real production tool. What you get is a comparative sample to make size and speed comparisons against. Also hacked my syntax so that the compiler accepts (almost) the same syntax as the other 4 kernels. The 32bit direct threaded subroutine code seems to be about 4 times faster than the 32bit implementation of the direct threaded address XT4 eforth, and 2 times faster than 16bit 4thIce/Ice. Part 3: XT4 XT4 is a 32bit conversion of the direct threaded address 16bit eforth. See the readme in the XT4 subdirectory for usage and implementation details. It, too, is just provided as a reference for size and speed and was useful for 1x efforts. The XT4 'bare' kernel is a complete 32bit eforth implementation and comes from the assembler at 14,292 bytes. It implements quasi-ANS compiler syntax and always looks for numbers in the dictionary before conversion. ------------- Part 4: 4word My latest take on putting all of the above together is 4word. It takes a stripped, minimal 32bit direct threaded subroutine call kernel, plus numeric I/O, DOS file I/O, asm command line, asm dictionary search and name creation, compiler primitives, top of stack in T register, indexed A register and indexed R register (uses top of return stack) and simple minded interpreter-only executive. In other words, almost everything talked about at UltraTechnology by Jeff and Chuck. The only thing lacking is non-ascii characters. 4word 'bare' kernel word list: ----------------------------------------------- go hex 2* a@+ preset u. +* a!+ abort d. um+ c@ fload cr or c! load type xor w@ bload ' and w! bsave find not @ create/hopen L over ! hopen ] dup a@ >asciiz ; swap a! carry call drop @a hclose , sp! !a hread w, sp@ execute hwrite c, >r ?branch (hopen/create) : r> doLIT (hopen) N r getxy @ds token rp! gxy x" cmove rp@ cls - word cr@+ tx! + number cr!+ rx r@ mu/mod ac@+ rx? query * ac!+ halt accept u* ac@ bye decimal 2/ ac! ----------------------------------------------- See the accompanying file MinWord.txt for a breakdown of the necessity of the various words used in 4word. It is conceivable, but is actually infeasible without a Forth assembler. This really is a more compact 32bit kernel at 3900 bytes. It has 95 words that were selected after my experiences implementing the Ice kernel. Cutting the kernel back to interpreting, only, is a worthwhile exercise. Useful for new environments, a fresh take on a stale problem, or as a specific exercise to hone ones programming skills. But 4word aint minimal. It is non-minimal MISC on CISC which takes full advantage of the rich opcode resources and registers of the 386 architecture using MASM. The 4word kernel makes novel use of lea (load effective address) in lieu of 'multiply' for numeric input (NUMBER), and has a very efficient use of the Intel 'div' instruction for single precision use in double precision numeric output(D.). The later short piece of assembly code was also used to implement 'mu/mod'. See 4word.asm source for full details. ----------------- Informal Results: All of the different versions of Forth have an associated Forth disassembler/debugger and an ASCII CR/LF text editor. These are provided for comparison of ease-of-programming, code size and speed. The combined total source (MASM kernel, boot, editor and debugger) ranges from 52k total for the smallest (4word) to 96k for the largest (XT4). Ice would be the smallest, but it shares IFDEFd source code with a quasi-complete 4thIce. Total binary code+dictionary sizes range from 9.5k (16bit Ice) to 19.8k (32bit XT4). 'Bare' kernel code+dictionary range from 2734 bytes (16bit Ice) to 14228 (32bit complete XT4). The 4thIce.com and Ice.com file sizes are misleading. The code is embedded in a 16k-$100 hex block. This is due to a split in code and dictionary and to fill out the Ice.com file to a known boundary (4000h) for the interpreter to know where to start interpreting in RAM. The file 4.com is the result of concatenating Ice.com + iBoot.p, i.e. from the DOS command line 'copy /b Ice.com+iBoot.p 4.com' 4thIce does not have a 4iBoot.p file because it's extensions are in the MASM source. Same for XT4. Prok32 and 4word have pBoot.p and 4wBoot.p, respectively, but because they have kernel DOS File I/O and command line, can load them from their Forth command lines, i.e. 'fload pBoot.p ' (Prok32 and 4word require the trailing space, after the file name.) File sizes: -------------------------------- 01/10/2001 10:04a 57,494 4thice.asm 01/17/2001 10:17a 2,189 4iDbg.p 01/18/2001 02:16a 6,653 4iDe.p 01/10/2001 10:05a 16,128 4thice.com 01/10/2001 10:04a 57,490 ice.asm 01/19/2001 01:31a 23,424 iBoot.p 01/17/2001 10:17a 2,189 iDbg.p 01/10/2001 10:05a 16,128 ice.com 01/19/2001 12:31a 39,552 4.com 01/17/2001 10:22p 62,486 prok32.asm 01/17/2001 05:33p 12,012 pboot.p 01/17/2001 10:48p 3,465 pDbg.p 01/18/2001 10:07a 6,445 pDe.p 01/17/2001 10:22p 8,506 PROK32.COM 01/04/2001 08:32p 56,569 XT4.ASM 01/18/2001 11:42p 3,894 xDbg.p 01/17/2001 11:44p 6,425 xDe.p 01/04/2001 08:32p 14,292 XT4.COM 01/18/2001 02:41p 30,067 4word.asm 01/18/2001 03:23p 7,310 4wBoot.p 01/18/2001 03:27p 3,852 4wDbg.p 01/18/2001 02:54p 9,304 4wDe.p 01/18/2001 02:41p 3,900 4WORD.COM ------------------------- Conclusions and Comments: --- 1x: 1) If one does not know the compiler, one CANNOT do 1x programming. You gotta be smarter than the computer. You don't like that? Go program in 'C' where the problem is even worse. 2) Factoring improves the design by forcing the programmer to inspect the code and recognize redundancies. This encourages the programmer to optimize these sequences. Factor, factor, factor translates to inspect, inspect, inspect the written code. Always worthwhile. Also gives the programmer a terminal condition to stop at, rather than having production shoot him so that the code _can_ go to production. 3) Went back and just casually inspected the 4word code. Noticed that the inline code for 'DUP' and 'DROP' was repeated many times. Factored those to either call32 or jmp and ended up saving 193 bytes. (The following 2 are caveats about factoring, NOT criticisms.) 4) However, factoring applied to the limit is counter productive when generating production code. Over factoring can add to the depth of subroutine nesting, thereby slowing down execution. 5) Factoring can also increase memory usage when applied without consideration of the dictionary overhead. Typically, factoring short sequences that occur only 1, 2 or 3 times may take more memory. Longer code sequences really lend themselves to factoring. ------------ Color Forth: a) Complex interpreter/compiler jump tables are irrelevant to well formulated (and formed) code. For a working Color Forth w/source, that implements these jump tables, go to my web site (below). The above mentioned colorless Color Forth compilers all use just a simple interpreter, i.e. (defined in old Forth syntax) BEGIN ( get input and interpret loop) QUERY TIB @ >IN ! ( get user input, reset text pointer) BEGIN ( start interpreter loop) TOKEN DUP ( delimit next input word) WHILE ( while not end of line, a 0 length token) FIND ( search dictionary) 0= IF ABORT THEN ( if error then reset on bad word) DROP @EXECUTE ( execute, otherwise) REPEAT ( continue interpreter loop) 2DROP ( 0 length token) . " OK" CR ( make user feel good) REPEAT ( forever) Using the following single character 'modes', with the above interpreter, enables colorless Color Forth programming: ':' := create new dictionary entry and (optionally) compile a call to doLST '[' := null definition. Used to visually indicate entering 'IMMEDIATE' mode. i.e. keep interpreting. Same thing as no prefix. ']' := compile a call to the immediately following string token. '$' := convert the following string token as a hex number. '&' := convert the following string token as a decimal number. (Never needed decimal input, just output in the text editor) ';' := exit/return from the current word. 'v' := create a named variable, initialized to zero. 'L' := compile number TOS as an inline literal '(' := comment; skip characters until terminating ')' found. Color, you ask? This is colorless Color Forth. All of the syntactic cues for the interpreter are single character and can be (tada!) interpreted as colors by the COLOR text editor _you_ write. Interpreting the above single character 'mode' tokens as individual colors, turns colorless programming, colorfull. That is, if you have display code that supports color output. b) 'IMMEDIATE' definitions in Color Forth and ANS Forth are just telling the compiler to interpret the word, not compile a call to it. So the programmer needs to know the interpreter well enough to know when to prefix conditional branch tokens with '[' as well as numeric input and strings, JUST LIKE IN COLOR SELECTION. See rule 1, above, in the 1x section. c) Minimal kernels allow a _new_ opportunity for re-writing the high-level structures that make up an implementation. They are also less memory efficient than a dedicated kernel (e.g. 15k self-extending kernal+source vice 9.5k pure object code, both including text editor). Keep the minimal kernel around for really new endeavors, but it is so much easier to use a dedicated kernel tuned to the environment at hand. d) (Really showing my ignorance on this one) Implementation of indexed A and R registers is not nearly as cost effective as I thought. The A and R registers are saved and restored across all kernel primitives, but the cost of continuing to do so in high level definitions can be prohibitive. Unless there is a specific hardware requirement (say the F21 cpu), judicious assembly language programming in the kernel (i.e. direction sensitive CMOVE) can compensate for their lack. The A register was used minimally and the R was never used in the kernel, debugger and text editor because they took an extra program step to load. I always found an effective way to use c@ and c!, instead. Specific apps that require significant block memory moves DO make indexed A and R cost effective on a speed basis, if nothing else. e) Command line input of Color Forth is a raving pain in the fingers. Having to remember to prefix each number with a color or a symbol gets in the way of exploratory development. Having kept and used a reference 'old' Forth, it was _always_ a pleasure to check something quickly on it. BUT, when forming a program in the text editor, the Color Forth structures well paid for themselves in forcing rigor to the programming process. My conclusion about Color Forth is good for design, bad for ad hoc hacking (sic). f) My _personal preference_ is to use standard ASCII (CR)LF text files. (Which explains why I include an ASCII text editor as an example.) Standard text enables both easy communication of the source to other people and the use of hi-level tools like full-blown text editors, grep and uniq during factoring. Embedded development systems are typically rich in memory and will support ASCII text. If production memory is limited, then a stripped binary is going to be delivered regardless of the source format. g) MISC on CISC or why I couldn't get rid of mult, divide and asm cmove. When implementing Color Forth on a limited resource CPU, there is no choice. But when you start implementing MISC on a CISC CPU, why waste the time and effort coding hi-level multiply, divide and string ops when they are intrinsic to the CPU. Almost certainly more efficient than any Forth code written to replace them. Once you understand how these opcodes work, spend that time and effort on optimizing things you can improve. Also, the kernel is written in MASM assembler. Writing extensions in the kernel allows not having a Forth assembler. Porting two small apps to five versions of Forth was bad enough. Did not even want to attempt to maintain Forth assemblers for all parties concerned! -------------------- Personal background: Male. Been doing systems analysis, embedded process control, Forth and ASCII text editors for over 25 years. Maybe one of these days I will get it right. ----------------- Feedback welcome. email: loveall@iinet.com Two operational color Color Forths, can be downloaded from my home web page: http://www.modest-proposals.com/ For the inspiration of this effort see: http://ultratechnology.com/dindex.htm especially the Fireside Chats. For everything else, including tutorials, Forth, try: http://www.taygeta.com/forth.html Terry L. Loveall Updated: Feb. 01, 2001