Overview
The goal of this project is to build a scanner for a version of the Core language, a pretend
language we will be discussing in class.
For this project you are given the following:
\3341 Project 1.pdf” – This handout. Make sure you read it completely and handle all
requirements in your implementation. You are encouraged to post any questions on
Piazza.
\main.c”, \core.h”, \scanner.h”, \scanner.c” – I have outlines the project in this les,
and give you some of the code you will need. Make no changes to to \core.h” le.
You can make changes to \main.c” as long as those changes do not change break my
tester.sh script.
You may create additional les to contain any additional classes or methods you want
to create.
\tester.sh” – This is a script I wrote to help you test your project. It is very similar
to the script that will be used to grade your project, so if your project works correctly
with this script you are probably doing well. The only guarantee I give is that this
script will work on stdlinux.
Folder \Correct” – This contains some correct inputs and their expected outputs. The
\tester.sh” script will test your code against these examples.
Folder \Error” – This contains some inputs that should generate error messages. The
\tester.sh” script will test your code against these examples.
The following are some constraints on your implementation:
Do not use scanner generators (e.g. lex, ex, jlex, j ex, ect) or parser generators
(e.g. yacc, CUP, ect)
Use only the standard libraries of C. This is the reference I like to use:
https://en.cppreference.com/w/c/header
Your submission should compile and run in the standard linux environment the CSE
department provides (stdlinux). I will leave it up to you to decide what IDE you will use or
if you will develope your code locally or remotely, but as a nal step before submitting your
code please make sure it works on stdlinux. Use the subscribe command – make sure
you are subscribed to GCC-10.1.0. The graders will not spend any time xing your
code – if it does not compile on stdlinux, your project will not be graded and you
will get a 0.
Your Scanner
You are responsible for writing a scanner, which will take as input a text le and output a
stream of \tokens” from the core.h enumeration. You scanner must implement the following
functions:
scanner open and scanner close: These functions open the le, nd the rst token, and
release memory when we are done scanning.
currentToken: This function should return the token the scanner is currently on, with-
out consuming that token.
nextToken: This function should advance the scanner to the next token in the stream
(the next token becomes the current token).
getId: If the current token is ID, then this function should return the string value of
the identi er. If the current token is not ID, behavior is unde ned.
getConst: If the current token is CONST, then this function should return the value
of the constant. If the current token is not CONST, behavior is unde ned.
All of these functions will be necessary for the parser you will write in the second project.
You are free to create additional functions.
To make things easier for you, you may assume no token is made of more than 20
characters. Also, I suggest using calloc to allocate memory, instead of malloc.
Input
The input to the scanner will come from a single ASCII text le. The name of this le will
be given as a command line argument to the main function.
The scanner should process the sequence of ASCII characters in this le and should
produce the appropriate sequence of tokens. There are two options for how your scanner can
operate:
(1) the scanner can read the entire character stream from the le, tokenize it, stores all the
tokens in some list or array and calls to currentToken and nextToken simply walk through
the list
or
(2) the scanner reads from the le only enough characters to construct the rst token, and
then later reads from the le on demand as the currentToken or nextToken functions are
called.
Real world scanners typically work as described in (2). In your implementation, you can
implement (1) or (2), whichever you prefer.
Once your scanner has scanned the entire le, it should return the EOS token (End Of
Stream).
Invalid Input
Your scanner should recognize and reject invalid input with a meaningful error message. The
scanner should make sure that the input stream of characters represents a valid sequence of
tokens. For example, characters such as ` ‘ and ‘%’ are not allowed in the input stream. If
your scanner encounters a problem, it should print a meaningful error message to standard
out (please use the format “ERROR: Something meaningful here”) and return the ERROR
token so the main program halts.
The Language
The Core language consists of 4 kinds of strings, which you will need to tokenize:
Keywords:
and begin do else end if in integer
is new not or out procedure record then while
Identi ers:
Begins with a letter (uppercase or lowercase) followed by zero or more letters/digits.
Refer to this regular expression once we cover regular expressions:
(aj : : : jzjAj : : : jZ)(aj : : : jzjAj : : : jZj0j1j : : : j9)*
Constants:
Integers from 0 to 1009 (inclusive)
Symbols:
+ – * / := = < : ; . , ( ) [ ]
Your scanner walk through the input character stream, recognize strings from the lan-
guage, and return the appropriate token from the enumeration in \Core.java” or \Core.py”.
If there is any situation in which it is unclear to you which token should be returned, please
ask for clari cation on Piazza.
Write your scanner with these rules in mind:
1. The language is case sensitive, and the keywords take precedence over the identi ers.
For example, \begin” should produce the token BEGIN (not ID), but \bEgIn” should
produce the token ID.
2. Strings in the language may or may not be separated by whitespaces. For example the
character stream may contain the string \x=10″ or the string \x = 10″, and both of
these should generate the token sequence ID EQUAL CONST.
3. Always take the greedy approach. For example, the string \whilewhile” should produce
an ID token instead of two WHILE tokens, string “123” should produce a single CONST
token, and string \:=” should produce ASSIGN.
4. Keyword/identi er strings end with either whitespace or a non-digit/letter character.
For example:
(a) the string \while (” and the string \while(” should both result in the WHILE and
LPAREN tokens.
(b) the string \while 12″ should result in the WHILE and CONST tokens, but the
string \while12″ should result in the ID token.
5. Constant strings end with any non-digit character. For example:
(a) the string \120while” or \120 while” should result in the CONST and WHILE
tokens.
6. Symbols may or may not be separated from other strings by whitespace. For example:
(a) String \++while<= =12=” should result in the token sequence ADD ADD
WHILE LESS EQUAL EQUAL CONST EQUAL.
Let me know if you think of any situations not covered here.
Testing Your Project
I have provided some test cases. For each correct test case there are two les (for example
4.code and 4.expected). On stdlinux you can redirect the output of the main program to a
le, then use the di command to see is there is any di erence between your output and the
expected output. For an example of how to do this you can take a look at the script le
“tester.sh”.
The test cases are weak. You should do additional testing with your own test cases. Feel
free to create and post additional test cases on piazza.
Project Submission
On or before 11:59 pm January 27th, you should submit to the Carmen dropbox for Project
1 a single zip le containing the following:
All your .java or .py les.
An ASCII text le named README.txt that contains:
{ Your name on top
{ The names of all les you are submitting and a brief description stating what each
le contains
{ Any special features or comments on your project
{ Any known bugs in your scanner If the time stamp on your submission is 12:00 am on January 28th or later, you will
receive a 10% reduction per day, for up to three days. If your submission is more than 3
days late, it will not be accepted and you will receive zero points for this project. If you
resubmit your project, only the latest submission will be considered.
Grading
The project is worth 100 points. Correct functioning of the scanner is worth 65 points. The
handling of errors is worth 20 points. The implementation style and documentation are
worth 15 points.