Our main objectives were to have a multi-thread environment with queues, mutex and read-write locks to protect shared data and queues integrity. We also decided to use real-time clock timers for accurate wake up time to verify if a timeout as occur or not.

In order, to have a fully working multi-thread environment, that is fast, efficient and reliable, where data integrity is guaranteed, we needed to use some protection mechanism that will insure that no global shared object will ever get corrupted. We used global shared object, so that we can split the task to be accomplished by many threads that are working together to get the job done. Those mechanisms are known as mutex and read-write locks. Mutex ensure that the only owner of the lock will be able to modify the data at a given time. Read-write locks ensure that only a single writer can modify the data or that many readers can read the data at a given time. In order, to achieved some degree of portability, we used the commonly known Posix Threads library, that we wrapped up to make it easy to used in a class named Basic_PThread. We did the same for Mutex and RWLock classes. Those classes permit us to forget about the Posix Thread library complexity and to make it as easy to use as it is in Java. Having those tools at hands, we divided the program in many sub tasks, where each sub task is a class, which inherit from Basic_PThread. We can therefore say that we used mainly the bottom-up approach, mainly by first designing the required tools needed by the program and then use those fully debugged and working classes to build up more complex classes.

For instance, we wrote down a brief list of needed classes for the program and decided to start designing and coding those classes, since everything else used those classes by after. For example, we cannot design multi-thread TFTP classes without having threads, mutex, RW locks and UDP sockets working.

It was pretty easy by after to create as many sub task as needed as the following list illustrates:

• TFTP_Packets is used to create and send packets.

• TFTP_Option parse the command-line and retrieve the appropriate information to initialize every global data structures that are used everywhere in the program.

• TFTP_Parser parses incoming packet sent by the appropriate UDP_Socket class and dispatches the packet to the appropriate function, so that the wanted TFTP task, asked by the other host, get executed.

• JFile is used to open, close, read, write and seek inside a file.

• DataFactory is used to create data packets from a JFile object.
• FileFactory is used to take data packet enqueued in the data_queue by the DATA_driver function in TFTP_Parser and dequeue them to write them down in the file, using JFile functions.

• TFTP_DataAck is called by the ACK function in TFTP_Parser that will look at the waitingAck_queue and increase the window size and remove the packet from the data queue when it’s been acknowledged.

• TFTP_Error creates all the JErrorHandler objects that points to the callback function, which will send an error packet to the other host, if the program is connected. It will append those JErrorHandlers to the global object pgmErrorController, which is of type JErrorController. It will also define all the macros that will be used to call these error handlers by other parts of the program.
• TFTP_Sender sends all the data packets created by the DataFactory until the queue is empty or that the window size has reached zero. The windowsizeLeft value is decremented every time a packet is sent. The windowsizeLeft will be incremented by TFTP_DataACK when the appropriate acknowledgement is received.
• Timer is a base class that defines a basic real-time clock timer that will be interrupted by a given amount of time specified in seconds and nanoseconds. When this event happen it callback a virtual function called timedRun implemented by the child class.
• TFTP_Timeout is a derived class from the Timer class, that is used mainly to check if a packet acknowledgment timeout event as occur, by searching through the waitingAck_queue for expired timestamp packets, so that they can be resent.
• UDP_Socket is a base class that defined everything in common between UDP_SocketClient and UDP_SocketServer.
• UDP_SocketClient is a derived class from the UDP_Socket class that implements every functionality of a client-side socket, as mentioned earlier.
• UDP_SocketServer is a derived class from the UDP_Socket class that implements every functionality of a server-side socket, as mentioned earlier.

• TFTP_Shell is a class that contains every user interactive functionality that a shell can provide, such as defining TFTP options and get/set/put/open/close commands. The current version does not implement the TFTP_Shell yet and therefore only used the command line approach as demonstrated during the demo, resulting in one file transfer at a time.
• TFTP is the main program, it creates and initialized every object and variables following what was given on the command line by asking TFTP_Option to parse it. It then identifies which kind of TFTP is running: server, client, client reading or client writing, and execute the appropriate function to get the job done. It will create and start every needed sub threads and ask them to do the required task for the current program state.

Run server on port 2000

Run a client to write a file call test.txt on the server DEA,

port 2000 with a blocksize of 600 bytes and a windowsize of 3.

As mentioned, when we send a data packet, we take it out from data_queue and send it through the socket and decrement the windowsizeLeft counter by one. We keep on sending data packet until the windowsizeLeft counter reaches zero. When we get an ACK packet, the ACK_driver function in TFTP_Parser will search in the queue and increment the counter by one for each element found.

This is fairly simple, we read from disk in binary read-only mode up to the given blocksize, if that much data is provided. We construct data packet from it using DataFactory, label them and queue them into data_queue. After each reading, a file seek of blocksize bytes is done if the data length is equal to the blocksize option, so that the next read to file will give us the next data packet, else we close the file. The current data transfer session is ended when a packet of less than the blocksize is received or get acknowledged. If the file size is a multiple of the blocksize number, we send a dummy packet, with no data attach to it, which acts as a flag to end the session.

As mentioned, when we send a data packet, we take it out from data_queue and send it through the socket, we keep a local copy that we timestamp with an issue time of currentTime + timeout and append it to the waitingAck_queue. Once the timer wakes up, we increment the currentTime value and scan the list for expired timestamp data packet. For instance, if the new current time is lower then or equal to the expired timestamp then this data packet timeouts and its resent. We send only one packet at a time, if by chance, we receive an acknowledgement before the next data packet expire then we know that everything was received. In the worst case, the next data packet will timeout and be resent and so on, until an acknowledgement for up to the last packet is received.

When we parse the command line in TFTP_Option, we call a function named build_OptionPacket() and build_RRQ_WRQ_TemplatePacket(). The first one creates a template packet that will be used by RRQ, WRQ and OACK statements. It simply insures that all initialized variables are appended to Option_template through memcpy(), the length is saved in Option_templateLength and the two templates are protected by the read-write lock named rwlock3_templates. The number 3 is here to avoid any deadlocks, by simply making sure that any sequence of RWLock or mutex requests are in ascending order. RRQ_WRQ_template is a template with a default RRQ opcode, the filename requested and the transfer mode. Once a RRQ or WRQ has to be sent, the function create_RRQ_WRQ() locks the RWLock for reading and checks the length and if it’s bigger than zero, it appends it to the RRQ, after copying the RRQ_WRQ_template and opcode in the appropriate byte location.

On the server side, when we receive RRQ or WRQ, we parse the raw packet and identify every null-terminated string using the strlen() function and by doing some pointer arithmetic. We copy all those null-terminated string to an array and store the size of the array for further processing. We then scan the array for keywords like "blksize", "window", "octet", "timeout", etc. knowing that the next parameter, except for "octet" shall be the value of that variable. Using strlwr(), strcmp(), atol() and few if statements, we can know exactly what is what and even know if some garbage was added and ignore it. We identify each variable in consequence and an Option_template is created. If OACK_arg is set to true, then we know that we must respond with an OACK, else we send DATA 1 or ACK 0. Due to some unknown bus error, we had to comment OACK_arg out, but the templates are still created and decoded accurately on both sides.

1. Connection Phase

• Transfer of file begins with Read Request (RRQ) or Write Request (WRQ) to a file
• Connection is established when server grants the request
• Octet mode is used to transfer a file (8-bit format)
• The option Acknowledgment (OACK) is used to acknowledge a client's option negotiation request

1. Data Transfer

• When the client appends options to the end of a Read Request packet, the server may return three possible responses:

OACK Acknowledge the Read Request and the options.

DATA 1 Acknowledge the Read Request but not the options.

ERROR The read request has been denied.

• When the client appends options to the end of a Write Request packet, the server may return three possible responses

OACK Acknowledge the Write Request and the options

ACK 0 Acknowledge the Write Request but not the options

ERROR The write request has been denied

• When the server implementation does not support option negotiation, the server ignore any options appended to the client's request

• File is divided into a data packet of size blocksize bytes. (e.g. 512 bytes)
• Each data packet is acknowledged by an acknowledgment packet

or by an acknowledgement packet for a data packet with a bigger block number.

• If the packet get lost in the network than the intended recipient will timeout and retransmit only the last packet that timed out.
• The sender has to keep all packet not acknowledged for retransmission.
• The transfer is terminated when an error packet is sent or when a data packet with a data size smaller then the blocksize or when we received an acknowledgement for the last packet of the session.
• When no activity takes place for a very long amount of time, both parties are said to be disconnected from each other, an error packet might be sent.

1. Disconnection Phase

• The transfer is completed when the data packet is less than the blocksize.
• The host sent the final ACK just before disconnecting.
• After the host sent the final ACK, the host will wait for a while before disconnecting.
• If the host time out, the host will retransmit the final DATA and wait for a final ACK. If this procedure repeat 3 times, the host will terminate the connection

The class Mutex contains the following POSIX C structures:

pthread_mutex_t

pthread_mutex_attr_t

pthread_cond_t

pthread_cond_attr_t

The class Mutex is used when one user wants to access a shared variable, (generic shared data inside the program) to prevent two users from accessing the same shared variable at the same time, to guaranteed data integrity and to avoid data corruption. By definition, Mutex works on a FIFO (first in first out) basis, which is the first user to request the Mutex, obtain the Mutex.

The class RWLock is used when one user wants to access a shared variable, (generic shared data inside the program) that might be read by many readers at a given time. It is implemented using Mutex and a bunch of counters. The first reader gets the mutex and grant access to every other read request but denying every write requests. The last reader to notify a release the RWLock releases the mutex. The first writer gets the mutex and prohibits any other action on the protected object until he’s done with the modifications.

The class UDP_SocketServer will listen at a well-known port for service requests. The UDP_SocketServer::ListenThread() process remains dormant until a connection is requested by a client's connection to the server's address. When a connection is requested to the server's address, the server process wakes up and services the client requests. The server address consists of its IP address and the service port where the request was made. Once a server has established its environment, it creates a socket and begins accepting service requests. The bind() function is required to ensure that the server listens for request at the expected location. In order to listen to a port number under 1024, the server must be run with root user ID.

The class UDP_SocketClient is an active entity process that initiates a connection when invoked. The client knows where to find the server by using the getservbyname() function. Next the destination host is looked up with a gethostbyname() call. At this point the address buffer is cleared, then filled in with the Internet address of the remote host and with the port number where the login process resides on the remote host. At this point a socket is created and a connection is initiated. The process UDP_SocketClient::ListenThread() is created to wait constantly for a reply from the server.

The class FileFactory will take care of writing the raw packets into the file, by parsing the data and the length and store it into the file at the appropriate location by calling the appropriate JFile functions. We currently use directWrite() instead of our first design which was to enqueue packet to be written into data_queue and then let the FileFactory thread take care of writing the packets to the file once it get some CPU cycles. The problem was that the FileFactory didn’t get enough CPU cycles and it was therefore simpler to just write the data packet to the file as soon as we receive them.

The class DList_pPacket is simply a double linked list with queue features where each element stored is a pointer to a TFTP_Packet object.

The class DataFactory will take care of reading the file and produce data packets from the content stored inside the file. After that, it will take the data packet and enqueue it at the end of the double linked list named data_queue. This class use JFile and TFTP_Option to read from the file.

The class TFTP_Options will take care of every option and mostly all global variables that we are using inside this program. Most of those variables are initialized by default and reinitialized later on as needed through the command line, the shell, a client request or a server acknowledgement. Few RWLock objects protect all the data stored inside this class. Each of them is assigned an ID number, so that any cascaded request to a RWLock must be in ascending order to avoid any deadlocks. For instance, you can lock 1 and 3, but you cannot ask for 2 before unlocking 3, else a deadlock will probably occur.

This way all the class users can simultaneously access the data as needed in parallel, but only one of them can write to it, but many of them can read it, at a given time. As seen in the demo, we often print the current program status by calling showOptions() to display the following global data and their length for debugging purposes:

• Host name
• Port
• Timeout value
• File name
• File size
• TFTP Access Mode
• Block size
• Window size
• JFile pointer to myFile
• UDP_Socket pointer to sender
• DataFactory pointer to dataFactory
• FileFactory pointer to fileFactory
• TFTP_Timeout pointer to timeoutThread
• TFTP_Sender pointer to sendFactory
• DList_pPacket pointer to data_queue
• DList_pPacket pointer to waitingAck_queue
• RRQ_WRQ_template
• Option_template
• TFTP_Status_t state ( State of the program )
• TFTP_Type_t state ( Type of the program )
• Boolean state variable Done
• LastData ( Indicate if the current block number is equal to

• currentTime ( Indicate the current time as a double value )
• windowsizeLeft ( A counter to know how much we sent to make sure

• openedFile ( The name of the file we opened )

Each of those data follows the RFC documentation, the range are respected and for the access mode any value other that octet is considered as non implemented and result in the termination of the program with an error code 2.

To make the implementation of the class TFTP_Packet easier, we added two special functions in TFTP_Option which are to build the option extension template using given defined parameters and to build the read or write request template without the option extension. This way, the TFTP_Packet simply have to take care of initializing packets with the default-established templates that we decided to use. The first operation is obviously to create those templates.

The class PThread and Basic_PThread are wrappers for the Posix Thread set of functions part of the POSIX Thread library. PThread is more evolved than Basic_PThread, since it contains the setting for scheduling and other pthread advanced features. This class is wonderfully design and very easy to used. It follows the Java idea, which is order to get a working PThread, you simply have to inherit from that class, implement the function virtual void run() and start() it to make the pthread running. The function start() will take care of initializing the pthread and create it with default scheduling parameters, it will pass a static function wrapper, since a callback cannot be done on virtual functions and a pointer to the class, since a static function cannot access members of a class. The wrapper takes a void* argument which will be cast back to PThread*, so that we can call the run function from it, as simple as that. If you ever programmed threads in Java, you will get the same easy to use API to make your life simple. Inherit from PThread, implement run() and start() it !

The class Timer uses signals and timer_create() to implement a real-time clock timer, it uses SIGUSR1 that will be catch by a timer handler. Since that timer_create() cannot pass a void* parameter and that only signal SIGUSR1 and SIGUSR2 can be used, we are forced to have a maximum of two timer for the entire program, unless we fork a timer for each packet, but that could slow down everything. This class use the same trick as in PThread, which is to call a static function that will call a virtual, unimplemented one that shall be implemented by the children class. The Timer class constructor takes as a parameter the number of nanoseconds and seconds which is the clock tick length of the timer square wave. The clock low duration and the clock high duration are set as equal, to get a real square wave at the specified frequency.

The class JFile can handle any kind of files, text or binary. The main role is to read data from it or to write to data into it. The class knows itself (the FILE* pointer), it’s size and it’s current position. The filename included in the path, access mode and the record size are given as parameters to the constructor. The JFile is able to parse the given path to retrieve the short filename, so that the TFTP_Option user can ask JFile what is the short filename, so RRQ and WRQ parameters can be established. It can also ask the file size for the option extension, if needed. Therefore, the class is able to append data or to read data as a stream, since it updates the pointer as it read or write into the file by incrementing the pointer by the specified block size stored internally.

JErrorNo is an enumeration of supported errors that are used in C programming. It follows the errno definition as declared in <sys/errno.h>. Since we used a lot of C functions inside this TFTP program, it was much more convenient, to use the returned errno value and simply match that value with a set of predefined error handlers. The error code returned by the function is converted to a JErrorNo type and passed to the JErrorController. Since different platforms have different error number, we standardized it, so that we can have a uniform array of function pointer. It converts incompatible error number to the one we defined as standard and then pass it to the call() function, which will do a look up in the array using the JErrorNo as an index. Each element of the array are a pointer to the head of a single linked list of JErrorHandler* elements.

The class JErrorController manages and formats the error received from the TFTP program. The JErrorController will find in the array the error received from the TFTP program and deliver the error to the error handler. The main task of the JErrorController is to react accordingly to that specific error by using the JErrorHandler for that error number. The call() function inside the JErrorController class will simply call the JErrorHandler* head, which will call a static function which will deal with the error. The default error handler is simply a text display, specifying the origin of the error and a short description of the possible reason to explain why this error happened. The line number and source code file name are displayed and also the theoretical JErrorLevel, which is a value out of five, which tell us how important this error is and if the program can be resumed, aborted or if the error can be solved or not.

The class JErrorHandler handles errors and try to resolve the error according to the description of the error and the error number. If the error can be recovered then JErrorHandler will resolve the current error. Otherwise, JErrorHandler will display by default the following error message on the standard error output ( stderr ):

***********************************************

An error of type #X of level Y/5 has occurred

at line L of file XYZ.cpp,

Here's a brief description of the error:

[………]

***********************************************

By default the JErrorHandler will point on the default error handler. New error handlers can be added to perform specific error handling management, i.e. different from the defaultErrorHandler() function, simply by adding a new JErrorHandler that points to the new error handler static function and then add this handler on top of the stack for that specific JErrorNo number. JErrorHandler is simply a container, so that we can do a link list of it. The JErrorHandler is used, for example, to inform us that the JFile couldn't be opened properly or that a Mutex, RWLock or PThread operation was not successful.

The class TFTP_Timeout scan the double link-list waitingAck_queue from the head and search for an expired timestamp stored inside a TFTP_Packet, if this is the case then a timeout occurs and the packet will be resent if less than three retries were attempts. We detect a timeout by storing the current time added to the timeout value to get a timestamp value. When a clock tick occurs, we update the current time and look for any timestamp lower then or equal to the current time. If such a thing happen, we resent the packet, update the timestamp and enqueue it back into the waitingAck_queue, just in case another timeout occurs.

The class TFTP_DataAck will acknowledge each packet in the queue during the TFTP transfer and will be called every time a valid ACK packet is received with a block number greater than zero.

The file TFTP_header is design to define all states, enumerations and C type structures used in the TFTP program, such as:

The class TFTP_Sender sends data packet created by the Data_Factory until the queue is empty or that the windowsizeLeft reached zero.

The functions in TFTP_Error send a TFTP error packet for the assigned errno values. The following error functions are defined:

• TFTP_NotDefinedErrorHandler

• TFTP_FileNotFoundErrorHandler

• TFTP_AccessViolationErrorHandler

• TFTP_DiskFullErrorHandler

• TFTP_IllegalOpErrorHandler

• TFTP_FileAlreadyExistErrorHandler

• TFTP_NoSuchUserErrorHandler

The last error should not occur because we are not using Mail option.

• Only octet mode is supported choosing another mode generates an error

and the program is aborted.

• All the queues can be as big as memory or virtual memory can offer.

• There is enough memory to hold both queues (double linked-lists).

• File name and path must be less than 160 characters all together.

• Host name must be less than 160 characters all together.
• Opened file name must be less than 160 characters all together.

• The block size must be between 8 and 65464. It must fits in long.
• The window size must be between 1 and 65464. It must fits in long.
• Timeout can be from 1 to 255 seconds. It must fits in long.

• The file size must fit in a size_t variable, normally unsigned long, the char[] version must be less than 100 TB. Due to the usage of atol(), the actual limit is long.

• This project is designed to work on any Posix-Unix compatible platform,

such as Solaris.

• The server receive only one connection request at a time. ( No multiple connection )
• The timer_create function with the option real-time clock is very accurate.
• We scan the waitingAck_queue each few seconds as specified by timeout.

• Packets can arrive on any side: corrupted, not aligned, out of order or meaning less.
• All C functions and system calls should work properly.
• All Posix library functionality used in this program should work properly.

• We accept command line parameter format that start with -, -- or / followed by the abbreviation or the full name of the option, followed by a space or = sign and the value for that option.

• The compiler will create assembly code that will not corrupt the heap or the stack of the program in any given situations. As demonstrated during the demo, we have a char array getting corrupted that has been fixed by copying the data twice back and forth.

• The scheduling done by the Solaris scheduler is fair and followed a Round-Robin approach for every thread including the main program contained in this project.

• RRQ, WRQ, OACK can have null-terminated garbage appended without affecting the operation of the program, it is simply ignored. If an option doesn’t exist, it is ignored by the given host and not included in its response.

• Our program will get enough CPU cycles to operate properly during its execution.
• By using RW Locks, both queues should not get corrupted in any given way.

• PThread concurrency shall be optimal and appropriately distributed by the Solaris scheduler among every thread.

• Every single part of the program that was heavily tested alone should work properly when added to the main TFTP program, without creating weird bugs or corrupting the stack or the heap.

• _free_unlock() and _malloc_unlock() errors should not occur

with the inclusion of the -lmalloc library in the Makefile.

• We are assuming that both parties might be malfunctioning and should send an error

packet, if connected, and display a meaningful message before exiting as long as the

error is covered by the Unix errno values, if not the program execution

is undetermined.

e.g. Bus error, segmentation fault are not covered by JErrorController.

• The default setting are:
• Port number = 2000
• Host name = fbi.ece.concordia.ca
• Block size = 512
• Timeout = 5
• Window size = 1
• Mode = octet

• The program should not erase any file while sending or creating it’s packet from that file in reading binary mode.

• The user of the TFTP program should have the permission set to read or write into the file in order to read or write into that file.

• We tested the UDP Socket with 2 port numbers unidirectional completely. (Not used)

• We also tested the new version of UDP Socket with a single port bi-directional completely. (The current version)

• We tested JErrorController, JErrorHandler and JError completely.

• We tested Posix Thread library completely ( PThreads, Mutex, RW Locks )

• We tested JFile alone and with a crazy bunch of PThreads, Mutex and RW Locks running with huge memory allocation and deletion, in a random fashion, during 30 minutes to discover that no memory faults or deadlock ever occurred !

• We tested argument parser separately.

• We tested ulltostr() and every other parts of argument parser separately.

• We tested parts of the shell and did few scanf() tests.

• We tested our program in many conditions and we had to solve a lot of memory problems, such as bus error, segmentation fault and _free_unlock problems. Most of them disappeared once we included the –lmalloc library.

• We tested DList alone, the search engine and the new added queue features.

• We did every possible test on the DList/Queue engine stepping into it with gdb.

• We debugged this program step by step about 200 times with gdb to solve many bugs that has passed through the grill and still we know that some bugs might still be hiding, even if the program seems to work fine. We currently can say that we mostly know gdb inside out after debugging this TFTP program !

• We tested the program with zero length files, one byte length, 512 bytes length, 1024 bytes length, etc.

• We also tested the program with binary files, executables and ASCII files of different lengths.

The project was a great learning experience for us. We encountered many problems during this project. Segmentation Faults were a naturally incurring error. This off course made the use of gdb essential for solving those errors. The decision to use threads and mutex proved to be wise decision since scheduling was left to the operating system. The use of Mutex caused some problems in the beginning but only because we had not included the proper libraries.

The design of project went through many changes throughout the course of the semester since we used a bottom up approach to the project. But that proved to be a good idea since most of our time was spent on making the low-level code work without flaws.

Working in a team caused some friction at some times but was dealt with like any team project. But the TFTP project proved to be an overall success at the end.