kat Online Bookmark Manager - Development Guide

By Mitch Stuart
Copyright © 2005 FullSpan Software  -  Usage subject to license
Software Version: 1.0  -  Document Version: $Revision: 1.3 $, $Date: 2005/08/14 02:38:48 $

Introduction

This document is the Development Guide for kat. See the Overview for general information and a list of other available kat documentation.

Architecture Overview

kat uses the following architectural and infrastructure elements:

• Struts web application framework.
• User interface: JSP, JSP-EL, JSTL, Struts JSP tags, custom JSP tags, HTML, CSS, JavaScript.
• Application code layers (described in detail later).
• Hibernate object/relational persistence library.
• MySQL database server.

Framework Separation

• The business logic should be usable from "clients" other than the mainstream web application. Examples of other clients include: JUnit test cases, Console (text mode) applications, GUI (e.g., Swing) applications, and Web services (non-browser-based) interfaces.
• Separation of concerns allows more code reuse, promotes good design, and allows more development scalability (by having separate developers work on each layer in parallel).
• Separation makes later changes in the web app framework simpler. For example, if we decide to change from Struts to Spring, it will be much simpler if Struts-specific code is not sprinkled throughout the application.

Action Separation

Problem: When your execute method is invoked, it is passed several parameters, including the Struts-specific ActionMapping and ActionForm, and the Servlet-specific [Http]ServletRequest and [Http]ServletResponse. If you write your business logic in the Action class, you will be tied to Struts and to the Servlet environment.

Solution: Introduce a Service layer that is not tied to the Struts or Servlet infrastructure. Your Action code becomes a thin layer that does any Struts/Servlet specific things (e.g., parameter management), and then invokes your Service layer. To support the Service approach, you will need to introduce context interfaces for Application, Session, and Request. These interfaces will have one concrete implementation for the Servlet environment, and one for the standalone (non-Servlet) environment.

The Service layer is similar to using stateless session beans in the EJB environment, but without the unnecessary baggage of EJB.

ActionForm Separation

Problem: If you pass the ActionForm to your Service layer, you end up contaminating that layer with Struts and Servlet related dependencies.

Solution A: Use DynaActionForm. DynaActionForm is a subclass of ActionForm that uses a Map to hold property names and values (as opposed to the individual property member variables found in a typical ActionForm). DynaActionForm is a subclass of ActionForm, so it would seem to have the same problem as mentioned above. However, DynaActionForm also implements the DynaBean interface (from Apache Jakarta Commons BeanUtils). So, you can write your Service layer to use DynaBeans, and you will be decoupled from Struts/Servlet dependencies. You will still have a coupling to the Apache Jakarta Commons library, but that is much less of a problem: it does not dictate the environment (Servlet vs. standalone) in which your code runs. In fact, the Commons library offers many nice utilities, so you might want to include it in your project anyway.

Solution B: Use FormData objects. You can create your own "POJOs" (Plain Old Java Objects) that hold the form data; I call these "FormData" classes. Then embed an instance of your FormData class in the ActionForm class (i.e., containment instead of inheritance). Since your FormData class has no dependencies on Struts or Servlets, you can freely pass it down to your Service layer. This does introduce one additional layer of indirection/complexity. For example, you always have to write two classes now (ActionForm and FormData) instead of just one (ActionForm). But I think the tradeoff in separation of concerns is worth it.

For the kat application, I implemented Solution B, FormData objects. At first, Solution A (DynaActionForms) seemed clean to me. But there is one major drawback: the DynaActionForms are not self-documenting. An ActionForm/FormData combination is self-documenting because of the property getter/setters and the JavaDoc that is generated for them (plus whatever additional JavaDoc that you write). For DynaActionForms, there is no automatic knowledge of what properties are in the form: you must write this manually in JavaDoc or some other format.

Both Solutions A and B require an indirection in JSTL expressions:

• For a non-Struts tag, you would normally write (for example):

     <c:out value="${myform.myproperty}" />

  With Solution A you have to write:

     <c:out value="${myform.map.myproperty}" />

  With Solution B you have to write:

     <c:out value="${myform.formdata.myproperty}" />

• For a Struts tag, you would normally write (for example):

     <html:hidden property="${myproperty}" />

  With Solution B you have to write:

     <html:hidden property="${formdata.myproperty}" />

Application Code Layers

The following diagram gives an overview of the layers and packages:

The following table provides more detail on the layers and packages. The columns in the table are:

• Package. The Java package of the classes described in this row of the table. There is a common package prefix that precedes all of the package names listed here: com.fullspan.kat.
• Class. In some cases, specfic classes will be described. In other cases this column will contain "Various" to indicate that the classes will be domain or application specific. This column also may list static resources (such as configuration XML files) that are accessed via the ClassLoader (as opposed to being accessed via the filesystem).
• Comments. The purpose of the package/class is described here.

Package	Class	Comments
app	App	Anchor for application-wide configuration, resource bundles, etc.
	IAppRequest AppRequestBase AppRequestStandalone	Representation of a single request. IAppRequest is the interface. AppRequestBase is an abstract base class. AppRequestStandalone is a concrete class for standalone clients, i.e., it represents requests that are not hosted inside the Servlet container.  Examples of standalone clients are JUnit tests or command line utilities.
	IAppSession AppSessionBase AppSessionStandalone	Representation of an application session. Follows the same pattern as the request classes described above.
	AppRequestStatus	A readonly object (its only public methods are getXXX) that provides request and session status.  For example, it tells whether the user is logged in, their loginid and userid, their role, etc. The AppServletFilter stores an instance of this object as a request attribute so that (e.g.) JSP pages can alter their display based on the request status.  For example, certain links should be displayed only if the user has the Admin role.
web	AppServletContextListener	Does application initialization in the contextInitialized method.
	AppServletFilter	Does access control checking, login / redirect handling, dispatching, error handling, and transaction management.
	AppRequestWeb	A concrete implementation of IAppRequest for Servlet container requests.  Extends AppRequestBase.
	AppSessionWeb	A concrete implementation of IAppSession for Servlet container sessions.  Extends AppSessionBase.
	AppHttpReqAttribKeys AppHttpReqParamKeys AppHttpSessAttribKeys	Constants used to store and retrieve values from various scopes. Not all values stored in these scopes require defined constants.  In particular, the Struts form names, and the properties in the forms (and corresponding HTML form parameters) have their own management and are not included with these constants.
	web-access-control.xml	A mapping between URL regex and the rolemask allowed to access that URL.
web.action	Various	Subclasses of Struts org.apache.struts.action.Action.
web.form	Various	Subclasses of Struts org.apache.struts.action.ActionForm.
web.jsptag	Various	Java code supporting our custom tags.
service	Various	There is one service class for each operation that a client can perform.
	service-access-control.xml	A mapping between service class name regex and the rolemask allowed to access that service.
service.form	Various	This is the non-Struts-specific form layer discussed in ActionForm Separation.
service.validate	Various	Validation utilities available to the service layer.
domain	Various	Domain and application specific objects.  The classes in this package are generally composites or utilities that interact with the domain.entity objects.  For example, for the entry tree listing, there is an Entry object in domain.entity, and an EntryTree object in domain.
domain.entity	Various	Core entities in the application domain.  Typically each of these map to a database table.
db	Various	Objects and utilities for managing database connection and transaction context.
db.dao	Various	The DAO object provides the glue to transfer the domain.entity objects to and from the database.
security	AccessChecker	Loads access control files (e.g., web-access-control.xml and service-access-control.xml). Determines whether a request for a resource is allowed based on the rolemask in the access control file and the current user's role.
resource	messages.properties, others	Message and other resources.
exception	ConcurrentModificationException DataAccessException DuplicateKeyException LoginRequiredException ServiceException plus others as required	Exceptions.

Control Flow Sequence

The following diagram shows the lower-level application logic flow of control sequence. As stated above, this example envisions a user viewing her account preferences and clicking "Save". In this lower-level diagram, we see the application logic to load the user record, apply the user edits, and save the record.

Validation

JavaScript

Having said that, it is essential that JavaScript validation only be used in addition to server-side validation, not in place of it. We always have to protect the server against data that is posted from non-JavaScript-enabled browsers, or from unusual or malicious user agents.

Action

Although the bulk of our validation is not in the Action layer, we have some validation there. There are some types of "hard" errors that need to be trapped in the Action layer and reported gracefully. For example, say there is a web page parameter that determines how the Action behaves, and it needs to be parsed as an integer. If it is not a valid integer, the Action must raise this as an error. Basically, anything that prevents the Action from instantiating or executing a Service, or computing the ActionForward to return from the execute method, must be reported by the Action.

Service

• Data type conversion. Checking if the FormData string properties be converted to the proper domain types (dates, numbers, etc.).
• Property value checking. Checking for missing values, or values outside the allowed range.
• Property interaction checking. Checking for invalid combinations of values across multiple properties.

Authentication

Authorization

Web Authorization

Checking web authorization is done in our ServletFilter, so it is done automatically for every request.

Strictly speaking, from an application point of view, there is not a lot of web authorization required. Requests will be checked at the service layer so there is not a strong requirement to check them at the web layer. However, there are a couple of reasons to do so. First, just the principle of protecting at the earliest possible point. Why allow a normal user to access an admin Action if that user will simply be rejected at the Service layer. Second, web access control allows us to protect static content such as images or HTML files, or in fact to protect entire directories or subsections of the application.

Service Authorization

Checking service authorization is done in the abstract base class ServiceBase. The execute method invokes the checkAccessToService method as part of processing the service request. All concrete service implementations automatically inherit this checking.

Database Authorization

For example, imagine a user editing a user profile. There might be a URL pattern containing editUserProfile.do that allows any standard or admin user to access it. And there might be a service class pattern containing MyAppEditUserProfile that allows any standard or admin user to access it.

But when we actually go to retrieve or store the user record, a standard user should only be able to work with his or her own user record, whereas an admin user should be able to work with any user record.

This can be achieved with a predicate in the SQL statement like this:

   user.id = :identityUserid or :isAdmin = 1

Data Access

Hibernate

kat uses the Hibernate Interceptor to initialize and update standard entity fields, like the create and update timestamps. See: KatEntityInterceptor.

Hibernate automatically updates each object's sequential version number for optimistic locking. When an object is displayed for editing, we store that version number in a hidden field in the HTML form, which is submitted with the data that the user has edited. The version number is checked in each entity's DAO implementation when the object is being updated. For example, KatEntryDao.get when the forUpdate flag is true. If the object has been edited between the time it was retrieved for display and the time it is being saved, a KatConcurrentModificationException is thrown.

MySQL

The table creation and initialization scripts are in src/db.

The kat code needs to do extra application-level value checking because MySQL does not strictly enforce all constraints. For example, if you try to store NULL into a column that doesn't take NULL values, MySQL Server instead stores a default value, such as 0 for numeric columns or an empty string for text columns. See the MySQL constraints documentation for more details.

There are uniqueness constraints on some of the kat database columns. For example, values in the t_user.loginid and t_user.email must be unique. To detect uniqueness violations and display the appropriate error message to the user, the SQLException text is parsed in each entity DAO class. For example, see KatUserDao.getPossibleDuplicateKeyException.

The relationship between parent categories and child entries is modeled in t_entry.parent_catid (and in several other columns, described in detail later). MySQL cascading operations (for example, cascade delete to delete all child entries recursively when the parent category is deleted) cannot be nested more than 15 levels. Therefore, kat does not allow entries to be created any deeper than 15 levels of nesting.

Entry Tree Data Model

 1)   Top
 2)      Computing
 3)         Java
 4)            Sun
 5)               J2EE
 6)               J2SE
 7)               Tutorial
 8)            XML
 9)               JDOM
10)               Xalan
11)               Xerces
12)         Python
13)            Python Cookbook
14)            wxPython
15)      News and Views
16)         Salon
17)         Slate

Tree Model Background

Terminology

• A tree is a node and all its children, recursively. In the above example, the Java tree consists of node numbers 3 through 11, inclusive.
• A shrub is a node and all its direct children. In the above example, the Java shrub consists of node numbers 3, 4, and 8.
• The breadcrumbs for a node are the parents of a node up to and including the root, but not including the node itself. In the above example, the breadcrumbs for the Java entry are node numbers 1 and 2.

Domain Objects
kat entries (categories and items) are stored in the t_entry database table. The KatEntryShrub and KatEntryTree domain objects represent the tree and shrub objects described above. Both KatEntryShrub and KatEntryTree also include the breadcrumbs for the entry, which are displayed in the user interface.

References
A lot has been written about representing trees in relational databases. Here are some references that I consulted:

• Joe Celko, Hierarchical SQL.
• Joe Celko's SQL for Smarties: Advanced SQL Programming. See also: Joe Celko's Trees and Hierarchies in SQL for Smarties.
• robvolk, More Trees and Hierarchies in SQL. See also this followup thread.
• James Crowley, Tree structures in ASP.NET and SQL Server.
• Vadim Tropashko, Trees in SQL: Nested Sets and Materialized Path.
• Hibernate blog posting.

After considering various options, I decided to use the "path notation" approach to represent the tree relationships (this is described in detail below).

Compared to parent/child modeling, path notation seems better because it is simpler and more efficient to answer queries like: get an entry and all its parents; and get an entry and all its children.

Compared to "edge notation", path notation seems easier to understand and maintain.

Tree Model Details

Name	Level	ID	Parent Cat ID
Top	0	8	null
Computing	1	23	8
Java	2	479	23
Sun	3	862	479

If we look at the Sun entry for example, there are several ways to represent its path:

• Variable length path segments with delimiter: 8/23/479/862.
• Fixed length path segments without delimiter: 008023479862.
• Fixed length path segments with delimiter: 008/023/479/862.

kat uses the third option: fixed length path segments with delimiter. Because the path segments are fixed length, the delimiter is not strictly necessary. However, the delimiters make the SQL and other code a bit easier to write, and they make the paths more human readable for troubleshooting.

For entry IDs, we use an integer sequence, starting with 1. We use a fixed-length, 12-digit ID, therefore we have a maximum value of 999,999,999,999. This is one trillion (minus one) IDs, it is enough to handle the scale of the system we are building. If we add 1 million rows a day, it would take 1 million days (over 2,700 years) to max out the sequence. Note that the commas are used here for ease of display, they are not present in the acutal data in the database, so they don't count towards its length.

To store the entry path, we use a varchar(255) column. Limiting the path length to 255 characters increases the portability of our software, since this column length is typically supported and fully indexable by most relational database engines (such as MySQL).

If we take our 12-character ID, and we use a 1-character separator, we have a total of 13 characters. 255 divided by 13 is 19.6, so in theory we can support a maximum of 19 levels of hierarchy.

However, we use referential integrity to maintain the relationships:

• t_entry.owner_userid is a foreign key to t_user.id
• t_entry.parent_catid is a foreign key to t_entry.id

As discussed earlier, there is a limitation in MySQL of 15 levels of nesting for the cascade of referential integrity. Therefore, although our data model is set up to handle 19 levels of nesting in theory, in practice we only support 15 levels.

Example Query

   <query name="GetEntryShrub">
      select shrub
      from KatEntry self, KatEntry shrub, KatUser user
      where user.loginid = :ownerLoginid
      and self.ownerUserid = user.id
      and self.id = :entryId
      and (   (left(shrub.entryPath, length(self.entryPath)) = self.entryPath)
           or (left(self.parentPath, length(shrub.entryPath)) = shrub.entryPath) )
      and shrub.entryLevel &lt;= self.entryLevel + 1
      and ((shrub.ownerUserid = :identityUserid)
           or (:isAdmin = 1)
          or (shrub.computedVisibility = 1))
      order by shrub.parentPath, shrub.name
   </query>

• The parent categories, ordered from highest to lowest position in the hierarchy.
• The entry itself.
• The child entries, ordered by name.

Notice the line:

         and shrub.entryLevel <= self.entryLevel + 1

Known Issues

• The unit test coverage is not thorough enough.
• No performance or scalability testing has been done.
• I wanted to use XHTML for the application pages, but I had to fall back to HTML due to a Struts bug that caused JavaScript failures in validation and focus setting. See the Struts bug report.

License and External Libraries

kat is released under the FullSpan license. kat uses the following FullSpan libraries, which are released under the same license: FSJUtil, JMailSend, and PropKeyConst.

In addition, kat uses several external libraries, which are used and distributed under license from their copyright holders:

Component	Description	License
Apache libraries	kat uses several Apache libraries including: Struts, Commons Codec, and others.	This product includes software developed by the Apache Software Foundation (http://www.apache.org/). apache-license.txt
Hibernate	Object/relational persistence library	hibernate-license.txt
c3p0	Connection pooling library	c3p0-license.txt
JavaMail and JavaBeans Activation Framework	Email and Activation APIs	mail-license.txt, activation-license.txt
JDBC 2.0 Optional Package API (formerly known as the JDBC 2.0 Standard Extension API)	JDBC extensions	jdbc2_0-stdext-license.txt
JDOM	A Java library for XML: "We intend to provide a solution for using XML from Java that is as simple as Java itself."	This product includes software developed by the JDOM Project (http://www.jdom.org/). jdom-license.txt
JTA	Java Transaction API	jta-license.txt
JUnit	Unit testing framework	junit-license.html
log4j	Logging framework	log4j-license.txt
MySQL Connector/J	JDBC driver for MySQL	mysql-connector-java-license.txt Note that Connector/J is distributed under the GPL, and kat is not distributed under the GPL. Normally, per the MySQL licensing terms, this would disallow the kat project from distributing the Connector/J library. However, because kat is distributed under the BSD license, distribution of Connector/J is allowed under MySQL's FLOSS License Exception.
Servlet Library	Servlet and JSP API	servlet-license.txt
Eclipse	Icon image files	This product includes software developed by the Eclipse Project (http://www.eclipse.org/). License text: eclipse-license.html