Speed up Linq to Sql with compiled queries

August 15, 2008

There is no doubt that tehre are some drawbacks to Linq to Sql. One of them is that the Sql statement is built dynamically so it is needed to be parsed and compiled each time you run it. Fortunately .Net 3.5 has a solution for this problem. System.Data.Linq namespace includes a class named CompiledQuery which is responsible for caching the compiled version of a Linq to Sql query. This class has a static method called Compile which takes a Func<T,S,R> delegate. In this signature, T is the type of a DataContext (i.e. HRMDataContext) , S is the type of a predicate to filter the query and R is the type of returned result. Needless to say that it must be IQueryable<T>.

In this article we will see how to pre-compile a query, its limitations and how it really improves the speed of a Linq query.

To pre-compile a query we must define a public static field of type Func<T,S,R> . What we assign to this field is the result of CompiledQuery.Compile method:

public static Func<testDataContext , SearchCriteria, IQueryable<Person>> FilteredResult …

In the above line, testDataContex is the type of a DataContext inside the project, SearchCriteria is type of a class or struct that is designed for passing search criteria to .Compile method. For example, suppose that in testDataContext, we have a Table named Person. We have also defined a class (or struct) named SearchCriteria as bellow:

public class SearchCriteria
{
public int id { set; get; }
public string FirstName { set; get; }
public string LastName { set; get; }
}

Now to get these definitions to work with a precompiled query we can write such a statement:

public static Func<testDataContext , SearchCriteria, IQueryable<Person>> FilteredResult =
System.Data.Linq.CompiledQuery.Compile(
(testDataContext dc , SearchCriteria criteria ) =>
from p in dc.Persons
where (p.id == criteria.id || criteria.id == -1)
&& (p.FirstName == criteria.FirstName || criteria.FirstName == string.Empty)
&& (p.LastName == criteria.LastName || criteria.LastName == string.Empty)
select p
);

That’s it. At this point, FilteredResult contains a pre-compiled query and can be used this way:

testDataContext dc = new testDataContext();
SearchCriteria criteria = new SearchCriteria();
criteria.id = -1;
criteria.FirstName = “Bill”;
criteria.LastName = “Gates”;
List<Person> p = FilteredResult(dc, criteria).ToList();

The above code creates instances of testDataContext (dc) and SearchCriteria (criteria) and passes them to FilteredResult as arguments. The result of FilteredResult is IQueryable<Person> we have called .ToList() extension method to get a List<Person> series.

One upsetting point about pre-compiled queries is that you can not use a stored-procedure to make a compiled query. In the above Linq to Sql code, if you write “from C in usp_GetPerson() …” you will get an error indicating that stored procedures are not allowed to be used.

Now let’s see how much precompilation can be helpful. I have written a small Console application that runs two version (one is compiled and one is not) of a query over a database for 1000 times. The time needed to run each query is as follows:

Compiled query takes 0 minutes, 1 seconds and 62 milliseconds.

Regular query takes 0 minutes, 13 seconds and 328 milliseconds.

As it is clear, the compiled query is greatly faster than a regular query. Notice that in a Linq model, nothing will really happen unless we iterate over the result of the query. Therefore, I have written a foreach statement to iterate over the result of queries. I also have written a small query at the beginning of the program to make Linq manager open a connection to Sql Server. If we do not do this, the compiled query will surprisingly takes longer!

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data.Linq;

namespace CompiledQuery
{
class Program
{
public static Func<testDataContext , SearchCriteria, IQueryable<Person>> FilteredResult =
System.Data.Linq.CompiledQuery.Compile(
(testDataContext dc , SearchCriteria criteria ) =>
from p in dc.Persons
where (p.id == criteria.id || criteria.id == -1)
&& (p.FirstName == criteria.FirstName || criteria.FirstName == string.Empty)
&& (p.LastName == criteria.LastName || criteria.LastName == string.Empty)
select p
);

static void Main(string[] args)
{
testDataContext dc = new testDataContext();
SearchCriteria criteria = new SearchCriteria();
IQueryable<Person> Q = null;

// The following code makes Linq manager to open a connection to Sql Server
var init = from p in dc.Persons select p;
foreach (Person person in init) ;

criteria.id = -1;
criteria.FirstName = “Bill”;
criteria.LastName = “Gates”;
DateTime BeginTime = DateTime.Now;

for (int i = 0; i < 1000; i++)
{
Q = FilteredResult(dc, criteria);
foreach (Person person in Q) ;
}

DateTime EndTime = DateTime.Now;
TimeSpan Diff1 = EndTime - BeginTime;

BeginTime = DateTime.Now;

for (int i = 0; i < 1000; i++)
{
Q = from p in dc.Persons
where (p.id == criteria.id || criteria.id == -1)
&& (p.FirstName == criteria.FirstName || criteria.FirstName == string.Empty)
&& (p.LastName == criteria.LastName || criteria.LastName == string.Empty)
select p;
foreach (Person person in Q) ;
}

EndTime = DateTime.Now;
TimeSpan Diff2 = EndTime - BeginTime;

Console.WriteLine(”Compiled query takes : {0}:{1}:{2}”, Diff1.Minutes, Diff1.Seconds, Diff1.Milliseconds);
Console.WriteLine(”Regular query takes {0}:{1}:{2}”, Diff2.Minutes, Diff2.Seconds, Diff2.Milliseconds);

Console.ReadKey();
}
}
}

You can download the source of of a full sample project from here.

A 3-Tier Architecture with LINQ TO SQL

July 28, 2008

Recently, I posted a 5-part article about developing a 3-tier architecture using ADO.NET. In this post, I am going to mention how to develop such an architecture using Linq to Sql. Since the last article was too long, I am going to make this one short. Therefore, the architecture is not changed and still includes four layers: Common, Data Access, Business and Presentation. Moreover, since the business and presentation layers do not change that much, I am going to focus on the development of Common and DAL layers.

The nature of Linq to Sql seems to be designed for 2-tier programs, especially when we use Visual Studio to visually create entities and DataContexts. However, we can seperate the definition of entities and the data access layer following the methods bellow.

First of all, inside Visual Studio 2008, create a new blank solution and then add a Class Library project to it, named Linq3TierCommon. Right-click on Linq3TierCommon, choose “Add new item” and then add a new “Linq to Sql classes” item to your project. Then design your entities visually or even drag-drop any table, view or stored procedure you like on to this .dbml surface (if the surface is not seeing, in Solution Explorer, righ-click on the .dbml file and choose “View Designer” ). The following image is an example:

Actually, you do not have to use .DBML (Linq to Sql classes item) and you can define your required entities by writing code or using SqlMetal tool. However, it is needless to say that using Visual Studio makes everything much easier.

Anyway, up to now we have created entities and the relationships, but the problem is that an automatically generated DataContext will let you access to database thorough the common layer. To avoid this, set the “Access” property of your DataContext class to “Internal”:

After completing the common layer, add a new class library to the solution and name it Linq3TierDAL. To shorten this post I have put the base DAL class and the concrete DAL classes in one class library, but for a real application, I strongly recommend to put them in different assemblies. However, add a class to this library and name it Linq3TierDALBase.cs

This class is the base DAL class and contains common DAL functionalities. First, we define a protecred property of type DataContext for CRUD actions:

private DataContext _innerDataContext = null;
protected DataContext innerDataContext
{
get
{
string ConnectionString = ConfigurationManager.ConnectionStrings["main"].ToString();
if (_innerDataContext == null)
_innerDataContext = new DataContext(ConnectionString);
return _innerDataContext;
}
}

Then, we will add a generic method named GetTables in order to return a Table<T> collection. This collection will be used by concrete data access classes to perform Linq query on them.

protected Table<T> GetTables<T>() where T:class
{
return innerDataContext.GetTable<T>();
}

You may also add a method to call the ExecuteCommand method of DataContext class in order to execute stored procedures.

we also add a method named Save to save all entity changes:

public virtual void Save()
{
innerDataContext.SubmitChanges();
}

Now we will add a new class, named UserDAL.cs which is inherited from Linq3TierDALBase.cs. This class may have several methods like, FetchAll, FetchByPK, Save and so on:

Notice that these methods are not constant and you may define as many as methods you like.

Afterwards, add a new class library project to this solution and call it Linq3TierBusiness, containing a class named UserBiz.cs . This class may have several business methods, like FetchAll, Save and so on. UserBiz is responsible for validations, concurrency control and so on. The following code is a very simple example of what a business class can be:

public class USERBiz
{
private USERDAL innerDAL = new USERDAL();
public IQueryable<USER> FetchAll()
{
return innerDAL.FetchAll();
}
public void Save()
{

// perform validations…
innerDAL.Save();
}
}

PLEASE NOTICE that you must call .SAVE method of a DAL object from which you have retrieved your entities. Otherwise, LINQ to Sql manager assumes that you have made no change and it will save nothing. The good thing about linq is that you may ever use Linq To Objects inside your business classes, withough interfering the data access layer.

For the presentation layer, I have written a small Console application, which retrieves all users, find a user whose login name is “admin” and changes it’s password:

class Program
{
static void Main(string[] args)
{
USERBiz biz = new USERBiz();
IQueryable<USER> Users = biz.FetchAll();

USER admin = Users.Single(x => x.LOGINNAME == “admin”);

if (admin.PASSWORD != “newpass”)
{
admin.PASSWORD = “newpass”;
biz.Save();
}

foreach (USER item in Users)
Console.WriteLine(”Login Name: {0} , Password={1}”, item.LOGINNAME, item.PASSWORD);

Console.ReadKey();
}
}

That’s it. Please notice that this so-called architecture needs to be improved a lot and what is mentioned here is just to give an idea.

You can download a full source code from here. The source code contains a SQL 2005 database backup, used to write the sample program of this post. you may use it or use your own database.

P.S. The password of the attached sample file is aspguy.wordpress.com

Implementing a 3-Tier architecture with C# - Part 5

July 22, 2008

Hi back,

In this post we will finalize the implementation of our 3-tier asp.net architecture, by developing a Web-based presentation layer.

So far, we have developed the data access and business layers. The implementation of UI, consists of methods and approaches for handling business and crud operations thorough a Web UI. In fact, there are several ways to present business objects and reflect the changes back to data base. Like:

  1. Binding controls (i.e. GridView) directly to DataTables coming from business classes (i.e. by FetchAll method)
  2. Binding controls (.e. GridView) to business classes using an IDataSource, like ObjectDataSouce control.
  3. Mixing up the above methods.

If you opt the first option, you need to handle sorting and paging manually. It means that you have to write code for PageIndexChanged and Sorted methods. This might take several minutes for you to handle every details and make your development process too long and not productive.

A better approach is using ObjectDataSource. For using it, we have to prepare our business classes first. ObjectDataSoruce works with classes that DataObject attribute has been applied to them. Thus, open your business class (i.e. PersonBiz.cs), and apply DataObject attribute on it:

[DataObject]

public class PersonBiz
{ …

Then we might specify four methods for doing Select/Insert/Update/Delete operations. However, I personally prefer to use Select method only and do the other operations manually. My reason is that, if I declare a method for Insert/Update methods, I will have to use GridView’s (or similar controls) editing features and have to pass a lot of params to the Insert/Update methods. and I do not like it! :-)

Anyway, to mark a method as a Select method, we have to apply DataObjectMethod attribute on it. This attribute takes a parameter indicating which kind of method is it. For example, a select method can be declared like this:
[DataObjectMethod(DataObjectMethodType.Select)]
public DataTable FetchAll(…

If you are intended to use ObjectDataSource, your Select method must have a return value of type IENumerable, and preferably, it would better return a DataTable or DataView.Thus, FetchAll method returns a DataTable. It also has two parameters as follows:

[DataObjectMethod(DataObjectMethodType.Select)]
public DataTable FetchAll(int startRowIndex, int maximumRows)
{
PersonEntity entity = new PersonEntity();
innerDAL.Fill(entity, startRowIndex, maximumRows);
return entity;
}

As you see, the startRowIndex and maximumRows are used for paging. If you do not wish to manage paging, you may omit these two arguments.

To bind your UI control (i.e. GridView) to this method, put an ObjectDataSource on your web form. Then, choose Configure DataSource from it’s smart tag. The Configure Data Source window will open:

Check the “Show only data components” and open the drop down list beside it. If nothing is seen, you have to build the BusinessLayer.Dll and put it in your website’s Bin folder.

Select your business class and click Next. At the next step, you have to specify which methods are used for Select, Insert, Update and Delete operations. The following image is an example :

After choosing all your methods, click Finish to close this dialog. If you are going to let the ObjectDataSource handle the paging operation, go to your page’s source code, find the code of GridView control and remove the SelectParameters. This is necessary because startRowIndex and maximumRows will be provided by ObjectDataSource control. If you do not remove them, you will get an error message.

In order to activate paging in ObejctDataSource, you have to set its EnablePaging property to True. you also have to set the AllowPaging property of your GridView control to true. In order for ObjectDataSource to do paging operation correctly, we have to tell it how many records are in database. Therefore, there must be a public method in our business class that return the total number of rows in DB. What we need is to assign it’s name to SelectCountMethod property of ObjectDataSource. You do not need to flag this property with an attribute:

public int GetTotalRecordCount()
{
return innerDAL.GetTotalRecordCount();
}

Notice that if you are filtering the results of your SelectMethod, for example by passing a filter string to it, you have to perform the same filter on the result of SelectCountMethod.

Now build and run your website. you will see that the data is being shown and the grid supports sorting and paging very well. To ensure that paging is working correctly, run Microsoft Sql Server Profiler, and check to see if the Select SQL statement has correct startRowIndex and maximumRowCount values.

To insert a new record, you may create a new entity (i.e. PersonEntity) , fill it and save it to DB.

To update a record, retrieve the record from database using it’s Primary Key, edit it and save it back to db.

To delete a record, perform a delete operation on database using the records PK.

But how to obtain the record’s Primary Key? You may save it somewhere in UI when the UI control is getting bound. To do this, I usually write such a code in RowDataBound event :

if (e.Row.RowType == DataControlRowType.DataRow)
{
int Serial = DataBinder.Eval(e.Row.DataItem, “Serial”);
e.Row.Cells[0].Attributes.Add(”Serial”, Serial.ToString());
}

In the above code, if the row being bound is a data row (not footer or header), we get the value of “Serial” column and store it in the 1st cell of the current row. Notice that whatever toy add yo .Attributes collection will be available to the rendered html page. Thus, never save passwords or other sort of crucial data with this method.

Now, for each row, you have the Primary Key value. Therefore, we may perform update and delete operations easily. Not keeping the whole data in memory (or in ViewState), increases the access to data base, but reduces the usage of server’s resources. However, it is up to you and your project’s conditions to decide that method should be used.

You may download a full example of this implementation HERE. Please note that the file is password protected and it’s password is : aspguy.wordpress.com

Bye for now..

Implementing a 3-Tier architecture with C# - Part 4

July 18, 2008

Up to now, we have finished implementing the data access layer. However, we can reduce the dependency between the deriver DAL classes and Web.config file. As you remember, we have specified the DB provider name in Web.config file so that DAL classes must have access to this file to read the provider name. Another method to specify the provider type is using custom attributes.

Firstly, in the DataAccessLayerBase namespace, we declare a public enum type named ProviderType:

public enum ProviderType
{
SqlServer,
Odbc,
OleDb
}

We also remove the ProviderName property from DALHelper class, and instead, add a new internal method, as bellow, to get a string representation of each ProviderType member:

internal static string GetProviderTypeName(ProviderType providerType)
{
switch (providerType)
{
case ProviderType.SqlServer: return “System.Data.SqlClient”;
case ProviderType.Odbc: return “System.Data.Odbc”;
case ProviderType.OleDb: return “System.Data.OleDb”;
default: return “System.Data.SqlClient”;
}
}

The custom attribute we need for specifying the provider type, must have a public property of type ProviderType. We also add a constructor, with a positional parameter of type ProviderType, to our custom attribute class:

[AttributeUsage(AttributeTargets.Class)]
public sealed class DbProviderTypeAttribute : Attribute
{
public ProviderType PrType
{
set;
get;
}
}

The AttributeUsage attribute indicates that this custom attribute can be applied to classes only.

To make this attribute effective, we need to alter the base DAL class’s constructor as bellow:

first we must check to see if the DbProviderTypeAttribute attribute has been applied to the derived DAL class:

if (!this.GetType().IsDefined(typeof(DbProviderTypeAttribute), false))
throw new System.Exception(”DbProviderTypeAttribute must be applied to DAL class”);

The above code examines the existence of DbProviderTypeAttribute and throws an exception of it does not exist.

Then, the following piece of code, extracts the instance of DbProviderTypeAttribute from the class’s metadata, and uses the value of it’s PrType property to set the ProviderName :

List<object> t = this.GetType().GetCustomAttributes(typeof(DbProviderTypeAttribute), false).ToList();
DbProviderTypeAttribute Provider = t[0] as DbProviderTypeAttribute;
ProviderName = DALHelper.GetProviderTypeName(Provider.PrType);
_ProviderFactory = DbProviderFactories.GetFactory(ProviderName);

That’s it. This way we have reduced the coupling between our UI (web.config file) and the DAL class. When developing a derived DAL class, we have to apply DbProviderType attribute to it:

[DbProviderType(PrType=ProviderType.SqlServer)]
public class PersonDAL : DALBase
{ …

Implementing the business classes

The implementation of business classes are fairly easy. A business class is a bridge between the presentation layer and the data access layer, perform business checks, controls concurrencies and even might control the business transactions.

The business classes can be either static or instance classes. Each business method can create one or more instances of the required DAL classes. However, since each DAL class has it’s own connection object (innerConnection property), system transactions would better be controlled in business methods. This may even solve the issue that is raised when a layered architecture needs to control both business and system transactions.

The easiest way to control the transactions, is using System.Transaction name space and TransactionScope class. The illustration of TransactionScope requires an indipendent post, but for now, lets say that transaction begins when an instance of TransactionScope class is created and it is commited when the .Complete() method is called:

using (TransactionScope scope = new TransactionScope())

{

…..

scope.Complete();

}

if an exception is thrown, the transaction will roll back :

using (TransactionScope scope = new TransactionScope())

{

PersonDAL PDAL= new PersonDAL();

….

ChildDAL CDAL= new ChildDAL();

PDAL.Update(personDT);

CDAL.Update(childDT);

scope.Complete();

}

Note that MSDTC service must be running on the server machine.

so far, the definition of business classes requires no additional settings. however, when developing the presentaion layer, we will come back and make some minor changes to make our bisuness classes compatible with UI components.

A sample business class might look like this:

public class PersonBiz
{

private PersonDAL innerDAL = new PersonDAL();
public DataTable FetchAll(int startRowIndex, int maximumRows)
{
PersonEntity entity = new PersonEntity();
innerDAL.Fill(entity, startRowIndex, maximumRows);
return entity;
}

public void Update(System.Data.DataTable table)
{
// validate fields/perform business checks and throw an exception if a criteria is not met
innerDAL.Update(table);
}
:

:

}

Implementing a 3-Tier architecture with C# - Part 3

July 15, 2008

The Fill method we have developed so far, executes the Select command on the underlaying server and picks up RowCount records from startRowIndex position. For example, if Select * from Person returnS 20 rows, and startRowIndex is 0, only the first RowCount records will be fetched. This means that the complete result set will be generated on server and will be fetched into an IDataReader, but ADO.NET transmits only RowCount number of records into the DataTable. If the results are too many, that will not be helpful that much, so we have to do something about it.

For doing this, we may define the Fill method virtual and each DAL class may override it to fetch records based on startRowIndex and RowCount from the data base. The 2nd way is to assume that every SelectCommand has two paramerers names startRowIndex and RowCount! This means that we always must supply two int parameters to Fill method. I propose you to have several overloaded Fill methods to use a proper one in different occasions.

Anyway, if we suppose that every SelectCommand executes a stored procedure having two int parameters, we might change the Fill method (or develop an overloaded one) this way:

public void Fill(DataTable table, int startRowIndex, int RowCount)
{
DbParameter p;
innerAdapter.SelectCommand.Parameters.Clear();

p = ProviderFactory.CreateParameter();
p.ParameterName = “StartIndex”;
p.Value = startRowIndex;
innerAdapter.SelectCommand.Parameters.Add(p);

p = ProviderFactory.CreateParameter();
p.ParameterName = “RowCount”;
p.Value = RowCount;
innerAdapter.SelectCommand.Parameters.Add(p);
if (table.Rows.Count > 0)
table.Rows.Clear();
innerAdapter.Fill(table);
}

This Fill method adds two parameters to SelectCommand property of the innerAdapter, fills the DataTable by calling the innerAdapter.Fill method. Notice that we must clear up the DataTable because the innerAdapter.Fill method will append new records to the DataTable if any records exists in it.

But how to implement the paging? The answer might differ fro different data bases. However, I will provide a simple method for SQL Server 2005. SQL Server 2005 provides Ranking funtions, i.e. Row_Number() funtion, so that we may easily rank the resulting records of each table. For more information about Row_number function, please check out the SQL Server 2005 Books Online or MSDN.

Anyway, for our sample database, I have written a stored procedure named FetchAllPerson which has two int parameters, called @StartIndex and @RowCount:

create procedure FetchAllPerson
@StartIndex int,
@RowCount int
As

with NoFilterRecords
as
(
Select Row_Number() over (order by Serial) as RIndex , Serial, FirstName, LastName from Person
)
Select * from NoFilterRecords where RIndex>=@StartIndex and RIndex <@StartIndex+@RowCount
order by 1
Go

This procedure first ranks all records, then fetch only those records positioned in the specific range. Therefore, the innerAdapter.SelectCommand.CommandType must be set to CommandType .StoredProcedure .

innerAdapter.SelectCommand.CommandText = “FetchAllPerson”;
innerAdapter.SelectCommand.CommandType = System.Data.CommandType.StoredProcedure;

Working with identity fields

The value of Identity fields is generated automatically. Therefore, a new row in a DataTable, may not have a value for its Identity fields or their value may not be identical to the mapping record in database after the record is inserted. Thus, we have to reflect the new value of Identity fields to the related row inside the DataTable. For doing this, add a Select statement to the end of Insert statements and seperate them with a semi colon :

innerAdapter.InsertCommand.CommandText = “Insert into Person (Serial, FirstName, LastName) \n” +
“Values (@Serial, @FirstName, @LastName); \n” +
“Select * from Person where Serial = SCOPE_IDENTITY()”;

The Select statement will fetch the new record from database and reflect any changes to the inserted row. To make this method work, we must set the UpdatedRowSource property to FirstReturnedRecord:

innerAdapter.InsertCommand.UpdatedRowSource = System.Data.UpdateRowSource.FirstReturnedRecord;

This may be done in the base DAL class. This setting will fetch one record and will synchronize the inserted row with the data base. The SCOPE_IDENTITY() funtion returns the last identity value assigned to a field. This works in SQL SERVER only and you must perform a similar task for other data bases.

Ok this is all about the Data Access Layer. We will discuss about the business layer on the next post.

Implementing a 3-Tier architecture with C# - Part 2

July 14, 2008

Hi again,

Let’s keep on going and work on the Data Access Layer. The DAL layer we are going to design can be used with, theoretically, any sort of RDBMS because it will take advantage of ADO.NET generic classes. For this reason, I have written a base class for DAL classes and have placed in into a seperate .dll. The reason is that the base DAL class is used for any database and is not being changed if the underlaying RDBMS is changed. But the DAL class must be replaced in order to change the target database. This will reduce coupling and will let you inherit your own DAL classes and make minor customizations only.

The base DAL class will do nearly everything and the derived DAL classes are used to customize CRUD commands only.

The following class diagram shows the relation between the base and derived DAL classes:

The DalBase class is using System.Data.Common namespace and it’s factory methods in order to create generic objects for different kind of databases. ADO.NET 2 introduces a class named DbProviderFactory, which is an implementation of Factory pattern, to produce a variety kind of database objects, like connection, command, adapter and so on. So far, it can create the following sort of objects:

  1. using System.Data.SqlClient
  2. using System.Data.Odbc
  3. using System.Data.OleDb

Most of the common ADO.NET classes are inherited from their bases classes defined in System.Data.Common namespace. For example. SqlConnection is derived from DbConenction, SqlDataAdapter is derived fom DbDataAdapter and so on.

To create a specific series of objects, we must get an instance of DbProviderFactory class, using DbProviderFactories.GetFactory method. The following code does this to obtain a factory for System.Data.SqlClient provider:

DbProviderFactory _ProviderFactory = DbProviderFactories.GetFactory(”System.Data.SqlClient”);

To make it more flexible, I will put the name of the desired provider in Web.Config file :

<appSettings>
<add key=”Provider” value=”System.Data.SqlClient”/>

</appSettings>

Access to Web.Config file will be through a helper class, called DalHelper. This class will provide us the connection string, provider name and so on :

public static class DALHelper
{
public static string ProviderName
{
get
{
return WebConfigurationManager.AppSettings["Provider"];
}
}
}

Let’s take a look at DalBase class. DalBase has three protected properties:

  1. innerAdapter : a protected and read only DbDataAdapter that the derived classes will use to customize it’s Insert/Udpate/Select/Delete commands.
  2. innerConnection: a protected and read only DbConnection that will be in available to the derived classes and will be used in case a derived class needs to perform a particular CRUD operation.
  3. ProviderFactory: a protected and read only DbProviderFactory that will help derived classes create a specific DbConnection, DbCommand, DbDataAdapter, …

DalBase also has a parameterless constructor which will be discussed shortly. It also provides the following methods:

  1. Fill : it fills a DataSet or DataTable using innerAdapter.
  2. Update : it reflects the updates made to a DataSet/DataTable to the underlaying database using innerAdapter.
  3. Initialize : This method is protected and is used to customze the Select/Update/Insert/Delete command of innerAdapter.
  4. GetTotalRecordCount : for the time being, forget about this method. We will come back and discuss about it later.

The DALBase() parameterless constructor creates Command, Adapter and Connection objects according to the provider name, specified in Web.Config file and finally calls Initialize method:

public DALBase()
{
ProviderName = DALHelper.ProviderName;
_ProviderFactory = DbProviderFactories.GetFactory(ProviderName);

_innerConnection = _ProviderFactory.CreateConnection();

_innerAdapter = _ProviderFactory.CreateDataAdapter();

_innerAdapter.SelectCommand = _ProviderFactory.CreateCommand();
_innerAdapter.UpdateCommand = _ProviderFactory.CreateCommand();
_innerAdapter.InsertCommand = _ProviderFactory.CreateCommand();
_innerAdapter.DeleteCommand = _ProviderFactory.CreateCommand();

_innerConnection.ConnectionString = DALHelper.ConnectionString;
_innerAdapter.SelectCommand.Connection = _innerConnection;
_innerAdapter.UpdateCommand.Connection = _innerConnection;
_innerAdapter.InsertCommand.Connection = _innerConnection;
_innerAdapter.DeleteCommand.Connection = _innerConnection;

Initialize();
}

Therefore, by specifying a different provider name in Web.Config file, you will have a different set of ADO.NET objects.

The fill method has three parameters:

  1. a DataTable (or DataSet) to be filled with innerDataAdapter.
  2. an int parameter called startRowIndex. This is used for paging.
  3. ab int parameter called RowCount. This will be used for paging too

If one or two of the two int parameters is -1, the paging will not be performed.

if (startRowIndex == -1 && RowCount == -1)
_innerAdapter.Fill(table);
else
_innerAdapter.Fill(startRowIndex, RowCount, table);

TO BE CONTINUED…

Implementing a 3-Tier architecture with C# - Part 1

July 12, 2008

Hi again,

Nearly two years a go I probed Internet to find out a good reference architecture for my ASP.NET applications. What I meant GOOD ARCHITECTURE was an architecture that:

  1. IS FAIRLY SIMPLE
  2. IS BASED ON N-TIED ARCHITECTURE
  3. IS PRODUCTIVE
  4. IS EASY TO IMPLEMENT
  5. DOES NOT NEED A NY 3rd PARTY CLASSES (I.E. NHIBERNATE)
  6. CAN BE USED FOR A WIDE RANGE OF DATABASE SERVERS
  7. IS WELL-DESIGNED AND DOES NOT CONTRADICT WITH OOD DICIPLINES

I tried out a variety of architectures I’d come across but I found them somehow faulty. For example, in some cases, typed datasets (datatable) was used with TableAdapters attached to them! This contradicts the rules of a N-Tier application because an Entity would carry out it’s CRUD functions, that must be placed in DAL layer, with itself anywhere (i.e. BIZ layer).

I finally designed a shining architecture (!) that is going to be shared and discussed here. Since a weblog is too small for discussing about all details, I will write about the main parts of the implementation and about main ideas. Therefore we may forget about the details and set our mind to the significant things. I invite you to leave comments and let’s grow this architecture and rectify it’s problems.

The implementation that I am going to discuss is simple and is based on a (somehow) standard 3-tier architecture. In this architecture, four layers play a part:

  1. Common Layer : includes entities and is common between other three layers
  2. Data Access Layer (DAL): includes CRUD operations
  3. Business Layer (BIZ) : does business operations, check business rules, manage concurrencies,…
  4. Presentation Layer (UI)

The presentation layer can be either Web-based or Windows-based. However, since my focus on this article is on the implementation, I will stick to a Web UI.

As the above image shows, each layer is dependent on the common layer and the underneath layer. For example, Biz layer can access the common and DAL layer but can not access UI layer. A layer is not necessarily a physical component. In the other hand, you may define a class for each layer and put all of them inside a single application (i.e. EXE). However, in order to reduce the coupling and increasing the cohesion, we will create a .DLL for each layer and separate them psychically.

The Common Layer

As mentioned before, the Common layer comprises all Entity definitions. An Entity definition is a class describing an entity. An entity is an object that will transmit information between layers and is, nearly, a representation of actual data in DB. Entities can be of any type, however, I strongly recommend to use Typed DataSets (DataTables) because you can work with them visually and ASP.NET UI components are astonishingly match with them.

In order to create the Common layer, first start a solution and name it ThreeTierArch. Afterwards, add a Class Library project to the solution and call it CommonLayer. This will, by default, produce an assembly named CommonLayer.dll and a namespace called CommonLayer. In Solution Explorer window, right-click the CommonLayer project, choose “Add new item” and add a DataSet to the project. An empty DataSet is added and you may drag a DataTable from Toolbox window inside it. DataSet designer let’s you design your DataSet visually and create any relationship needed between tables.

The above image displays a sample DataSet with two DataTables having a Master/Detail relationship.

That’s it. Add as many as needed data tables and datasets to your project. These are our Entities. Something that I usually do, to shorten my entity names, is to inherit a class from their class. For example, I might add a class like the following one to CommonLayer in order to shorten my entity name:

public class PersonEntity : PersonDataSet.PersonEntityDataTable
{
}

Needless to say that this is not necessary and it is just to get rid of long names!

Ok, it is enough for the common layer. Tomorrow, I will talk about the DAL layer and we will see how to implement it.

cheers :)

Reducing the page size by storing ViewState on Server

July 9, 2008

ASP.NET uses ViewState in order to maintain the value of controls during post-backs. ViewState, by default, is stored in client side. Infact, objects that are ViewState-Enabled and their state is going to be saved, are serialized and stored in a hidden field. This works fine until the size of serialized objects is not too large. If the size of objects is too large, the size of the page goes up and it means more data transfer during a round-trip. In order to reduce the page size we may use either one or two of the following methods:

  1. Compressing the page content
  2. Storing ViewState on server

In this article I am going to provide a short description on how to store ViewState on a server location. In ASP.NET 1.1 ViewState was just save inside the page but, in ASP.NET 2.0, ViewState is saved by a descendant of PageStatePersister class. This class is an abstract class for saving and loading ViewsState and there are two implemented descendants of this class in .Net Framework, named HiddenFieldPageStatePersister and SessionPageStatePersister. By default HiddenFieldPageStatePersister is used to save/load ViewState information, but we can easily get the SessionPageStatePersister to work and save ViewState in Session object. The only thing to do this is to override PageStatePersister property of Page class and ask it to return an instance of SessionPageStatePersister class:

protected override PageStatePersister PageStatePersister
{
get
{
return new SessionPageStatePersister(Page);
}
}

Let’s create a small website and see what happens if we make this change. In the attached sample website, I have created a class named Person, having two string properties named First Name and LastName. Then a list of 10,000 Person object is created, added to ViewState, and displayed with a GridView control:

if (!IsPostBack)
{
List<Person> PersonList = new List<Person>();
for (int i = 0; i < 1000; i++)
{
PersonList.Add(new Person { FirstName = “Aref”, LastName = “Karimi” });
}
ViewState.Add(”PersonList”, PersonList);
GridView1.DataSource = PersonList;
GridView1.DataBind();
}

The size of Default.aspx page in this state is 165,952 bytes.

If I override PageStatePersister property of the page, with the code bellow, the size of the page will be 80,152 bytes, nearly half of the previous page size!

protected override PageStatePersister PageStatePersister
{
get
{
return new SessionPageStatePersister(Page);
}
}

Please note that Session State is, by default, stored in-process and consumes memory. Therefore, this approach is good for the websites with few or medium visitors. If the size of Session object goes beyond the server memory, your application will crash.

A solution for this problem is to persist Session State on Sql Server. This is possible only if, all the objects stored in Session are serialize able. For example, if for any reason, a SqlConnection is stored in Session object, you can not persist it in SQL Server . Furthermore, in many cases, we want to persist ViewState only and not the complete Session content.

A better solution is to write our custom persister class by deriving a new class from PageStatePersister class. In the new persister class we may store/retrieve the ViewState in any medium, like a file or Sql Server. PageStatePersister is an abstract class with two abstract methods named Save and Load. Save, stores ViewState and Load, retrieves it.

Before saving and loading the session object, we must notice that the state of each individual page must be identical. This can be reached by concatenating the SessionID and the name of the page. Moreover, a mechanism to remove all the information saved for a session, is required when a session is about to closed.

PageStatePersister provides any required information for saving and loading the ViewState. All we have to do is to Serialize and DeSerialize ViewState by PageStatePersister.StateFormatter property:

IStateFormatter formatter = this.StateFormatter;

StateFormatter, can Deserialize a serialized ViewState object and return a Pair of objects for ViewState and ControlState:

public override void Load()
{
IStateFormatter formatter = this.StateFormatter;
string VSContent = LoadFromDB(Page);
Pair statePair = (Pair)formatter.Deserialize(VSContent);
ViewState = statePair.First;
ControlState = statePair.Second;
}

In the above code, LoadFromDB is a custom method that should retrieve the serialized ViewState from database. I will not go thorough the implementation of this method because it is not necessary.

public override void Save()
{
Pair statePair = new Pair(ViewState, ControlState);
IStateFormatter formatter = this.StateFormatter;
string VSContent = formatter.Serialize(statePair);
SaveToDB(Page, VSContent);
}

The mechanism for Save method, is somehow the same. We just need to create a Pair of ViewState and ControlState properties, Serialize it and save it to the desired medium. The code above is a sample Save method.

After writing our custom PagePersister class, we need to override the Page.PageStatePersister property to return an instance of this class.

You may download the sample website from HERE.

Thanks,

Asp guy

A practical use of Serialization and Extension methods in C# 3.0

July 3, 2008

Recently, reviewing a colleague’s design diagrams, I noticed that in order to keep the history of changes of a particular field, he had created a small table in SQL Server! For example, in a contract management program, we are asked to keep track of changes in Due Date and Contract Payments. A primitive way for doing this, as my friend had done, is to save changes in a table that has a SmallDateTime field name DueDate. A better way that I proposed him is to design a table like this:

Field Name

Type

ID

Bigint, Identity(1,1)

Key

Varchar(10)

XMLBody

NVarchar(8000)

In the above table, ID is the Primary key, Key is a string field used to group up objects and XMlBody is a text field (Char, Varchar, Text…) used to keep the history of changes! In order to save an object in this table, the only job for us is to serialize it! For example, we can serialize the objects to XML format (my preference).

How to serialize an object?

To serialize an object, it must be serialize able. Only instances of a class that has been marked with [Serializable] attribute are surely serialize able but, due to my experience, many simple objects can be serialized without this attribute!

For example suppose we are going to keep track of changes in a contract’s due date. The first step is to define a class like the following:

private class ContractChange

{

public DateTime DueDate { get; set; }

public DateTime ChangeDate { get; set; }

}

This class has not been marked with Serializable attribute but is serialized into XML properly. To serialize and desterilize objects we might create a generic class named SerializationHelper<T> :

private static class SerializationHelper<T> where T: class

{

public static string SerializeObject(T obj)

{

string XmlString = String.Empty;

try

{

MemoryStream MemStream = new MemoryStream();

XmlSerializer Serializer = new XmlSerializer(obj.GetType());

XmlTextWriter XmlText = new XmlTextWriter(MemStream, Encoding.Default);

Serializer.Serialize(XmlText, obj);

XmlString = Encoding.Default.GetString(MemStream.ToArray());

MemStream.Close();

}

catch

{

throw;

}

return XmlString;

}

public static T DeserializeObject(string xmlbody)

{

byte[] Buffer = Encoding.Default.GetBytes(xmlbody);

MemoryStream input = new MemoryStream(Buffer);

XmlTextWriter XMlWriter = new XmlTextWriter(input, Encoding.Default);

XmlSerializer Serialzier = new XmlSerializer(typeof(T));

return (T)Serialzier.Deserialize(input) ;

}

}

The above two methods are private because they are going to be used internally. The class will have the following methods in order to save/load objects:

public static class SerializationHelper<T> where T: class

{

private static string SerializeObject(T obj) …

private static T DeserializeObject(string xmlbody)…

public static void Save(T obj, string key)

{

string XmlBody = SerializeObject(obj);

// CODE FOR SAVING XMLBODY INTO DATABASE GOES HERE

}

public static List<T> Load(string Key)

{

// retreive all records from DB regarding to Key parameter.

// for each record, call DeserializeObject

return null; // replace null with deserialized object.

}

public static List<T> LoadByFilter(string Key, Func<T, bool> predicate)

{

return Load(Key).Where(predicate).ToList();

}

public static void DeleteKey(string Key)

{

// A code to delete all records whose Key field value is equal to

Key param.

}

}

Now we can save/load any type of information with this class and with a simple table. In an advanced condition, you might design a hierarchical structure for better organization of stored objects.

Anyway, the following code is a crude example of how to use this approach:

static void Main(string[] args)

{

ContractChange C = new ContractChange { ChangeDate = DateTime.Now ,

DueDate= DateTime.Parse(”2000/10/11″)

};

// storing

SerializationHelper<ContractChange>.Save(C, “DueDate”);

// retrieving

List<ContractChange> changes = SerializationHelper<ContractChange>.Load(”DueDate”);

}

As shown in the above code, we might save/load any kind of information only by defining a new class for it.

Using extension methods to extend this approach…

C# 3.0 introduces extension methods. Using extension methods we might add new methods to a class without deriving new classes. For example if a class can calculate sum of a series of numbers, we might add a method to it in order to calculate the average of them:

public class extended

{

public int sum()

{

return 7+3+2;

}

}

public static class extending

{

public static float average(this extended extnd)

{

return extnd.sum() / 3;

}

}

As you see, the class Extending is adding a method named average to class Extended. To get the average, you call average method, as it belongs to extended class:

extended ex = new extended();

Console.WriteLine(ex.average());

A complete description of how extended methods work is out of the scope of this article. Anyway, we may use this new C# feature to add Serialize methods to any object:

public static class SerializeExtender

{

public static string Serialize (this object obj)

{

string XmlString = String.Empty;

MemoryStream MemStream = new MemoryStream();

XmlSerializer Serializer = new XmlSerializer(obj.GetType());

XmlTextWriter XmlText = new XmlTextWriter(MemStream, Encoding.Default);

Serializer.Serialize(XmlText, obj);

XmlString = Encoding.Default.GetString(MemStream.ToArray());

MemStream.Close();

return XmlString;

}

}

To examine this code we may write a sample program like the one bellow:

class Program

{

static void Main(string[] args)

{

ContractChange C = new ContractChange { ChangeDate = DateTime.Now ,

DueDate= DateTime.Parse(”2000/10/11″)

};

Console.WriteLine(C.Serialize ());

Console.ReadKey();

}

}

Unfortunately extension methods can not be generic or can not belong to a generic class. Therefore, I  attached Serialize method to Object class but needless to say that not all classes are serialize able! Thus, calling this method on a non-serializable object will cause a run-time error.

Exporting a DataTable to Excel

June 26, 2008

In many cases Microsoft Excel can do much for you and reduce your programming efforts. For example, if you must provide a printable version of a data collection or you need to provide a report with grouping, you may send all information to Excel and let the end-user do it with Excel. Therefore, knowing how to export a data collection (i.e. a DataTable) to Microsoft Excel can be helpful and sometimes reliving!

In ASP.NET, you may export a GridView control to excel but I do not think it is a good way (except in certain cases) because we mostly do not display the whole columns and rows in a grid (i.e. for paging reason). Plus, exporting a GridView control needs to re-render the grid into response output stream and it might be time/resource consuming.

In the rest of this article we will write a handy function to export a DataTable to Microsoft Excel. The features of this function are as follows:

1. Is very quick

2. Is very lightweight

3. You may choose which columns to be exported

4. You may specify your desired encoding

Ok let’s get started. The method we are going to write is named ExportToSpreadsheet and have the following signature:

public static void ExportToSpreadsheet(DataTable table, string name, string ColumnList)

The 1st argument is the DataTable object we are going to export. The 2nd one is the name of generated target file and ColumnList is a comma-separated list of columns to be exported. Not always do we export all columns. Thus we should be able to choose which columns to be exported.

The file we are going to generate is a Comma Separated Value (.csv) file because Excel knows this file and opens it like a spread-sheet. In the other hand, we do not have to create a real .xls file.

Ok, for the implementation part we first convert the ColumnList argument to an IList collection. This is necessary because we need to find a column name in the list later.

IList<string> collist = new List<string>();

string[] columnstr = ColumnList.ToUpper().Split(’,');

foreach (string s in columnstr)

collist.Add(s);

The above code splits the ColumnList argument into a string array and adds each string item to an IList object. I had to use foreach statement because the code was written in C# 2.0. In C# 3.0 you may omit Foreach and use .ToList() extension instead. I mean the above code will get shorter in this way:

IList<string> collist = ColumnList.ToUpper().Split(’,').ToList();

Since we are sending the .CSV file to HttpResponse, we create a new HttpContext instance, and set the ContentEncoding property. We would better set ContentEncoding since the encoding used in clients might be different from the response encoding! For example suppose you have a website whose response encoding is UTF-8 but your clients use Windows-1256 encoding. If you export the .CSV file in UTF-8 format clients will not show the strings correctly. In this code I have used Encoding.Default but you may let the the encoding to be chosen as a new argument for ExportToSpreadsheet method.

context.Response.ContentEncoding = Encoding.Default;

ontext.Response.Clear();

Afterwards, we start to loop over the DataTable.Columns collection to create the titles. For column titles I have used DataColumn.Caption property (and not DataColumn.ColumnName) since the title is not necessarily equal to the column name. For instance a column name might be Cust_Title but the caption should be “Customer’s Full Name”. I recommend to use typed datasets because setting such properties are very easy in VS.:

foreach (DataColumn column in table.Columns)

{

if (ColumnList==”" || collist.Contains(column.ColumnName.ToUpper()))

context.Response.Write(column.Caption + “;”);

}

After creating column titles, we loop over rows and add them to the .csv :

foreach (DataRow row in table.Rows)

{

foreach(DataColumn column in table.Columns)

{

if (ColumnList == “” || collist.Contains(column.ColumnName.ToUpper()))

context.Response.Write(row[column].ToString().Replace(”;”, string.Empty) + “;”);

}

context.Response.Write(Environment.NewLine);

}

At the end, we alter the http header in following way to get the download process started:

context.Response.ContentType = “text/csv”;

context.Response.AppendHeader(”Content-Disposition”, “attachment; filename=” + name);

context.Response.End();

That’s it. You may download a .PDF version of this article HERE.