Large Data Transfer over SQL server DB

Transferring a large data set from server to server or even from database to another on the same server or even from table to table in the same database from SQL server is very big problem concerning time, efficiency, and memory and space together.

I was facing this problem in transferring large data set, contains 1.5 million records per table and each table set is a query from source tables. so the output is transferring query results to a tables. It sounds simple but it is not!

when doing it by SQL import wizard it will take one query per time!!!! how could they think i will need only this!
when trying to do it by SSIS this where i went to hell.... although SSIS has some benefits at least outlining the task flow of the work, but transferring large amount of data is much bigger than it.
my server specs are:
Core 2 due intel processor
6 G.B Memory
win 2008 64 bit
60 GB disk space

It sound OK for transferring 1.6 million records from a query, but it took nearly 1 hour to do so.
Sure I added indices and all the tuning it should take, but with no big improvement.

The best solution I found is: "Bulk insert"...
but how could i dump 1.5 million to a text file quickly as  SSIS also fails doing that. the solution to this problem was bcp utility. "bulk copy" how could not Microsoft integrate it into SSIS or in the import wizard. it is like hiding a treasure.

bcp is brilliant in that, minimizing the time into seconds instead of hours in both dump in files and import into table. the only concern about that was to have a good HDD space to dump files on and then deletes it.

It solved my problem. hooray.

Entity Framework

Entity framework is designed to separate the usage of the data layer elements like table names columns types, database type, and some added functionalities like super-set and subset.

It is good in the aspects mentioned above. But as Microsoft says, it is not a OR mapper. So, you cant get the benefits of the OR mapper like caching and database column validation, custom load, foreign keys values, and some like that.

All you have is the designer that you can do many things in it. and you have to edit this file for further subclassing conditions value - it saves the hexadecimal value and you have to change it manually, hope it will be solved at the final release- also, you can manage whether this table can be accessed via stored procedure or by generated query - if you select a stored procedure for one function like delete, you have to supply *sp for the rest update, insert :S - It is not complete now. Hope that it will be fixed in the final release. I made a comparison between Entity framework and .net Tier, and I found this

Entity Framework	.net Tier
one file contains entities and relations	contains many projects and files to represent the entities and its relations	No. of files generated
you can subclass entities based on condition easily from designer	you have to create the subclass file and modify the super-class manually	Super-class and subclass
entity object references not accessible outside domain [web service, WCF]	entity object accessible anywhere	entity object accessibility
queries accessible via linq and e-sql	accessible via defined functions	data retrieved
select query generated from linq and e-sql	select query defined in sp	select query
using the facade generated class for update and delete and insert, no e-sql	using the sp for all the operations	DML functions

This is all as far as I used the Entity framework. it is good but not as good as hibernate.


*sp: stored procedure

Microsoft Bussiness Intelligent solution

Microsoft evolved itself in the industry of "BI" solutions. First edition was attached to SQL Server 2000. it was there for the DTS, Data Transformation Service, and For the OLAP, OnLine Analytical Processing, that make highly definitions of the business requirements. And the Reporting Service solution.


Now, with SQL server 2005. It makes the complete solution. that includes the SSIS, SQL Server Integration Service, it is more complex and good implementation for the ETL solutions, Extract Transform Load. And the SSAS, SQL Server Analysis Serivce, it is the OLAP but with more functionality, and the SSRS, SQL Server Reporting Service, with added features that suits the BI solution not just the Database perspective.

How the things work:

we might raise some questions like, Why all of these? Do I need this solution?

First: the purpose as we mentioned, is to make a business perspective to the data you have in the database. This perspective can be viewed from Top Level Management, Mid Level Management, and Lower level Management.

Second: If you make a solution for some company, person, or anything that deals with huge amount of data, and want to make a decision according to the behavior of the data. Then you need a BI solution.

Now we will say some sort of work flow that every BI solution have it.

First you migrate the data into a database using SSIS, then you need to analyze these data with SSAS, that creates the business definitions, then you need to represent it in something useful to read SSRS, so you are now getting the big picture.

the Next time we will talk the SSIS, features and lakes. and how to solve it.