DataSet Diffgrams To Manage Dynamic Localization in C#

By Robbe D. Morris

Printer Friendly Version

Robbe Morris
Robbe & Melisa Morris
The .NET Framework offers a fairly complete architecture for managing localization via resource files and the classes in the System.Globalization namespace.  The resource files are used to translate static content and the System.Globalization classes are primiarily geared towards number and date formatting of strings based on CultureInfo settings.  The framework, however, has no real mechanism for translating one language to another dynamically at runtime and this article aims to provide at least one alternative.
This past week I received a request that I've tried to avoid addressing for over a year.  Our product managers want to discuss the viability of converting a complex North American research and analysis website tool to a multi-language tool.  Their reasoning is simple: money.  Users from other countries would love to have the tool but their people simply aren't used to working in English.  Like most other apps, we discussed the costs and development when the application was originally created and determined it was just too expensive.  Now that the tool is very successful, they naturally want to make more money by going global.
 


To give you some background on the challenge, here is a very simplistic summation of what the tool does.  The tool essentially asks a user to data enter a rather elaborate set of measurement values both as numbers as well as text concerning their business and it returns risk/reward information about the likely degree of success.  As part of the analysis process, the studies are sent from user to analyst and back to user again to ensure the study results are as accurate as possible.  Naturally, the user may speak Spanish and the analyst may only speak English.  So, virtually everything must be translated at runtime so that each user can effectively use the tool.
Rather than cower to such a daunting endeavor, I opted to research the likely success of trying to implement this level of dynamic localization before saying "It can't be done".  The first item I opted to try and come up with was a way of filtering data both to and from the database access layer rather than try to totally rearchitect the application.  This filter would need to be able to provide easy access to the original value, the translated value, and a means to determine which language was currently held in each column.  While pouring through my ADO.NET books, I remembered the Diffgram capability of the DataSet object.  A Diffgram is used primarily in situations where you want a sort of do/undo capability when editing multiple rows such as working in a grid environment for instance.  It keeps track of original values, new values, and any potential errors that arose on a column by column, row by row basis.
This seemed like a potential match to my language filter requirements.  So, I put together a test database with one table that contained common datatypes and populated it with some junk data.  Then, wired up a test harness console application (classes shown below) to experiment a bit by posting data to Google's language translation form and parse the returned results.  As I had hoped, this little test application utilizing the DataSet as a means to hold data translations worked out relatively well.  So much so that it really opened my eyes to even more challenges to implementing dynamic data translation that I hadn't thought of.  The following is a short summary of what I found:
 
1.Column size constraints
You've sized one of your nvarchar columns as 100 and allow your web application UI to contain 100.  Your English user's type in data that comes close to reaching the max length.  During the translation process, you'll find that other languages often take up more space.  Thus, you haven't allocated enough space in the column to use the translated value much less store it in the database.  You'll find that this sort of conflict with the table schema can be found and dealt with "outside" of the database itself during the filter process and you can create your own custom business rules for what to do when it occurs in different sitations.
2.Flaws in automated translation software
You'll find that English translated to Spanish and then back to English again can return results that are different than the original.  In some cases, this too can generate column size constraint issues.  This occurs most often when the original value contains slang.
3.Language recognition on a column by column basis
There is no standardized way to flag which columns in the DataTable have been translated successfully as well as what their current language is.  I opted to use the rather unorthodoxed method of tagging the column error field for each column on each row with the current language.  I could now prevent the translation process from running on undesired column data types as well as those columns that have already been translated.  This does, however, eliminate the column error capability for use with actual data editing and determining problems.  One field can't handle more than one functionality after all.  You'll also need to remember to call the .RejectChanges() method on the DataTable prior to update if you don't want the translation to be saved.
4.Existing database values
This tool proved useful in executing translation queries against a copy of actual production data.  It gave me a good feel for how many of the current records would be a problem.  I could now quantify to management the likely level of inaccurate translations.  You'll need to decide what level of inaccuracy in the translations is acceptable.  This may or may not be a show stopper...
5.UI considerations
You may want to review your method for capturing dates, numbers, and currency from users.  Look for ways to standardize values to make it easier to avoid translation problems.
 
In the test harness, I only addressed Sql Server data types converted to .NET Framework strings.  It automatically bypasses all other data types.  I think you'll find that the .NET framework's ability to translate numbers and dates is pretty solid and may cause fewer headaches if you perform the translation outside of the filter.  I also put in a few lines of code to write out the current status of the Diffgram to a file as well as the contents of the DataSet to allow you to review the process as you step through the code.  The source code comments will give you more detail.
In the end, I found the use of DataSet Diffgrams to be pretty powerful at isolating database related problems from "outside" the database itself.  It also became apparent that more than one translation engine could be used or more than one filter needed to be available for use.  Thus, I kept the actual translation in a separate method.  If you really wanted to get fancy, you could get translations from multiple sources and hold them in datasets and let the user pick which one is best.  I hope you find this test harness as helpful as I did in researching your own localization implementation.
In the following zip file, I've include the complete C# project and table script in this zip file: download.  You'll need to populate the table with your own test values.  The source code to the two primary classes in the sample are shown below:
 
Class1.cs
 
using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using System.Xml;  
using System.Diagnostics;
using System.IO;
namespace LanguageFilter
{
class Class1
{
[STAThread]
static void Main(string[] args)
{
Class1 oClass = new Class1();
try
{
oClass.Run();
}
catch (Exception e) { Console.WriteLine(e.Message); }
Console.WriteLine("");
}
  #region Run
  public void Run()
  {
    // Test class to translate English to Spanish and back.  I used language
    // identifiers that matched Google (en,es,etc...).  They purposely
    // don't match the .NET framework identifiers.  
    // Remember to adjust the Folder and ConStr variables accordingly.
    LanguageFilter.DataBaseFilter oFilter = new LanguageFilter.DataBaseFilter();
    SqlConnection oConn = new SqlConnection();
    DataSet oDataSet = new DataSet();
    SqlDataAdapter oAdapter;
    string Folder = @"C:\temp\";
    string FileName = Folder + "sample.txt";
    string ConStr="Data Source=(local);User ID=myuser;Password=mypassword;Initial Catalog=mydatabase";
    string Cols = " ID,Description,Message,ProductCode,LastUpdateTime,Amount,Quantity";
    string Sql=" Select top 3 " + Cols + " from sample";
    FileStream fs = new FileStream(Folder + "sample.txt",FileMode.Create,FileAccess.Write);
    StreamWriter w = new StreamWriter(fs);
    try
    {  
      oConn.ConnectionString = ConStr;
      oConn.Open();  
      oAdapter = new SqlDataAdapter(Sql,oConn);
      oAdapter.FillSchema(oDataSet,SchemaType.Mapped,"Sample");
      oAdapter.Fill(oDataSet,"Sample");
      oConn.Close();
      DataTable oTable = oDataSet.Tables[0];
  // Show the contents of the original DataSet diffgram
      oDataSet.WriteXml(Folder + "ds1.xml",XmlWriteMode.DiffGram); 
  // Process table through the filter assuming all columns
  // are in English to start from the database.
      oFilter.FilterDataSet(ref oTable,"en","en","es"); 
  // Show the current state of the DataSet diffgram for review
      oDataSet.WriteXml(Folder + "ds2.xml",XmlWriteMode.DiffGram); 
  // Write the contents of the table out to a file for review.
      this.ShowTable(ref w,"First Run After Translation",FileName,oTable);
 /* this.UpdateTable(ref oTable,"select " + Cols + " from Sample",ConStr); */
  // Run the translated DataSet back through the filter to "undo" the 
  // translation to Spanish".
      oFilter.FilterDataSet(ref oTable,"en","es","en"); 
  // Show the current state of the DataSet diffgram for review
      oDataSet.WriteXml(Folder + "ds3.xml",XmlWriteMode.DiffGram);
  // Append the translation "undo" to our existing file for review
      this.ShowTable(ref w,"Second Run After Translation",FileName,oTable);
    }
    catch (Exception) { throw; }
    finally { if (oConn.State == ConnectionState.Open) { oConn.Close(); } w.Close(); }
   }
   #endregion
 #region Show Table
  public void ShowTable(ref StreamWriter w, string FileDescription, string FileName, DataTable oTable)
  {
    w.WriteLine("**************************************");
    w.WriteLine(FileDescription);
    w.WriteLine("**************************************");
    try
    {
      foreach(DataRow oRow in oTable.Rows)
      {  
        if (oRow.RowState == DataRowState.Modified)
        {
          w.WriteLine("");
          w.WriteLine("Row Modified");
          w.WriteLine("");
        }
        foreach(DataColumn oColumn in oTable.Columns)
        {
          w.Write("   " + oColumn.ColumnName + ": ");
          w.WriteLine(oRow[oColumn.Ordinal].ToString());
          w.WriteLine("");
          if (oRow.GetColumnError(oColumn).Length > 0)
           {
             w.WriteLine("     " + oRow.GetColumnError(oColumn));
           }
           w.WriteLine(""); 
        }
      }
   }
   catch (Exception) { throw; } 
 }
 #endregion
 #region Update Table
 public void UpdateTable(ref DataTable oTable, string Query, string ConStr)
 {
    SqlConnection oConn = new SqlConnection();
    SqlDataAdapter oAdapter;
    SqlCommandBuilder oBuilder;
    try
    {
       oConn.ConnectionString = ConStr;
       oConn.Open();  
       oAdapter = new SqlDataAdapter(Query,oConn);
       oBuilder = new SqlCommandBuilder(oAdapter); 
       oAdapter.Update(oTable);
       oConn.Close();
    }
    catch (Exception) { throw; }
    finally { if (oConn.State == ConnectionState.Open) { oConn.Close(); }}
 }
 #endregion
 }
}
 
DataFilter.cs
 
using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using System.Xml;  
using System.Diagnostics;
using System.Web;
using System.IO;
using System.Net;
using System.Text;
namespace LanguageFilter
{
 public class DataBaseFilter
 {
public DataBaseFilter() { }
 #region Filter DataSet
  public void FilterDataSet(ref DataTable oTable, string StoredLanguage, string SourceLanguage, string TargetLanguage)
  {
    string Value="";
    string Translated="";
    string CurLanguage="";
    try
    { 
      foreach(DataRow oRow in oTable.Rows)
      { 
        foreach(DataColumn oColumn in oTable.Columns)
        {
          Translated = "";
          Value = oRow[oColumn.Ordinal].ToString();
          CurLanguage = oRow.GetColumnError(oColumn).Length;
          // Assume column has not been translated away from StoredLanguage
          // and set a flag indicating the language by default.
          if (CurLanguage.Length < 1) {oRow.SetColumnError(oColumn,StoredLanguage); }
          // If already translated to correct language, skip column.
          if (CurLanguage==TargetLanguage) {continue; }
          // No need to run expensive translation process if no value exists.
          if (Value.Length < 1) { continue; }
          // Let .NET Framework handle numeric, datetime, and currency
          // formatting via the Locale or CultureInfo classes outside
          // of this filter.  Also skip other non-string oriented data types.
          if(oColumn.DataType.ToString() != "System.String") {continue; }
          Translated = this.Translate(Value,SourceLanguage,TargetLanguage);     
          // If Translated is not populated, it will be empty.  We only
          // want to populate the column with a translated result and
          // set the current language used in this column.
          // This is where you may need to add more restrictions to ensure
          // that the translated value doesn't violate column constraints
          // other than size which is handled here.
          if ((Translated.Length >0) && (Translated.Length <= oColumn.MaxLength))
          {
            oRow[oColumn.Ordinal] = Translated;
            oRow.SetColumnError(oColumn,TargetLanguage);
          }
        }
      }
    }
catch (Exception) { throw; }
  }
 #endregion
 #region Translate 
  private string Translate(string Value, string SourceLanguage,string TargetLanguage)
  {
    // This method is used for demo purposes only.  It posts form values to
    // Google's translation form and parses the return values.  In a production
    // environment, you'd want a local piece of software to handle the language
    // translation for you.
       string StartText = "<textarea name=q rows=5 cols=45 wrap=PHYSICAL>";
       string StartText2 = "<textarea name=q rows=5 cols=45 wrap=PHYSICAL readonly>";
        string EndText = "</textarea>";
       string Url="http://translate.google.com/translate_t";
        string Form="";
        string H="";
        string Translated="";
try
{
  Value = Value.Trim();
   Form= "langpair=" + SourceLanguage + "|" + TargetLanguage;
  Form += "&text;=" + Value.Replace(" ","+");    
   WebRequest oRequest = WebRequest.Create(Url);
  oRequest.ContentType = "application/x-www-form-urlencoded";
  oRequest.Method = "POST";
  StreamWriter oWriter = new StreamWriter(oRequest.GetRequestStream());
   oWriter.Write(Form);
  oWriter.Close();
  WebResponse oResponse = oRequest.GetResponse();
  H = new StreamReader(oResponse.GetResponseStream(),Encoding.ASCII).ReadToEnd();
  oResponse.Close();
  int Start = H.IndexOf(StartText);
  if (Start <1) { Start = H.IndexOf(StartText2); }
  int End = H.IndexOf(EndText);
          if ((Start>0) && (End>0))
          {
            H = H.Substring(Start,End - Start);
           H = H.Replace(StartText,"");
            Translated = H.Replace(StartText2,"");
           Console.WriteLine("Translated: " + Translated);
          }
}
catch (Exception) { throw; }
    return Translated.Trim();
  }
 #endregion
 }
}
 

Robbe has been a Microsoft MVP in C# since 2004.  He is also the co-founder of NullSkull.com which provides .NET articles, book reviews, software reviews, and software download and purchase advice.