Mental Paging Space

RealTime and Rails

2011-08-28T01:12:00.000-07:00

Sometimes you have a need to process things in real-time. Stock trading apps for example,
or real-time video. EventMachine, in Ruby is great. On top of this you want a gui or a
web page, so Rails naturally fits in. The problem is trying to do Real-Time and Rails in
the same sentence is not quite there. In a application I've done, I had exactly that,
I receive multiple HD cctv feeds in UDP, and handled the whole RTSP protocol using eventmachine.
The problem was every 5 minutes I wanted to save the current file information into
the database. I tried a few methods, but the most reliable turned out to use a em_http request.
This allows you to do async http in eventmachine. I of course targeted my rails stack that
was already running the application.

So this way, my realtime work can continue (async), and the rails stack gets a tickle
every 5 minutes to stay alert. Result: No more dropped packets.

EventMachine.run {

      http = EventMachine::HttpRequest.new('http://gshift-app/').get :query => {'keyname' => 'value'}



      http.errback { p 'Uh oh'; EM.stop }

      http.callback {

        p http.response_header.status

        p http.response_header

        p http.response



        EventMachine.stop

      }

    }

Saving My Bacon when things go Wrong - Veeam Backup In Action

2011-03-24T23:14:00.000-07:00

Sometimes you click something inoculus, and the world ends.
The button was not red, and it didnt have a safety cover.
Yet it did something really bad, and totally unexpected.

Had that happen to me this morning. Shutdown a file server,
and the datastore was gone. I understand why, as I was moving data
off the datastore, but I didnt expect it to go at that point.

What to do Now !?!?!

We had updated to Veeam 5.0 Vmware backup few months ago.
I decided to give it a try, as I had a backup from just hours ago
and no data had been updated.

I traditional tape, a restore is a multi-day affair with logs of downtime
and other unplesant activity.

With Veeam I still expected a 5-8 hour downtime as veeam did the restore.

Instead I decided to use the Instant Recover Mode. From start to having the
server running again was 2 minutes counting boot time. The Restore only took a minute and a half. At that point the server was running over a
virtulized nfs server automatically attached to the blade of my choice,
automatically added to my vcenter, and automatically started.

For the non-believers I've included a screen shot just to prove the point.
The server in question has been running now most of the day without issue
and its 60% migrated back to production storage using Storage vMotion.

This is really unbeliveable how well and easy it worked.

Parsing dates and times in english free form

2011-03-16T17:46:00.000-07:00

I've been looking for this a while. I want to be able to take
something like "Every tuesday, starting on the 5th of april" and
convert it to computer readable form. Now in Ruby, you there is a
library to do it.

Its a ruby gem/add-on called Nickel by Natural Inputs.

Nickel is an API that extracts date, time, and message information from naturally worded text.
Why you should use it.

Simplify any form with date and time inputs
Increase your website's usability
Handles recurring date and time information

In my case I wanted to create calendar entries via a email message.

This way the subject line alone can give me alot of information.

http://naturalinputs.com/
https://github.com/lzell/nickel

A Example of _BeforeDataInitialize for Microsoft Lightswitch Collections

2011-02-27T00:01:00.000-08:00

When you have a Lightswitch edit screen, its not clear from the doc what
you need to do to init data. You have to create a instance of the item,
and put it into the collections selected item field.
Like So.

partial void PurchaseRequests_BeforeDataInitialize()
{
// Write your code here.
PurchaseRequest theitem = new Portal.PurchaseRequest();
theitem.PrState = "Unsubmitted";
theitem.Requestor = Application.Current.User.Name;
theitem.Subtotal = 0;
theitem.Total = 0;
theitem.CurrentcyType = "SGD";
theitem.DateCreated = DateTime.Now;
theitem.DateUpdated = DateTime.Now;
PurchaseRequestCollection.SelectedItem = theitem;
}
}

End of Nokia

2011-02-11T01:57:00.000-08:00

Sometimes I really wonder if people understand the market. While generally I agree in most areas Microsoft does a great job, mobile devices as in phones is not one of them. Windows phone 7,has died to a large degree, even close partners are not happy by the market reality of windows 7. The market share for Windows mobile/windows CE continues to fall, with the iphone and android winning fast.

Actually palms offering/hp is far stronger than windows 7, and its the "third" choice as it is for the consumer. Android phones are cheap, and getting cheaper, with a wealth of applications, and reasonably quick software updates.

Nokia hardware and design is not bad, the problem is one of a company being in the business for too long a time, too many models, to many different os's
To the point that change management is just broken. (Try doing application development in Nokia space, its a nightmare of screen sizes and versions)

So the choice Nokia had were:
1. There new Linux based in-house effort, to long in coming, that its dead
2. Use android/google and have a close partnership with the other giant.
3. Use Microsoft - The underdog.

I can understand Nokia does not want to compete with HTC, but the reality they are.
HTC produces both Windows as well as android models. The android ones are incredible popular. So Nokia thinks they can make windows mobile better? After how many versions? Please, dont waste your money. Even on my industrial environment devices (Bar code scanners) I'd prefer to have android, it would be easier to develop.

I really feel nokia is missing the boat. Them joining the Android universe would be a smart move, it would put them in the game. Even offering ports to the existing hardware platforms(phones) would allow them quick entry into the market, without big retootling on the factory side. Newer versions of android really dont need big skinning jobs, so very quick to see a result.

I love nokia hardware, till iphones came, every phone I bought was a Nokia. Solid
hardware, cost effective, great ability to customize the products. The iphone changed
the game. Android changed it again. With android you can compete with iphone, with windows 7 dont bother to try.

Attention Nokia Board, u need to wake up. I agree that the "house" is on fire, and you have to take action, but dont use gasoline to put out the fire. Windows Mobile/Windows CE has not delivered, and really dont see it delivering. It had years, and never even touch much of your symbian market share. Look at what android has done?

I hope I'm wrong and your going to do both android and windows, and windows came first. If so. They you have a chance. But dont make windows mobile your final eulogy.

New widgets for rails 3

2010-12-07T23:49:00.000-08:00

I'm used activescaffold in countless rails projects. Rails 3, resets the clock a bit.
With each major revision of the framework, I always re-eval my toolkit to see what is a better or more up to date fix.

While I'm happy with the work I've done combining rails and silverlight, and I've also tried some Silverlight WCF RIA projects, a pure rails project has some appeal.

Looks like there is another choice. Instead of using activescaffold, look at NetZke.
It leverages ExtJs, and gives very nice gui, with better look and feel than activescaffold plus widgets. Overall, I'd say the base site for a business app in Rails 3 plus NetZke will look so much better than AS plus Widgets Menu I've been using a while for backend ops.

Of course if you want smooth, go silverlight for the Gui with REST interface.

http://netzke.org/

Returning to Roots - Comparison of Ruby and C-Sharp

2010-10-19T17:13:00.000-07:00

Ran into this while doing my morning reading.
This is a great video that compares C-Sharp and Ruby.
As I'm currently doing both side by side, this is really great review
of the plus and minus of each.

The presenter is great, and there lots of code.

http://vimeo.com/12803005

Limiting Data by User In Microsoft LightSwitch

2010-10-13T03:17:00.000-07:00

I wanted to be able to limit my queries by user. In order to do this you need to add additional query code. This can only be done, if you create the query under the datasource. So for example, say I have a table, called
appointments, and the table also has a relationship to users. Right Click on Appointments, and Add A Query. If you try to do this under the screen, you will not be able to add the code.

Once you have the query, and rename it, you can click the Edit Additional Query Code button, on Properties.

The gorgous part of this, you can have a complex query, in my case there are three groups or'd togeather, and then on top of that we limit by user.

And it only takes a couple of lines.

 partial void MyTodaysAppointments_PreprocessQuery(ref IQueryable<Appointment> query)
  
     {
  
       query = from Appointments in query
  
           where Appointments.User.Name == Application.Current.User.Name
  
           select Appointments;
  
     }

Minimal Telnet Parsing in C#

2010-10-06T19:52:00.000-07:00

Ever wonder what the minimal parsing is needed to do a telnet server?
While the socket/listen is easy stuff, and life for low session counts
is easy using a thread per session, the issue is, what do I need to do
to get to the point I can talk to my device. In this case a scangun
with a telnet client.

Telnet actually is a protocol. The PC Micro link is a good write-up
which wikipedia was so nice to provide a link to.

I google for a while, and could not find anything other than expensive .net libraries, or basic protocol documentation.

Here is the code, so you dont have to search any longer.

private void HandleClientComm(object client)
        {
            TcpClient tcpClient = (TcpClient)client;
            NetworkStream clientStream = tcpClient.GetStream();

            byte[] message = new byte[4096];
            byte[] cmd = new byte[4096];
            int cmdcnt;
            int bpos;
            int bytesRead;
           
            const int tmode_normal = 0;
            const int tmode_iac = 1;
            const int tmode_option = 2;
            const int tmode_do = 3;
            const int tmode_will = 4;
            int mode;
            byte thebyte;
            BarCodeClient bci;
            
            cmdcnt = 0;
            bpos = 0;
            mode = tmode_normal;

           bci = new BarCodeClient(client);
       
            while (true)
            {
                bytesRead = 0;

                try
                {
                    //blocks until a client sends a message
                    bytesRead = clientStream.Read(message, 0, 4096);
                }
                catch
                {
                    //a socket error has occured
                    break;
                }

                if (bytesRead == 0)
                {
                    //the client has disconnected from the server
                    break;
                }
               
                bpos = 0;
                while( (cmdcnt < 4096) && (bpos < bytesRead)){
                    thebyte = message[bpos++];
                    switch (mode)
                    {
                        case tmode_normal:
                            switch (thebyte)
                            {
                                case telnet_iac:
                                    mode = tmode_iac;
                                    continue;
                                case 0x0a: // LineFeed
                                    break;
                                case 0x0d:
                                    string thecmd = Encoding.ASCII.GetString(cmd,0,cmdcnt);
                                    if (thecmd.Length > 0)
                                    {
                                        bci.Cmd(thecmd);
                                    }
                                    cmdcnt = 0;
                                    continue;
                                default:
                                    break;
                            }
                            cmd[cmdcnt++] = thebyte; 
                            break;
                        case tmode_iac:
                            switch (thebyte)
                            {
                                case telnet_do:
                                     mode = tmode_do;
                                     break;
                                case telnet_se:  // End Subnegotiation
                                     mode = tmode_normal;
                                     bci.Cmd(""); // Let the lower level know to force a screen refresh
                                     break;
                                case telnet_sb:
                                    mode = tmode_option;
                                    break;
                                case telnet_will:
                                    mode = tmode_will;
                                    break;
                                default:
                                    mode = tmode_normal;
                                    break;
                            }
                            break;
                        case tmode_do:
                            switch (thebyte)
                            {
                                default:
                                    mode = tmode_normal;
                                    break;
                            }
                            break;
                        case tmode_will:
                            switch (thebyte)
                             {
                                 default:
                                     mode = tmode_normal;
                                     break;
                             }
                            break;
                        case tmode_option:
                            switch (thebyte)
                            {
                                default:
                                    mode = tmode_normal;
                                    break;
                            }
                            break;
                        }
                    }
                
            }

            tcpClient.Close();
        }

Creating a debian iscsi shared raid for backup of vmware esx systems

2010-10-05T02:35:00.000-07:00

I usually keep a backup image on one of my test boxes for the production
systems, so I can have a easy way of recoverying in event of disaster.
This previous was a openfiler but found it managed to lose its disk.
So I decided to setup a equivlent system using debian. So using
the minimal network install, I setup a base debian, then added software raid, and finally iscsi target support. Note this is all running as a VM on top of esxi, with raw disks mapped to local sata drives.

First, map your raw disk using vmkfstools

vmkfstools -r /vmfs/devices/disks/vml.0100000000202020202020202020202020395653334e595335535433313530 disk1.vmdk

Next you need to go into the vm, and add the raw vmdk into the vm's config

Now in debian, you will see the raw devices at boot

Next install mdadm

apt-get install mdadm

and you need to install the iscsi target kernel modules

apt-get install iscsitarget-modules-2.6.26-2-686

Now setup the raid

mdadm --create /dev/md0 --level=5 --raid-devices=5 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf

To check the status:
mdadm --detail /dev/md0

Now add the disk into the /etc/ietd.conf

Change the following:

#Assuming you want some security, a chap password on incoming user is useful
IncomingUser sinadmin xxyyzz



Target iqn.2001-04.com.rutledge:storage.disk0.sys1.md0
        # Fixup the security  
         IncomingUser sinadmin xxyyzz
        # Block devices, regular files, LVM, and RAID can be offered
        # to the initiators as a block device.
        Lun 0 Path=/dev/md0,Type=fileio

And finally:
invoke-rc.d iscsitarget restart

The get everything using our new config.

Surprisingly I found this no worse than using openfiler, and I know what is happening
behind the covers.

A Debuggable Windows Service in Csharp

2010-09-30T01:35:00.000-07:00

This is a great example of a windows service. I use it in a app that reads email from pop, and updates a database/webapp written in SilverLight for the FrontEnd.

One note, this uses a Timer Approach to initiate the check. I found unless I put a lock around the worker class,I would see the timer fire again, and I get get a breakpoint in another "thread", while debugging the first. The lock fixed the issue. But take note if you using this or something similar.

 using System;
  
 using System.Collections;
  
 using System.ComponentModel;
  
 using System.Configuration.Install;
  
 using System.Reflection;
  
 using System.ServiceProcess;
  
 using System.Threading;
  
 using System.Management;
  
 using Microsoft.Practices.EnterpriseLibrary.Logging;
  
 namespace WhereAmIEmail
  
 {
  
   public class WhereAmIEmail : ServiceBase
  
   {
  
     private Worker myworker;
  
     private Timer serviceTimer;
  
     private Container components = null;
  
     public WhereAmIEmail()
  
     {      
  
       InitializeComponent();     
  
     }
  
     // The main entry point for the process
  
     private static void Main(string[] args)
  
     {
  
       string opt = null;
  
       // check for arguments
  
       if (args.Length > 0)
  
       {
  
         opt = args[0];
  
         if (opt != null && opt.ToLower() == "/install")
  
         {
  
           TransactedInstaller ti = new TransactedInstaller();
  
           ProjectInstaller pi = new ProjectInstaller();
  
           ti.Installers.Add(pi);
  
           String path = String.Format("/assemblypath={0}",
  
                         Assembly.GetExecutingAssembly().Location);
  
           String[] cmdline = {path};
  
           InstallContext ctx = new InstallContext("", cmdline);
  
           ti.Context = ctx;
  
           ti.Install(new Hashtable());
  
         }
  
         else if (opt != null && opt.ToLower() == "/uninstall")
  
         {
  
           TransactedInstaller ti = new TransactedInstaller();
  
           ProjectInstaller mi = new ProjectInstaller();
  
           ti.Installers.Add(mi);
  
           String path = String.Format("/assemblypath={0}",
  
                         Assembly.GetExecutingAssembly().Location);
  
           String[] cmdline = {path};
  
           InstallContext ctx = new InstallContext("", cmdline);
  
           ti.Context = ctx;
  
           ti.Uninstall(null);
  
         }
  
       }
  
       if (opt == null) // e.g. ,nothing on the command line
  
       {
  
 #if ( ! DEBUG )
  
         ServiceBase[] ServicesToRun;
  
         ServicesToRun = new ServiceBase[] {new WhereAmIEmail()};
  
         ServiceBase.Run(ServicesToRun);
  
 #else
  
         // debug code: allows the process to run as a non-service
  
         // will kick off the service start point, but never kill it
  
         // shut down the debugger to exit
  
         WhereAmIEmail service = new WhereAmIEmail();
  
         service.OnStart(null);
  
         Thread.Sleep(Timeout.Infinite);
  
 #endif 
  
       }
  
     }
  
     /// <summary> 
  
     /// Required method for Designer support - do not modify 
  
     /// the contents of this method with the code editor.
  
     /// </summary>
  
     private void InitializeComponent()
  
     {
  
       components = new Container();
  
       this.ServiceName = "WhereAmIEmail";
  
     }
  
     /// <summary>
  
     /// Clean up any resources being used.
  
     /// </summary>
  
     protected override void Dispose(bool disposing)
  
     {
  
       if (disposing)
  
       {
  
         if (components != null)
  
         {
  
           components.Dispose();
  
         }
  
       }
  
       base.Dispose(disposing);
  
     }
  
     /// <summary>
  
     /// Set things in motion so your service can do its work.
  
     /// </summary>
  
     protected override void OnStart(string[] args)
  
     {
  
       myworker = new Worker();
  
       TimerCallback timerDelegate = new TimerCallback(myworker.DoWork);
  
       serviceTimer = new Timer(timerDelegate, null, 10000, 10000);
  
     }
  
     /// <summary>
  
     /// Stop this service.
  
     /// </summary>
  
     protected override void OnStop()
  
     {
  
       // test.Stop() ;
  
     }
  
   }
  
 }

Hacking LightSwitch User Database Entry using Csharp .net and LinqToSQL

2010-09-30T01:28:00.000-07:00

I want to be able to dynamically create Users in my Microsoft Ligthswitch applications. One of the issues is they use a spearate database.

Since my app accepts email messges, and creates records for the users, it rather important they exist in the system.

So the background Windows service uses LinqToSql to update the tables.

Each user in Lightswitch is in the aspnet_table, there is a record for each application sharing the same database. So if your creating a user, you also need to know which application guid you need to connect the user too.

  public int CreateUser(string UserName)
  
   {
  
     User myUser = new User();
  
     myUser.Name = UserName;
  
     db.Users.InsertOnSubmit(myUser);
  
     db.SubmitChanges();
  
     return (myUser.Id);
  
   }
  
   public Guid CreateAspUser(string UserName)
  
   {
  
     aspnet_User myUser = new aspnet_User();
  
     myUser.ApplicationId = MyAppId;
  
     myUser.UserId = Guid.NewGuid();
  
     myUser.UserName = UserName;
  
     myUser.LoweredUserName = UserName.ToLower();
  
     myUser.IsAnonymous = false;
  
     myUser.LastActivityDate = System.DateTime.Now;
  
     db.aspnet_Users.InsertOnSubmit(myUser);
  
     db.SubmitChanges();
  
     return (myUser.UserId);
  
   }
  
   public Guid GetMyAppId(string appName)
  
   {
  
     var query = from aApp in db.aspnet_Applications where aApp.ApplicationName == appName select aApp;
  
     return (query.First().ApplicationId);
  
   }
  
   public Guid GetUserID(String UserName)
  
   {
  
     var query = from aUser in db.aspnet_Users where aUser.UserName == UserName where aUser.ApplicationId == MyAppId select aUser;
  
     if (query.Count<aspnet_User>() == 0)
  
     {
  
       return (Guid.Empty);
  
     }
  
     return (query.First().UserId);
  
   }
  
   public int GetPortalUserID(String UserName)
  
   {
  
     var query = from aUser in db.Users where aUser.Name == UserName select aUser;
  
     if (query.Count<User>() == 0)
  
     {
  
       return (0);
  
     }
  
     return (query.First().Id);
  
   }

So then it just a matter of calling the code:

 Guid theUserID = GetUserID(UserName);
  
          if (theUserID == Guid.Empty)
  
          { // No Such User
  
            theUserID = CreateAspUser(UserName);
  
          }
  
          int thePortalID = GetPortalUserID(UserName);
  
          if (thePortalID == 0)
  
          {
  
            // Need to create local user
  
            thePortalID = CreateUser(UserName);
  
          }
  
          CreateAppointment(theUserID, thePortalID, Email.Subject, theMessage);

ActiveDirectory and Csharp - Finding the domain user via email

2010-09-30T01:09:00.000-07:00

Sometimes, how many examples, and none seem to work.

I wanted to lookup and verify a user exists via email, in a process that reads email via pop, and then creates records in Silverlight/Lightswitch application.

So how do it?

I've included the code below. Not that a normal user is fine for the username and password, and you must supply the domain controller. I used the IP of mine.

1:  public DirectoryEntry find_by_email(string email)
  
2:    {
  
3:      DirectoryEntry dirEntry = new DirectoryEntry("LDAP://10.0.7.100","ausername","apassword",AuthenticationTypes.Secure);
  
4:      DirectorySearcher Dsearch = new DirectorySearcher(dirEntry);
  
5:      Dsearch.Filter = "(&(objectCategory=person)(sAMAccountName=*)(mail="+email+"))";
  
6:      SearchResult sResult = Dsearch.FindOne();
  
7:      if (sResult == null)
  
8:      {
  
9:        return null;
  
10:      }
  
11:      EventLog.WriteEntry("DoneEmailLookup");
  
12:      DirectoryEntry user = sResult.GetDirectoryEntry();
  
13:      return (user);
  
14:    }

Doing a Offline Client Using Silverlight 4 and Rails Backend

2010-08-26T19:53:00.000-07:00

I've prevously did a MultiMedia Silverlight app for a Video.
Now I'm doing a data driven app, more in line with Rails.
The issue with any web app is no internet = no work.
But how to have a nice web client and have a local database?
Espesially that can operate with Rails backend?

I Found a Great Solution for Doing this,in the form of a Database Product
called Siaqodb. I managed create a web app, with a datagrid in silverlight
that can run on the web or as a local app, and have local tables, and did it in a few hours.

For your local tables you can create a model in silverlight such as:

SendItAttachment.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Sqo;

namespace SendIt
{
    public class SendItAttachment : ISqoDataObject 
    {
        public string Name { get; set; }
        public string Path { get; set; }
        public DateTime StartTime { get; set; }
        public DateTime EndTime { get; set; }
        public int TransferTime { get; set; }
        public Boolean Done { get; set; }
        public Boolean Failed { get; set; }
        public string Hash { get; set; }
        public int FileSize { get; set; }
        public int OID { get; set; }


        public object GetValue(System.Reflection.FieldInfo field)
        {
            return field.GetValue(this);
        }
        public void SetValue(System.Reflection.FieldInfo field, object value)
        {

            field.SetValue(this, value);

        }
    }

}

Its Important that you make sure your grid is set to autogenerate the columns

 <sdk:DataGrid AutoGenerateColumns="True" Height="273" HorizontalAlignment="Left" Margin="13,321,0,0" Name="SendingGrid" VerticalAlignment="Top" Width="780" />

Add a reference to SiaqodbSL so that you can use the database.

Next thing is to Instance the database, and load your data

       public MainPage()
        {
            InitializeComponent();
            this.loginContainer.Child = new LoginStatus();
            this.Loaded += new RoutedEventHandler(MainPage_Loaded);
        }
        Siaqodb siaqodb;
        void MainPage_Loaded(object sender, RoutedEventArgs e)
        {

            if (Application.Current.IsRunningOutOfBrowser && Application.Current.HasElevatedPermissions)
            {
                //use MyDocuments folder on client machine to store data
                siaqodb = new Siaqodb("siaqodb",Environment.SpecialFolder.MyDocuments);
            }
            else
            {
                //use IsolatedStorage to store data
                siaqodb = new Siaqodb("siaqodb");
            }

            SendItAttachment attachment = new SendItAttachment();
            attachment.Name = "";
            attachment.StartTime = new DateTime(2010,1,1);
            attachment.EndTime = new DateTime(2010,1,1);
            siaqodb.StoreObject(attachment);
            siaqodb.Flush();

            IObjectList<SendItAttachment> senditattachments = siaqodb.LoadAll<SendItAttachment>();
            this.SendingGrid.ItemsSource = senditattachments;

Thats the basics, we will get a bit facier in new articles coming soon.

ESXi - Resizing Disk

2010-07-12T23:20:00.000-07:00

Wow, I was running in circles. I needed to resize a system disk on
one of my development machines, that running on the HA ESX/Vsphere
cluster. It was "grayed" out, and I could not change it.

It puzzled me, as I've resized dozens of times on other machines.
All my machines were created using the latest VM spec, but yet
no resize. I even pushed the disk back and forth between datastores
and it didnt come back.

What was the problem?

If you resize, you can have no snapshots.

Sure enough, deleted my snapshots, and everything was good.

Vmware - Leasons Learned - View ESX vs ESXi ISCSI Backup

2010-07-08T17:59:00.000-07:00

Wow, Its been interesting.

Designing and Deployment of a fully virtualized enterprise.

This includes:
1. Replacement of all desktops with thin clients.
2. Move from hosted email to a virtulized exchange environment.
3. iscsi based storage for everthing.
4. Blade Based Servers

Leasons Learned:
1. When your designing your blades, use at least a single SSD
drive to host ESX hypervisor. Yes, you can boot iScsi, yes you can
use a SD card.

But:
a. A SD Card means your running ESXi, your updates consists of re-installating. I've had to do it three times. Vs ESX on a SSD drive you
can use update manager.

b. iscsi boot - Its a rather "flakey" feature. I put some time on this
there are two problems
1) Only a select number of iscsi HBA are supported.
2) It consumes a LUN on your raid for each blade.
c. If I use SSD, I can even ghost one to another using FOG.

2. Storage
My primary storage is a Dell Equal Logic Raid. A PS4000 to be
exact.
Leasons:
a) Get the biggest that will not get you ask to leave :)
I'm buying my second one, people tend to keep a lot of garbage
b) Make sure you put the latest firmware on the device Day 1
c) Be very careful about your storage Design. The EqualLogic
has great features. Such as overcommit at the LUN level.
The problem is when the Volume/LUN gets full, its gets marked
offline. And any VM's in the VMFS associated are pretty much
dead. So you key servers put in separate LUN's. A DC, and
exchange for example in my case should be allocated a dedicated volume. Then you dont have to worry that one overcommit takes out all. I would not recommend you do it for everything but some good division can save your bacon.
d) I would recommend for key servers, Domain Controller, Exchange, your Vcenter you dont thin provision them. You make sure space
is allocated all the way down. In that way you know there is not a lurking storage outage waiting.

3. Memory - Get lots, 8 x what you need. Blads + Vmware is not cheap, the
reality is vmware on the CPU side is great and pretty good on the memory side as well. What I found on the blades, is I could put 10x more vm's from a cpu point of view, but memory was red-lined. So I could get more value buying more memory than buy cpu's. So when I expand, I'll up my memory from 12G per blade to at least 48G.

4. Backup - Vreeam Rocks - Make sure you setup jumbo packets, and be a bit
patient on setup to learn the product. But it really works quite well. You need scratch space to dump the backup file. And use Yosemite to write the backup image to the tape library. Make sure the NICs can run jumbo frames it make a big difference, and get you a nice quad port gig-e card or a 10g interface.

5. What not to buy - Dont use a Dell R200 for anything for a couple of reasons. No Jumbo Frames, and second the CPU is just under powered.
Also it would not leat me use some ESX features, so ended up uses native.

Finding Identical and Different Images

2010-06-11T21:11:00.000-07:00

Its a common problem to have lots of images, and when you do, it even more common to
have lots of duplicates. The problem with images, is they can be the same to the eye, but vastly different at the binary level. Different resolutions, slightly different cropping, but yet at a glance the same. In IT its common to calculate a hash, or magic number that changes radically for slight difference. What more useful in imaging duplication is changing slightly with near identical images.

This ruby plugin, does exactly that. I'm looking at using in a upcoming project, and will update the article as I go forward. What's even more exciting is the ability to do the same in video.

http://github.com/mperham/phashion

Moving thin vm's between datastores on esxi

2010-05-27T01:29:00.000-07:00

Wow, I wanted to move my vm's from testbed running on esx-i 4.0 to my production servers, which support iscsi.

The trick is to move them quick, without letting them ballon.

To do this you have to move the disk images: *.vmdk and the snapshots using
vmkfstools, plus the .vmx file and the .nvram file.

So as a example:

The original directory:
/vmfs/volumes/4bbda047-6fa7b452-4280-6cf049e2d29a/sin-fs-1 # ls -l -h
-rw------- 1 root root 80.0G May 27 01:37 sin-fs-1-flat.vmdk
-rw------- 1 root root 8.5k May 27 01:37 sin-fs-1.nvram
-rw------- 1 root root 501 May 26 07:14 sin-fs-1.vmdk
-rw------- 1 root root 0 Apr 30 10:07 sin-fs-1.vmsd
-rwxr-xr-x 1 root root 3.2k May 27 01:37 sin-fs-1.vmx
-rw------- 1 root root 263 May 5 01:33 sin-fs-1.vmxf
-rw------- 1 root root 600.0G May 27 01:37 sin-fs-1_1-flat.vmdk
-rw------- 1 root root 504 May 26 07:15 sin-fs-1_1.vmdk
-rw-r--r-- 1 root root 234.7k May 11 09:21 vmware-2.log
-rw-r--r-- 1 root root 135.3k May 12 06:29 vmware-3.log
-rw-r--r-- 1 root root 135.3k May 13 02:55 vmware-4.log
-rw-r--r-- 1 root root 135.3k May 25 04:03 vmware-5.log
-rw-r--r-- 1 root root 132.5k May 25 04:14 vmware-6.log
-rw-r--r-- 1 root root 135.2k May 26 07:07 vmware-7.log
-rw-r--r-- 1 root root 136.5k May 27 01:37 vmware.log
/vmfs/volumes/4bbda047-6fa7b452-4280-6cf049e2d29a/sin-fs-1 #

Note the disk image is 600G. Its actually a sparse file, that is very "thin". So we only want to copy the data, not the empty space. Also VM can have snapshot disk images, so you also have to copy them.

So in this case lets first copy the disk images:

vmkfstools -i /vmfs/volumes/datastore1/sin-fs-1/sin-fs-1.vmdk -d thin sin-fs-1.vmdk
vmkfstools -i /vmfs/volumes/datastore1/sin-fs-1/sin-fs-1_1.vmdk -d thin sin-fs-1_1.vmdk

Now we copy the vmx files:
cp /vmfs/volumes/datastore1/sin-fs-1/sin-fs-1.vmx .
cp /vmfs/volumes/datastore1/sin-fs-1/sin-fs-1.nvram .

You can now use the vSphere client datastore browser to import, or import at the command line.

Reverting a Cisco LAP1142N to Atonomyous Mode

2010-04-15T01:51:00.000-07:00

Wow, this was fun.

A friend of mine got in half-a-dozen Cisco AP's, and they all came with the wrong firmware. These are nice, slim flying saucers Wifi AP's with 2 radios supporting
2.4G and 5.0G operation. The focus of the line is to use a controller, my friend
chose to not buy a controller, but to buy autonomous units.

So when they came in. IOS was missing in action.

While there are notes on the web to do this, not appeared to work.

The basics is, to setup a tftp server at a magic address (10.0.0.2),
the ap will boot, setting it self to 10.0.0.1, and check for a default image
based on model.

Didnt work at all, despite shooting a day.

I tried several different tftp implementions, including linux tftp, atftpd, and
finally two different ones on windows.

So after two days of pain, and some inspiration from one of my friends, I figured out
the secret sause.

1. Press and hold the mode button
2. Let it boot till it notices (You can see the messages if you plug in the cisco cable) It will say the mode button is pressed in the logs
3. It will fail to find the image need and dump you to the bootloader
4. At the bootloader erase the flash - format flash:
5. Reboot
6. set the IP address, network mask, and router, as the instructions are given in the log
7. tftp_init
8. tar -xtract tftp://thefirmwarefilename.tar flash:

If you mess up start back at 1

tftp is very sensitive to traffic, and to extra overhead in the server side,
so be careful about switches.

Mac Printing - Getting Color to PCL Printers

2010-01-17T17:44:00.000-08:00

Wow, what a pain. My friend has a couple of Docucentre III printers, and now they
have a quite a few macs in the office. Getting black and white printing is easy.
Just use the generic Black and white driver, and your printing. Now of course they
want to print color.

So after trying everything I can think I come to find out the standard printing subsystem, which uses Gutenprint. I upgraded to the latest, which is version 5.2.4,
and still no-soap. Just black and white printing. After reading the forums, seems like the generic Gutenprint 5.2.4 cannot print color.

So the choice is to add the second printing subsystem.
Its called foomatic, plus you need the RIP(RasterImageProcessor) which is a version
of ghostscript, and finally the print driver which is called pxmono.

For my case I downloaded the following:
foomatic-rip-4.02.211.dmg
gplgs-8.64so-ub.dmg
pxlmono-1.6.dmg

The url for getting foomatic and gplg is:
http://www.linuxfoundation.org/collaborate/workgroups/openprinting/macosx/foomatic

The driver you need is pxlcolor, and its contained in pxlmono installer.
http://www.linuxfoundation.org/collaborate/workgroups/openprinting/macosx/pxlmono

You must install all three, start with the gplgs, which is a shared ghostscript,
followed by foomatic, and finally by pxmono.

In your printer settings you now will have a printdriver for:
Generic PCL6/PCL XL Printer Foomatic/pxlcolor

After you added the printer, if it does not print, you must go to the web interface
and change the print-out-mode setting. To do that, you must go to set-printer-options, using http://localhost:631

Select your printer, the set printer option tab and change the printout mode from normal-graysacle to normal.
using your

This will print nicely on docucentre in color, and I suspect most PCL capable printers.

Finding missing controllers and models

2010-01-10T17:38:00.000-08:00

Ever get aggravated you didn't create a model or a base controller for your rails app.
I find its a common issue when large changes are afoot. Like when I create dozens
of tables at a sitting.

For those wondeful times, I've created a rake take that checks my models and controllers, and creates basic versions of the code.

 require 'rubygems'  
 require 'pp'  
 require 'find'  
 def controller_exists?(thefilename)  
   Find.find("./app/controllers") do |path|  
      if FileTest.directory?(path)  
       if File.basename(path)[0] == ?.  
         Find.prune  
        else  
         next  
       end  
      else  
       if path.include?(thefilename)  
         return TRUE  
         end  
        end  
      end  
   return FALSE  
 end  
 namespace :check do  
      desc "Check Model, and create missing ones"  
      task :modelfile => :environment do  
           pp ActiveRecord::Base.connection.tables  
           ActiveRecord::Base.connection.tables.each do |tname|  
               begin  
                themodel = tname.classify.constantize  
                model_exists = TRUE  
               rescue  
                model_exists = FALSE  
                end  
               case tname  
                when "schema_migrations"  
                   model_exists = TRUE # Special table  
                when "tag_cross_ref"  
                   model_exists = TRUE # Special table  
                end  
            if model_exists == FALSE # Dont exist  
             model_name = tname.classify  
             puts "Missing #{model_name} for table #{tname} - created\n"  
             mfilename = "./app/models/" + tname.singularize + ".rb"  
             if File.exists?(mfilename)  
               puts "#{mfilename} exists - Bad NEWS\n"  
               raise  
               end  
             mfile = open(mfilename,"w")  
             mfile.puts "class #{model_name} < ActiveRecord::Base\n"  
             mfile.puts "end\n"  
             mfile.close  
             else  
             puts "Exists #{model_name}(#{tname})\n"  
             end  
        end  
     end  
  desc "Check Controllers, and create missing ones"  
  task :controllerfile => :environment do  
           ActiveRecord::Base.connection.tables.each do |tname|  
               controller_file_name = tname.singularize + "_controller.rb"  
               ctl_missing = !controller_exists?(controller_file_name)  
               case tname  
                when "schema_migrations"  
                   ctl_missing = FALSE #Special Table  
                when "tag_cross_ref"  
                   ctl_missing = FALSE # Special table  
                end  
               if ctl_missing  
             controller_name = (tname.singularize + "_controller").camelize  
             puts "Missing #{controller_name} for table #{tname} - created\n"  
             cfilename = "./app/controllers/adminspace/" + controller_file_name  
             if File.exists?(cfilename)  
               puts "#{cfilename} exists - Bad NEWS\n"  
               raise  
               end  
             cfile = open(cfilename,"w")  
             cfile.puts "#generated by checker\n"  
             cfile.puts "class Adminspace::#{controller_name} < ApplicationController\n"  
             cfile.puts "layout 'tier1admin'\n"  
             cfile.puts "before_filter :login_required\n"  
             cfile.puts 'require_role "admin"' + "\n"  
             cfile.puts "\n"  
             cfile.puts "   active_scaffold :#{tname.singularize} do |config|\n"  
             cfile.puts "       config.actions = [:nested, :create, :update, :show, :list, :search]\n"  
             cfile.puts "       end\n"  
             cfile.puts "\n"  
             cfile.puts "end\n"  
             cfile.close  
             else  
             #puts "Exists #{controller_name}(#{tname})\n"  
             end  
        end  
     end  
 end

Virtual Models and Parsing in Ruby

2009-10-12T17:15:00.000-07:00

Ever want to process something external to your database or your rails app, such
as the contents of files. But you think rails is only good for database items?
Actually rails can easily adapt to handle any kind of data.

In these examples, I process all my controllers, to give a view and a RESTful interface
for testing of my controllers. I also do the same for the menus. All without a database
table in sight.

First, lets take a look at the controllers "model".

This allows us to view our controllers from within rails, useful if you want to know what code is running, or
in my case to use watir to test. The lovely part is I can actually RESTfully pull the list of controllers, and make
sure each is hit in watir.

The next problem is menu's, how do I validate that each menu does something, a little of the same
technique. Here is a bit more advanced, we actually build a parent menu table, and a child menu item table.

Here is the menu model. The Menu Model actually creates all the child menu items at startup. It does this by
parsing tabnav/widget format menus. While I was looking at a way of using erb to do this for me, I ended up just
doing the parsing myself.

And finally the MenuItem Model

After this I do simple activescaffold based controllers, and more importantly add routes so I can pull xml files
for testing.

ActiveJquery - Without git

2009-08-21T05:09:00.000-07:00

If you pull the files, and install them without using rails plugin code, there
is a "install" script that copies things into the right place.

The file will be in: #{projectroot}/vendor/plugins/activejquery

So for me, its in the following directory:
/Users/gwest/mymrp2/vendor/plugins/activejquery

As you can see it, actually copies a few files into the right place in your project.

directory = File.dirname(__FILE__)
copy_files("/public/css", "/public/css", directory)
copy_files("/public/javascripts", "/public/javascripts", directory)
copy_files("/app/views/activejquery", "/app/views/activejquery", directory)

I love to help, so if you have questions, feel free to email me.
or if your commenting, please give me your email, as when you comment on the
blog, the system does not give me a email to respond.

Call Center Voice Recording

2009-06-15T00:22:00.000-07:00

Ever wonder how you do audit in a large call center?

Here's the basics:

First, most modern call centers are using Voip. It makes it easy to setup, and even better
makes it easy to monitor. Also call-centers tend to be in places like india, or phillipines
serving the US or Europe markets. So VoIp plays a role in getting the traffic from one
country to another.

There are several pieces to this puzzle:

First, you need to capture the raw data:
So to do that, we need a "tee" off the ethernet channel that is connecting to the VoIP gateway.
This will allow us to see all the VoIP IP Traffic that is going back and forth.

It's sometimes called 'port mirroring', 'port monitoring', 'Roving Analysis' (3Com), or 'Switched Port Analyzer' or 'SPAN' (Cisco).

http://wiki.wireshark.org/CaptureSetup/Ethernet
http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008015c612.shtml#descp

Now that we have traffic, a good way to "capture" the voice is Oreka
http://oreka.sourceforge.net/

Oreka lets you capture the packets, and convert them to audio files. It actually consists
of three parts.

- OrkAudio: This is the workhorse that processes the calls and does the actual recording
- OrkWeb: This is the XML based Web U/I to access and manage the system
- OrkTrack: This is the master database (MySQL) that records the call records, metadata, etc.

Oreka lets you get a basic system up quickly.

The next issue, is setting up a system of audit, and to connect and understand the calls into transactions,
customers, call-agents, and supervisors.

For a good GUI on top, I'd switch to a Rails App. This lets me do several things:

1. I can tie agents to calls
2. I can allow agents access to review there own calls
3. I can allow supervisors to view/listen to calls for agents reporting to them
4. I can let audit department audit calls as appriate.
5. Ease of interfacing existing call center apps togeather

Now that we have a phase 1, and phase 2 completed, lets add a phase 3.
One of the emerging technologies is Speach to text.
Using the latest SDK's from Dragon Speaking, we can get very high 99% plus on our agents,
and not bad recognition on our clients/customers calling in. This will allow audit to search
for key words, such as scam, theft, etc that would audit to be better aware of things going
on.

The idea way of implementing the whole system is break things down to separate VM's in a VmWare system.

1. OakCapture - 1 VM per major Trunk (One Per - Multiple DS1, One DS3)
2. OakGui - 1 VM For system - For reference/admin control only
3. Database - In a call center, a Oracle or DB2 may be better suited, depending on the company
4. VoiceCenterManager - A Rails Server using Nginx/Passenger that provides interfaces to agents, supervisors, audit,
and managers. Number of VM determine by staff size
5. Recognition Engines - A separate VM that is running a Ruby Agent tied to a Dragon Speaking SDK allowing speaker dependent and Speaker Independent Speech to text conversion. This allow for the creation of easily searchable as well
as readable results of the calls in text form. Using a rack of blades, and VmWare we can realtime convert all calls to text
index them, and make them searchable.

Note that each call center is different but leveraging Rails, and Ruby a AGILE and fast solution can be combined that
gives the best in Audit and accountability.

Ghosting NTFS/Windows using Knoxppix

2009-06-11T07:43:00.000-07:00

I have a USB 1 Gig stick that is on my key ring.
Its really handy, as I have a complete copy of Knoxppix.
Knoxpix is a Linux Distribution designed to boot and run
from CDROM or DVD. It can also run from a handy USB stick.
With the right options it can also in "RAM", which is handly if you want to
run a few parallel operations, and only need one usb stick.

While you can use clonezilla, if you want something more packaged,
but in a shop that is windows centric, ghosting to a windows server
may make more sense.

Boot your usb stick, then at the bash shell, mount your server:

mount -t cifs //server-name/share-name /mnt/cifs -o username=shareuser,password=sharepassword,domain=nixcraft

On the share, I usually will the have my ntfscloning scripts

On a dell machine, there is really only one partition that has "data",

So to the "dobackup" script is very trival.
This takes the OS partition, in most of the environments I was working in this
would have been /dev/sda2 and uses ntfsclone, which takes only the in-use
sectors, compresses them, and put them in a file on the server.

dobackup script

echo backup $1 to $2
mkdir $2
ntfsclone -s -o - /dev/$1 | gzip > $2/$1.ntfsclone.gzip

The reverse, to restore is:

dorestore script

echo restore $1 to $2
cd $1
cat $2.ntfsclone.gzip | gunzip - | ntfsclone --restore-image --overwrite /dev/$2 -

The only other madatory item is the partition table:

doptablebackup script

echo partion backup $1 to $2
sfdisk -d /dev/$1 > $2/ptable.doc
dd if=/dev/$1 bs=512 count=1 of=$2/ptable.sector

doptablerestore script

echo partion restore $1 to $2
dd of=/dev/$2 bs=512 count=1 if=$1/ptable.sector

Dynamically Generating Javascript using Parameters

2009-06-07T22:07:00.000-07:00

I love re-using code, and really believe in DRY.
In doing ActiveJquery, I want my grids to be very re-usable.
For one, I need to use multiple grid/sub-grids to handle Rails
relationships, second I need to be able to have multiple grids
in one page. And want to do all of this with the least amount of code.

In ActiveJquery, the Grid code, is generated on the fly. While it looks like a normal
call, its actually hitting controller code tucked inside the plugin.

So when we request the javascript code for a particular controller, the ActiveJquery
code generates the appropriate javascript on the fly.

In doing, the sub-table, I need to have parameters passed in, so I rely on standard
Rails Semantics when handling HTML.

<script src="location.js?subof=Company&div=Company_Location" type="text/javascript"></script>

This technique lets me generate customized javascript on the fly, making the generator re-usable, and
the grid re-usable any number of times in the same application.

Finding ActiveRecord Associations

2009-06-06T00:08:00.001-07:00

Im working away on ActiveJquery, and the next thing on the list, is to add support for relationships. This gets to be really interesting, because your Inside the box, in a plugin, and you need to know what associations the user has
defined.

I was scanning the documentation, and looking around for the api to find the information.

After searching for a while, I discovered railway, a gem for rails that does diagraming of rails models using dot. So grab the gem, and look thru the source.

So the key to finding associations is:

@associations = table.reflect_on_all_associations

This results in:

@associations=
  [#ActiveRecord::Reflection::AssociationReflection:0x26305cc
    @active_record=
     Company(id: integer, name: string, location_id: integer, created_at: datetime, updated_at: datetime),
    @macro=:has_many,
    @name=:user,
    @options={:extend=>[]}>,
   #<ActiveRecord::Reflection::AssociationReflection:0x262faa0
    @active_record=
     Company(id: integer, name: string, location_id: integer, created_at: datetime, updated_at: datetime),
    @macro=:has_many,
    @name=:location,
    @options={:extend=>[]}>,
   #<ActiveRecord::Reflection::AssociationReflection:0x262f0c8
    @active_record=
     Company(id: integer, name: string, location_id: integer, created_at: datetime, updated_at: datetime),
    @macro=:has_many,
    @name=:division,
    @options={:extend=>[]}>],

From my models:

class Company < ActiveRecord::Base
  has_many :user
  has_many :location
  has_many :division
  
end
class Department < ActiveRecord::Base
      belongs_to :division
end
class Division < ActiveRecord::Base
belongs_to :company
end
class Location < ActiveRecord::Base
end
class User < ActiveRecord::Base
end

ruby script/plugin git does not work in Rails 2.3.2 and Ruby 1.9

2009-05-14T21:30:00.001-07:00

Lots of people are scratching there head over why they cannot
install rails plugins using git.

I'm working with Ruby 1.9.1 and Rails 2.3.2. And git is a pretty natural
for working with version control.

So I wanted to pull from my own respository, but kept getting
a error on tryping to do the install.

sin-gwest-laptop:testjq gwest$ ruby script/plugin --verbose install git://github.com/glennswest/activejquery.git
Plugins will be installed using http
Plugin not found: ["git://github.com/glennswest/activejquery.git"]
#<TypeError: can't convert Array into String>

Come to find out, this is a known but as the mkdir_p call has changed its
return parameter in Ruby 1.9 and that messes up the install of git plugins
unless you do the patch

https://rails.lighthouseapp.com/attachments/90768/plugin_mkdir_p.diff

ActiveJquery Goes to version .011

2009-05-14T19:57:00.001-07:00

I've updated ActiveJquery on github to .011

That was fun. I thought I'd put the authtoken issue to bed.
On doing test for delete and add, found I still was getting authtoken
issues. So I moved from using editData jqgrid parameters to adding
it to the editurl.

Also found a bug in the page based xml pull of the controller component.
This would result in you not seeing the last 10 records. (You notice that
when your adding records and they dont show up).

ActiveJquery consists of:

1. active_jquery - The controller plugin
2. active_jquery_runtime - Generates the dynamic jqgrid javascript
3. Dynamic Javascript - The code that runs in the browser
4. jQuery/Jquery UI/jqGrid

I've also combined all the needed public css and javascript into the plugin.

Need to add a rake task and init that makes sure they get copied.

ActiveJquery Reaches Version .010

2009-05-14T02:20:00.001-07:00

ActiveJquery, which is designed to integrate Jqgrid, Jquery UI into rails, is
now at Version 0.010.

It now is a Rails 2.3.3 Controller Plugin. You can invoke it in a single
line in your controller.

The plugin automatically generates javascript for the grid, based on your table,
with inline editing. As well as a full REST server to serve the data to the browswer.

I've implemented the ForgeryProtection on Posts, and that is working. Seems like a bit of comfusion in BLogSpere about weather the URI Components need to be encoded or not. Least in Ruby 1.9.1 and Rails 2.3.2 They do NOT need to be encoded.

Todo:
1. Allow Customizations
2. Allow Relationships and SubTables
3. JQuery UI Menus

Test and More TEst

Also I will implement a rails DEMO app that is based on data from the JQGRID site.

I'll make a separate git repository for the demo.

ActiveJquery - Status

2009-04-29T18:58:00.000-07:00

Current Things That Are Working:

Controller/Server
1. Can read full table
2. Can read using JqGrid Pager
3. Added Sort support, so server will honor the grid sort request
4. Added Delete,Add and Update support.
5. Auto Generates JqGrid Javascript from ActiveRecord
6 Added Total Records to XML so Pager works properly.

ActiveJquery Library/Client
1. Added support for string, integer, date.
2. Generates JSON Based Reader compatible with Rails JSON Format
3. Uses Humanize to handle automatic column names.
4. Uses JqueryUI for theming
5. InLine Edit Support

Things to Do:
1. Paste Controller code into prepared plugin
2. Add static data support
3. Add a bit of DSL(Domain Specific Language) to allow easy configuration
4. Add Master/Detail Support
5. Add Date Picker Plugin
6. Add Parent Table Dynamic Data Selector

ActiveJquery - Features

2009-04-29T18:05:00.000-07:00

ActiveJquery is a Rails Plugin that combines the goodness of Jquery, Jquery UI, and JqGrid.

Feature	Description
REST	ActiveJquery breaks your GUI into a javascript REST Client, and a Rails based REST JSON Server. This reduces the overhead of our web app, and gives you better expandability, and better response time
DRY	Tired of repeating yourself, dont. ActiveJquery will find your Field Name, and configure the Grid for you.
Customizable	You can configure grids in countless ways, allowing you easy control over what you want to see.
MultiGrid	Need a butch of grids on one page? Not a problem. You can have any number of grids in one page. Each is named automatically for you.
Master/Detail	You Rails Associations are used to generate sub-grids or master-detail views of your data.
Theming	ActiveJquery uses Jquery UI themes, so you can easily customize the look and color scheme of the grid
Menu's/Tabs	Want to have a easy to use menu system. ActiveJquery supports JqueryUI tabs, so you can do complex, but easy to use apps.
Dynamic Data	By default data is read a screenful at a time, using the grid pager. This reduces the need to read large amounts of a big table. Also makes loading multiple grids very fast.
Static Data	ActiveJavascript can embed the table data directly in the javascript. This is great for tables that 1000 rows or less, and are read-only.

Jdgrid Pager Problem

2009-04-29T04:47:00.000-07:00

Trying out the latest versions of jquery/jquery ui/jqgrid, and just cannot get the formatting right.

Lets see if we can figure out whats going on.

The html for the grid:

<html>
<head>
  <title> Airstate </title>
</head>
<body>
<h2> Airstate </h2>
<script src="javascripts/jquery-1.3.2.min.js" type="text/javascript"></script>
<script src="javascripts/jquery-ui-1.7.1.custom.min.js" type="text/javascript"></script>
<script src="javascripts/jquery.layout.js" text="text/javascript"></script>
<script src="javascripts/jqModal.js" type="text/javascript"></script>
<script src="javascripts/jqDnR.js" type="text/javascript"></script>
<script src="javascripts/jquery.jqGrid.js" type="text/javascript"></script>
<link rel="stylesheet" type="text/css" media="screen" href="themes/redmond/jquery-ui-1.7.1.custom.css" />
<link rel="stylesheet" type="text/css" media="screen" href="themes/ui.jqgrid.css" />
<style>
html, body {
    margin: 0;            /* Remove body margin/padding */
    padding: 0;
    overflow: hidden;    /* Remove scroll bars on browser window */    
    font-size: 75%;
}
/*Splitter style */


#LeftPane {
    /* optional, initial splitbar position */
    overflow: auto;
}
/*
 * Right-side element of the splitter.
*/

#RightPane {
    padding: 2px;
    overflow: auto;
}
.ui-tabs-nav li {position: relative;}
.ui-tabs-selected a span {padding-right: 10px;}
.ui-tabs-close {display: none;position: absolute;top: 3px;right: 0px;z-index: 800;width: 16px;height: 14px;font-size: 10px; font-style: normal;cursor: pointer;}
.ui-tabs-selected .ui-tabs-close {display: block;}
.ui-layout-west .ui-jqgrid tr.jqgrow td { border-bottom: 0px none;}
.ui-datepicker {z-index:1200;}
</style>

<script type="text/javascript">
var listlastsel;jQuery(document).ready(function(){
jQuery("#list").jqGrid({
   url:'airstate.xml',
   editurl:'airstate',
   datatype: 'xml', 
   mtype: 'GET',
   colNames:['Id','Created on','Whom','Logmessage'],
   colModel :[
      {name: 'id',index:'id',key:true,width:80,align:'right'},
      {name: 'created_on',index:'created_on',key:false,width:90},
      {name: 'whom',index:'whom',key:false,width:300,align:'left',editable:true},
      {name: 'logmessage',index:'logmessage',key:false,width:300,align:'left',editable:true}],
   pager: jQuery('#list-pager'),
   onSelectRow: function(id){
     if(id && id!==listlastsel){
        jQuery('#list').restoreRow(listlastsel);
        jQuery('#list').editRow(id,true);
         listlastsel=id;
         }
      },
   autowidth: true,
   rowNum:10,
   rowList:[10,20,30],
   sortname: 'id',
   sortorder: "desc",
   viewrecords: true,
   imgpath: 'themes/basic/images',
   caption: 'Airstate',
   xmlReader: {root: "root",
                row: "syslog",
                page:"root>page",
                total:"root>total",
                records:"root>records",
                repeatitems:false
},
  });
jQuery("#list").navGrid('#list-pager',{ edit:true,add:true,del:true,search:true });
});
</script>
<div id="list-pager" class="scroll"></div>
<table id="list" class="scroll"></table>

</body>
</html>

Now lets see if we can figure out whats going on.

Ok, Great support from jdquery. Seems like its a a bug in the latest v2 alfa test.
Alfa 3 should solve the problem.

Getting only the columns you want in Rails

2009-04-07T05:25:00.000-07:00

A thing that is often missed when your doing a web service or a rails app is selecting only
the columns you need.

A traditional find:

user = User.find :all
render :xml => user.to_xml

Will give you the bloat of the whole user table.
If you only hae a few small columns then thats not so bad,
but I've seen legacy apps that have very large number of columns,
so it makes since then to control this better.

user = User.find(:all, :select = 'email')
render :xml => user

This also will make your resulting xml file alot more managable.
And since bandwidth is not free, it will help you handle more users
for less money.

Network Monitor as a Windows Service in Ruby

2009-04-07T00:56:00.000-07:00

In reading thru the Ruby Google Group, there seems to be a lot of misconceptions about what should be done inside
of a Rails process and what should be done outside it. Generally if it take any time at all, you should be it outside
the Rails Web Server. In windows you can do a Windows Service, in Linux/Unix you would do a daemon.

This is a example of a windows service that scans cisco switches, finds new nodes, and keep track of there mac
address and there port. Even builds a network map.

The app consists of a Rails app to display the data, and take in configuration information. Which is put in a database,
such as sqlite3, and updated by this Windows Service.

As long as the Service is running, the data will be updated. If I was doing it again, I'd use rufus-scheduler to
wrap the scanning and the mapping.


require 'rubygems'
require 'win32/daemon'
include Win32
require 'logger'
require 'win32/process'
require 'net/ping'
#Note that most of your requires need to go in the service init

class IPAddr
   def succ()
       return self.clone.set(@addr + 1)
       end
end

class AirstateDiscovery

def doscan()
    ProcessNetworks()
    ProcessDiscovery()
    return
    end


def ProcessDiscovery()
    Mac.find(:all).each do |mymac|
          mydevice = mymac.device
          if mydevice
             mynet = Network.find_network_by_ip(mymac.ip)
             if mynet #Ok we have a valid network
                Device.find_device_type(mymac.ip,mydevice)
             end
          end
    end
end


def ProcessSwitches()
    Switch.find(:all).each do |myswitch|
            if myswitch.enabled
               if myswitch.switch_type == "cisco"
                     cisco = CiscoSwitch.new(myswitch)
                     #begin
                       cisco.update_macs()
                     #rescue
                     #  end # Rescue
                     cisco = nil
                  end #if myswitch.switch_type == "cisco"
               end # myswitch.enabled
            thegraph = SwitchGraph.new(myswitch)
            thegraph.graph()
            thegraph = nil
            end # find do
end

def ProcessNetworks()
  ProcessSwitches()
  return
  
  # Brute Force Scan
    Network.find(:all).each do |mynet|
            ProcessSwitches() # Networks is "long" running, Switch is "fast"
            if mynet.enable
               startip = mynet.ip_start
               endip   = mynet.ip_end
               ip = IPAddr.new(startip)
               while ip.to_s != endip 
                  ProcessWmi(mynet,ip.to_s)
                  ip = ip.succ()
                  end
               end
            end
end # def Process Networks

end # class AirStateDiscovery 
   
class Daemon
      def service_init
          end

      def service_main
     
         require 'ipaddr'
          Dir.chdir("\\projects\\airstate")
          require File.dirname(__FILE__) + '/../config/environment.rb'
          require 'lib/cisco_phone'
          require 'lib/managepc'
          require 'lib/switchgraph'
          @mywork = AirstateDiscovery.new
          mylog = Syslog.new
          mylog.whom = "AirStateDiscovery"
          mylog.logmessage = "Discovery Starting"
          mylog.save
         while running?
            @mywork.doscan()
            sleep 3
         end
      end
end


# Test Code
#          require 'ipaddr'
#          Dir.chdir("\\projects\\airstate")
#          require File.dirname(__FILE__) + '/../config/environment.rb'   
#          require 'lib/managepc'
#          require 'lib/cisco'
#          require 'lib/switchgraph'
#          require 'lib/cisco_phone'
#          @mywork = AirstateDiscovery.new
#          while 1
#            @mywork.doscan()
#            end
#          exit

Daemon.mainloop

Rails - Useful Plugins

2009-04-05T18:21:00.001-07:00

In updating myself I found some really useful Rals plugins for my next project.

Hoptoad Notifier
A Rails app is a very dynamic thing. Finding out when your users have errors, and what the pattern of errors is a must have. The traditional approach is to view the log files from time to time, or use a notifier that sends the errors to your email. Then go thru it as they come in. First, it not uncommon for a error to repeat. So you get log of junk in your email. A better approach is to use the hoptoad service/plugin. This way, you get nice reporting and analysis, and for a single project, its free. It consists of a hosted site web gui, and a rails plugin, and a optional local mac gui to show your croak's, ie the errors in your apps.

http://www.hoptoadapp.com/welcome
http://blog.railsrumble.com/2008/10/6/hoptoad
http://github.com/thoughtbot/hoptoad_notifier/tree/master
http://github.com/bricooke/croak-app/tree/master

Simple Ruby Rest Client
REST is a lovely way of doing Remote Procedure Calls. It the rails way of creating web services. Sometime its handy to have a simple REST-client.
There is a simple to Rest client done by Adam Wiggins

http://github.com/adamwiggins/rest-client/tree/master
http://rest-client.heroku.com/rdoc/

Now you can call a REST api even from IRC or rails console

C Code - Bucket Based Allocator

2009-03-21T20:38:00.001-07:00

In realtime systems, malloc/free/garbage collection can take a very long time.

For a real-time system I'm doing, I want to have very fast response time, I call malloc/free often.
Traditional malloc/free will fragment memory quickly, and garbage collection will just take
forever.

The solution I've used many times is a bucket allocator, setting pre-defined sizes of memory, and putting them in a queue, or in this case a "bucket". The idea being find the bucket, pull out a entry and return it.

Also something I've found useful is to keep track of busy memory elements as well. That way if I suspect that memory is getting corrupted, I can easily add a tag at the end of the allocation, and run thru the busy memory looking for the corruption. Very useful in debug.

#include <stdio.h>
#include <stdlib.h>
#include "include/queue.h"



struct bucket_struct {
       struct entry_struct entry;
       struct queue_struct free_q;
       struct queue_struct busy_q;
       int    size;
       };

struct alloc_struct {
       struct entry_struct entry;
       struct bucket_struct *bucket;
       };

struct queue_struct buckets;

int gkmalloc_init_needed = 1;

void gkmalloc_newbucket(thesize)
int thesize;
{
struct bucket_struct *mybucket;

     mybucket = malloc(sizeof(struct bucket_struct));
     mybucket->free_q.head = NULL;
     mybucket->free_q.tail = NULL;
     mybucket->busy_q.head = NULL;
     mybucket->busy_q.tail = NULL;
     mybucket->entry.next = NULL;
     mybucket->entry.prev = NULL;
     mybucket->size = thesize;
     queue(&buckets, &mybucket->entry);
     return;
}    


void gkmalloc_init(void)
{
    gkmalloc_init_needed = 0;
    buckets.head = NULL;
    buckets.tail = NULL;
    gkmalloc_newbucket(32);
    gkmalloc_newbucket(64);
    gkmalloc_newbucket(128);
    gkmalloc_newbucket(256);
    gkmalloc_newbucket(512);
    gkmalloc_newbucket(1024);
    gkmalloc_newbucket(2048);
    gkmalloc_newbucket(4096);
    gkmalloc_newbucket(8192);
    gkmalloc_newbucket(32768);
    gkmalloc_newbucket(12000000); // Big Buf for jpegs
    return;
}

char *gkmalloc(thesize)
int thesize;
{
struct bucket_struct *mybucket;
struct alloc_struct  *myalloc;
void *ptr;

     if (thesize == 0)
        return(NULL);
     if (gkmalloc_init_needed){
        gkmalloc_init();
        gkmalloc_init_needed = 0;
        }
     mybucket = (struct bucket_struct *)buckets.head;
     while(mybucket->size < thesize){
          if (mybucket == NULL){ // Should never happen
              gkfatal("gkmalloc: Allocation bigger than max bucket");
              return(NULL);
              }
          mybucket = (struct bucket_struct *)mybucket->entry.next;
          }
     myalloc = (struct alloc_struct *)unqueue(&mybucket->free_q);
     if (myalloc == NULL){ // The allocation queue is empty
         // Dynamically allocate from the system
         myalloc = malloc(sizeof(struct alloc_struct) + mybucket->size);
         if (myalloc == NULL){
            gkfatal("gkmalloc: Cannot malloc new buffer");
            return(NULL);
            }
         myalloc->bucket = mybucket;
         }
           
      queue(&mybucket->busy_q, &myalloc->entry);
      ptr = (void *)(myalloc + 1); // Add the size of the header to get Memory After
      return(ptr);
}
       
void gkfree(ptr)
void *ptr;
{
struct bucket_struct *mybucket;
struct alloc_struct  *myalloc;

      if (ptr == NULL){
         gkfatal("gkfree: Bad Free - Null Ptr Passed");
         return;
         }
      myalloc = (struct alloc_struct *)ptr;
      myalloc = myalloc - 1; // Go back to the header info
      mybucket = myalloc->bucket;
      if (mybucket == NULL){
         gkfatal("gkfree: Bad Bucket");
         return;
         }
      queue(&mybucket->free_q,&myalloc->entry);
      return;
}

C Code - Link List or Queues

2009-03-18T19:45:00.001-07:00

One of my favorite real-time system tricks is to use queues, or linked-list
to handle message passing. Then the whole system can be lots of small co-routines, without the overhead of task-switching. Small and tight code really can run fast. Usually the problem you get with this is malloc/free overhead.

A pre-allocated chunk based system solves that. I'll post it to a future blog.

## queue.c
#include <stdio.h>
#include <stdlib.h>

#include "include/queue.h"
/*
** Queue.c
*/

struct entry_struct *unqueue(q)
struct queue_struct *q;
{
struct entry_struct *entry;

  if (q==NULL)         return(NULL);
  if (q->head == NULL) return(NULL);
  entry = q->head;
  q->head = entry->next;
  if (q->head == NULL) q->tail = NULL;
  entry->next = NULL;
  entry->prev = NULL;
  return(entry);
}

void queue(q,entry)
struct queue_struct *q; 
struct entry_struct *entry;
{
  if (q==NULL) return;
  if (entry == NULL) return;
  entry->next = NULL;
  entry->prev = NULL;
  if (q->head == NULL){
     q->head = entry;
     q->tail = entry;
    return;
    }
  entry->prev = q->tail;
  q->tail->next = entry;
  q->tail = entry;
 return;
}

struct queue_struct *init_queue(void)
{
struct queue_struct *q;

    q = (struct queue_struct *)malloc(sizeof(struct queue_struct *));
        q->head = NULL;
        q->tail = NULL;
        return(q);
}

## queue.h
/*
** Queue.h
*/

struct entry_struct {
       struct entry_struct *next;
       struct entry_struct *prev;
};

struct queue_struct {
     struct entry_struct  *head;
     struct entry_struct *tail;
};

struct entry_struct *unqueue(struct queue_struct *);
void queue(struct queue_struct *, struct entry_struct *);
struct queue_struct *init_queue(void);

Scraping Digikey using Ruby and ScRUBYt

2009-02-25T04:36:00.001-08:00

Sometimes the information you need in your application is already out there on a web site. In a BOM (Bill-of-Materials) app I'm doing, I want to be able
to lookup distributor part numbers from a manufacture part number.

In this example I use Digikey, as they have the vast majority of parts I use in projects.

I use Scrubyt, a ruby library, by Peter Szinek, and Glenn Gillen.
It does the heavy lifting.

The data returned in this example, is in the form of a table. ScRUBYt makes
parsing the table straight forward. You can use firebug in firebird to find your
data if your doing a different website.

Here is the base code

require 'rubygems'
require 'scrubyt'
require 'pp'
digikey_data = Scrubyt::Extractor.define do
thepartnumber ='C0805C104K3RACTU'
thesearch = 'http://search.digikey.com/scripts/DkSearch/dksus.dll?
lang=en&site=US&x=0&y=0&keywords=' + thepartnumber
fetch thesearch
table "/html/body/div[2]/table" do
row "//tr" do
dist_pn "/td[1]"
manuf_pn "/td[2]"
pn_description "/td[3]"
pn_oem "/td[5]"
end
end
#
end
part_data = digikey_data.to_flat_hash
pp part_data

The result:
sin-gwest-laptop:digikey gwest$ ruby digikey.rb
[{:pn_oem=>"Kemet",
:dist_pn=>"399-1168-2-ND",
:manuf_pn=>"C0805C104K3RACTU",
:pn_description=>"CAP .10UF 25V CERAMIC X7R 0805"},
{:pn_oem=>"Kemet",
:dist_pn=>"399-1168-1-ND",
:manuf_pn=>"C0805C104K3RACTU",
:pn_description=>"CAP .10UF 25V CERAMIC X7R 0805"},
{:pn_oem=>"Kemet",
:dist_pn=>"399-1168-6-ND",
:manuf_pn=>"C0805C104K3RACTU",
:pn_description=>"CAP .10UF 25V CERAMIC X7R 0805"}]
sin-gwest-laptop:digikey gwest$

ActiveScaffold - Example of using Roles for security

2009-02-19T17:44:00.001-08:00

Recently I was ask for examples of using rolerequirement with activescaffod,
so here is the examples.

For security, two areas to consider.

1. Manage what menus appear.
2. Verify access in controllers.

You can also control what columns appear, and what actions are available as well. While I have done this all in one controller, to follow DRY, I find that having a adminspace controller, and a userspace controller a bit cleaner.

Note if you want to configure activescaffold dynamically, AS does some caching, and you need to specifically reconfigure AS each time, to make sure
of cache consistency.

For menus, I use widgets/tabnav

<%
# this partial renders a tabnav, you can call it in your views with:
# <%= tabnav :admin % > (just the tabnav)
# or, if you want a boxed tabnav:
# <% tabnav :admin do % >
# your html here
# <% end % >
# (remove the space between % and >, we don't want to mess up your brand new tabnav :-))
#
# you can pass render_tabnav a few options:
# :generate_css => true|false #=> generates a default inline css for the tabnav, defaults to false
# :html => aHash #=> sets html options for the tabnav's div (es :html => {:class=> 'myCssClass', :id=>'myCssId'})
#
render_tabnav :admin,
:generate_css => true do
if @current_user.has_role?('admin')
add_tab do |t|
t.named 'Admin'
t.titled 'Administration'
t.links_to :controller => 'adminspace/admin'
end
add_tab do |t|
t.named 'Users'
t.titled 'Users'
t.links_to :controller => 'adminspace/user'
end
add_tab do |t|
t.named 'Roles'
t.titled 'Roles'
t.links_to :controller => 'adminspace/role'
end
end
end

2. For the controllers:

class Adminspace::AdminController < ApplicationController
layout 'tier2admin'
before_filter :login_required
require_role "admin"
#
# Generated by GenRails
#

active_scaffold :Logmsg do |config|
config.label = 'AdminMessages'
config.actions = [:search, :show, :list]
end
end

Converting a rails app to db2

2009-02-08T00:28:00.001-08:00

DB2 has a long history, having ran on IBM mainframes, and mid-range for decades. Now it easy to move your rails app using it.

The lovely thing about the latest version 9.5 DB2 is that you can use
it for free, if your using the DB2-ExpressC version. It has most all the features you will need, and the database size is not limited. You can add support, or add-ons easily.

The issue you will have, if you have a existing rails app, is moving your data from mysql, or other rails database into db2.

The idea solution for this is a rails plugin, called YamlDb.

In Rails 2.1 its as easy as:
script/plugin install git://github.com/adamwiggins/yaml_db.git

rake db:data:dump
rake db:schema:dump

then update your database config file to point to db2:

development:
adapter: ibm_db
database: baseapp
username: gwest
password: xxxxxxxx

The rake db:create does not work in db2, so:
db2 create database baseapp

Then:
rake db:schema:load
rake db:data:load

And your ready to go.

Not that database names can only be 8 characters, and underlines and spaces may cause you problems. Otherwise, its painless.

Mac DBI - dbi binary will not list gem drivers but will work

2009-02-01T19:27:00.001-08:00

Thanks to the creator of DBI, I found out that if you install gem based dbi drivers, that the dbi command will not list them, but they will work properly in your ruby code.

So to list tables in our sqlite3 development database.

require 'rubygems'
gem 'dbi'
require 'dbi'
require 'pp'

# Assume dbd-sqlite3 is installed
# sudo gem install dbd-sqlite3
# Assumes we are in our project directory

fromdb = DBI.connect('dbi:SQLite3:db/development.sqlite3', nil, nil)

tables = fromdb.tables
tables.each { |tablename|
pp tablename
}

I tend to use DBI in migrating from foreign environments into a rails app.
I also use DBI to write automatic generators for controllers, models, and
menus. It really makes getting going on a legacy app very fast.

Ruby DBI does find DBD drivers on Mac when using Gem

2009-01-25T02:08:00.000-08:00

I found a very nice problem when trying to use DBI on a mac, with the sqlite3 driver
I've been a avid your of dbi, which is a direct database API in Ruby, that allow
low level access to databases.

I've used it on Linux, and Windows against Oracle extensively, and several other databases.

So expected it to work the same. Seems like I hit a small problem in that oh.

My script was telling me no driver. When I ran the dbi command, it showed there was no drivers.
If I used irb, and did a DBI.drivers_available, it told me none.

First, a quick review of the installed gems:

sh-3.2# gem list

*** LOCAL GEMS ***

actionmailer (2.2.2)
actionpack (2.2.2)
activerecord (2.2.2)
activeresource (2.2.2)
activesupport (2.2.2)
dbd-sqlite3 (1.2.4)
dbi (0.4.1)
deprecated (2.0.1)
gem_plugin (0.2.3)
hpricot (0.6.164)
mechanize (0.9.0)
nokogiri (1.1.1)
rails (2.2.2)
rake (0.8.3)
scrubyt (0.4.06)
sqlite3-ruby (1.2.4)
wxruby (1.9.9)

Yep, Its there, everything looks ok.

Now lets track down the source to find out what is going on.
# Returns a list (of String) of the currently available drivers on your system in
# 'dbi:driver:' format.
#
# This currently does not work for rubygems installations, please see
# DBI.collect_drivers for reasons.
def available_drivers
drivers = []
collect_drivers.each do |key, value|
drivers.push("dbi:#{key}:")
end
return drivers
end
That looks ok.

Lets go and see what collect_drivers()
#
# Return a list (of String) of the available drivers.
#
# NOTE:: This is non-functional for gem installations, due to the
# nature of how it currently works. A better solution for
# this will be provided in DBI 0.6.0.
def collect_drivers
drivers = { }
# FIXME rewrite this to leverage require and be more intelligent
path = File.join(File.dirname(__FILE__), "dbd", "*.rb")
Dir[path].each do |f|
if File.file?(f)
driver = File.basename(f, ".rb")
drivers[driver] = f
end
end

Ok, the FIXME gives us the clue.
Looks like it does not bother to search thru rubygems.
You must manually copy the dbd drivers into dbi.

The Solution:
Copy the drivers into the dbi gem.
cd /opt/local/lib/ruby/gems/1.8/gems/dbi-0.4.1/lib
cp -r ../../dbd-sqlite3-1.2.4/lib/dbd dbd

A Review of Authentication and Access Control for Ruby On Rails

2009-01-24T17:37:00.001-08:00

A review of Authentication and Access Control for Ruby On Rails

Previous Articles That Are Related
http://mentalpagingspace.blogspot.com/2007/11/getting-started-faster-in-rails.html
http://mentalpagingspace.blogspot.com/2008/12/rails-to-windows-integration-single.html

First, to give some context, when building Rails apps, its common practice to use
a common set of plugins. In "EDGE" rails, there is even a template system so that
when you create a rails apps, it can load your standard plugins.

For my basic set, I use the following:

My base set is:
ActsAsAuthenticated (Login)
http://www.railslodge.com/plugins/1-acts-as-authenticated
RestfulAuthentication (The Replacement for the above)
http://github.com/technoweenie/restful-authentication/tree/master
RoleRequirement (Access Control in controllers and menus)
http://code.google.com/p/rolerequirement/
Tabnav/Widgets (Menus)
http://wiki.github.com/paolodona/rails-widgets
ActiveScaffold (Controller/View for CRUD)
http://activescaffold.com/

In rails there are "two" different problems, first there is authentication, then there is access control.

Authentication is:
1. How do we login to the app
2. User Model
a. User Name
b. Email Address
3. What Access do we have? (Roles)

Do do the heavy lifting for this, we use a plugin

There are several different plugs for authentication:
1. ActsAsAuthenticated (The original)
3. RestfulAuthentication (The new "standard")
2. OpenId
3. NTLM (Use this when your app is only in a ActiveDirectory environment)

So once your logged in, using one of the above, we ad the rolerequirement plugin.

I find RoleRequirement handles my access control very nicely. It enhances your user model, with the ability to have
multiple role. You can then use the "role" to verify in your menus and controllers if a user can perform or use
certain actions.

If you look at rolerequirement, in your code, you can do something like the following:

class Adminspace::AdminController < ApplicationController
before_filter :login_required
require_role "admin"

layout 'basic'
def index
end
end

Notice the require_role above.
When you setup rolerequirement, You can login to your app, and add additional roles.
You can also add a migration to add these roles. (Useful in a empty database setting.

Also notice the before filter. This makes sure the user is logged in.

The next issue you will get is, updating menus on the fly.
I use tabnav typically for my menus.

So in order to make menus come and go based on roles, you can check roles in tabnav.

<%
# this partial renders a tabnav
render_tabnav :aircenter,
:generate_css => true do

add_tab do |t|
t.named 'ReqCenter'
t.titled 'ReqCenter'
t.links_to :controller => '/reqcenter'
end
if @current_user.has_role?('admin')
add_tab do |t|
t.named 'Admin'
t.titled 'admin'
t.links_to :controller => 'adminspace/admin'
end
end
add_tab do |t|
t.named 'Requests'
t.titled 'Requests'
t.links_to :controller => 'requestspace/requesthome'
end
add_tab do |t|
t.named 'Logout'
t.titled 'Logout'
t.links_to :controller => 'adminspace/account', :action => 'logout'
end
end
%>

A example of a Admin CRUD, using ActiveScaffold:

class Adminspace::UserController < ApplicationController
layout 'tier1admin'
before_filter :login_required
require_role "admin"

active_scaffold :users do |config|
config.list.columns = [:id,:display_name,:businessunit,:department,:location]
config.columns[:businessunit].ui_type = :select
config.columns[:location].ui_type = :select
config.columns[:department].ui_type = :select
config.columns[:roles].ui_type = :select
end
end

So now, using namespaces, we can have a admin controller, with a role of admin for access, that allows the admin
to change anything in all users, and another controller, that allows only the user to change his own fields.

Rails verse Microsoft Access

2009-01-19T05:33:00.000-08:00

Like everyone with significant Microsoft experience I've done some Access Applications.
The reality is, I'd prefer to do it in Rails. Ruby on Rails let me gain the power of using
any ruby gem or rails plug-in. I have access to practically every database in Rails.
And I have access to nice Menus as well. Plus the ability to define a web api,
using REST.

For me, for a small app, I can host it as a windows service directly on Windows with Mongrel. I can
use single sign-on, so if your in the domain, your automatically authenticated. Using role-requirement,
I can define user roles, and even have dynamic menus.

Even better I can use PRAWN or Pdf::Writer to directly generate reports. Or Scruby to scrape web pages.
And Add to the Data on the fly.

The other thing I've seen with Access applications, it too easy to get the file locked by one user, and
no one else able to access it. Scaling on Rails is Easy compared to Access. And on top of all of this is
easy cross platform functionality.

The nice thing is that the language of Rails is Ruby. And Ruby is significantly easier and faster to develop on than VBA.

Windows 7 - The Bleeding Edge

2009-01-15T22:21:00.000-08:00

Well I finally bit the bullet. While I had tried vista on my work laptop, and after sweeting it for 3 months, went back to xp.
Microsoft Windows 7 seemed to be the answer to what I was looking for.

So I finally decided to use the Beta on my main gaming machine. My main work machine is a MAC.

First, Lets see how well the beta behaves. So I created a new VM on my Mac, and did the install from the ISO.
Wow, it went smooth and easy. Very fast install compared to XP. And so much better than Vista.

So after playing a bit with that, I decided that my gaming machine needs a cleanup. So its a great time to
give Windows 7 a try. In my house I currently support 2 desktop for my kids under Vista, another of the same desktop for my wife, and a desktop for myself, plus a laptop. I've migrated most things to my MAC.

For me on the Gaming Machine, WOW is very important. Vista generally left something to be desired on Gaming performance.
So Windows 7, I thought there was hope.

So first making sure that I had copies of everything, I decided to do a "new" install without formating the disk.
(Not something I generally like doing by the way). Windows 7, renamed the windows folder, and moved the Document and Users, as well as the Program files under the Windows.old. I like that implementation. I then went into the Windows.old, and dragged the documents folder back to the Windows 7 location. Wow, it noticed and put the photos in the right place.

I then turned on my Pixma 5000 USB printer, and Windows 7, went out and grabbed the right printer driver and installed that as well. Nice.

Now to do a first go/no-go test. I sent a shortcut from my WOW.exe to the desktop. And ran it. It worked right out of the box.
WOW client is actually very good at copy and go. And with Windows 7 it was no different. Good job Microsoft. I'll do some detailed testing tonight, but so far the performance and drivers look good

Now lets move on and try a older version of office. Installed was clean.
And to my shock, the old version of Office was FAST. Now that is a improvement.

So I'll update the post over the next few days. I really want to try some of the new VM offerings from VMware,
and Microsoft.

And Max ask how to find the user name? (Automatic Logon)

2009-01-08T22:46:00.000-08:00

This answer varies depending on if you doing linux or windows.
The key is NTLM. If the client is coming from the local net, and part of a domain,
you can use NTLM based login, this is also know as Integrated Windows Authentication.

If you using NTLM to automatically logon, there is a nice plugin you can using with
mongrel on windows.

See my post
http://mentalpagingspace.blogspot.com/2008/12/rails-to-windows-integration-single.html

This is idea for light-duty applications hosted on a windows machine or vm. I use this when my
application needs to make WMI or other windows specific calls. Your application login
is automatic if the user is coming from the local network and part of a domain. The mongrel
can run as a service that auto starts at boot time.

In Linux you can also do the same trick, there is a apache plugin to allow you to use NTLM.
Since I've seen apache articles on using NTLM in a domain, then I'll leave those to the reader.

And as a note, I prefer LINUX for applications that do not need significant usage of Window API.
Its faster and more stable than running it on XP. My fav at the moment is to use Passenger to manage
linux hosting of Rails

On the other hand, If you have a app a handful of people are going to use sporadically, which
is common in the enterprise. A XP based Implementation has been problem free for me.

Virtual Manufacturing Control - VmWare To the Factory Floor

2008-12-30T04:19:00.001-08:00

In the industry, VmWare has taken over the Data Center. Server Consolidation has been going on for a while. Multiple servers
that used to run on dedicated hardware, are now Virtual Machines, multiple setting on one rack or blade server. Its not uncommon to see 10's to 100's of virtual server running on one physical server. This has resulted in great cost saving, easy upgrades, easier provisioning, and a reduction of power consumption. This technology is proven, and works.

To give you a example, before VmWare, it would take a week to get a quote, get the approvals for purchase, then another week for delivery, and then a day to set it up, install the OS, and then your ready to start thinking about putting a application on the server. You also had to worry about backup, and KVM (Keyboard and monitor switching).

With VmWare, I log into VmCenter, create a new Virtual machine, selecting how much memory and CPU I need, and pointing it to a ISO image of the OS. And Say Go. The new machine "boots" up, and starts installing the OS. Since a Dell Blade, or 1U server could support upwards of 32 machine, the physical act of buying hardware is reduced significantly. The reality is, most servers are under-utilized, and VmWare lets us use these otherwise wasted resources better.

Another layer of this puzzle is high-availability. We really dont want our applications to go down. It doent matter if its a DHCP server, or a Oracle Database, being up means people are working, and having a hardware issue can really spoil your day when you manage 100's of servers, and almost as many applications. The answer to this is has a few components.

First, we break the Storage component using iSCSI into a separate device. The RAID is now a external device we talk to over TCP/IP using SCSI over our ethernet connections. With 10Gig Ethernet, and hardware based switching, IP is the fastest SAN. Most blade give us 4 1-Gig links which allows 512Megabytes a second of capacity for getting data in and out. This concept means our blades or rack based servers now have no disk locally.

This separation means we can now migrate any virtual machine into any physical cpu, using what VmWare calls vMotion. So if you want to pull a blade server, and replace memory, or reprogram its bios, you can use VmCenter to move a running virtual machine to another physical cpu, without the application even knowing it. In the event of a failure the VM can be run on any physical CPU available. Also Snapshots can be taken, and the allows a virtual ghost type functionality where we can take a virtual photo of what is running, and go back to it at any time.

In the above example, it could have been even easier. In that you load your server-os into a VM, do your patching and updates, then convert it into a template. After you have a template, you can then create more servers using it by just cloning it, and giving it a new name.

None of this so far is new. I've been doing this for a whle. The next step is called VmView.

In manufacturing for example, its not uncommon that you need PC near or in the production areas. So for example lets say we need 320 PC to handle our ERP or SAP type application. These would be scattered over a large area of factory floor. The condition in a factory are not office environment. Heat, Temp, and Humidity are not comfortable for the workers, and worse for the equipment. There are other problems with this approach. 1) Updates - While WSUS and Microsoft tries to automate this, all to often you end up walking to every pc. 2) Security - USB Drives, Thumb Drives, CD-ROM, Phones, and Ipod are all avenues for data leakage, as well as virus to come into the system. The solution is a thin-terminal.

Traditionally with thin-terminals, you need a terminal-server. A terminal server gives you a shared environment to run your applications. This works reasonably well, but not every application can install in such a shared environment. Forcing us back to the common pc approach. With VmView, we can now apply the same technology from the DataCenter to the desktop.

We can create one or more template machines, that are automatically created, re-used, removed or restored that run in the data-center. We can choose what USB devices are connected to the VM on the fly. This allows us to support scanners, barcode guns, rfid readers, and printers that are connected via thin-terminals to the VM in the data-center.

To give a example. For 320 Users we could put those on six M905 Dell Blades. Each user would be allocated a 20Gig Virtual hard disk. Using templates, the OS and common apps could be shared. We would customize AD to do small roaming profiles to avoide any corruption issue, and re-direct MyDocuments etc to proper file server. Now we can "install" a new pc for a new staff in a minute instead of hours. We can snapshot, audit, and reboot all centrally. We can update and touch every pc in the shop with a script easily. And we can deploy new patches, even change from XP to Vista in minutes.

Even changing the desktop hardware is easy. We buy new servers for VmView. Mount them in the data center, and vMotion them over to the new servers.

Using proper network design, we can have a high-availability redundant solution from server to desktop onto the factory floor that is secure and easy to maintain.

Ruby on Rails - Are you a Member?

2008-12-10T23:51:00.001-08:00

Large Enterprises that use Windows are typically in a Domain environment.
Often application access is controlled by group membership in ActiveDirectory.

The easy and fast way of finding this out, is via ruby-net-ldap, a native ruby implmenation.

So first install ruby-net-ldap
gem install ruby-net-ldap

You will get:

C:\>gem install ruby-net-ldap
Successfully installed ruby-net-ldap-0.0.4
1 gem installed
Installing ri documentation for ruby-net-ldap-0.0.4...
Installing RDoc documentation for ruby-net-ldap-0.0.4...

Example:
ad = ActiveDirectory.new('user','password','server','dc=company,dc=com')
pp ad.GetMembers('user1')
pp ad.GetMembers('user2')
if ad.MemberOf?("user1","myappgroup")
puts "YES We can do work"
end

You will need the following library:

#lib\active_directory.rb
require 'rubygems'
require 'net/ldap'
require 'pp'

class ActiveDirectory

#server has to be full name.domain.com
#treebase is dc=domain,dc=com
def initialize(username,password,server,treebase)
@username = username
@password = password
@server = server
@treebase = treebase
@ldap_con = Net::LDAP.new( {:host => @server, :port => 389, :auth =>
{ :method => :simple, :username => @username, :password => @password}})
end

def cleanup_members(members)
mymembers = Array.new
mymembers << '-none'
members.each do |member|
cmc = member[3,64]
cmcs = cmc.split(',')
mymembers << String.new(cmcs[0].delete(' '))
end
return(mymembers)
end

def GetEmail(username)
op_filter = Net::LDAP::Filter.eq( "samaccountname", username)

entries = @ldap_con.search( :base => @treebase, :filter => op_filter,:attributes=>
['samaccountname','mail'])
entry = entries[0]
emailaddress = entry.mail
return(emailaddress)
end

def GetMembers(username)
op_filter = Net::LDAP::Filter.eq( "samaccountname", username)

entries = @ldap_con.search( :base => @treebase, :filter => op_filter,:attributes=>
['samaccountname','memberof'])
entry = entries[0]
begin
themembers = entry.memberof
rescue
themembers = Array.new
end
membership = cleanup_members(themembers)
return(membership)
end

def MemberOf?(username,thegroup)
groups = GetMembers(username)
return groups.include?(thegroup)
end
end #ActiveDirectory

Rails to Windows Integration - Single Signon with NTLM and Mongrel

2008-12-10T18:18:00.000-08:00

I was moving a application for network management in Rails into a production environment.
It had originally started out as something for my own use.
Of course I did not do security for a single user app.

In previous apps I'd used authenticated system with ldap to give a sign-on.
I wanted easier and tighter integration. NTLM comes to mind, as you can just clink a link,
and your windows login is re-used for your web application.

The problem has always been that mongrel does not support NTLM.
Well that solved now. Thanks to the work of Seggy Umboh, there is a new
NTLM filter for mongrel.

http://github.com/secobarbital/mongrel-ntlm/tree/master

This actually passes you the windows domain user name
directly into your controller.

To get mongrel to use the filter, you must pass a config file, as
documented in the readme.

Then you need a few things in your app:

First in your routes:

map.connect 'ntlm/login', :controller => 'account', :action => 'login'

I added this so the ntlm mongrel filter will chain to my account controller. That way
your "user" controller can remain unchanged. I just prefer to keep them separate.
Easier to add into existing systems that way.

The in the account controller you need the following:
class AccountController < ApplicationController
layout 'tier1'

def login
my_user_name = request.env['REMOTE_USER']
if my_user_name.nil?
session[:logged_in] = :false
return
end
myuser = User.find(:first, :conditions => ["name = ?", my_user_name])
if myuser.nil?
myuser = User.new
myuser.name = my_user_name
myuser.save
end
session[:current_user] = my_user_name
session[:user_id] = myuser.id
session[:logged_in] = :true
redirect_back_or_default('/myapp')
end
end

Then you need a little helper lib. This one is cut down from authenticated system.
This should be in you lib directory.

#ntlm_system.rb
module NtlmSystem
protected
# Returns true or false if the user is logged in.
# Preloads @current_user with the user model if they're logged in.
def logged_in?
if session[:logged_in].nil?
return false
end
if session[:logged_in] == :false
return false
end
return true
end

def login_required
if logged_in?()
return
end
pp "Access Denied"
access_denied()
end

# Redirect as appropriate when an access request fails.
# Our application only uses ntlm based login - you must be part of the domain
def access_denied
redirect_to('/ntlm/login')
end

# Store the URI of the current request in the session.
#
# We can return to this location by calling #redirect_back_or_default.
def store_location
session[:return_to] = request.request_uri
end

# Redirect to the URI stored by the most recent store_location call or
# to the passed default.
def redirect_back_or_default(default)
session[:return_to] ? redirect_to(session[:return_to]) : redirect_to(default)
session[:return_to] = nil
end
end

Finally in your application.rb
# Filters added to this controller apply to all controllers in the application.
# Likewise, all the methods added will be available for all controllers.

class ApplicationController < ActionController::Base
helper :all # include all helpers, all the time
include NtlmSystem # Add this for NTLM to work
# See ActionController::RequestForgeryProtection for details
# Uncomment the :secret if you're not using the cookie session store
protect_from_forgery :secret => '6e2f000403020a0d9041a7bfb069239d'

# See ActionController::Base for details
# Uncomment this to filter the contents of submitted sensitive data parameters
# from your application log (in this case, all fields with names like "password").
filter_parameter_logging :password
end

When you start your application, make sure you do it like this:
mongrel_rails start -S config\mongrel_ntlm.conf

After reading, on NTLM, and playing with it, I found the URL are a bit picky, if you
want to use the app, without needing to key your username and password

http://servername:3000/ work perfectly fine.

If you put the whole name, it tends to ask you the user name and password

Simple Rails Goodness - Annotate_Model

2008-12-08T16:28:00.000-08:00

Ever have trouble remembering the fields in your rails models?
Now you dont have too.

This is really great time saver. Used to I'd keep up a SQL query windows
just to be able to look at fields. Now with this plugin I dont have too.

gem install annotate-models

Will get you the gem version.

Usage:

cd [your project]
annotate

And you will get something like:

# == Schema Information
#
# id :integer(11) not null
# quantity :integer(11)
# product_id :integer(11)
# unit_price :float
# order_id :integer(11)
#

class LineItem < ActiveRecord::Base
belongs_to :product

end

This is so Useful when your writing model code

http://github.com/ctran/annotate_models/tree/master

RPG really still exists

2008-11-23T19:54:00.001-08:00

I was looking thru some surveys and surprised that in hometown of Cleveland/Chattanoga TN, one of the more common environments is AS/400
or iSeries midrange machines. And the language of choice is RPG.

I've done a few projects in my early years with RPG, find it incredible that
its still alive and kicking.

The nice thing is now, there is ruby support on AS/400(iSeries) as well as
database support.

Using this, new applications can be developed rapidly. They can be viewed from standard pc, or mac, inside or if you choose outside the company. One of the
the truely powerful elements is creating Iphone applications, that can be access in a mobile environment. Using the DB2 adapter, exiting applications can be enhanced
or new functionality added, without changing the core infrastruture.

http://www.ibmsystemsmag.com/i5/july07/tipstechniques/16330p1.aspx

Why Use Ruby? Why Not Java? Why Not C?

2008-11-03T17:09:00.001-08:00

Its a common question, that is always ask when you start a project. What
tool to use. Having long experience, with all languages practically, I've programmed
in multiple assemblers, Basic, Pascal, Cobol, Fortran, RPG, VB, C, C++, Java Python, VHDL, Veriflog, and Ruby.

I agree with the comment that CPU get cheaper, and people get more expensive.
Also timelines are decreasing. Market timeframes get smaller. Also for me, as a
senior manager of technology, I have no time for the likes of C. (Yes I need
to go back and do a project just to keep the skill, that is coming soon)
But if I want a functioning prototype system, and I need it in a day, even
at my best I cannot do it in C. I can do it in Ruby ( on Rails).

Then I can go in the meeting with my programming staff, and tell them, ok this
is how I expect it to work. At that point, little room for discussion. With Ruby
you can do a system in the time it takes to do the powerpoint to describe it. Thats
a major statement as I eat and breath powerpoint.

The other thing that people know ruby for is perfomance, but to a large degree that is just startup time. With the new implementations using VM based technology there is no reason ruby cannot run on a 8 bit processor, and now we have 64bit ones almost for free.

So next time you need to fix 20 year old code, consider using Ruby, some nice plugin and gems, and a few days of your time. Its a lot better than spending 5-10 man years in java.

Reference:
http://engineroom.mixx.com/2008/08/10/ruby-on-rails-vs-java-vs-c-vs-assembler/

Solving Large Scale Company Software Issues

2008-11-02T17:24:00.000-08:00

What to do when your dealing with complex enterprise issues,
with multiple legacy systems?

If you look at most large companies today, you find a long organic history
of IT software and hardware.

The hardware side of things is easy enough to fix over time. I find that
DELL has fastastic warranty, and great service, and moderate price. Get a
five year warranty, and forget about hardware problems for your servers and
pc's. Likewise on your network gear, Cisco does a great job.

That solves infrasturure, but not the complexity of software.
Some have current investment in java, some even in cobol still.
I could write a 100 page essay on the wide variety of systems in
large companies.

Going forward, what you want is the ability to integrate and have a standard
across the enterpise. As CTO and Technology managers, we need something that
will tie everything togeather. Something that will last, and has good performance
and that we will not have to spend 100Million on just to get thing consistent.

For this "glue" consider using Ruby on Rails. Its mature, It fast, and its efficient.
It can scale from 1 User to Millions of users. And it can be developmed in small
to large teams.

Add to this a development center in the phillipines for even more cost effective solution. The cost of english speaking staff in Manilla is 1/3 that of the US.
The people are smart, and hard workers. With good video conferencing, and
VOIP telephony they can be virtually down the hall from you. And my experience
they can work on US timezone allowing for no time lag.

Be Gentle Admins(Gods) - Be slow to judge

2008-09-22T17:07:00.000-07:00

I recently tried the site getacoder.com The site offers contractors/programmer/engineers for work-for-hire, at good rates.
Thats the good news. The bad news, is there administration is heavy
handed to say the least.

I posted a request, and having done lots of outsourcing gave as much
information as I could, even siting the a open-source reference board.

Seems like the terms of service specify that you cannot provide contact info,
which I'm perfectly fine with. But the Open Source project has contact info for there project. And within a hour of posting, my account was suspended, for violation of terms-of-service.

I've done no violation of terms of service. As my contact details were not included.

-----------------------------------------------
Dear Glenn,

Your account was suspended due to the fact that you have provided contact information in a project page; Please be reminded that this is not allowed by GetACoder.

Statements like: \"For any questions, concerns, or issues submit them to gerald@BeagleBoard.org\"- Will not be tolerated.

If you wish to reactivate your account send us an email, informing that you understand our Terms of Use agreeing to abide by them at all times.

--------------------------------------------------

Funny dont you think that the person knows my name is Glenn, yet she suspended my
account for a unrelated person called gerald.

I've withheld her name to protect the guilty.

Its now been more than 16 hours, account suspended, with no response.

So just to make sure.

I Am Informing Getacoder.com that I AGREE TO THE TERMS OF SERVICE, HAVE NOT, NOR WILL EVER VIOLATE THEM IN THE FUTURE.

Finally I get a response:
---------------------------------------------------------------------------------
Dear Glenn,

Glad to know that you are aware of the situation and plan to follow a professional conduct with GetACoder. We have reopened your account.

Feel free to contact us if you have further questions.

NOTE: If you wish to send additional information regarding this ticket, please do not create a new email. Instead, reply to this one and do not change subject line.

Sincerely,
Anna Kesel
---------------------------------------------------------------------------------

I really dont know about these people. The quality of responses is 4 so far, 2 of
which were proposing web designs for a hardware design, ie they dont have a clue
of what is being talked about. Waiting on the other two to "bid" and see what the price is. Since this is a hobby project, cannot be so high.

Making Polywell Magnets Using Ceramics and Liquid Metals

2008-09-14T00:30:00.000-07:00

One of my areas of interest is fusion. Since tradition tokamak fusion devices dont seem to have a future. I tend torwards the Inertial Confinement Fusion Camp, with
Polywell based technology being the area I believe would work the best with the lowest cost.

One of the cost issues of polywell that needs to be solved from engineering point of view is the magnet design. In polywell you use a sperical array of magnets to create a virtual grid.

The magnets needless to say need to create a strong magnetic field. So for continious operation you really need cooling. Water does not work because its going to flash to steam way to easily.

The concept I had was to use EoPlex Technologies (http://www.eoplex.com/index.html) to create a ceramic layered magnetic structure combining the magnet and a integrated channel for ibm's liquid metal as well as the copper magnetic wire. The magnets mounted in the reator chanber would connect to a liquid metal circulation system, carrying the liquid metal to a large heat exchanger capable of 1600 degree's C down to 85 degree C, as stated the IBM article.

Since the liquid metal has magnetic properties we may even be able to get rid of the copper conducter and have a liquid conductor. Need a bit more research, but believe this may offer the ability to produce 100Megawatt plants without the need to have cryo cooling.

http://www.nextenergynews.com/news1/next-energy-news5.16.08d.html
http://www.eoplex.com/index.html
http://www.talk-polywell.org/bb/index.php
http://en.wikipedia.org/wiki/Polywell

Questons Questions Questions - A meme inspired by David Mohundro

2008-09-09T00:15:00.000-07:00

How old were you when you started programming?

15 Years old.

How did you get started in programming?

After following many articles in Byte Magazine. (Where I learned electronics
from the wondering writing of Steve Ciarcia (http://www.circellar.com/)
I saved my pennies and got me the first TRS80 PC from Radio Shack.

I did lots of small apps, like programs to teach me primary school spelling. Speach Output. Light Pen Input.

Then started doing commercial projects around 16 years old

What was your first language?
BASIC (Microsoft Basic)
Z80 Assembler

What was the first real program that you wrote?
Real Program, means I got paid for it.
First paying job was a mail-merge program for the TRS80 Word Processor
Followed by a automation system for a beer disti.

What languages have you used since you started programming?
Cobol
Fortran
RPG
C
TurboPascal
8086 Assembler
68K Assembler
8031/8051 Assembler
VB
C++
Java
Python
Ruby

If you knew what you know now, would you have started programming?
Absolutely. But I'd skip down to ruby.

What's the most fun you've ever had... programming?
Several that come to mind.
1. LightPen App for the TRS80.
2. Banking Analysis APP - Took it from a month of run-time to a weekend.
3. PDFtoRuby - A Ruby Gem to create ruby code from pdf files
4. Network Monitor APP I did recently in Ruby/Rails

I can’t think of any one specific instance, but I think one of the best feelings is, after having spent literally hours trying to debug some problem and then giving up and going home, waking up the next day and having the light bulb come on with the solution to this problem. I love solving problems with software.

Tag, you’re it!

Ruby and Rails can do almost anything

2008-08-21T16:55:00.000-07:00

I'd like to expose a few things that Rails is not just for Web Applications.

By definitions of traditional web applications I mean, your traditional shopping cart, or social networking sites.

Applications I've done with Rails

Network Management - This maps all nodes in a network, grabs there macs, find what ports they are on switches, gets there service tag, and even for phones find outs there extensions. Yes you use a IE or Firefox to access it, but its not a social networking site

Regional Aviation Maint. Mangament Applicaton
This manages the overhaul process for a company that repair aircraft components and does the regulatory paperwork. Its a very controlled environment. Again not your normal shopping cart

A Proof-of-concept Data Deduplication Backup system
This uses ActiveRecord and sqlite3 to do the tracking part of the backup. And Rails to be a GUI for the server.

Embedded Device Managment.
In my hobby, I'm building a 360 degree camera.
All the device management, and the gui code is done using ruby and
rails, running on a tiny avr32 cpu.

Database Conversion(s) - More than I care to count

Thing I could do with Rails
Any Commecial Program that you can think of. From Cash Registers in food courts, to Factory Management applications

So you see, Rails is not just about yet-another-web-site. Its a GUI for any application you want to write.

WIth Ruby maning the heavy-lifting, and Rails handling the GUI, there very few things that cannot be done with the combination.

dotr - drawing network maps

2008-08-21T00:48:00.000-07:00

Ever want to draw a network map.
Seems like there is a "great" way to do this.
The ruby 'dotr" library. It can be used for diagraming code, to diagramming
the whole internet.

DotR provides a simple Ruby interface for contructing directed graphs in a natural syntax and plotting them via the 'dot' utility to PNG, PS and other graphic formats.

the ruby dotr drives Graphviz Which is a mapping
application done by ATT. Its open source, and you can "write" dot files using
the dotr gem.

I call the SwitchGraph in the "background" of a "rails" app, generating "maps" of all my network switches. In my rails app, they appear to "update" by themselves.
I use Rmagick to create smaller thumbnails, that can be "clicked" to a large version.

Example of driving dotr/graphviz

#switchgraph.rb
require 'tempfile'
require 'dotr'
require 'RMagick'
include Magick

class SwitchGraph

def initialize(theswitch)
@me = theswitch
end #init

def graph()
d = DotR::Digraph.new(@me.name,:rankdir=>"LR",:ranksep=>"2.75") do |g|
switchname = @me.name
switchlabel = @me.site.name + "/" + @me.name
g.node(switchname, :label => switchlabel)
@me.switch_ports.each do |port|
port.macs.each do |mac|
if Device.exists?(mac.device_id)
begin
nodename = mac.device.name
nodelabel = mac.device.name + "(" + mac.ip + ")"
rescue
nodename = "-Unknown-"
nodelabel = "(" + mac.ip + ")"
end
g.node(nodename,:label => nodelabel) do |n|
portlabel = port.name
n.connection(switchname, :label => portlabel)
if mac.device.user
username = mac.device.user.name
userlabel = username
g.node(username, :label => userlabel) do |u|
u.connection(nodename)
end # do u
end #mac.device.user
end #g.node mac
end #if mac.device
end #port.macs.each
end #@me.switch_ports
end #DotR

#C:\projects\airstate\public\switch\graph\4
thebase = 'public/switch/graph/' + @me.id.to_s + "/"
thefilename = thebase + @me.name + ".png"
thumbdir = thebase + "thumb"
FileUtils.mkdir_p thumbdir

File.open(thefilename, 'wb') { |f| f.write(d.diagram(:png)) }
img = Magick::Image::read(thefilename).first
thumb = img.resize(250,250)
thumb.write thumbdir + '/' + @me.name + ".png"
end

end #class switchgraph

Scraping Web Pages with Soup

2008-08-21T00:41:00.001-07:00

I wanted to "grab" the contents of a "management" web page off a cisco 7941 phone.

After reading up on the variety of ruby gems to use for scrapping.
My choice was "soup".

http://www.crummy.com/software/RubyfulSoup/

This seems to work nice. I've managed to pull serial numbers, extensions, and
phone names as well as models from close to 100 phones this way.

This is a "class" for getting the phone information.

#cisco_phone.rb - Class to "Scrap" Cisco Phone
class CiscoPhone
require 'rubygems'
require 'open-uri'
require 'rubyful_soup'
require 'cgi'
require "pp"

def initialize(theip)
valuestack = Array.new
@values = Hash.new
url = "http://#{theip}"

begin
open(url) {
|page| page_content = page.read()
soup = BeautifulSoup.new(page_content)
result = soup.find_all('td')
valuename = nil
result.each do |item|
content= item.b
if content
value = content.string
if value
value.strip!
if !valuestack.include?(value)
valuestack.push(value)
end
end
end
end
}
rescue
@values = nil
return nil
end
thename = nil
thevalue = nil
valuestack.each do |content|
if thename.nil?
thename = content
else
thevalue = content
@values[thename] = thevalue
thename = nil
end
end

# Validations
if !@values.has_key?('Model Number')
@values = nil
return nil
end
if !@values.has_key?('Phone DN')
@values = nil
return nil
end
return self
end

def isphone?()
if @values.nil?
return false
else
return true
end
end

def getvalue(thename)
if !@values.include?(thename)
return nil
end
return @values[thename].downcase
end

end

#Test Cases
#thetest = CiscoPhone.new("192.168.1.20")
#pp thetest

Data Deplucation In Ruby - Backup Terabytes In Minutes

2008-08-21T00:15:00.000-07:00

Recently I went looking at the "perpetual" problem of "backups".

I wanted to "know" how some of the commerical products, do there "magic".

So I wrote a "simulation", in ruby, with activerecord for my data storage.

The theory of this, is your "average" server/workstation has "lots" of duplicated files. They may have "different" names, and be in "different" directories, but the
contents is the same.

This is even more the case spread over multiple machines, or for machines that have been around for a while.

I created the following script to "explore" dedup backup.

First, terminology explained:

1. Chunk is a "unique" file, as indentified by a "hash" - In my case a SHA-512 hash.
2. Backup is a "typical" full "backup" of everything on my workstation.

Logic Flow:
1. Loop Thru All Available Files.
2. Compute a SHA-512 for the current file
3. Look up the hash, do we have a "chunk" that exists.
4. If we do, "connect" the file record to the chunk.
5. else create a new chunk, and link the file record

The first run, without "compression" we get a 15% saving
If you run this a second time, the second backup will go be reduced from 37+ gig
to 200Meg or less. Resulting in a deduplication ration of 99%+

The nice "tricks" that can be done with this, is to reduce your "backups" from days to "minutes", since the "normal" rotation of "removable" backups already have "data" that we can delta.

In addition we could keep 2 years of backup at a cheap price.
And a slew of other features.

Stats of the "backup" actually done on my workstation.

Chunks 382899
Backups 1
Backup Files 510960
Total Backup Size 40031553419 = 37.28G
Total Chunk Size 33727083825 = 31.41G
Savings 6304469594 = 5.87G = 15% for first backup

Second Backup would have a savings of 99%

Now the code:

First we need a "place" to track our progress.

#migration
class CreateChunks < ActiveRecord::Migration
def self.up
create_table "backupchunks", :force => true do |t|
t.string :filehash
t.integer :filesize
t.string :ctype
t.datetime :created_at
end
create_table "backups", :force => true do |t|
t.string :host
t.string :description
t.datetime :created_at
end
create_table "backupfiles", :force => true do |t|
t.integer :backup_id
t.string :filename
t.integer :filesize
t.integer :backupchunk_id
end
end

def self.down
drop_table :backupchunks
drop_table :backups
drop_table :backupfiles
end
end

Now our "agent"

#scandisk.rb
#---------------
require 'digest'
require 'find'
require 'pp'
require "rubygems"
require "activerecord"

ActiveRecord::Base.establish_connection(
:adapter => "sqlite3",
:database => "db/development.sqlite3"
)

class Backupchunk < ActiveRecord::Base
has_many :Files
end

class Backup < ActiveRecord::Base
has_many :Files
end

class Backupfile < ActiveRecord::Base
belongs_to :Backupchunk
end

def hash_file(file, hash_kind)
digest, chunk = hash_kind.new, ''
open(file, 'rb') do |f|
digest << chunk while f.read(4096, chunk)
end
digest
end

def handlebackup(f)

begin
thehash = hash_file(f, Digest::SHA512).to_s
rescue
puts "Cannot read #{f}"
return
end

thechunk = Backupchunk.find(:first,:conditions => {:filehash => thehash})
if thechunk.nil?
thechunk = Backupchunk.new
thechunk.filehash = thehash
thechunk.filesize = File.size(f)
thechunk.save
else
pp f
end
thefile = Backupfile.new
thefile.filename = f
thefile.filesize = File.size(f)
thefile.backupchunk_id = thechunk.id
thefile.save

end

target_dir = "h:/dedup/store/"

mybackup = Backup.new
mybackup.host = "sinub-glenn2"
mybackup.save

Find.find('/') do |f|
if File.file?(f)
handlebackup(f)
end
end

Corrupted Flash Card - Recovery of FAT32 Files

2008-07-27T06:03:00.000-07:00

Recently, I was getting ready to copy some images from a "full" memory card, to my pc for archive. Somewhere during the process, the fat32 file system on the card, got confused, and the directory where the images were simply disappeared.

What to do now? I tried chkdsk? Nothing
I tried the "Trash" folder, Again nothing.
Over 540+ irreplaceable photos from my Nikon D70 were
gone.

So doing some research, I found many a "pay" solutions.
But not really wanting to pay, I wanted to find a "free"
solution, without running into a virus, spyware, etc.

After looking at three or four, I found my solution.

PhotoRec.
The interface is a bit primative, but it worked perfectly.
The first time. Every photo was recovered.

So if you hit that problem, give it a try.
Cost is $0.00. Very good perfomance for the price.

http://www.cgsecurity.org/wiki/PhotoRec

Dynamic Tabnav Widget menus

2008-07-10T23:28:00.000-07:00

I wanted to do a "dynamic" menu that is filled from data in my Rails Application.
I have a "table" (Model in rails speak) that has the values. Then I can "overload"
the controller to add more functionality.

The "gotcha" in coding this is, there is "no" each method for activerecord.

The other "gotcha" is a "site" can have mutliple networks, so since we are selecting my "site", and network does not have it, we do a lookup in the conditions_for_collections code. Note that active_scaffold can also "view" the devices via the site controller. But I find having "buttons" on the device controller a nice interface.

Using this technique, I can "dry" up the controller, and reuse it for several "purposes". No need to repeat the "controller".

Note that we have to "pass" the level3 status up thru the controller, and then back down thru the partials.

First the widgets for tabnav

File: app\views\widgets\_device_tier2_tabnav.rhtml
<%
level3 = nil
render_tabnav :device_tier2,
:generate_css => true do

add_tab do |t|
t.named 'All'
t.titled 'All'
t.links_to :controller => 'device'
end
add_tab do |t|
t.named 'Sites'
t.named 'Sites'
t.links_to :controller => 'device?level3=device_sites_tier3'
end
end
%>

File: app/views/widgets/_device_sites_tier3_tabnav.rhtml
<%
render_tabnav :device_sites_tier3,
:generate_css => true do
Site.find(:all).each do |thesite|
add_tab do |t|
t.named thesite.name
t.links_to :controller => "device?site=#{thesite.id}&level3=device_sites_tier3"
end
end
end
%>

File app\views\layouts\tier2device.rhtml
<html>
<%= javascript_include_tag :defaults %>
<%= active_scaffold_includes %>
<%= stylesheet_link_tag "/javascripts/jscalendar-1.0/calendar-win2k-cold-1.css" %>
<%= javascript_include_tag "jscalendar-1.0/calendar.js" %>
<%= javascript_include_tag "jscalendar-1.0/lang/calendar-en.js" %>
<%= javascript_include_tag "jscalendar-1.0/calendar-setup.js" %>
<head>
<title>AppName</title>
</head>

<body>
<br>

<%= tabnav :airstate %>
<%= tabnav :device_tier2 %>
<% if @level3 %>
<%= tabnav @level3 %>
<% end %>
<%= @content_for_layout %>
</body>
</html>

The controller, which is using activescaffold looks like this:
class DeviceController < ApplicationController
layout 'tier2device'

active_scaffold :device do |config|
config.actions = [:delete,:show, :list, :nested, :search]
config.list.columns = [:network, :name, :serialnum, :model, :macs, :purchase_date, :warranty_end]
config.columns[:macs].ui_type = :select
config.columns[:network].ui_type = :select
config.columns[:user].ui_type = :select
end

def conditions_for_collection

@level3 = nil
if params.include?("level3")
@level3 = params.delete("level3")
end
if params.include?("site")
mysite = params.delete("site")
mynets = Network.find(:all,:conditions => "site_id = #{mysite}")
mywhere = ['network_id IN (?)', mynets]
end

return mywhere
end

end

References - No .each in active record
http://weblog.jamisbuck.org/2007/4/6/faking-cursors-in-activerecord

Reading the Service Tag or Serial Number in Ruby

2008-07-06T17:20:00.000-07:00

I want to "do" a inventory of "all" my equipment. (Yes It will eventually be a Rails app), but the first issue is gettig all the "service" tags, and updating them on the fly. If you search, you'll find plenty of examples in vb or python, but not very many. (Read none) in ruby. So I decided to see if I could do it.

First I liked the ruby-wmi package, that gives a activerecord flavor to wmi. WMI is windows standard "Instrumentation" library, where you can find out all too many facts about a windows system.

Next a little "code" of my own to show "how" to do it. Implemented as a class on top of ruby-wmi, with some basic "caching" to make the overhead smaller.

require 'ruby-wmi'
require 'pp'

# Reference URL's
# http://www.vbforums.com/showthread.php?t=326425
# http://ruby-wmi.rubyforge.org/doc/classes/WMI/Base.html
# http://mentalpagingspace.blogspot.com/2008/01/ruby-class-tutorial-using-unify-dbi.html

class ManagePC

def initialize(ipaddress)

@myip = ipaddress
@biosv = WMI::Win32_BIOS.find(:all,:host=>@myip)
@csysv = WMI::Win32_ComputerSystem.find(:all,:host=>@myip)
@netv = WMI::Win32_NetworkAdapterConfiguration.find(:all,
:conditions => 'IPEnabled = True')
return

end #initialize

def prtnetvalues()
@netv.each do |netx|
netx.properties_.each do |p|
value = netx[p.name]
if value
if value.class == Fixnum
value = value.to_s
end
if value.class == Array
value = value.to_s
end
if value.class == TrueClass
if value
value = "True"
else
value = "False"
end
end
theprop = p.name + "="
theprop << value
puts theprop
end
end
end
end # prtnetvalues()

def prtcsysvalues()
@csysv.each do |csys|
csys.properties_.each do |p|
printf "#{p.name} = #{csys[p.name]}\n"
end
end
end # prtcsysvalues()

def netvalue(name)
@netv.each do |netx|
netx.properties_.each do |p|
if p.name == name
return netx[name]
end
end
end
return nil
end

def csysvalue(name)
@csysv.each do |csys|
csys.properties_.each do |p|
if p.name == name
return csys[name]
end
end
end
return nil
end

def macs()
mymacs = Array.new
current_mac = nil
@netv.each do |netx|
netx.properties_.each do |p|
if p.name == "IPAddress"
current_mac << netx[p.name].to_s
end
if p.name == "MACAddress"
current_mac << netx[p.name]
end
if p.name == "Caption" # A new network interface
if current_mac
mymacs << current_mac
end
current_mac = Array.new
end
end
end
mymacs << current_mac
return(mymacs)
end

def domain()
return csysvalue("Domain")
end

def model()
return csysvalue("Model")
end

def name()
return csysvalue("Name")
end

def domain?()
return csysvalue("PartOfDomain")
end

def username()
return csysvalue("UserName")
end

def memory()
return csysvalue("TotalPhysicalMemory")
end

def servicetag()
@biosv.each do |bios|
bios.properties_.each do |p|
if p.name == "SerialNumber"
return(bios[p.name])
end
end
end
return(nill)
end #GetServiceTag

end #ManagePC

mypc = ManagePC.new("127.0.0.1")
puts mypc.servicetag
puts mypc.domain
puts mypc.model
puts mypc.name
puts mypc.username
puts mypc.memory
pp mypc.macs

Annoucing PDFTORUBY Support Forum

2008-07-01T17:33:00.000-07:00

PDFtoRuby now has a support "forum" hosted via Google Groups.
If your having issues or problems with PDFtoRuby, or its "just" working
please feel free to drop a message on the board.

I'll be glad to help "resolve" any issues you have

http://groups.google.com/group/pdftoruby?hl=en

Rapid easy loading of CSV data in a Rails migration

2008-06-25T23:25:00.000-07:00

Well, it had to happen sometimes.
I was sent some nice "huge" database tables, that
had been exported as "csv" files. So new there
were several "ways" of handling this in ruby,
so I wanted to do it in a migration, and since the
tables have "lots" of columns, didnt want to have to "write" them
out.

I use both "fastercsv" in combination with ar_extensions
I could not find a "blog" article that had a "simple" migration using
the combination.

Not that I load "ar_extensions" in my enviornment.rb
Also note that the table is "created" in another migration.

# 099_load_myrecipes.rb
require 'fastercsv'

class LoadRecipes < ActiveRecord::Migration
def self.up
mycnames = Recipes.columns.map{ |column| column.name }
mycnames.delete("id")
value_sets = FasterCSV.read("#{RAILS_ROOT}/mdata/Recipes.csv", :col_sep => ",",
:row_sep => "\n")
options = { :validate => false }
Recipes.import( mycnames, value_sets, options)
end

def self.down
Recipes.delete_all()
end
end

Resources:
FasterCSV Doc - http://fastercsv.rubyforge.org/
FasterCSV and arext article - http://www.jobwd.com/article/show/31
AR Extention - http://www.continuousthinking.com/tags/arext

Great new ruby "tool" - Free Message queue

2008-06-24T17:21:00.000-07:00

Wow, now this is something that is easy to use, and works out of the "can".

gem install fmq

Gets you started. This is a "message" queue system. It lets you implement a "fire" and forget type messaging system. (In my old Stratus or VOS days, we called them "server" queues.)

So you set up a "server", which in this can is a "mongrel" rails application, that manages multiple "server" queues. The communications method is "http", which means it will go thru firewalls easy. Your "client", which could be a pc, or a "ruby" capable device then can "fire" work to be done at the server, such as request a report, or perform some "action".

For example, in "device" management, it could be to "get" outstanding commands, or "update" status. The work is "queued" and processed as quick as possible.

The nice thing about fmq that I can see is the "dependencies" are "tiny" to "none". There's a cute little demo/management system that lets you create "queues", and fire messages, and receive messages.

In doing a lot of "missing" critical applications I find a good message system is critical. I've even coded my own "system" to extend "server/message" queues to a pc.
This looks "simple" and "good".

So I'll give this a spin in a "real" project, and see how it flies.

http://fmq.rubyforge.org

Annoucing pdftoruby

2008-05-20T00:12:00.000-07:00

Did you ever need to "modify" pdf files, or "recreate" them with
fields changed or added. Do you ever want to generate nice "invoices"
or other "forms" documents, but dont want to break out the ruler to
"recreate" the form.

Now there's a tool that can take a pdf form, and convert it to ruby code.
This "unifies" pdf::reader and pdf::writer libraries.

pdftoruby generates ruby code from a supplied pdf.
You can then "edit" the ruby code to add/delete/modfiy and run the code,
it will generate the pdf file using pdf::writer.

I've used the result code in a rails production application for months now, without issue.

http://code.google.com/p/pdftoruby/

Creating PDF::Writer Compatible PNG Files

2008-05-20T00:07:00.000-07:00

I've fought PNG for a while. Scanning google countless times to
figure out how to get a PNG to write, in as few as lines of possible.

So my final choice is to use a "clone" of png gem, with a few changes.

One, make sure to write in "binary", so your file opens in the library must
be in a "b" option. The default install of PNG, does not have the "b" option,
which means it will work perfectly fine on linux/unix, and not at all in windows.

The second problem is the "color" space type. The default is 6, which means there
is a transparency byte per pixel. This of course does not work with PDF::writer.
So My version uses "2", which drops the transparency byte, and writes 3 bytes.

The later versions of png gem, switch to using inline, which require a compiler to work. Since it makes installing pdftoruby a pain, I wanted something simple.

class String # :nodoc:
##
# Calculates a CRC using the algorithm in the PNG specification.
def png_crc
unless defined? @@crc then
@@crc = Array.new(256)
256.times do |n|
c = n
8.times do
c = (c & 1 == 1) ? 0xedb88320 ^ (c >> 1) : c >> 1
end
@@crc[n] = c
end
end

c = 0xffffffff
each_byte do |b|
c = @@crc[(c^b) & 0xff] ^ (c >> 8)
end
return c ^ 0xffffffff
end

end

## gsw - Make sure we have version 1 - as version 2 will require major havoc - also fix the "binary" issue
# A pure Ruby Portable Network Graphics (PNG) writer.
#
# http://www.libpng.org/pub/png/spec/1.2/
#
# PNG supports:
# + 8 bit truecolor PNGs
#
# PNG does not support:
# + any other color depth
# + extra data chunks
# + filters
#
# = Example
#
# require 'png'
#
# canvas = PNG::Canvas.new 200, 200
# canvas[100, 100] = PNG::Color::Black
# canvas.line 50, 50, 100, 50, PNG::Color::Blue
# png = PNG.new canvas
# png.save 'blah.png'

class PNG

##
# Creates a PNG chunk of type +type+ that contains +data+.

def self.chunk(type, data="")
[data.size, type, data, (type + data).png_crc].pack("Na*a*N")
end

##
# Creates a new PNG object using +canvas+

def initialize(canvas)
@height = canvas.height
@width = canvas.width
@bits = 8
@data = canvas.data
end

##
# Writes the PNG to +path+.

def save(path)
File.open(path, "wb") do |f|
f.write [137, 80, 78, 71, 13, 10, 26, 10].pack("C*") # PNG signature
f.write PNG.chunk('IHDR',
[ @height, @width, @bits, 2, 0, 0, 0 ].pack("N2C5"))
# 0 == filter type code "none"
data = @data.map { |row| [0] + row.map { |p| [p.values[0], p.values[1], p.values[2]] } }.flatten
#data = @data.map { |row| [0] + row.map { |p| p.values } }.flatten
f.write PNG.chunk('IDAT', Zlib::Deflate.deflate(data.pack("C*"), 9))
f.write PNG.chunk('IEND', '')
end
end

##
# RGBA colors

class Color

attr_reader :values

##
# Creates a new color with values +red+, +green+, +blue+, and +alpha+.

def initialize(red, green, blue, alpha)
@values = [red, green, blue, alpha]
end

##
# Transparent white

Background = Color.new 0xFF, 0xFF, 0xFF, 0x00

White = Color.new 0xFF, 0xFF, 0xFF, 0xFF
Black = Color.new 0x00, 0x00, 0x00, 0xFF
Gray = Color.new 0x7F, 0x7F, 0x7F, 0xFF

Red = Color.new 0xFF, 0x00, 0x00, 0xFF
Orange = Color.new 0xFF, 0xA5, 0x00, 0xFF
Yellow = Color.new 0xFF, 0xFF, 0x00, 0xFF
Green = Color.new 0x00, 0xFF, 0x00, 0xFF
Blue = Color.new 0x00, 0x00, 0xFF, 0xFF
Purple = Color.new 0XFF, 0x00, 0xFF, 0xFF

##
# Red component

def r; @values[0]; end

##
# Green component

def g; @values[1]; end

##
# Blue component

def b; @values[2]; end

##
# Alpha transparency component

def a; @values[3]; end

##
# Blends +color+ into this color returning a new blended color.

def blend(color)
return Color.new((r * (0xFF - color.a) + color.r * color.a) >> 8,
(g * (0xFF - color.a) + color.g * color.a) >> 8,
(b * (0xFF - color.a) + color.b * color.a) >> 8,
(a * (0xFF - color.a) + color.a * color.a) >> 8)
end

##
# Returns a new color with an alpha value adjusted by +i+.

def intensity(i)
return Color.new(r,b,g,(a*i) >> 8)
end

def inspect # :nodoc:
"#<%s %02x %02x %02x %02x>" % [self.class, *@values]
end

end

##
# PNG canvas

class Canvas

##
# Height of the canvas

attr_reader :height

##
# Width of the canvas

attr_reader :width

##
# Raw data

attr_reader :data

def initialize(height, width, background = Color::White)
@height = height
@width = width
@data = Array.new(@width) { |x| Array.new(@height) { background } }
end

##
# Retrieves the color of the pixel at (+x+, +y+).

def [](x, y)
raise "bad x value #{x} >= #{@height}" if x >= @height
raise "bad y value #{y} >= #{@width}" if y >= @width
@data[y][x]
end

##
# Sets the color of the pixel at (+x+, +y+) to +color+.

def []=(x, y, color)
raise "bad x value #{x} >= #{@height}" if x >= @height
raise "bad y value #{y} >= #{@width}" if y >= @width
@data[y][x] = color
end

##
# Iterates over each pixel in the canvas.

def each
@data.each_with_index do |row, y|
row.each_with_index do |pixel, x|
yield x, y, color
end
end
end

##
# Blends +color+ onto the color at point (+x+, +y+).

def point(x, y, color)
self[x,y] = self[x,y].blend(color)
end

##
# Draws a line using Xiaolin Wu's antialiasing technique.
#
# http://en.wikipedia.org/wiki/Xiaolin_Wu's_line_algorithm

def line(x0, y0, x1, y1, color)
dx = x1 - x0
sx = dx < 0 ? -1 : 1
dx *= sx # TODO: abs?
dy = y1 - y0

# 'easy' cases
if dy == 0 then
Range.new(*[x0,x1].sort).each do |x|
point(x, y0, color)
end
return
end

if dx == 0 then
(y0..y1).each do |y|
point(x0, y, color)
end
return
end

if dx == dy then
Range.new(*[x0,x1].sort).each do |x|
point(x, y0, color)
y0 += 1
end
return
end

# main loop
point(x0, y0, color)
e_acc = 0
if dy > dx then # vertical displacement
e = (dx << 16) / dy
(y0...y1-1).each do |i|
e_acc_temp, e_acc = e_acc, (e_acc + e) & 0xFFFF
x0 = x0 + sx if (e_acc <= e_acc_temp)
w = 0xFF-(e_acc >> 8)
point(x0, y0, color.intensity(w))
y0 = y0 + 1
point(x0 + sx, y0, color.intensity(0xFF-w))
end
point(x1, y1, color)
return
end

# horizontal displacement
e = (dy << 16) / dx
(x0...(x1-sx)).each do |i|
e_acc_temp, e_acc = e_acc, (e_acc + e) & 0xFFFF
y0 += 1 if (e_acc <= e_acc_temp)
w = 0xFF-(e_acc >> 8)
point(x0, y0, color.intensity(w))
x0 += sx
point(x0, y0 + 1, color.intensity(0xFF-w))
end
point(x1, y1, color)
end

end

end

Ruby Png Gem - Getting Rid of Inline

2008-04-23T20:04:00.000-07:00

Hmm, I know I grew up cutting teeth on C, but the "overhead" of getting
decent development environment, and "syncing" your compiler version
to that of who-ever wrote the code originally gets to be fun.

Its even worse when your going cross platform. (Yes I now prefer to
do my coding in ruby whenever possible).

I was reading thru ruby group on goole, about how to speed up the "png"
gem without using inline. I'm going to give that a try. Then I can use
the resolve in my pdftoruby tool.

Should be interesting, will update the article after I finish.

Geting Ruby png1.0.0 to work in windows

2008-04-23T02:00:00.000-07:00

I need to be able to save "png" files out of my pdftoruby application.
My big problem was to take a "rgb" file, and write it, without using something
heave like ImageMagik. (Which seems like everyone complains about)

So my first choice was "png", but the latest version of png is not "pure" ruby.
Version 1.0.0 is "pure". The reason this is important, is that with the 1.1.0
release of png gem, you have to use "inline" and that then requires you to
use the "latest" compiler.

I've installed, and reinstalled png several times, but it seems to never work,
then finally I found a "mention" in a old ruby-talk message thread that on windows
you need the "b" open on the open. (A following comment said it had been patched).

Well decicded to go look at the source, and sure enough, its not got the "b" option
on the open.

So do the following:
gem install --version 1.0.0 png

Then go to the png directory
cd \ruby\lib\ruby\gems\1.8\gems\png-1.0.0\lib

And edit png.rb

vi png.rb # Use the editor of your choice

def save(path)
File.open(path, "wb") do |f| #Must add the "b" next to the "w"
#here for windows to work
f.write [137, 80, 78, 71, 13, 10, 26, 10].pack("C*") # PNG signature
f.write PNG.chunk('IHDR',
[ @height, @width, @bits, 6, 0, 0, 0 ].pack("N2C5"))
# 0 == filter type code "none"
data = @data.map { |row| [0] + row.map { |p| p.values } }.flatten
f.write PNG.chunk('IDAT', Zlib::Deflate.deflate(data.pack("C*"), 9))
f.write PNG.chunk('IEND', '')
end
end

Now png examples will work, and I can get back to business.

(I've seen some nice pure ruby optizations in the threads, will implemented those in pdfto ruby)

The Evils of EBAY - Never never never use them

2008-03-30T04:56:00.001-07:00

Well you know from the title I had a bad experience from ebay.]
Two years ago, I decided to "try" ebay by buying a lens.
The "seller" had a great reputation. (looks like there are tricks
to give you a false "reputation", if anyone know how, feel free to post them to a comment.)

So after reading the ad, the seller wanted a "money order" or a "cashier" check. Nevermind, I can do that. So I went to the bank, paid the fee, and sent off my check. I "have" a copy of the check laying in my library to prove it. I mailed it off to the individuals address, and waited and waited and after a month got nothing. THe person saying "he" never got the check. Needless to say I complained to ebay. I gave him "negative" reputation. He returned the favor.

So I havent tried to use ebay from that bad experience for two years. I did write in and ask for help. I always got "wonderfully" worded emails, but no action. Worse come to find out, after two years I want to "try" it again, and guess what, I still have the "-1" reputation from the "bad" seller. And cannot do a transaction.

So for me, I'me giving up ebay for life, and will tell everyone I know its a untrustworty company, that there not to be trusted. That you will lose your money, that you will get "sweet" email, when you complain from them, and nothing more.

Well looks like they (Ebay) will not change the post, but if you see the comment,
they gave a nice invitation.

So any lawyer in california interested in class action lawsuit against ebay?
I'm intersted in joining a class action suit for people abused, and mentally abused by them.

ICC Profiles - Decoding Using BitStruct And Ruby

2008-03-24T00:23:00.001-07:00

I wanted to be able to explore ICC Color Profiles.
Since the data is binary, I wanted to be able to define nice "Structures",
and see the data.

So I disovered "Bitstruct". Its a library that lets you define
binary data easily.

Example o fdecoding ICC Profiles:

require 'bit-struct'
require 'pp'

class IccHeader < BitStruct
unsigned :size, 32, "Profile Size"
char :type, 32, "CMM Type Signature"
octets :version, 32, "Profile Version Number"
char :pdclass, 32, "Profile/Device Class"
text :colorspace, 32, "Color Space of Data"
text :pconnspace, 32, "Profile Connection Space"
char :date, 96, "Creation Date"
text :sign, 32, "Profile Signature"
text :platform, 32, "Primary Platform"
text :flags, 32, "CMM Flags"
text :devmanuf, 32, "Device Manufacture"
text :devmodel, 32, "Device Model"
text :devattr, 64, "Device Attributes"
text :renderintent,32, "Rendering Intent"
char :xyz1, 32, "XYZ Number 1"
char :xyz2, 32, "XYZ Number 2"
char :xyz3, 32, "XYZ Number 3"
char :creator, 32, "Profile Creator"
text :id, 128, "Profile ID"
text :reserved, 224, "Reserved"
unsigned :tagcount, 32, "Tag Count"
end

class TagTableEntry < BitStruct
text :sign, 32, "Signature"
unsigned :offset, 32, "Offset"
unsigned :size, 32, "Size"
end

puts "Dump of ICC Profile"
pfile = File.open("profile","r")
icchdr = IccHeader.new(pfile.read(132))
pp icchdr
cnt = icchdr.tagcount
offset = 133
tags = Array.new
while cnt > 0
tags << TagTableEntry.new(pfile.read(12))
cnt -= 1
end
pp tags

Rake Script to Load Seed Data or Update Existing Data

2008-02-16T23:41:00.001-08:00

In the previous blog we descussed loading of seed data using rake/active record.
What was missing was examples.

So First a example of our "rake" file.

namespace :project do
desc 'parts_master_load'
task :parts_master_load => :environment do
PartsMaster.uoc :pn => '11-1111-11',
:description => 'Part description for part 11-111-11',
:application_code => ApplicationCode.foc('appcode1'),
:pn_type_code => PnTypeCode.foc('REF')
PartsMaster.uoc :pn => '22-2222-22',
:description => 'Part Description for part 22-222-22',
:application_code => ApplicationCode.foc('apcode2'),
:pn_type_code => PnTypeCode.foc('REF')
end
end

Rake has the advantage in that it runs with all of rails environment, and for us that means activerecord, so we can use our "models". Another advantage is if there is a error, if you have the rake trace option, you get exactly the line of "data" that caused the issue.

In the above example, there are "three" models that are directly mentioned.
One is the PartsMaster, Two is the ApplicationCode, and the Third is the PnTypeCode.

The "method" foc is one of my own, that means "find" or "create". Very handly for those "simple" tables that are used for "catagories".

So as a example of "foc" as used in the "application" code module.

class ApplicationCode < ActiveRecord::Base
set_table_name "APPLICATION_CODES"

defaults :application_code => "",
:cannot_modify => "F",
:converted => "F",

public

def self.foc(thevalue) #find_or_create
if thevalue.nil?
thevalue = "_"
end
if thevalue == ""
thevalue = "_"
end
therecord = find(:first,:conditions => "application_code = '#{thevalue}'")
if therecord
return therecord
end
therecord = self.new
therecord.application_code = thevalue
therecord.description = thevalue
therecord.save
return therecord
end

end

Here you can see the power of "foc". We actually use the "application" key to do a "lookup", on existing data. We can "also" cleanup any "issues" with the "incoming" data, to make it more "consistent". If the record "does" not exist, then the record is created. This also is good illustration of "default" values to make usage of the model "easier".

Now lets look at a "stippet" of the "update or create" or uoc method. For the main
table in the application, we need to update existing records, since the file has constraints, so we cannot "truncate" the table, and aways do a "create".

def self.uoc(options = {}) # update or create
thevalue = options[:pn] #fixme - Need to also look at manufactuer
record = find(:first,:conditions => "pn = '#{thevalue}'")
if record.nil?
record = new
end
record.attributes = options
record.save!

record
end

This illustrates a lovely "concept" in ruby of allowing us to use a hash, allowing us to take variable arguments in, and pass those variable arguments down. Code looks simple. I could code the same in "C" and it would be rather ugly.

In this code, the "new" will create a record, and establish defaults (The record has a couple hundred fields, so defaults is "not" small.) and then "update" it with what we pass. That allows the "create" rake script to be very small.

So there you have it. From now own, I'm going to use active record to do my data load and migration. It works very nice.

Integrating Legacy Applications with Rails

2008-02-15T19:00:00.001-08:00

I'm current involved in a large software project which is taking a "legacy" applicaton to a "new" generation application. Of course for me, ruby is my "main" tool. So I'm doing the migration with ruby, and looking to use rails to "expand" the canned application with rails functionality.

So the issues are many, and countless.

1. Schema - While the database is oracle, the schema of the new system is "huge", 400+ tables is nothing to sneeze at. So first we need some good documentation of it. Does it come with the system? Nope, no schema documentation, so lets create some documentation for easy reference.

http://schemaspy.sourceforge.net/

Schemaspy will let us "create" a map of existing schema, so we can explore it, and get familiar with internal standards.

2. Walking around the data. Next we need to "explore" the data of the new system. For this in a oracle environment, Oracle Sql Developer is free, and lets us look at the data easily.

3. Write a "script" in ruby to create models automatically. Typing any normal of models, getting plurization right, and creating controllers is not for the faint of heart. I create a "gen" script that generates "missing" models, and controllers. (I use activescaffold plugin for handling the controller side, and models can be started without relationships if you wish. (If I do this a second time, I'll have the relationships also done by the script. Not much fun doing tables that have 20 relationships, where the keyname does not match the tablename.) (Now you know why you need schemaspy).

4. For importing your "existing data, assuming its not terabytes, I find that it works best in a "two" step process. In previous projects I would write a "dbi" ruby script taking one table to the "new" rails table. That works fine, but now I have destination tables that have "rich" relationships, writing dbi is no fun. ActiveRecord can save the day.
a. Step 1 - dbi process - Read existing table, and generate a rake script, calling active record "models"

Reference: http://railspikes.com/2008/2/1/loading-seed-data

b. Step 2 - Run your rake script

Reference:
http://www.slashdotdash.net/articles/2007/01/18/using-activerecord-outside-rails-part-ii

I find that this gives you the easy ability to handle the multi-tier adds
while keeping things simple.

When your creating your models, I find that often in "large" applicatons the tables from tradiitional applications can be "huge" number of fields. Of course the reality is that the majority of fields are always" default" values. To keep your create and update clean, use a "defalts" plugin. This will allow you to write you "create" or "create-and-update" call in your rake script to be "very" small, while all the flags and other values get set to the right values.

Reference:
http://agilewebdevelopment.com/plugins/activerecord_defaults

This makes my "rake" task incrediably small.

Finally consider in your models adding a method of either:

1. uoc - Update or Create - This will let you take a hash, and find a existing record and update it, or create a new one based on the "primary" key that is "application" specific. While my target application does have a numeric primary key, the "real" key is two other fields. using uoc allows me to do a find on the record and update it. (Yes I could "truncate" the table and always do creates, but remmeber its a complex app with a spiderweb of connections, updates work better)

2. foc - Find or create - This is great for those "simple" tables like unit of measure, or other tables that typically reasly are for "catorgories". It simply "finds" the right record or "create" a new one and returns it for the relationship your building.

This also makes the rake script re-usable. And the code in your modeles can also be used in "expansion" applications.

Next week I'll add a few examples.

Rails - The Menu Problem - Handling Large Scale Applications moving to Rails

2008-02-05T21:50:00.000-08:00

It was interesting. I'm doing a large progject, and there are 500+ tables to deal with. This is my first "legacy" as it is project. And of course the "same" old tricks dont always work.

One of the problems, to check my models, of course I generate activescaffold controllers.

If your've read my previous post, my converts/imports usually do the following:

1. Move the table.
a. grab the schema
b. add missing fields
c. fix field names and table names for destination database
d. create the sequence number
e. create the table in the new schema
2. Create a model
3. Create a controller
4. Update the menu

This strategy has worked great. You get a reasobly native rails app,
and cleanup some of the database/schmema issues in the process.

New project is a bit different. Its all in oracle all ready. And its a "off-the-shelf" application.

So now I need to not "touch" the database schema, but man, I need a extra table, and
how to do menus etc when the functionality is so "rich".

So I decided to create a separate sqlite3 database just to do menus and functions.
(and maybe some other things along the way).

So first, need a do a migration (001_system_menu.rb). This will create our new
sqlite3 database, not touching our oracle database that is already working.
I did a trick to get this to work, I created a "empty" sqlite3 database file,
so the schema_version table would already exist, otherwise you get a "nice" error message saying you "cannot" create it.

Next wanted to get some data "into" the tables, so I created a second migration
(002_add_base_menus.rb) that creates some data so the menu system can be tweaked further.

So now how to "fill" up the tables in the operating "application"?
So in the models, I added the ability to "walk" the "controller" directory,
and any associated namespaces, and create entries. This will allow you then
to "play" them into the "menu" tree and "treak" the names, and the roles.

The "update_functions" method does a few things:
1. Verify "functions(controllers)" actually exist,
then enable or disable as appropriate
(If we delete we loose the custimization)
2. Find any new controllers and "add" them into the system

For now thats what we got, next will be to automatically generation of
menu (tabnav) partials for each level of the menu, and arrange for "hooks" in
the application to "know" what his menu is.

The system_function model has a very good example of "walking" directories.

#-----------------------001_system_menu.rb ------------------------
class SystemMenu < ActiveRecord::Migration

class SystemMenu < ActiveRecord::Base
ActiveRecord::Base.establish_connection(
:adapter => "sqlite3",
:database => "db/system.sqlite3"
)
acts_as_tree
has_many :SystemFunction
end

class SystemFunction < ActiveRecord::Base
ActiveRecord::Base.establish_connection(
:adapter => "sqlite3",
:database => "db/system.sqlite3"
)
belongs_to :SystemMenu
end

def self.up
create_table :system_functions do |t|
t.column :system_menu_id, :number
t.column :present, :boolean
t.column :enable, :boolean
t.column :name, :string, :limit =>32, :null => false
t.column :title, :string, :limit =>64, :null => false
t.column :controller, :string, :limit =>128
t.column :role, :string
t.column :created_at, :datetime
t.column :updated_at, :datetime
end
create_table :system_menus do |t|
t.column :name, :string, :limit =>32, :null => false
t.column :title, :string, :limit =>64, :null => false
t.column :role, :string, :limit =>32
t.column :enable, :boolean
t.column :parent_id, :number
t.column :created_at, :datetime
t.column :updated_at, :datetime
end

end

def self.down
drop_table :system_menus
drop_table :system_functions
end
end

class AddBaseMenus < ActiveRecord::Migration
class SystemMenu < ActiveRecord::Base
establish_connection :adapter => "sqlite3", :database => "db/system.sqlite3"
acts_as_tree
has_many :system_function
end

#-------------------- 002_add_base_menus.rb ------------------------
class SystemFunction < ActiveRecord::Base
establish_connection :adapter => "sqlite3", :database => "db/system.sqlite3"
belongs_to :system_menu
end

def self.up
homemenu = SystemMenu.create :name => "home", :title => "Home"
adminmenu = SystemMenu.create :name => "admin", :title => "Admin", :role => "Admin", :parent_id => homemenu.id
SystemFunction.create :system_menu_id => adminmenu.id, :name => "Menus", :title => "System Menus", :controller => "adminspace/system_menu", :role => "admin", :present => TRUE, :enable => TRUE
SystemFunction.create :system_menu_id => adminmenu.id, :name => "Functions", :title => "System Functions", :controller => "adminspace/system_function", :role => "admin", :present => TRUE, :enable => TRUE
end

def self.down
end
end

#----------------------- system_menu.rb ----------------------
class SystemMenu < ActiveRecord::Base
establish_connection :adapter => "sqlite3", :database => "db/system.sqlite3"
acts_as_tree
has_many :system_function

end

#---------------------- system_function.rb -------------------
class SystemFunction < ActiveRecord::Base
establish_connection :adapter => "sqlite3", :database => "db/system.sqlite3"
belongs_to :system_menu

public
def self.update_functions
app_directory = "./app/controllers"
verify_functions(app_directory)
system_function_process_dir(app_directory,"")
end

def self.verify_functions(thedir)
SystemFunction.find(:all).each do |record|
thepath = thedir + "/" + record.controller + "_controller.rb"
if File.exists?(thepath)
if record.present == FALSE
record.present = TRUE
record.save
end
else
if record.present == TRUE
record.present = FALSE
record.save
end
end
end
end

def self.system_function_process_entry(theparent,thedir, theentry)
if theentry == "."
return
end
if theentry == ".."
return
end
if theentry == "archive"
return
end
fullpath = thedir + "/" + theentry
if File.directory?(fullpath)
system_function_process_dir(fullpath,theentry)
return
end
if theentry.include? "_controller.rb"
thelen = theentry.index('_controller.rb')
thename = theentry[0,thelen]
if theparent == ""
mycontroller = thename
else
mycontroller = theparent + "/" + thename
end
myfunction = SystemFunction.find(:first, :conditions => "controller = '#{mycontroller}'")
if myfunction == nil # We dont exist
myfunction = SystemFunction.new
myfunction.name = thename
myfunction.system_menu_id = nil
myfunction.title = thename
myfunction.controller = mycontroller
myfunction.enable = FALSE
myfunction.present = TRUE
myfunction.role = ''
myfunction.save
return
end
end
end

def self.system_function_process_dir(thedir,theparent)
Dir.foreach(thedir){ |theentry| system_function_process_entry(theparent,thedir,theentry)}
end

end

Rails Security - Is there passwords waiting for the world to read

2008-01-30T21:18:00.000-08:00

Recently I was promoting a "Rails" app, that was going to be used by another division. And was asking my "boss" to login, and it got me thinking about log file "security". One of the things that has been bothering me for a while was when I was running a app, and looking at the log, either in production or development I would always "say" a few words when a user was loggin in while I was watching. One of my hero's has a favorite saying, "To much information".

To illustrate:
Processing AccountController#login (for 127.0.0.1 at 2008-01-31 13:10:58) [POST]
Session ID: 9a85051d00c5aee3dc20305ac0dd315e
Parameters: {"commit"=>"Log in", "action"=>"login", "controller"=>"adminspace/account", "login"=>"user", "password"=>"secret"}

And you wll see this in both production and development.

No there was a few solutions when I was googling:

1. Turn the logging to warning
in config/environments/production.rb
# Fix the logging so sensitive information is not logged
config.log_level = :warn

Cons: You loose all your request, now you log analyser and you have very little information on what is going on in your application.

2. "Filtering"
Now this is "nice". And is good even in development side. (ie I use ldap based authentication, so in development the account and password is REAL). So it
lets me still see whats going on, without printing out the password (My password)
1000 plus times.

The fix:
in app/controllers/application.rb

filter_parameter_logging "password"

This will get rid of any filtered parameter.

The result (in the development log):

Processing AccountController#login (for 127.0.0.1 at 2008-01-31 13:10:58) [POST]
Session ID: 9a85051d00c5aee3dc20305ac0dd315e
Parameters: {"commit"=>"Log in", "action"=>"login", "controller"=>"adminspace/account", "login"=>"user", "password"=>"[FILTERED]"}

Lovely.

You can also hide other things as well if you want to keep them secure, credit card numbers etc are good examples.

I tend to think this should be your "default" setting in any application.
It will be in mine from now on.

This works in 2.0.2 rails very nicely. (It was actually added quite a while back)

If you ask me it should be the "default" in the template when generating a rails applicaton.

Oracle and Rails - Moving tables to new tablespaces

2008-01-27T18:32:00.000-08:00

Hmmm. Things are growing in my oracle tables, and I really want to
move things around a bit. So moving a "application" or "user" to a new
tablespace is resonably common.

For me, language for first resort is ruby, so I have quite
a few "little" scripts to do oracle tasks.

Here's today's, move all of a users tables to a new tablespace.

require 'DBI'
require 'pp'
#movetablepace.rb
#***************************************
#make a Oracle Connection

fromdbconstr = ARGV[0] # For example - DBI:OCI8://myoracleserver/oracleinstance
fromdbconuser = ARGV[1] # MYRAILSDBUSER-DEVELOP
fromdbconpass = ARGV[2] # MYRAILSDBUSER-PASSWORD
dataspacename = ARGV[3] # APPLICATION-DATASPACE

fromdb = DBI.connect(fromdbconstr, fromdbconuser, fromdbconpass)

tables = fromdb.tables
tables.each do |tablename|
tablename = tablename.strip
if tablename.length == 0
next
end
if tablename.length > 4
if tablename[0,4] == "BIN$"
next
end
end
puts "Move Table [#{tablename}] to new tablespace"
sqlmove = "alter table " + '"' + tablename + '" move tablespace "' + dataspacename + '"'
puts sqlmove
begin
result = fromdb.execute(sqlmove)
rescue
puts "Table [#{tablename}] not moved"
end

end

And after you do that, you need to "rebuild" the indexes into the new tablespace

require 'DBI'
require 'pp'

#make a Oracle Connection

fromdbconstr = ARGV[0] # For example - DBI:OCI8://myoracleserver/oracleinstance
fromdbconuser = ARGV[1] # MYRAILSDBUSER-DEVELOP
fromdbconpass = ARGV[2] # MYRAILSDBUSER-PASSWORD
dataspacename = ARGV[3] # APPLICATION-DATASPACE

fromdb = DBI.connect(fromdbconstr, fromdbconuser, fromdbconpass)

tables = fromdb.tables
sqlget = "select index_name from user_indexes"
indexs = fromdb.execute(sqlget)
indexs.each do |theindex|
indexname = theindex[0]
printf "Rebuilding Index %s\n", indexname
sqlrebuild = 'alter index "' + indexname + '" rebuild tablespace "' + dataspacename + '"'
begin
result = fromdb.execute(sqlrebuild)
rescue
printf "[%s] Index not rebult\n", indexname
end
end

When the C-LEVEL(CTO,CEO,CFO) ask so why should we use Rails

2008-01-25T21:57:00.000-08:00

Ok Mr. Big CTO, why shuld we use RAILS?
A question that was recently asked in the Ruby forums.

So having done a few C-Level jobs, I thought I'd answer.

1. Is it going to be around a long time - Yes - Ruby itself has, and
Rails has hit 2. Plus recognition from
major software firms. (Oracle offered class on it on the Shanghai
Training Session,), oracle has support.
Major media companies are using it. Other MNC are using. Trading(As in
stock, on wallstreet) are using it.
Look at the "books" available. Need one every week. Look at the number
of web articles. By every metric its
everywhere.

2. Is training available? Yes, on multiple fronts. Also fresh java
grads tend to pick it up easier.

3. Does it marry with my "enterprise" systems. Now this one is
interesting. Ruby can marry to just
about anything, so thus "rails" can. I am just finishing a "marriage"
between Unify and Rails. The original
applicaiton is over 15 years old, and the data is still worth
millions. Of course I "moved" the data over to
oracle. (With ruby). I've also did "bridges" between applications.

3. Is it secure? If its running on "Linux" would be my answer. Look at
the security guide. Put a https proxy(Nginx)
in fromt. 100x more secure than windows.

4. Is it scalable. Big question. But in a short answer, yes yes yes.
For our users, 3000-4000 is max user count.
I even was helping a friend with handling million plus user bases.
Hardware is cheap, with nginx, mongrel clusters, and
good hardware its falling off a log. (If your public hosting, even a
single server can handle your average load").

5. Can I get consultants? Yes. Look at the tavnav guys for one. There
great.

6. But does it take "long" to do the job?
Its the "fastest" system I've found to get thing done fast, neat, and quick.
Look up "DRY" under Rails. It stands for "Dont Repeat Yourself".

7. Does it do "Windows"
Yes very well. How about a "Applicatoin-to-Application" Bridge done
in a week as a Windows "Service"

8. Does it do "Clusters"
Yes

9. Does it do "Linux"
Yes

10. Is it better than Java
Yes - very. So much more concise.

Advanced Rails Migrations - Adding and Updating Related Table ID's

2008-01-22T18:49:00.001-08:00

Ever have that "rails" problem.
You imported a butch of "existing" tables to create a rails application,
and you dont have appropriate index/id's to help with your belong_to, and
associated tables?

Well Migrations can do it for you.

Example migration using two tables that are in ActiveRecord.

This one builds a link between a inventory table, and a customer table.

class UpdateXco < ActiveRecord::Migration
def self.up
add_column :REG_INV_SERS, :REG_CUST_ID, :integer
RegInvSer.reset_column_information
RegInvSer.find(:all).each do |record|
regcustomer = RegCust.find(:first, :conditions => "CUST_NBR = '#{record.cust_nbr}'")
if not regcustomer.nil?
record.reg_cust_id = regcustomer.id
record.save
end
end
end
def self.down
remove_column "REG_INV_SERS", "REG_CUST_ID"
end
end

Another one, that illustrates doing a table import from a dbase/foxpro table, with a id update as well.

class Updatesalesorder < ActiveRecord::Migration
def self.up

thepathname = "h:xxxxx.DBF"
table = DBF::Table.new(thepathname)
table.records.each do |record|
mysalesorder = salesorder.create(record.attributes)
mycustomer = customer.find(:first, :conditions => "custno = '#{record.custno}'")
if not mycustomer.nil?
mysalesorder.customer_id = mycustomer.id
mysalesorder.save
end
end
end
def self.down
drop_table "salesorders"
end
end

Ruby - A "class" tutorial using a Unify DBI interface as a example

2008-01-21T23:02:00.001-08:00

This is a "good" example of a "class".
It implemnets a simple "dbi" like interface to a unify IDS database.
(This is a embedded database used in linux, and embedded devices)

Recently needed to access data from a unify database in "rails".

So of course, needed a "dbi" like layer to work with one of my scripts to
pull the data. (See previous posts on data conversion and migrations).

So it was natural to create a "class". A class is a "object" consisting of code,
or methods, and "data". The data can come in two types "instance" and class data.
Here everything is "instance" data. So when we create a new "Unify" object, we
get another set of "data" for that instance.

In our example, we use two temp files per instance to communication to Unify sql command via a "shell" command. One is for the SQL statement, and one is for the "results".

Also the lovely thing about "instances" is it makes it easy to see all your "instance" data, simply by doing a "pp" (Pretty Print) on the "instance" variable.

So:
mydb = Unify.new(database,user,password)
pp mydb (Will let you see what going on with the instance)

Note that all the "@" instance varibles are recreated on each "new".

This allows this simple example to have multiple outstanding "queries", without
getting confused.

class Unify
def make_tmpname(type,instance)
return 'unify-' + type + '-' + instance + '.tmp'
end
def execute_unify_sql(thesql)
thequery = File.open(@uqueryfilename,"w")
thequery.puts "lines 0"
# thequery.puts "separator ','
thequery.puts thesql
thequery.close
thecommand = 'SQL ' + @uqueryfilename + '> ' + @udatafilename
result = system(thecommand)
if result == false
sleep(10)
result = system(thecommand)
end
theresult = File.open(@udatafilename)
@udatafile = theresult
return theresult
end
def initialize(database,user,password)
@udatabase = database
@uuser = user
@upassword = password
@udatafile = nil
@udatafilename = nil
@uqueryfilename = nil
queryinstance = rand(10000).to_s
@udatafilename = make_tmpname('result',queryinstance)
@uqueryfilename = make_tmpname('query',queryinstance)
end
def fields(tablename)
thesql = 'fields ' + tablename
thedata = execute_unify_sql(thesql)
header = thedata.gets
header = thedata.gets #throw away the two header lines
result = Array.new
thedata.readlines.each { |line|
theline = line.chomp!
# theline <= '|' # last element does not terminate with |
element = theline.split
result << element
}
self.finish()
return result
end
def tables
thedata = execute_unify_sql("tables")
thecontents = thedata.read
self.finish
mytables = thecontents.split
return mytables
end
def execute(sql)
#For execute create a new "instance"
unifyinstance = Unify.new(@udatabase,@uuser,@upassword)
if sql[-1,1] == ';'
sql[-1,1] = '/'
end
theresultfile = unifyinstance.execute_unify_sql(sql)
return unifyinstance
end
def fetch()
@udatafile.pos = 0 #Make sure we are at the beginning
thetable = Array.new
while thedata = @udatafile.gets
thedata.chomp!
therow = thedata.split('|',-1) # No supressed fields
thetable << therow
end
return thetable
end

def finish()
@udatafile.close
File.delete(@udatafilename)
File.delete(@uqueryfilename)
end

end

Rails 2.0.2 Issues and Problems Resolved

2008-01-21T01:31:00.000-08:00

Now this was fun, my main Rails box is now running Rails 2.0.2

First problem, when running Rake task, I get complaints about Active Recrd

C:\projects\ror\tempco>rake extract_fixtures
(in C:/projects/ror/tempco)
rake aborted!
can't activate activerecord (= 1.15.3), already activated activerecord-2.0.2]

This one stumped me for a moment, but then realized that the "old" gems may
be confusing issues. So a little cleanup is in order.

gem cleanup

This command cleans up all "the" old gems.
And solved my problems

The next problem I had, was:

C:\projects\ror\tempco>rake extract_fixtures
(in C:/projects/ror/tempco)
rake aborted!
Please install the oracle adapter: `gem install activerecord-oracle-adapter` (no
such file to load -- active_record/connection_adapters/oracle_adapter)

Rails 2 does not by default install ActiveRecord adapter for oracle.
Of course if you try the gem install command the error message recommends,
it still does not give you gratification.

The proper command that works is:

C:\projects\ror\tempco>gem install activerecord-oracle-adapter
--source http://gems.rubyonrails.org
Bulk updating Gem source index for: http://gems.rubyonrails.org
Successfully installed activerecord-oracle-adapter-1.0.0
1 gem installed

And now, I try to do a rake db:schema:dump
And the next problem occurs, seems like there is a "bug" in the
oracle adapter.

The fix seems to be here:
http://dev.rubyonrails.org/attachment/ticket/9062/fix_oracleadapter_problem_indexes.diff

But does not match the code in the 1.0 adapter. Hmmm.

Ok, lets try to setup a "session" migration.

Hmm same error

C:\projects\ror\tempco>
C:\projects\ror\tempco>rake db:migrate
(in C:/projects/ror/tempco)
== 1 CreateSessions: migrating ================================================
-- create_table(:sessions)
-> 38.5940s
-- add_index(:sessions, :session_id)
-> 0.0930s
-- add_index(:sessions, :updated_at)
-> 0.0630s
== 1 CreateSessions: migrated (38.7500s) ======================================

rake aborted!
select_rows is an abstract method

Ok, do a bit more google work, and what do we find:

http://dev.rubyonrails.org/ticket/10415

Insert the code into the oracle adapter, and now no error on our migration.

And our db:schema:dump works properly.

RedhatOnRails or CentosOnRails

2007-11-27T18:54:00.000-08:00

Wow, I finally did it.
A automatied script to get NGINX, Mongrel/MongrelCluster, and Rails up on Redhat/Centos.
The gotcha that was bad was I assumed the Centos/Redhat ruby was up to date. Its not.
And causes intersting problems. Even using yum to remove it does not get rid of the old binary.

So make sure you dont have ruby installed, and if you do you have to go and remove it both with yum and manually. Then you should be fine.

The script install ruby, ImageMagic, Nginx, and rails, and gem.
I found that gem install for rails was not working for linux. But installing with
the gem manually worked fine.

#install_rails - dont forget the chmod +x install_rails and run as root
yum install -y gcc
yum install -y buildsys-build
wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p110.tar.gz
tar -xzvf ruby-1.8.6-p110.tar.gz
cd ruby-1.8.6-p110
./configure && make && make install
cd ..
rm -r -f ruby-1.8.6-p110
rm ruby-1.8.6-p110.tar.gz
yum install -y openssl-devel
wget http://rubyforge.org/frs/download.php/28174/rubygems-0.9.5.tgz
mkdir rubygems
mv rubygems-0.9.5.tgz rubygems
cd rubygems
tar xzvf rubygems-0.9.5.tgz
cd rubygems-0.9.5
ruby setup.rb
cd ..
cd ..
rm -r -f rubygems
yum -y install libpng-devel libjpeg-devel libtiff-devel freetype-devel ghostscript-devel
wget ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz
tar -xzvf ImageMagick.tar.gz
cd ImageMagick-6.3.7
./configure
make && make install
cd ..
rm -rf ImageMagick-6.3.7
cp /etc/ld.so.conf /etc/ld.so.conf.backup
sed '$a\
/usr/local/lib' /etc/ld.so.conf > ld.so.conf.new
mv ld.so.conf.new /etc/ld.so.conf
/sbin/ldconfig -v
/sbin/ldconfig
gem install rmagick
yum -y install pcre pcre-devel
wget http://sysoev.ru/nginx/nginx-0.5.33.tar.gz
tar -xzvf nginx-0.5.33.tar.gz
cd nginx-0.5.33
./configure --with-http_ssl_module
make
make install
cd ..
rm -r -f nginx-0.5.33
rm nginx-0.5.33.tar.gz
wget http://rubyforge.org/frs/download.php/28337/rails-1.2.6.gem
gem install ./rails-1.2.6.gem
gem install mongrel mongrel_cluster

Getting started faster in rails

2007-11-06T01:25:00.000-08:00

Had a comment on my blog for "create xyz application" did not work.
I suspect the cut/paste must of went astray. Just to make sure
your start in rails is a bit faster, I wrote a quick ruby script.

So install ruby and rails per the previous instructions, then this script
will create your "rails" app tree, and automatically install the three plugins
that I recommend to get started in Rails.

1. ActiveScaffold
2. RoleRequirement
3. Widgets/Tabnav

To run the script

myrails appname

#myrails.rb
# Usage: myrails appname
myappname = ARGV[0]
puts "Creating Initial App - " + myappname
system ("rails " + myappname)
Dir.chdir(myappname)
#Each of these must be a single line, no returns/newline allowed - Must be 3 system lines
system ('ruby ./script/plugin install http://activescaffold.googlecode.com/svn/tags/active_scaffold')
system ('ruby ./script/plugin install http://rolerequirement.googlecode.com/svn/tags/role_requirement/')
system ('ruby ./script/plugin install svn://svn.seesaw.it/widgets/trunk')

Can you create xyz application in rails

2007-10-30T20:05:00.000-07:00

Yes you can. And its easy.

First oh, Go to your favorate book seller and buy the following books:

1. The Ruby Way - Second Edition By Hal Fulton
2. Rails Recipes - Fowler
3. Ruby Cookbook - Lucas Carlson & Leonard Richardson
4. Agile Web Development with Rails - Thomas Heinemeler Hansson

Then your ready to get started:

Its very easy.

1. Install Rails
gem install rails --include-dependencies
2. Add ActiveScoffold PlugIn (http://activescaffold.com/)
./script/plugin install http://activescaffold.googlecode.com/svn/tags/active_scaffold
3. Add RoleRequirement (http://code.google.com/p/
rolerequirement/)
./script/plugin install http://rolerequirement.googlecode.com/svn/tags/role_requirement/
4. Add TabNav/Widgets (ruby script/plugin install
svn://svn.seesaw.it/widgets/trunk) (Information: http://www.seesaw.it/en/toolbox/widgets/)
ruby script/plugin install svn://svn.seesaw.it/widgets/trunk
5. Define your takes in a migration

6. Build your model files (3-4 lines each)

7. Buld your controllers (for each table)

8. Build your menus (using tabnav)

9. Add your relationships and tweak tabnav.

10. Add email confirmation, and notification.

Rails Tabnav - Fixing Menu Spacing

2007-10-30T18:13:00.000-07:00

For my applications, I want a good "look" and feel, with standadized menus.
My "menu" of choice is Tabnav/widget done by seesaw.

In "Enterprise" applications, the number of menus sometimes can be "large".
For example if you want a "menu" per department, there could be 20 departments
in a large company.

While tabnav does not "limit" you in the number or levels of menus, there
is a minor overlap problem that can occur if the screen is swunken below
a certan point, or you have "to" many menus on one "level.

There's a "easy" fix for handling this.

In your project/vendor/plugins/widgets/lib/widgets/tabnav.css.erb file, you
can adjust the overlap.

If you look at the "top" of the tabnav.css.erb file, you will notice the "line-height", I found by setting this to 170% my overlap problems disappeared.

Modified tabnav.css.erb
color: #000;
line-height: 170%;
border-bottom: 2px solid black;
margin: 13px 0px 0px 0px;
padding: 0px;
z-index: 1;
padding-left: 10px

Original tabnav.css.erb (First few lines)
color: #000;
border-bottom: 2px solid black;
margin: 13px 0px 0px 0px;
padding: 0px;
z-index: 1;
padding-left: 10px

Ruby and ActiveDirectory - Finding Servers and Dead Computer Accounts

2007-10-30T01:38:00.001-07:00

Lets expand our example to find servers, and dead computers.
One thing, the "servers" dont always appear when we search for "computer" accounts, in AD. Didnt find a known bug for this, but servers I know are "out" there just didnt come back.

So a few more "gotcha" when using AD from Ruby/LDAP

1. Servers dont "come-back" from a search of objectype = Computer
2. "Server" objectype does not give you the OS. Must search for server, then find it specifically to get the OS.
3. You might have duplicates if you add these togeather.
4. AD Date and Time is very different format than ruby's, so thanks to google, I found some nice routeines for converting between the two.

Example code:

require 'ldap'
require "pp"

def fetch_computers(domain_controller,the_username, the_password)
conn = LDAP::Conn.new( domain_controller, 389 )
conn.set_option( LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )
conn.bind( the_username, the_password ) do |conn|

base = 'DC=company,DC=com'

results = conn.search2(base, LDAP::LDAP_SCOPE_SUBTREE, "objectClass=Computer")
if results == LDAP::LDAP_SIZELIMIT_EXCEEDED
printf "To Much Data\n"
end
return results
end
end

def fetch_servers(domain_controller,the_username, the_password)
conn = LDAP::Conn.new( domain_controller, 389 )
conn.set_option( LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )
conn.bind( the_username, the_password ) do |conn|

base = 'DC=company,DC=com'

results = conn.search2(base, LDAP::LDAP_SCOPE_SUBTREE, "objectClass=Server")
if results == LDAP::LDAP_SIZELIMIT_EXCEEDED
printf "To Much Data\n"
end
return results
end
end

def fetch_computer(domain_controller,the_username, the_password,thecomputername)
conn = LDAP::Conn.new( domain_controller, 389 )
conn.set_option( LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )
conn.bind( the_username, the_password )

base = 'DC=company,DC=com'

results = conn.search2(base, LDAP::LDAP_SCOPE_SUBTREE, "cn=#{thecomputername}")
return results
end

#[["Windows 2000 Server"],
# ["Windows Server 2003"],
# ["Windows XP Professional"],
# ["Windows 2000 Professional"],
# ["Windows NT"],
# ["Windows Vista\342\204\242 Business"],
# ["Windows XP Tablet PC Edition"]]

AD_EPOCH = 116_444_736_000_000_000
AD_MULTIPLIER = 10_000_000

# convert a Time object to AD's epoch
def time2ad(time)
(time.to_i * AD_MULTIPLIER) + AD_EPOCH
end

# convert from AD's time string to a Time object
def ad2time(time)
begin
thetime = Time.at((time.to_i - AD_EPOCH) / AD_MULTIPLIER)
return thetime
rescue
return 0
end

end

thedc = "myserver" # The "nearest" domain controler name
theaccount = 'domain\username' # Change this to your domain and username
thepassword = 'password' # Gota have a password

operating_systems = Array.new
w2000servers = Array.new
w2003servers = Array.new
winservers = Array.new
oldaccounts = Array.new

result = fetch_computers(thedc, theaccount, thepassword)
today = Time.now
result.each {|entry|
#printf "%s - %s\n", entry["cn"], entry["operatingSystem"]
thePasswordTime = ad2time(entry["pwdLastSet"].to_s)
theAge = (today - thePasswordTime) / 86400 # Number of days
if theAge > 90
oldaccounts << entry["cn"]
end
if entry["operatingSystem"] != nil
operating_systems << entry["operatingSystem"]
operating_systems.uniq!
if entry["operatingSystem"] == ["Windows 2000 Server"]
w2000servers << entry["cn"]
winservers << entry["cn"]
end
if entry["operatingSystem"] == ["Windows Server 2003"]
w2003servers << entry["cn"]
winservers << entry["cn"]
end
end
}

result = fetch_servers(thedc, theaccount, thepassword)
today = Time.now
result.each {|entry|
winservers << entry["cn"]
# The server AD entry does not contain the OS, so fetch the Servers ObjectType = Computer entry
myservers = fetch_computer(thedc,theaccount,thepassword,entry["cn"].to_s)

myservers.each { |myserver|
if myserver["operatingSystem"] == ["Windows 2000 Server"]
w2000servers << entry["cn"]
end
if myserver["operatingSystem"] == ["Windows Server 2003"]
w2003servers << entry["cn"]
end

}
}

w2000servers.uniq!
w2003servers.uniq!
winservers.uniq!

puts
pp operating_systems
puts
puts "Win2000 Servers "
pp w2000servers.sort!
puts
puts "Win2003 Servers "
pp w2003servers.sort!

puts "Windows Servers"
pp winservers.sort!

puts
puts "Old/Unused Computer Accounts"
pp oldaccounts.sort!

Ruby LDAP Example - How to find Computers

2007-10-29T18:19:00.000-07:00

Ever want to find "all" your computers in a large "domain".
Having trouble find examples?

Some gochas from using ruby ldap

1. Make sure the Active Directory Domain Controller is on the same "network" segment as you. I've seen this not work if your going over a WAN. And no error message is given. This happend when I was doing my first Active Directory login authentication. Once I used a local domain controller everything was fine.
2. On the "ObjectClass" string extra parens and quotes cause it not to work. and no error message is given.

Here's a working one

require 'ldap'
require "pp"

conn = LDAP::Conn.new( 'sinsiw2k1', 389 )
conn.set_option( LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )
conn.bind( 'domain\myname', 'mypass' ) do |conn|

base = 'DC=company,DC=com'

results = conn.search2(base, LDAP::LDAP_SCOPE_SUBTREE, "objectClass=Computer")
if results == LDAP::LDAP_SIZELIMIT_EXCEEDED
printf "To Much Data\n"
end
pp results
pp results.controls
pp results.referrals

results.each { |entry| puts entry }

end

Importing from ODBC sources into Oracle and Rails

2007-10-22T18:37:00.000-07:00

Having lots of data in a "third" party database, I needed a way of moving it over into a "rails" friendly environment. ODBC provided the "interface". This should work for most odbc/dbi compliant databases.

Note that you might have to work a bit on "dates".

require 'DBI'
require 'pp'
require 'active_support'
require 'fileutils'

def numeric?(thenumber)
if thenumber =~ /^[0-9]+$/
return(TRUE)
else return(FALSE)
end
end

def tableexists?(wantedtablename)

tables = $existing_tables
tables.each do |tablename|
if wantedtablename == tablename
return TRUE
end
end
return FALSE

end

def sanitize(thevalue)
thevalue.gsub(/[^[:alnum:]]+/, '_')
end

def XlateColumnName(thename)
thename = thename.delete(" ")
thename = thename.upcase
if numeric?(thename)
thename = "X" + thename
end
thename = sanitize(thename) # Get rid of / and \ and other goodies
if thename == "LOCK"
thename = "XLOCK"
end
if thename == "SCOPE"
thename = "XSCOPE"
end
if thename == "TYPE"
thename = "XTYPE"
end
if thename == "CLASS"
thename = "XCLASS"
end
if thename == "COST"
thename = "XCOST"
end
if thename == "FIXED"
thename = "XFIXED"
end
if thename == "CURRENT"
thename = "XCURRENT"
end
if thename == "DATE"
thename = "XDATE"
end
if thename == "TIME"
thename = "XTIME"
end
if thename == "NUMBER"
thename = "XNUMBER"
end
if thename == "CONDITION"
thename = "XCONDITION"
end
if thename == "SEQ"
thename = "XSEQ"
end
if thename == "COMMENT"
thename = "XCOMMENT"
end
return(thename)
end

def dotable(fromdb, todb, tablename, prefix)

tablename = tablename.strip
if tablename.length == 0
return
end
stmt = "select * from #{tablename};"
begin
tabledata = fromdb.execute(stmt)
rescue
return
end
row = tabledata.fetch # Get one
if row == nil #maybe we dont have data:)
tabledata.finish #on empty data dont bother to create it
return
end

tablename = sanitize(tablename) #Get rid of nasties
railsname = prefix + "_" + tablename
railsname = Inflector.pluralize(railsname)
railsname = railsname.upcase
if tableexists?(railsname)
printf "Exists %s\n",railsname
tabledata.finish
return
end
puts "Created #{railsname}"
#sqldrop = "DROP TABLE " + '"' + railsname + '" PURGE'
#begin
# result = todb.execute(sqldrop)
#rescue
# puts "Table [#{railsname}] not dropped"
# end
sqlcreate = String.new()
sqlcreate = "CREATE TABLE "
sqlcreate << '"'
sqlcreate << railsname
sqlcreate << '" ('
sqlcreate << '"ID" NUMBER(38,0) NOT NULL'
# Good reference for data conversion to oracle types
# http://www.indiana.edu/~dss/Services/DataWarehouse/Oracle/Sybase/conversion.html
fieldnames = Array.new
tabledata.column_info.each_with_index do |info, i|
sqlcreate << ", "
sqlcreate << '"'
thename = info["name"]
thename = XlateColumnName(thename)
sqlcreate << thename
sqlcreate << '" '
thetype = info["type_name"]
theprecision = info["precision"]
thescale = info["scale"]
if thetype == "CHAR"
thetype = "VARCHAR2(" + theprecision.to_s() + " BYTE)"
end
if thetype == "DOUBLE"
thetype = "NUMBER(38,0)"
end
if thetype == "BIT"
thetype = "CHAR(1)"
end
if thetype == "TINYINT"
thetype = "NUMBER(5,0)"
end
sqlcreate << thetype
fieldnames << thename
end
sqlcreate << ', PRIMARY KEY ("ID") '
sqlcreate << ")"
begin
result = todb.execute(sqlcreate)
rescue
puts "convertdata: Cannot create #{railsname}"
puts sqlcreate
raise
end

#Drop any "old" sequence
sqldrop = "DROP SEQUENCE " + '"' + railsname + '_SEQ"'
begin
result = todb.execute(sqldrop)
rescue
#puts sqldrop
#puts "Sequence [#{railsname}] not dropped"
end
#Now create the RAILS Sequence number
#CREATE SEQUENCE "AIRSTATION-DEVELOP"."BUSINESSUNITS_SEQ"
# MINVALUE 1 MAXVALUE 999999999999999999999999999 INCREMENT BY 1 START WITH 10040 CACHE 20 NOORDER NOCYCLE ;
sqlseq = String.new()
sqlseq << "CREATE SEQUENCE "
sqlseq << '"'
sqlseq << railsname
sqlseq << '_SEQ" '
sqlseq << "MAXVALUE 999999999999999999999999999 INCREMENT BY 1 START WITH 10000 CACHE 20 NOORDER NOCYCLE"
begin
result = todb.execute(sqlseq)
rescue
puts sqlseq
raise
end

if row == nil #maybe we dont have data:)
tabledata.finish
return
end
doinsert(todb,railsname,fieldnames,row) # We already "Fetched" One

tabledata.fetch do |row|
doinsert(todb,railsname,fieldnames,row)
end

tabledata.finish

end

#
def doinsert(thedb, tablename, fieldnames, row)
if row == nil
return
end
sqlinsert = String.new
sqlinsert = "INSERT INTO "
sqlinsert << '"'
sqlinsert << tablename
sqlinsert << '" ('
sqlinsert << "ID,"
fieldnames.each do |myfield|
sqlinsert << myfield + ","
end
#Must get rid of trailing comma and space
sqlinsert = sqlinsert[0,sqlinsert.length - 1]
sqlinsert << ') VALUES ('
sqlinsert << tablename + '_SEQ.NEXTVAL, '
row.each_with_name do |val, name|
val = val.to_s.strip
if val.length == 0
sqlinsert << "NULL,"
else
sqlinsert << thedb.quote(val.to_s.strip)
sqlinsert << ","
end
end
#Must get rid of trailing comma and space
sqlinsert = sqlinsert[0,sqlinsert.length - 1]
sqlinsert << ')'
begin
result = thedb.execute(sqlinsert)
rescue
puts sqlinsert
raise
end

end

argcount = ARGV.size
if argcount < 7
puts "command line is convertdata.rb FromDbDbi Username Password ToDbDbi Username Password Prefix"
puts " ie: convertdata.rb DBI:ODBC:PRODDATA . . DBI:OCI8:XE . . MAG"
puts " . = a empty string"
exit
end
fromdbconstr = ARGV[0]
fromdbconuser = ARGV[1]
fromdbconpass = ARGV[2]
todbconstr = ARGV[3]
todbconuser = ARGV[4]
todbconpass = ARGV[5]
prefix = ARGV[6]

if fromdbconuser == '.'
fromdbconuser = ''
end
if fromdbconpass == '.'
fromdbconpass = ''
end
if todbconuser == '.'
todbconuser = ''
end
if todbconpass == '.'
todbconpass = ''
end

printf "FromDB: [%s] User [%s] Password [%s]\n",fromdbconstr,fromdbconuser,fromdbconpass
printf "ToDB: [%s] User [%s] Password [%s]\n",todbconstr, todbconuser, todbconpass
printf "Prefix: [%s]\n", prefix

#make a ODBC Connection
fromdb = DBI.connect(fromdbconstr,fromdbconuser,fromdbconpass);
todb = DBI.connect(todbconstr,todbconuser,todbconpass)
#fromdb = DBI.connect('DBI:ODBC:PRODDATA','','')
#todb = DBI.connect('DBI:OCI8:XE', 'AIRSTATION-DEVELOP', 'AIRSTATION-DEVELOP')
#prefix = "MAG"

tables = fromdb.tables
$existing_tables = todb.tables

tables.each do |tablename|
dotable(fromdb, todb, tablename, prefix)
end

Import and Cleanup Dbase Files into Rails with Oracle Support

2007-10-18T20:45:00.000-07:00

Wow, so many files, and how to clean them up.
I can do migrations (See my previous post) with DBF,
and do some nice cleanup as I go, but I really want to "get"
rid of the dbase files, and make them native as fast as possible.

So finally wrote a dbf-dbi-to-oracle converter.

The issues you get with dbase files into rails are numerous. Lack
of ID field, and names that are reserved words in SQL.

Even "splits" of tables into multiple files is resonbly common.
And worse the "schema" could change over the life of the dbf files,
with some "dbf" files having one schema, and newer ones having signficant
new fields.

So here is the solution to all of this.

A importdbf script

require 'DBI'
require 'pp'
require 'active_support'
require 'fileutils'
require 'dbf'
require 'date'

def sanitize(thevalue)
thevalue.gsub(/[^[:alnum:]]+/, '_')
end

def numeric?(thenumber)
if thenumber =~ /^[0-9]+$/
return(TRUE)
else return(FALSE)
end
end

def tableexists?(wantedtablename)

tables = $existing_tables
tables.each do |tablename|
if wantedtablename == tablename
return TRUE
end
end
return FALSE

end

def FileExtension(thepath)
thedot = thepath.rindex('.')
thelen = thepath.length - thedot
theextension = thepath[thedot,thelen]
theextension = theextension.downcase
return theextension
end

#
def doinsert(fromdb, thedb, tablename, fieldnames, row)
if row == nil
return
end
sqlinsert = String.new
sqlinsert = "INSERT INTO "
sqlinsert << '"'
sqlinsert << tablename
sqlinsert << '" ('
sqlinsert << "ID,"
fieldnames.each do |myfield|
sqlinsert << myfield + ",\n"
end
#Must get rid of trailing comma and space
sqlinsert = sqlinsert[0,sqlinsert.length - 2]
sqlinsert << ') VALUES ('
sqlinsert << tablename + '_SEQ.NEXTVAL, '
fromdb.columns.each_with_index do |info, i|
val = row.attributes[info.name].to_s
if info.type == "D" #DATE
# "Mon Jan 01 00:00:00 UTC 2001" - dbf date
dbfformat = "%a %b %d %H:%M:%S %Z %Y"
#to_date('12-31-2007 12:15','MM-DD-YYYY HH:MI)
if val != ""
thedate = Date.parse(val)
val = "to_date('" + thedate.strftime("%m-%d-%Y %H:%M") + "', 'MM-DD-YYYY HH24:MI')"
sqlinsert << val
else
sqlinsert << "NULL"
end
elsif info.type == "M"
sqlinsert << thedb.quote(val.to_s.strip)
elsif info.type == "L"
if val == "true"
val = "Y"
end
if val == "false"
val = "N"
end
sqlinsert << thedb.quote(val.to_s.strip)
else
sqlinsert << thedb.quote(val.to_s.strip)
end
# printf "Name = %s Type = %s Value = %s\n", info.name, info.type, val
sqlinsert << ",\n"
end
#Must get rid of trailing comma and space
sqlinsert = sqlinsert[0,sqlinsert.length - 2]
sqlinsert << ')'
begin
# puts sqlinsert
tabledata = thedb.execute(sqlinsert)
rescue
puts sqlinsert
thesql = File.open("convertdbf.sql","w")
thesql.puts sqlinsert + ";"
thesql.close
raise
end
tabledata.finish

end

def ImportTable(fromdb,todb,railsname,fieldnames)

fromdb.records.each do |record|
doinsert(fromdb, todb,railsname,fieldnames,record)
end
end

def XlateColumnName(thename,thetype)
thename = thename.delete(" ")
thename = thename.upcase
if thetype == "D" #DATE Should have it in the name
if !thename.include?("DATE")
thename << "DATE"
end
end
if numeric?(thename)
thename = "X" + thename
end
if thename == "FILE"
thename = "XFILE"
end
if thename == "LOCK"
thename = "XLOCK"
end
if thename == "SCOPE"
thename = "XSCOPE"
end
if thename == "TYPE"
thename = "XTYPE"
end
if thename == "CLASS"
thename = "XCLASS"
end
if thename == "COST"
thename = "XCOST"
end
if thename == "FIXED"
thename = "XFIXED"
end
if thename == "CURRENT"
thename = "XCURRENT"
end
if thename == "DATE"
thename = "XDATE"
end
if thename == "TIME"
thename = "XTIME"
end
if thename == "NUMBER"
thename = "XNUMBER"
end
if thename == "CONDITION"
thename = "XCONDITION"
end
if thename == "SEQ"
thename = "XSEQ"
end
if thename == "COMMENT"
thename = "XCOMMENT"
end
return(thename)
end

def XlateColumntype(thetype, theprecision, thescale)

if thetype == "C"
thetype = "VARCHAR2(" + theprecision.to_s() + " BYTE)"
end
if thetype == "L" #Logical - false or true
thetype = "CHAR(1)"
end
if thetype == "N"
thetype = "NUMBER(38,0)"
end
if thetype == "D"
thetype = "DATE"
end
if thetype == "B"
thetype = "CHAR(1)"
end
if thetype == "M"
thetype = "CLOB"
end
return(thetype)
end

def CreateTable(fromdb,todb,railsname)
sqlcreate = String.new()
sqlcreate = "CREATE TABLE "
sqlcreate << '"'
sqlcreate << railsname
sqlcreate << '" ('
sqlcreate << '"ID" NUMBER(38,0) NOT NULL'
# Good reference for data conversion to oracle types
# http://www.indiana.edu/~dss/Services/DataWarehouse/Oracle/Sybase/conversion.html
fieldnames = Array.new

fromdb.columns.each_with_index do |info, i|
sqlcreate << ", "
sqlcreate << '"'
thename = XlateColumnName(info.name,info.type)
sqlcreate << thename
sqlcreate << '" '
thetype = info.type
theprecision = info.length
thescale = info.decimal
thetype = XlateColumntype(thetype,theprecision, thescale)
sqlcreate << thetype
fieldnames << thename
end
sqlcreate << ', PRIMARY KEY ("ID") '
sqlcreate << ")"
begin
tabledata = todb.execute(sqlcreate)
rescue
puts "convertdata: Cannot create #{railsname}"
thesql = File.open("convertdbf.sql","w")
thesql.puts sqlcreate + ";"
thesql.close
puts sqlcreate
raise
end
tabledata.finish
#Drop any "old" sequence
sqldrop = "DROP SEQUENCE " + '"' + railsname + '_SEQ"'
begin
tabledata = todb.execute(sqldrop)
tabledata.finish
rescue
#puts sqldrop
#puts "Sequence [#{railsname}] not dropped"
end
#Now create the RAILS Sequence number
#CREATE SEQUENCE "AIRSTATION-DEVELOP"."BUSINESSUNITS_SEQ"
# MINVALUE 1 MAXVALUE 999999999999999999999999999 INCREMENT BY 1 START WITH 10040 CACHE 20 NOORDER NOCYCLE ;
sqlseq = String.new()
sqlseq << "CREATE SEQUENCE "
sqlseq << '"'
sqlseq << railsname
sqlseq << '_SEQ" '
sqlseq << "MAXVALUE 999999999999999999999999999 INCREMENT BY 1 START WITH 10000 CACHE 20 NOORDER NOCYCLE"
begin
tabledata = todb.execute(sqlseq)
tabledata.finish
rescue
puts sqlseq
raise
end
$existing_tables << railsname
return(fieldnames)
end

def create_master_content_table(thedb,mastercontent_table)
sqlcreate = "CREATE TABLE " + mastercontent_table + "( SUBFILENAME VARCHAR2(32) )"
begin
tabledata = thedb.execute(sqlcreate)
rescue
puts sqlcreate
raise
end
tabledata.finish
$existing_tables << mastercontent_table
end

def is_in_mastertable?(thedb, mastertable,thename)
mastercontent_table = mastertable + "_FILES"
if tableexists?(mastercontent_table) == FALSE
create_master_content_table(thedb,mastercontent_table)
return(FALSE)
end
sqlquery = "SELECT * FROM " + mastercontent_table +
" WHERE SUBFILENAME = " +
thedb.quote(thename.to_s.strip)
tabledata = thedb.execute(sqlquery)
row = tabledata.fetch
if row == nil
tabledata.finish
return(FALSE)
end
tabledata.finish
return TRUE
end

def add_to_file_table(thedb, mastertable, thefilename )
sqlinsert = "INSERT INTO " + mastertable + "_FILES " + " (SUBFILENAME) VALUES ("
sqlinsert << thedb.quote(thefilename) + ")"
begin
tabledata = thedb.execute(sqlinsert)
rescue
puts sqlinsert
raise
end
tabledata.finish
end

def DescribeTable(todb,railsname)
fieldnames = Array.new
sqlquery = "SELECT * FROM " + railsname
tabledata = todb.execute(sqlquery)
for column in tabledata.column_info()
if column['name'] == "ID" # We "assume" that ID is present
next
end
fieldnames << column['name']
end
tabledata.finish
return(fieldnames)
end

def UpdateTableDef(fieldnames,fromdb,todb,railsname)

#alter table author add (author_last_published date);
sqlmodify = "ALTER TABLE "
sqlmodify << '"'
sqlmodify << railsname
sqlmodify << '" ADD ('
thistablefields = Array.new
newfields = Array.new
fromdb.columns.each_with_index do |info, index|
thename = XlateColumnName(info.name,info.type)
thistablefields << thename
if fieldnames.include?(thename)
next
end
newfields << thename
sqlmodify << '"'
sqlmodify << thename
sqlmodify << '" '
thetype = info.type
theprecision = info.length
thescale = info.decimal
thetype = XlateColumntype(thetype,theprecision, thescale)
sqlmodify << thetype
sqlmodify << ", "
fieldnames << thename
end
if newfields.empty? # Table definition is identical
return(thistablefields)
end
# pp newfields
#Must get rid of trailing comma and space
sqlmodify = sqlmodify[0,sqlmodify.length - 2]
sqlmodify << ")"
begin
tabledata = todb.execute(sqlmodify)
rescue
puts "convertdata: Cannot modify #{railsname}"
thesql = File.open("convertdbf.sql","w")
thesql.puts sqlmodify + ";"
thesql.close
puts sqlmodify
raise
end
tabledata.finish
return(thistablefields)
end

def ProcessFile(thename, thepath,todb,prefix,mastertable)
filetype = FileExtension(thepath)
if filetype != ".dbf"
return
end
tablename = thename.slice(0,thename.length - 4)
tablename = sanitize(tablename)
if numeric?(tablename)
tablename = mastertable
railsname = Inflector.pluralize(tablename)
railsname = railsname.upcase
if is_in_mastertable?(todb,mastertable,thename)
return
end
add_to_file_table(todb,mastertable,thename)
puts "Adding #{thename} to #{railsname}"
else
tablename = prefix + '_' + tablename
tablename = tablename.upcase
railsname = Inflector.pluralize(tablename)
railsname = railsname.upcase
if tableexists?(railsname)
printf "Exists %s\n",railsname
return
end
end
fromdb = DBF::Table.new(thepath,:in_memory => false)
if tableexists?(railsname) == FALSE
fieldnames = CreateTable(fromdb,todb,railsname)
puts "Created #{railsname}"
else
fieldnames = DescribeTable(todb,railsname)
fieldnames = UpdateTableDef(fieldnames,fromdb,todb,railsname)
end
ImportTable(fromdb,todb,railsname,fieldnames)
fromdb = nil
end

def DoDir(mydirectory,todb,prefix,mastertable)
Dir.foreach(mydirectory) do |entry|
# Dont bother with . and ..
next if [".",".."].include? entry
fullname = mydirectory + "\\" + entry
if FileTest::directory?(fullname)
DoDir(fullname,todb,prefix)
elsif FileTest::file?(fullname)
ProcessFile(entry,fullname,todb,prefix,mastertable)
else
puts "Unknown file type - #{fullname}"
return;
end
end
end

argcount = ARGV.size
if argcount < 6
puts "command line is convertdbf.rb FromPath ToDbDbi Username Password Prefix"
exit
end
fromdir = ARGV[0]
todbconstr = ARGV[1]
todbconuser = ARGV[2]
todbconpass = ARGV[3]
prefix = ARGV[4]
mastertable = ARGV[5]

if todbconuser == '*'
todbconuser = ''
end
if todbconpass == '*'
todbconpass = ''
end

#make a ODBC Connection
todb = DBI.connect(todbconstr,todbconuser,todbconpass)

$existing_tables = todb.tables

DoDir(fromdir,todb,prefix,mastertable)

A Production generator for Rails

2007-10-18T20:31:00.000-07:00

As I get into Rails, I'm generating "large" applications, and I really like the RAILS DRY principle. So for production I'm using a variety of "plug-ins"

ActiveScaffold
RoleRequirement
Widgets/Tabnav

In addition I'm using a Oracle Database.

So I wrote a generator that "reads" the Oracle Database, and generates any
"missing" controllers and models. The lovely part is, when I change the basic
controller, I just "erase" the ones I want updated, and fire the script again.

This also handles the pluraziation, making sure the controllers, models are named the same. Also tries to do some cleanup so the menu names are a little more humanized. In addition good example of using name spaces to help organize a large project better.

require 'DBI'
require 'pp'
require 'active_support'
require 'fileutils'

def domodel(tablename)
modelname = tablename.downcase
modelname = modelname.singularize

modeldir = "app\\models\\"
FileUtils.makedirs(modeldir)
modelpath = modeldir + modelname + ".rb"
if FileTest::exists?(modelpath)
printf "exists %s\n", modelpath
return
end
themodel = File.open(modelpath,"w")

themodel.puts "class " + modelname.camelize + " < ActiveRecord::Base"
themodel.puts "# "
themodel.puts "# Created By GenRails"
themodel.puts "# "
themodel.puts "end"
themodel.close
printf "create %s\n", modelpath
end

def docontroller(thenamespace,tablename, layout, prefix)
controllername = tablename.downcase
controllername = controllername.singularize
controllerfilename = controllername + "_controller.rb"
controllerdir = "app\\controllers\\" + thenamespace + "\\"
FileUtils.makedirs(controllerdir)
controllerpath = controllerdir + controllerfilename
if FileTest::exists?(controllerpath)
printf "exists %s\n", controllerpath
return
end
thecontroller = File.open(controllerpath,"w")

thecontroller.puts "class " + thenamespace.camelize + '::' + controllername.camelize + 'Controller < ApplicationController'
thecontroller.puts " layout '" + layout + "'"
thecontroller.puts " before_filter :login_required"
thecontroller.puts "# "
thecontroller.puts "# Generated by GenRails"
thecontroller.puts "# "
thecontroller.puts " "
thecontroller.puts "active_scaffold :" + controllername.camelize + " do |config|"
menuname = tablename.slice(prefix.length+1,tablename.length - prefix.length + 1)
menuname = menuname.camelize
thecontroller.puts " config.label = '" + menuname + "'"
thecontroller.puts " config.actions = [:search, :show, :list, :create, :delete]"
thecontroller.puts " end"
thecontroller.puts "end"
thecontroller.close
printf "create %s\n", controllerpath
end

def dotable(fromdb, namespace, tablename, layout, prefix)

tablename = tablename.strip
if tablename.length == 0
return
end
# begin
# row = fromdb.select_one("select count(*) from #{tablename}")
# rescue
# return
# end
# record_count = row[0]
# if record_count == 0
# return
# end

domodel(tablename)
docontroller(namespace,tablename, layout, prefix)
end

#make a ODBC Connection

fromdb = DBI.connect('DBI:OCI8:XE', 'AIRSTATION-DEVELOP', 'AIRSTATION-DEVELOP')
prefix = "MAG"
appname = 'overhaul'
namespace = appname + "space"
layout = "tier2" + appname

tables = fromdb.tables

tabnavname = '_' + appname + '_tabnav.rhtml'
tabnavdir = 'app\\views\\widgets\\'
FileUtils.makedirs(tabnavdir)
tabnavpath = tabnavdir + tabnavname
thetabnav = File.open(tabnavpath,"w")

thetabnav.puts '<%'
thetabnav.puts '# GenRails - Tabnav based menu'
thetabnav.puts 'render_tabnav :' + appname + ","
thetabnav.puts ' :generate_css => true do'
thetabnav.puts ' '
thetabnav.puts ' if @current_user.has_role?(' + '\'' + appname + '\')'
thetabnav.puts ' add_tab do |t|'
thetabnav.puts ' t.named ' + '\'' + appname.capitalize + '\''
thetabnav.puts ' t.titled ' + '\'' + appname.capitalize + ' Home\''
thetabnav.puts ' t.links_to :controller => ' + '\'' + namespace + '/' + appname + 'home' + '\''
thetabnav.puts ' end'

tables.each do |tablename|
thetableprefix = tablename.slice(0,prefix.length)
if thetableprefix == prefix
dotable(fromdb, namespace, tablename, layout,prefix)
menuname = tablename.slice(prefix.length+1,tablename.length - prefix.length + 1)
menuname = menuname.downcase
menuname = menuname.singularize
controllername = tablename.downcase
controllername = controllername.singularize
thetabnav.puts ' add_tab do |t|'
thetabnav.puts ' t.named ' + '\'' + menuname.capitalize + '\''
thetabnav.puts ' t.titled ' + '\'' + menuname.capitalize + '\''
thetabnav.puts ' t.links_to :controller => ' + '\'' + namespace + '/' + controllername + '\''
thetabnav.puts ' end'
end
end

thetabnav.puts ' end'
thetabnav.puts 'end'
thetabnav.puts '%>'
thetabnav.close
printf "updated %s", tabnavpath

Ruby Rails - Importorting Dbase and Foxpro Files

2007-10-04T19:41:00.000-07:00

Did you have foxprofiles that you just want to move into a rails environment.
Then dbf is your friend.

It lets you do "mirgrations" that move dbase into the rails database of your choice.

So for example, one I just did:

class CreateMrosalesorder < ActiveRecord::Migration
def self.up
create_table "mrosalesorders", :force => true do |t|
t.column "mrocustomer_id", :integer
t.column "sono", :string, :limit => 8
....
t.column "tot_mcmosr", :float
t.column "tot_labosr", :float
end
thepathname = "...\SOMAST01.DBF"
table = DBF::Table.new(thepathname)
table.records.each do |record|
mysalesorder = Mrosalesorder.create(record.attributes)
mymrocustomer = Mrocustomer.find(:first, :conditions => "custno = '#{record.custno}'")
if not mymrocustomer.nil?
mysalesorder.mrocustomer_id = mymrocustomer.id
mysalesorder.save
end
end
end
def self.down
drop_table "mrosalesorders"
end
end

Now the issue will occur that some "field" names in dbase/foxpro may be invalid in your database of choice.

So its easy to resolve that in your model.
When "fields" are set they call a "method" of varname= in your model.
So take care of your "renames" there. If your a "cobol" programmer this is
like a "move" corresponding, with a "rename" in the middle, without needeing
to handle all the fields that are not renamed.

class Mrosalesorder < ActiveRecord::Base
belongs_to :customer
belongs_to :mrocustomer

def comment=(value)
self.socomment=value
end

def current=(value)
self.curr=value
end

def to_label
"#{sono}"
end

end

Fetcher Typos Resolved

2007-10-02T00:47:00.000-07:00

Well, somehow it happened. A few typo's entered my perfect program. Why did it even work? :) The story of code. The new version starts faster, and more reliable as well.

Fetch your Exchange Mail, and call recievemail function directy in your ActiveRecord Models.

require 'rubygems'
require 'win32/service'
require 'logger'
require 'net/imap'
require 'net/pop'
include Win32

Dir.chdir("\\projects\\ror\\AirCenter")
require File.dirname(__FILE__) + '/../config/environment.rb'

SERVICE_NAME = 'AirFetchService'
SERVICE_DISPLAYNAME = 'AirCenterEmailFetcherService'
$logger = Logger.new("c:\\projects\\ror\\AirCenter\\log\\AirFetchService.log")

class Mailman < ActionMailer::Base
def receive(email)
sender = email.from.to_s()
subject = email.subject.to_s()
body = email.body.to_s()
subjectparts = subject.split(/ /)
classname = subjectparts[0]
$logger.info("Mailman: Sender = #{sender} Class = #{classname} Subject = #{subject}")
myclass = Object.const_get(classname)
if myclass
myclass.ReceiveEmail(sender,subject,subjectparts,body)
else $logger.warn("Mailman: Invalid class #{classname}")
#Request.ReceiveEmail(sender,subject,body)
end
end
end

if ARGV[0] == "register"
#Start The Service
svc = Service.new
svc.create_service do |s|
s.service_name = SERVICE_NAME
s.display_name = SERVICE_DISPLAYNAME
s.binary_path_name = "c:\\ruby\\bin\\ruby.exe c:\\projects\\ror\\AirCenter\\script\\AirFetchService.rb"
# + File.expand_path($0)
s.dependencies = []
end
svc.close
puts "Registered Service - " + SERVICE_DISPLAYNAME
elsif ARGV[0] == "test"
$logger.info "Service Starting"
$logger.info "Loading EMAIL Config"
@config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
$logger.info "Setting up config"
@config = @config[RAILS_ENV].to_options
$logger.info "Setting up sleep time"
@sleep_time = @config.delete(:sleep_time) || 60
$logger.info "Service Init Complete"
$logger.info "AirFetchService Establish mail processor"
# Add your own receiver object below
@fetcher = Fetcher.create({:receiver => Mailman}.merge(@config))
@fetcher.fetch
exit
elsif ARGV[0] == "delete"
if Service.status(SERVICE_NAME).current_state == "running"
Service.stop(SERVICE_NAME)
end
Service.delete(SERVICE_NAME)
else
if ENV["HOMEDRIVE"] !=nil
puts "Usage: ruby airfetchservice.rb [options]"
puts " where options are: register or delete"
exit
end

class MyDaemon < Win32::Daemon

def service_init
sleep 1
$logger.info "Service Starting"

end

def service_stop
$logger.info "AirFetchService Stopped"
end

def service_main
$logger.info "Loading EMAIL Config"
@config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
$logger.info "Setting up config"
@config = @config[RAILS_ENV].to_options
$logger.info "Setting up sleep time"
@sleep_time = @config.delete(:sleep_time) || 60
$logger.info "Service Init Complete"
$logger.info "AirFetchService Establish mail processor"
# Add your own receiver object below
@fetcher = Fetcher.create({:receiver => Mailman}.merge(@config))
$logger.info "AirFetchService Mail Processor Ready"

$logger.info "AirFetchService: Service in Running State"
while state == RUNNING
#begin
@fetcher.fetch
#rescue
#end
sleep(@sleep_time)
end
end
end

d = MyDaemon.new
d.mainloop

end

Getting Email from Exchange Into Rails

2007-09-28T20:19:00.000-07:00

So here I was, lost on a windows box, looking for a way to get supplies
that were badly needed. And what do you know, but Dan Weinand came to the rescue. Dan had written a plug-in for rails that "does" fetching of a IMAP server.

Dan Weinand site: http://www.danweinand.com/

In my application, I wanted my "clients" existing email infrastructure which consisted of Exchange 2003 soon to be exchange 2007. Just wanted to fetch a "email" address from the existing server.

So after asking permision we enable IMAP on exchange. And I pulled in the "fetcher"
into the application. Of course I wanted it to run as a "service" on windows. The original was designed for Linux. So I converted to a windows service. In addition
I found that the "standard" authentications in the fetcher for IMAP needed a "tweak" to do a "login" at the IMAP level, rather than "authorize".
So I added "dumb" authentication to the code.

The imap.rb in vendors/plugins/fetcher/lib needed the two patches
1. Added dumb authentication
2. Fixed the "recovery" so it would not cause a exception on a empty mailbox

--- imap.rb ---
module Fetcher
class Imap < Base

protected

# Additional Options:
# * :authentication - authentication type to use, defaults to DUMB
def initialize(options={})
@authentication = options.delete(:authentication) || 'DUMB'
super(options)
end

# Open connection and login to server
def establish_connection
@connection = Net::IMAP.new(@server)
if (@authentication == "DUMB")
@connection.login(@username,@password)
elsif
@connection.authenticate(@authentication, @username, @password)
end
end

# Retrieve messages from server
def get_messages
@connection.select('INBOX')
@connection.search(['ALL']).each do |message_id|
msg = @connection.fetch(message_id,'RFC822')[0].attr['RFC822']
begin
process_message(msg)
rescue
# handle_bogus_message(msg)
end
# Mark message as deleted
@connection.store(message_id, "+FLAGS", [:Deleted])
end
end

The Fetcher is designed with a bit of "meta" programming so it can "submit" the email to the activerecord model.

-- script/AirFetchservice.rb
require 'rubygems'
require 'win32/service'
require 'logger'
require 'net/imap'
require 'net/pop'
include Win32

Dir.chdir("\\projects\\ror\\AirCenter")
require File.dirname(__FILE__) + '/../config/environment.rb'

SERVICE_NAME = 'AirFetchService'
SERVICE_DISPLAYNAME = 'AirCenterEmailFetcherService'
$logger = Logger.new("c:\\projects\\ror\\AirCenter\\log\\AirFetchService.log")

class Mailman < ActionMailer::Base
def receive(email)
sender = email.from.to_s()
subject = email.subject.to_s()
body = email.body.to_s()
subjectparts = subject.split(/ /)
classname = subjectparts[0]
$logger.info("Mailman: Sender = #{sender} Class = #{classname} Subject = #{subject}")
myclass = Object.const_get(classname)
if myclass
myclass.ReceiveEmail(sender,subject,subjectparts,body)
else $logger.warn("Mailman: Invalid class #{classname}")
end
end
end

if ARGV[0] == "register"
#Start The Service
svc = Service.new
svc.create_service do |s|
s.service_name = SERVICE_NAME
s.display_name = SERVICE_DISPLAYNAME
s.binary_path_name = "c:\\ruby\\bin\\ruby.exe c:\\projects\\ror\\AirCenter\\script\\AirFetchService.rb"
# + File.expand_path($0)
s.dependencies = []
end
svc.close
puts "Registered Service - " + SERVICE_DISPLAYNAME
elsif ARGV[0] == "test"
$logger.info "Service Starting"
$logger.info "Loading EMAIL Config"
@config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
$logger.info "Setting up config"
@config = @config[RAILS_ENV].to_options
$logger.info "Setting up sleep time"
@sleep_time = @config.delete(:sleep_time) || 60
$logger.info "Service Init Complete"
$logger.info "AirFetchService Establish mail processor"
# Add your own receiver object below
@fetcher = Fetcher.create({:receiver => Mailman}.merge(@config))
@fetcher.fetch
exit
elsif ARGV[0] == "delete"
if Service.status(SERVICE_NAME).current_state == "running"
Service.stop(SERVICE_NAME)
end
Service.delete(SERVICE_NAME)
else
if ENV["HOMEDRIVE"] !=nil
puts "Usage: ruby airfetchservice.rb [options]"
puts " where options are: register or delete"
exit
end

$stderr = $logger
$stdout = $logger

class MyDaemon < Win32::Daemon

def service_init
sleep 1
$logger.info "Service Starting"
$logger.info "Loading EMAIL Config"
@config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
$logger.info "Setting up config"
@config = @config[RAILS_ENV].to_options
$logger.info "Setting up sleep time"
@sleep_time = @config.delete(:sleep_time) || 60
$logger.info "Service Init Complete"
$logger.info "AirFetchService Establish mail processor"
# Add your own receiver object below
@fetcher = Fetcher.create({:receiver => Mailman.receive}.merge(@config))

end

def service_stop
$logger.info "AirFetchService Stopped"
end

def service_main
$logger.info "AirFetchService: Service in Running State"
while state == RUNNING
@fetcher.fetch
sleep(@sleep_time)
end
end
end

d = MyDaemon.new
d.mainloop

end

My Life On Rails

2007-09-28T20:07:00.000-07:00

I've been looking for a "idea" environment to do "applications".
I used to do alot of VB6, and felt that Microsoft "messed" things up
after that. My apps didnt move, and my mind was even less willing to
face the "explosion" of technology and "false" starts that Microsoft
offered in the way of programming.

My background covers C,assembler,Cobol,VB, PL1 and a variety of others.
I found Python about a year and a half ago, and did lots of small backend
projects with it for "backend" processes. Everything from colecting router
logs to database conversion, and realtime "interconnect" between legacy
applications. Python could not be beat. My problem was doing a "gui". The
gui must connect to a "database" and our database of choice is Oracle. In addition it should connect to a variety of others. As well as have libraries for everything else.

Enter "Stage-left" ruby and its framework ruby-on-rails.

Now I can develop a variety of applications with lovely "web" based gui in no time flat.

My technology stack:
active-scaffold
This take care of all the java-script for me. :)
tabnav/widgets
ruby-on-rails
activerecord - Manages the database
activemailer - Handles outgoing and incoming mail
ruby

Supporting cast:
Oracle (10g, Oracle XE)
win32-service - Lets me make my background rails functions run
as windows services

Malaria Killer

2007-08-12T23:25:00.000-07:00

Hmm. Here's a interesting weekend project. (Wish I had time, but got two others ahead of this one).

Use one of the NEW high performance IR(Infrared LEDS) they now can handle 1-2 watts sustained, maybe more in flash mode. Use a "microcontroller", such as a Cypress PSOC to do a "simulation" of human breething using the IR LED. Using the PSOC capacitive sensing detect with a Mosquito has "entered" the kill zone, and then bump up the power to max power (Can add a few LEDS to increase the "flash" to 100watts) to
"fry" the insect, then return to "breathing" mode.

This should reduce the population of mosquitos very nicely and reduce the chance of malaria and other mosquito born infections.

I know mosquitos are "drawn" to CO2 and IR, the question is a "IR" simulation of breathing may be enough to get them going. The other question is the kill ration.
Its a simple and inexpensive design. Considering the "breathing" can be realtive low level and kill is "pulse" mode, you may even be able to solar power it.

If your a mosquito researcher, drop me a line.

References:
LED Example - Hmm What Frequency will have best effect?
http://catalog.osram-os.com/catalogue/catalogue.do?favOid=000000030002856201230023&act=showBookmark

Processor with Cap Sense
http://www.cypress.com/portal/server.pt?space=CommunityPage&control=SetCommunity&CommunityID=209&PageID=215&gid=13&fid=24&category=All&showall=false&CID=ILC-shortlinks&shk=psoc

New "Water" Economy

2006-03-14T19:44:00.000-08:00

Everyone is talking about the new "Hydrogen" Economy, and what changes is will make.
In reality this is a misnomer, its actually the new "Water" economy. When you burn "hydrogen" and when I say burn that applies to "fuel" cells, or just converting a internal combustion engine (read "Car" engine) to burn hydrogen you get water. The flip side is to get hydrogen the best source is water.

There are several ways of doing this, "electrolysis" being the first that comes to mind, but the far better is algae. Electrolysis is sticking "two" wires, (Actually electrodes) in water and separating the two. Here, unless you want to build lots of new Nuclear plants, is rather costly. (And even if you are into pebble bed n-plants, its still not cheap, just cheaper than tradional plants). The far better approach, where rather progress is algae. There is already algae that does this on a small scale, and with genetic engineering, its gotten about 100,000 times better already. Another 100x improvement will make it "production" ready.

So projecting the future, you will have "algae" farms producing hydrogen, at affordable rates.

So what is needed, and how do we get there.

Things that are needed:

1. Water - Its our raw material, high-quality, pure in large amounts
Solution - Think desalination or better yet "membranes", check out http://www.hyflux.com While there are others, these guys are smaller in size and hungry. Idea "site" is near a ocean if possible

2. Land - Lots and lots of it. Hmm. It would be handy and generally favorable to use deserts for this. The middle-east has plenty, as well as dont mind a large scale investment to make things happen. Of course that means the oil-producing nations have three of the things needed to do this, land, water, and lots of sunshine.

3. Investment - Need to start looking at "all" the needs now, building pilot plants, building "foundation" for the whole eco-system

4. Farming "System" - A system of encapsulating the algae, circulating, and growing them on huge scale with exposure to sunshine. This is the "hardware" of the whole system. Since we are looking at desert locale, doing this in safety glass may be idea. A "plant" producing the panels is one of the pieces of infrastrucutor that is needed. The control software, prodcess monitoring, etc also need to be developed. All of this is "sellable" to all other parties wanting to do the farming.

5. Storage/Transport - Lots of work has been done on the foundation of "storage" of hydrogen, further applications need to be developed. ie. containers ("Think the standard size container used in the "container" industry), do user replaceble fuel modules, to large scale tankers. A plan needs to be done to address all the products for the ecosystem.

6. Algae - Lots of groups are already doing this, operational knowledge and license needs to be built. This will allow the customization of #4 to increase production. Not that generally the algae can be changed and readly updated to allow the farm production to increase over time. So you build a "plant" that covers 100 square miles, and every year you will get better production as you improve the algae.

Hydrogen can be safely transported today, but we have to start on the products that will make it useful everywhere. Corporate, utilities and the public are interested in "clean" energy today. Large scale hydogen production can start, its just a matter of "doing" now, and making it happen.

Python - Microsoft Excel And Telnet = MRTG

2006-01-02T23:18:00.000-08:00

Hmm. Got a unreliable T1 line in one of my sites, so I wanted to monitor it. Of course I could use MRTG, but then I want to run it on windows xp machine. (And yes I know I can do that with MRTG as well)

Ok, lets use Excel for our data repository, and formating. For gatering the data, lets use telnet. (Yes, I know I could also use ssh, but the router is a old one) And python to put it all togeather.

This "example" uses .split as the basis of parsing the telnet output, and updates a excel spreadsheet with the results. The spreadsheet is saved. The "whole" script is run once a hour to collect the data. For simplicity, I use the microsoft task scheduler.

Of course, this can easily be expanded to support multiple routers and multiple serial interfaces.

If anyone wants to "see" the final spreadsheet, drop me a email.

## check-the-line.py
port sys
import telnetlib
from win32com.client import Dispatch
import pywintypes
import datetime
import time
import pythoncom

prevupdate = 6
lastupdate = 5
deltaslot = 7

tn = telnetlib.Telnet("1.2.3.4") # Change your ip address

tn.read_until("Password: ")
tn.write("your.password.here.for.telnet\n")
tn.read_until(">")
tn.write("show interface serial 0\n") # The interface
router_status = tn.read_until(">")
tn.write("exit\n")

thelines = router_status.split("\n")

for theline in thelines:
thelen = len(theline) - 1
theline = theline[:thelen]

recvcnt = thelines[16].split()
recverr = thelines[17].split()
sentcnt = thelines[18].split()
senterr = thelines[19].split()

x1 = Dispatch("Excel.Application")
x1.Visible = 0
x1.Workbooks.Open("mn-line.xls")

myc = x1.ActiveSheet.Cells

# Received 11045 broadcasts, 0 runts, 0 giants, 0 throttles
# 3 input errors, 3 CRC, 0 frame, 0 overrun, 0 ignored, 3 abort
# 332207 packets output, 54998720 bytes, 0 underruns
# 0 output errors, 0 collisions, 15 interface resets

now = datetime.datetime.now()

#save the old
for thecol in range(1,18):
myc(prevupdate,thecol).Value = myc(lastupdate,thecol).Value

myc(lastupdate,1).Value = pythoncom.MakeTime(time.time())

#recvcnt
myc(lastupdate,2).Value = recvcnt[1]
myc(lastupdate,3).Value = recvcnt[3]
myc(lastupdate,4).Value = recvcnt[5]
myc(lastupdate,5).Value = recvcnt[7]
#recverr
myc(lastupdate,6).Value = recverr[0]
myc(lastupdate,7).Value = recverr[3]
myc(lastupdate,8).Value = recverr[5]
myc(lastupdate,9).Value = recverr[7]
myc(lastupdate,10).Value = recverr[9]
myc(lastupdate,11).Value = recverr[11]
#sentcnt
myc(lastupdate,12).Value = sentcnt[0]
myc(lastupdate,13).Value = sentcnt[3]
myc(lastupdate,14).Value = sentcnt[5]
myc(lastupdate,15).Value = senterr[0]
myc(lastupdate,16).Value = senterr[3]
myc(lastupdate,17).Value = senterr[5]

#put the deltas in the hour slot
thehour = now.hour
hourslot = 10 + thehour
for thecol in range(2,18):
myc(hourslot,thecol).Value = myc(deltaslot,thecol).Value
myc(hourslot,1).Value = myc(lastupdate,1).Value

x1.ActiveWorkbook.Save()
if thehour == 23: # Witching hour - Save a copy for reference
filename = "ml-line-" + now.strftime("%y-%m-%d")
x1.ActiveWorkbook.SaveAs(filename)

if thehour == 0: # Reset the Spreadsheet Date
myc(1,2).Value = myc(lastupdate,1).Value

x1.ActiveWorkbook.Close(0)
x1.Quit()
del x1

PIX Wars - CISCO Pix Bug

2005-11-17T00:17:00.000-08:00

Unrelated Comment: If your going to leave me a comment, and want a answer, please add your email address. The normal form does not give me the address.

In helping out a friend of mine, I was debugging a CISCO firewall. Of course the normal thing in the process is to setup a SYSLOG server to capture the bulk of what is going on. CISCO now recommends the TCP syslog. Well, of course prefer to follow there recomendation, so I set it up. Used there Windows Based SysLog server. Of course when I rebooted I didnt notice the logger process did not restart. Now my firewall was not taking any new connections.

"What did I do?" was what I was thinking. Check every setting, and come to fine out
there is a bug, if the destination is not available, it keeps spawning logging connections till all available connections are used up. After switching to UDP based logging, everything is fine.

Comming soon, Python scripts to automatically parse PIX logs.

Thermal Radiance - A way to cool your PC using Quantum Mechanics

2005-10-26T01:43:00.000-07:00

Your PC has lots of "glow", that really increases the bling factor, but does nothing to keep it actually cooler. There may be a way to "convert" the heat to light, giving that "nice" glow, while actually helping with the cooling.

In a recent invention, "Quantum" dots, which have 25 +/- electrons per dot, converted a blue LED, into a "white" led. No big deal to most people, but the physics of it, opens up several possibilities.

The first is a modification of your heat-sink, with tuning of the Quantum Dots to allow the conversion of heat to light. Since the light will stream nicely out of the case, it allows another way of getting heat out of the case quicker.

The technique should be relatively low-cost, and massed produced.

Note that another technique is to excite the quantum dots with electrical standing waves. (Think tesla coil). This should allow the production of large amounts of light, at high efficiently.