Ever want to process something external to your database or your rails app, such
as the contents of files. But you think rails is only good for database items?
Actually rails can easily adapt to handle any kind of data.
In these examples, I process all my controllers, to give a view and a RESTful interface
for testing of my controllers. I also do the same for the menus. All without a database
table in sight.
First, lets take a look at the controllers "model".
This allows us to view our controllers from within rails, useful if you want to know what code is running, or
in my case to use watir to test. The lovely part is I can actually RESTfully pull the list of controllers, and make
sure each is hit in watir.
The next problem is menu's, how do I validate that each menu does something, a little of the same
technique. Here is a bit more advanced, we actually build a parent menu table, and a child menu item table.
Here is the menu model. The Menu Model actually creates all the child menu items at startup. It does this by
parsing tabnav/widget format menus. While I was looking at a way of using erb to do this for me, I ended up just
doing the parsing myself.
And finally the MenuItem Model
After this I do simple activescaffold based controllers, and more importantly add routes so I can pull xml files
for testing.
Monday, October 12, 2009
Friday, August 21, 2009
ActiveJquery - Without git
If you pull the files, and install them without using rails plugin code, there
is a "install" script that copies things into the right place.
The file will be in: #{projectroot}/vendor/plugins/activejquery
So for me, its in the following directory:
/Users/gwest/mymrp2/vendor/plugins/activejquery
As you can see it, actually copies a few files into the right place in your project.
directory = File.dirname(__FILE__)
copy_files("/public/css", "/public/css", directory)
copy_files("/public/javascripts", "/public/javascripts", directory)
copy_files("/app/views/activejquery", "/app/views/activejquery", directory)
I love to help, so if you have questions, feel free to email me.
or if your commenting, please give me your email, as when you comment on the
blog, the system does not give me a email to respond.
is a "install" script that copies things into the right place.
The file will be in: #{projectroot}/vendor/plugins/activejquery
So for me, its in the following directory:
/Users/gwest/mymrp2/vendor/plugins/activejquery
As you can see it, actually copies a few files into the right place in your project.
directory = File.dirname(__FILE__)
copy_files("/public/css", "/public/css", directory)
copy_files("/public/javascripts", "/public/javascripts", directory)
copy_files("/app/views/activejquery", "/app/views/activejquery", directory)
I love to help, so if you have questions, feel free to email me.
or if your commenting, please give me your email, as when you comment on the
blog, the system does not give me a email to respond.
Monday, June 15, 2009
Call Center Voice Recording
Ever wonder how you do audit in a large call center?
Here's the basics:
First, most modern call centers are using Voip. It makes it easy to setup, and even better
makes it easy to monitor. Also call-centers tend to be in places like india, or phillipines
serving the US or Europe markets. So VoIp plays a role in getting the traffic from one
country to another.
There are several pieces to this puzzle:
First, you need to capture the raw data:
So to do that, we need a "tee" off the ethernet channel that is connecting to the VoIP gateway.
This will allow us to see all the VoIP IP Traffic that is going back and forth.
It's sometimes called 'port mirroring', 'port monitoring', 'Roving Analysis' (3Com), or 'Switched Port Analyzer' or 'SPAN' (Cisco).
http://wiki.wireshark.org/CaptureSetup/Ethernet
http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008015c612.shtml#descp
Now that we have traffic, a good way to "capture" the voice is Oreka
http://oreka.sourceforge.net/
Oreka lets you capture the packets, and convert them to audio files. It actually consists
of three parts.
- OrkAudio: This is the workhorse that processes the calls and does the actual recording
- OrkWeb: This is the XML based Web U/I to access and manage the system
- OrkTrack: This is the master database (MySQL) that records the call records, metadata, etc.
Oreka lets you get a basic system up quickly.
The next issue, is setting up a system of audit, and to connect and understand the calls into transactions,
customers, call-agents, and supervisors.
For a good GUI on top, I'd switch to a Rails App. This lets me do several things:
1. I can tie agents to calls
2. I can allow agents access to review there own calls
3. I can allow supervisors to view/listen to calls for agents reporting to them
4. I can let audit department audit calls as appriate.
5. Ease of interfacing existing call center apps togeather
Now that we have a phase 1, and phase 2 completed, lets add a phase 3.
One of the emerging technologies is Speach to text.
Using the latest SDK's from Dragon Speaking, we can get very high 99% plus on our agents,
and not bad recognition on our clients/customers calling in. This will allow audit to search
for key words, such as scam, theft, etc that would audit to be better aware of things going
on.
The idea way of implementing the whole system is break things down to separate VM's in a VmWare system.
1. OakCapture - 1 VM per major Trunk (One Per - Multiple DS1, One DS3)
2. OakGui - 1 VM For system - For reference/admin control only
3. Database - In a call center, a Oracle or DB2 may be better suited, depending on the company
4. VoiceCenterManager - A Rails Server using Nginx/Passenger that provides interfaces to agents, supervisors, audit,
and managers. Number of VM determine by staff size
5. Recognition Engines - A separate VM that is running a Ruby Agent tied to a Dragon Speaking SDK allowing speaker dependent and Speaker Independent Speech to text conversion. This allow for the creation of easily searchable as well
as readable results of the calls in text form. Using a rack of blades, and VmWare we can realtime convert all calls to text
index them, and make them searchable.
Note that each call center is different but leveraging Rails, and Ruby a AGILE and fast solution can be combined that
gives the best in Audit and accountability.
Here's the basics:
First, most modern call centers are using Voip. It makes it easy to setup, and even better
makes it easy to monitor. Also call-centers tend to be in places like india, or phillipines
serving the US or Europe markets. So VoIp plays a role in getting the traffic from one
country to another.
There are several pieces to this puzzle:
First, you need to capture the raw data:
So to do that, we need a "tee" off the ethernet channel that is connecting to the VoIP gateway.
This will allow us to see all the VoIP IP Traffic that is going back and forth.
It's sometimes called 'port mirroring', 'port monitoring', 'Roving Analysis' (3Com), or 'Switched Port Analyzer' or 'SPAN' (Cisco).
http://wiki.wireshark.org/CaptureSetup/Ethernet
http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008015c612.shtml#descp
Now that we have traffic, a good way to "capture" the voice is Oreka
http://oreka.sourceforge.net/
Oreka lets you capture the packets, and convert them to audio files. It actually consists
of three parts.
- OrkAudio: This is the workhorse that processes the calls and does the actual recording
- OrkWeb: This is the XML based Web U/I to access and manage the system
- OrkTrack: This is the master database (MySQL) that records the call records, metadata, etc.
Oreka lets you get a basic system up quickly.
The next issue, is setting up a system of audit, and to connect and understand the calls into transactions,
customers, call-agents, and supervisors.
For a good GUI on top, I'd switch to a Rails App. This lets me do several things:
1. I can tie agents to calls
2. I can allow agents access to review there own calls
3. I can allow supervisors to view/listen to calls for agents reporting to them
4. I can let audit department audit calls as appriate.
5. Ease of interfacing existing call center apps togeather
Now that we have a phase 1, and phase 2 completed, lets add a phase 3.
One of the emerging technologies is Speach to text.
Using the latest SDK's from Dragon Speaking, we can get very high 99% plus on our agents,
and not bad recognition on our clients/customers calling in. This will allow audit to search
for key words, such as scam, theft, etc that would audit to be better aware of things going
on.
The idea way of implementing the whole system is break things down to separate VM's in a VmWare system.
1. OakCapture - 1 VM per major Trunk (One Per - Multiple DS1, One DS3)
2. OakGui - 1 VM For system - For reference/admin control only
3. Database - In a call center, a Oracle or DB2 may be better suited, depending on the company
4. VoiceCenterManager - A Rails Server using Nginx/Passenger that provides interfaces to agents, supervisors, audit,
and managers. Number of VM determine by staff size
5. Recognition Engines - A separate VM that is running a Ruby Agent tied to a Dragon Speaking SDK allowing speaker dependent and Speaker Independent Speech to text conversion. This allow for the creation of easily searchable as well
as readable results of the calls in text form. Using a rack of blades, and VmWare we can realtime convert all calls to text
index them, and make them searchable.
Note that each call center is different but leveraging Rails, and Ruby a AGILE and fast solution can be combined that
gives the best in Audit and accountability.
Thursday, June 11, 2009
Ghosting NTFS/Windows using Knoxppix
I have a USB 1 Gig stick that is on my key ring.
Its really handy, as I have a complete copy of Knoxppix.
Knoxpix is a Linux Distribution designed to boot and run
from CDROM or DVD. It can also run from a handy USB stick.
With the right options it can also in "RAM", which is handly if you want to
run a few parallel operations, and only need one usb stick.
While you can use clonezilla, if you want something more packaged,
but in a shop that is windows centric, ghosting to a windows server
may make more sense.
Boot your usb stick, then at the bash shell, mount your server:
mount -t cifs //server-name/share-name /mnt/cifs -o username=shareuser,password=sharepassword,domain=nixcraft
On the share, I usually will the have my ntfscloning scripts
On a dell machine, there is really only one partition that has "data",
So to the "dobackup" script is very trival.
This takes the OS partition, in most of the environments I was working in this
would have been /dev/sda2 and uses ntfsclone, which takes only the in-use
sectors, compresses them, and put them in a file on the server.
dobackup script
echo backup $1 to $2
mkdir $2
ntfsclone -s -o - /dev/$1 | gzip > $2/$1.ntfsclone.gzip
The reverse, to restore is:
dorestore script
echo restore $1 to $2
cd $1
cat $2.ntfsclone.gzip | gunzip - | ntfsclone --restore-image --overwrite /dev/$2 -
The only other madatory item is the partition table:
doptablebackup script
echo partion backup $1 to $2
sfdisk -d /dev/$1 > $2/ptable.doc
dd if=/dev/$1 bs=512 count=1 of=$2/ptable.sector
doptablerestore script
echo partion restore $1 to $2
dd of=/dev/$2 bs=512 count=1 if=$1/ptable.sector
Its really handy, as I have a complete copy of Knoxppix.
Knoxpix is a Linux Distribution designed to boot and run
from CDROM or DVD. It can also run from a handy USB stick.
With the right options it can also in "RAM", which is handly if you want to
run a few parallel operations, and only need one usb stick.
While you can use clonezilla, if you want something more packaged,
but in a shop that is windows centric, ghosting to a windows server
may make more sense.
Boot your usb stick, then at the bash shell, mount your server:
mount -t cifs //server-name/share-name /mnt/cifs -o username=shareuser,password=sharepassword,domain=nixcraft
On the share, I usually will the have my ntfscloning scripts
On a dell machine, there is really only one partition that has "data",
So to the "dobackup" script is very trival.
This takes the OS partition, in most of the environments I was working in this
would have been /dev/sda2 and uses ntfsclone, which takes only the in-use
sectors, compresses them, and put them in a file on the server.
dobackup script
echo backup $1 to $2
mkdir $2
ntfsclone -s -o - /dev/$1 | gzip > $2/$1.ntfsclone.gzip
The reverse, to restore is:
dorestore script
echo restore $1 to $2
cd $1
cat $2.ntfsclone.gzip | gunzip - | ntfsclone --restore-image --overwrite /dev/$2 -
The only other madatory item is the partition table:
doptablebackup script
echo partion backup $1 to $2
sfdisk -d /dev/$1 > $2/ptable.doc
dd if=/dev/$1 bs=512 count=1 of=$2/ptable.sector
doptablerestore script
echo partion restore $1 to $2
dd of=/dev/$2 bs=512 count=1 if=$1/ptable.sector
Sunday, June 07, 2009
Dynamically Generating Javascript using Parameters
I love re-using code, and really believe in DRY.
In doing ActiveJquery, I want my grids to be very re-usable.
For one, I need to use multiple grid/sub-grids to handle Rails
relationships, second I need to be able to have multiple grids
in one page. And want to do all of this with the least amount of code.
In ActiveJquery, the Grid code, is generated on the fly. While it looks like a normal
call, its actually hitting controller code tucked inside the plugin.
So when we request the javascript code for a particular controller, the ActiveJquery
code generates the appropriate javascript on the fly.
In doing, the sub-table, I need to have parameters passed in, so I rely on standard
Rails Semantics when handling HTML.
This technique lets me generate customized javascript on the fly, making the generator re-usable, and
the grid re-usable any number of times in the same application.
In doing ActiveJquery, I want my grids to be very re-usable.
For one, I need to use multiple grid/sub-grids to handle Rails
relationships, second I need to be able to have multiple grids
in one page. And want to do all of this with the least amount of code.
In ActiveJquery, the Grid code, is generated on the fly. While it looks like a normal
call, its actually hitting controller code tucked inside the plugin.
So when we request the javascript code for a particular controller, the ActiveJquery
code generates the appropriate javascript on the fly.
In doing, the sub-table, I need to have parameters passed in, so I rely on standard
Rails Semantics when handling HTML.
<script src="location.js?subof=Company&div=Company_Location" type="text/javascript"></script>
This technique lets me generate customized javascript on the fly, making the generator re-usable, and
the grid re-usable any number of times in the same application.
Saturday, June 06, 2009
Finding ActiveRecord Associations
Im working away on ActiveJquery, and the next thing on the list, is to add support for relationships. This gets to be really interesting, because your Inside the box, in a plugin, and you need to know what associations the user has
defined.
I was scanning the documentation, and looking around for the api to find the information.
After searching for a while, I discovered railway, a gem for rails that does diagraming of rails models using dot. So grab the gem, and look thru the source.
So the key to finding associations is:
@associations = table.reflect_on_all_associations
This results in:
From my models:
defined.
I was scanning the documentation, and looking around for the api to find the information.
After searching for a while, I discovered railway, a gem for rails that does diagraming of rails models using dot. So grab the gem, and look thru the source.
So the key to finding associations is:
@associations = table.reflect_on_all_associations
This results in:
@associations=
[#ActiveRecord::Reflection::AssociationReflection:0x26305cc
@active_record=
Company(id: integer, name: string, location_id: integer, created_at: datetime, updated_at: datetime),
@macro=:has_many,
@name=:user,
@options={:extend=>[]}>,
#<ActiveRecord::Reflection::AssociationReflection:0x262faa0
@active_record=
Company(id: integer, name: string, location_id: integer, created_at: datetime, updated_at: datetime),
@macro=:has_many,
@name=:location,
@options={:extend=>[]}>,
#<ActiveRecord::Reflection::AssociationReflection:0x262f0c8
@active_record=
Company(id: integer, name: string, location_id: integer, created_at: datetime, updated_at: datetime),
@macro=:has_many,
@name=:division,
@options={:extend=>[]}>],
From my models:
class Company < ActiveRecord::Base
has_many :user
has_many :location
has_many :division
end
class Department < ActiveRecord::Base
belongs_to :division
end
class Division < ActiveRecord::Base
belongs_to :company
end
class Location < ActiveRecord::Base
end
class User < ActiveRecord::Base
end
Thursday, May 14, 2009
ruby script/plugin git does not work in Rails 2.3.2 and Ruby 1.9
Lots of people are scratching there head over why they cannot
install rails plugins using git.
I'm working with Ruby 1.9.1 and Rails 2.3.2. And git is a pretty natural
for working with version control.
So I wanted to pull from my own respository, but kept getting
a error on tryping to do the install.
sin-gwest-laptop:testjq gwest$ ruby script/plugin --verbose install git://github.com/glennswest/activejquery.git
Plugins will be installed using http
Plugin not found: ["git://github.com/glennswest/activejquery.git"]
#<TypeError: can't convert Array into String>
Come to find out, this is a known but as the mkdir_p call has changed its
return parameter in Ruby 1.9 and that messes up the install of git plugins
unless you do the patch
https://rails.lighthouseapp.com/attachments/90768/plugin_mkdir_p.diff
ActiveJquery Goes to version .011
I've updated ActiveJquery on github to .011
That was fun. I thought I'd put the authtoken issue to bed.
On doing test for delete and add, found I still was getting authtoken
issues. So I moved from using editData jqgrid parameters to adding
it to the editurl.
Also found a bug in the page based xml pull of the controller component.
This would result in you not seeing the last 10 records. (You notice that
when your adding records and they dont show up).
ActiveJquery consists of:
1. active_jquery - The controller plugin
2. active_jquery_runtime - Generates the dynamic jqgrid javascript
3. Dynamic Javascript - The code that runs in the browser
4. jQuery/Jquery UI/jqGrid
I've also combined all the needed public css and javascript into the plugin.
Need to add a rake task and init that makes sure they get copied.
ActiveJquery Reaches Version .010
ActiveJquery, which is designed to integrate Jqgrid, Jquery UI into rails, is
now at Version 0.010.
It now is a Rails 2.3.3 Controller Plugin. You can invoke it in a single
line in your controller.
The plugin automatically generates javascript for the grid, based on your table,
with inline editing. As well as a full REST server to serve the data to the browswer.
I've implemented the ForgeryProtection on Posts, and that is working. Seems like a bit of comfusion in BLogSpere about weather the URI Components need to be encoded or not. Least in Ruby 1.9.1 and Rails 2.3.2 They do NOT need to be encoded.
Todo:
1. Allow Customizations
2. Allow Relationships and SubTables
3. JQuery UI Menus
Test and More TEst
Also I will implement a rails DEMO app that is based on data from the JQGRID site.
I'll make a separate git repository for the demo.
Wednesday, April 29, 2009
ActiveJquery - Status
Current Things That Are Working:
Controller/Server
1. Can read full table
2. Can read using JqGrid Pager
3. Added Sort support, so server will honor the grid sort request
4. Added Delete,Add and Update support.
5. Auto Generates JqGrid Javascript from ActiveRecord
6 Added Total Records to XML so Pager works properly.
ActiveJquery Library/Client
1. Added support for string, integer, date.
2. Generates JSON Based Reader compatible with Rails JSON Format
3. Uses Humanize to handle automatic column names.
4. Uses JqueryUI for theming
5. InLine Edit Support
Things to Do:
1. Paste Controller code into prepared plugin
2. Add static data support
3. Add a bit of DSL(Domain Specific Language) to allow easy configuration
4. Add Master/Detail Support
5. Add Date Picker Plugin
6. Add Parent Table Dynamic Data Selector
Controller/Server
1. Can read full table
2. Can read using JqGrid Pager
3. Added Sort support, so server will honor the grid sort request
4. Added Delete,Add and Update support.
5. Auto Generates JqGrid Javascript from ActiveRecord
6 Added Total Records to XML so Pager works properly.
ActiveJquery Library/Client
1. Added support for string, integer, date.
2. Generates JSON Based Reader compatible with Rails JSON Format
3. Uses Humanize to handle automatic column names.
4. Uses JqueryUI for theming
5. InLine Edit Support
Things to Do:
1. Paste Controller code into prepared plugin
2. Add static data support
3. Add a bit of DSL(Domain Specific Language) to allow easy configuration
4. Add Master/Detail Support
5. Add Date Picker Plugin
6. Add Parent Table Dynamic Data Selector
ActiveJquery - Features
ActiveJquery is a Rails Plugin that combines the goodness of Jquery, Jquery UI, and JqGrid.
| Feature | Description |
|---|---|
| REST | ActiveJquery breaks your GUI into a javascript REST Client, and a Rails based REST JSON Server. This reduces the overhead of our web app, and gives you better expandability, and better response time |
| DRY | Tired of repeating yourself, dont. ActiveJquery will find your Field Name, and configure the Grid for you. |
| Customizable | You can configure grids in countless ways, allowing you easy control over what you want to see. | MultiGrid | Need a butch of grids on one page? Not a problem. You can have any number of grids in one page. Each is named automatically for you. |
| Master/Detail | You Rails Associations are used to generate sub-grids or master-detail views of your data. |
| Theming | ActiveJquery uses Jquery UI themes, so you can easily customize the look and color scheme of the grid | Menu's/Tabs | Want to have a easy to use menu system. ActiveJquery supports JqueryUI tabs, so you can do complex, but easy to use apps. |
| Dynamic Data | By default data is read a screenful at a time, using the grid pager. This reduces the need to read large amounts of a big table. Also makes loading multiple grids very fast. |
| Static Data | ActiveJavascript can embed the table data directly in the javascript. This is great for tables that 1000 rows or less, and are read-only. |
Jdgrid Pager Problem
Trying out the latest versions of jquery/jquery ui/jqgrid, and just cannot get the formatting right.
Lets see if we can figure out whats going on.
The html for the grid:
<html>
<head>
<title> Airstate </title>
</head>
<body>
<h2> Airstate </h2>
<script src="javascripts/jquery-1.3.2.min.js" type="text/javascript"></script>
<script src="javascripts/jquery-ui-1.7.1.custom.min.js" type="text/javascript"></script>
<script src="javascripts/jquery.layout.js" text="text/javascript"></script>
<script src="javascripts/jqModal.js" type="text/javascript"></script>
<script src="javascripts/jqDnR.js" type="text/javascript"></script>
<script src="javascripts/jquery.jqGrid.js" type="text/javascript"></script>
<link rel="stylesheet" type="text/css" media="screen" href="themes/redmond/jquery-ui-1.7.1.custom.css" />
<link rel="stylesheet" type="text/css" media="screen" href="themes/ui.jqgrid.css" />
<style>
html, body {
margin: 0; /* Remove body margin/padding */
padding: 0;
overflow: hidden; /* Remove scroll bars on browser window */
font-size: 75%;
}
/*Splitter style */
#LeftPane {
/* optional, initial splitbar position */
overflow: auto;
}
/*
* Right-side element of the splitter.
*/
#RightPane {
padding: 2px;
overflow: auto;
}
.ui-tabs-nav li {position: relative;}
.ui-tabs-selected a span {padding-right: 10px;}
.ui-tabs-close {display: none;position: absolute;top: 3px;right: 0px;z-index: 800;width: 16px;height: 14px;font-size: 10px; font-style: normal;cursor: pointer;}
.ui-tabs-selected .ui-tabs-close {display: block;}
.ui-layout-west .ui-jqgrid tr.jqgrow td { border-bottom: 0px none;}
.ui-datepicker {z-index:1200;}
</style>
<script type="text/javascript">
var listlastsel;jQuery(document).ready(function(){
jQuery("#list").jqGrid({
url:'airstate.xml',
editurl:'airstate',
datatype: 'xml',
mtype: 'GET',
colNames:['Id','Created on','Whom','Logmessage'],
colModel :[
{name: 'id',index:'id',key:true,width:80,align:'right'},
{name: 'created_on',index:'created_on',key:false,width:90},
{name: 'whom',index:'whom',key:false,width:300,align:'left',editable:true},
{name: 'logmessage',index:'logmessage',key:false,width:300,align:'left',editable:true}],
pager: jQuery('#list-pager'),
onSelectRow: function(id){
if(id && id!==listlastsel){
jQuery('#list').restoreRow(listlastsel);
jQuery('#list').editRow(id,true);
listlastsel=id;
}
},
autowidth: true,
rowNum:10,
rowList:[10,20,30],
sortname: 'id',
sortorder: "desc",
viewrecords: true,
imgpath: 'themes/basic/images',
caption: 'Airstate',
xmlReader: {root: "root",
row: "syslog",
page:"root>page",
total:"root>total",
records:"root>records",
repeatitems:false
},
});
jQuery("#list").navGrid('#list-pager',{ edit:true,add:true,del:true,search:true });
});
</script>
<div id="list-pager" class="scroll"></div>
<table id="list" class="scroll"></table>
</body>
</html>
Now lets see if we can figure out whats going on.
Ok, Great support from jdquery. Seems like its a a bug in the latest v2 alfa test.
Alfa 3 should solve the problem.
Tuesday, April 07, 2009
Getting only the columns you want in Rails
A thing that is often missed when your doing a web service or a rails app is selecting only
the columns you need.
A traditional find:
user = User.find :all
render :xml => user.to_xml
Will give you the bloat of the whole user table.
If you only hae a few small columns then thats not so bad,
but I've seen legacy apps that have very large number of columns,
so it makes since then to control this better.
user = User.find(:all, :select = 'email')
render :xml => user
This also will make your resulting xml file alot more managable.
And since bandwidth is not free, it will help you handle more users
for less money.
the columns you need.
A traditional find:
user = User.find :all
render :xml => user.to_xml
Will give you the bloat of the whole user table.
If you only hae a few small columns then thats not so bad,
but I've seen legacy apps that have very large number of columns,
so it makes since then to control this better.
user = User.find(:all, :select = 'email')
render :xml => user
This also will make your resulting xml file alot more managable.
And since bandwidth is not free, it will help you handle more users
for less money.
Network Monitor as a Windows Service in Ruby
In reading thru the Ruby Google Group, there seems to be a lot of misconceptions about what should be done inside
of a Rails process and what should be done outside it. Generally if it take any time at all, you should be it outside
the Rails Web Server. In windows you can do a Windows Service, in Linux/Unix you would do a daemon.
This is a example of a windows service that scans cisco switches, finds new nodes, and keep track of there mac
address and there port. Even builds a network map.
The app consists of a Rails app to display the data, and take in configuration information. Which is put in a database,
such as sqlite3, and updated by this Windows Service.
As long as the Service is running, the data will be updated. If I was doing it again, I'd use rufus-scheduler to
wrap the scanning and the mapping.
of a Rails process and what should be done outside it. Generally if it take any time at all, you should be it outside
the Rails Web Server. In windows you can do a Windows Service, in Linux/Unix you would do a daemon.
This is a example of a windows service that scans cisco switches, finds new nodes, and keep track of there mac
address and there port. Even builds a network map.
The app consists of a Rails app to display the data, and take in configuration information. Which is put in a database,
such as sqlite3, and updated by this Windows Service.
As long as the Service is running, the data will be updated. If I was doing it again, I'd use rufus-scheduler to
wrap the scanning and the mapping.
require 'rubygems'
require 'win32/daemon'
include Win32
require 'logger'
require 'win32/process'
require 'net/ping'
#Note that most of your requires need to go in the service init
class IPAddr
def succ()
return self.clone.set(@addr + 1)
end
end
class AirstateDiscovery
def doscan()
ProcessNetworks()
ProcessDiscovery()
return
end
def ProcessDiscovery()
Mac.find(:all).each do |mymac|
mydevice = mymac.device
if mydevice
mynet = Network.find_network_by_ip(mymac.ip)
if mynet #Ok we have a valid network
Device.find_device_type(mymac.ip,mydevice)
end
end
end
end
def ProcessSwitches()
Switch.find(:all).each do |myswitch|
if myswitch.enabled
if myswitch.switch_type == "cisco"
cisco = CiscoSwitch.new(myswitch)
#begin
cisco.update_macs()
#rescue
# end # Rescue
cisco = nil
end #if myswitch.switch_type == "cisco"
end # myswitch.enabled
thegraph = SwitchGraph.new(myswitch)
thegraph.graph()
thegraph = nil
end # find do
end
def ProcessNetworks()
ProcessSwitches()
return
# Brute Force Scan
Network.find(:all).each do |mynet|
ProcessSwitches() # Networks is "long" running, Switch is "fast"
if mynet.enable
startip = mynet.ip_start
endip = mynet.ip_end
ip = IPAddr.new(startip)
while ip.to_s != endip
ProcessWmi(mynet,ip.to_s)
ip = ip.succ()
end
end
end
end # def Process Networks
end # class AirStateDiscovery
class Daemon
def service_init
end
def service_main
require 'ipaddr'
Dir.chdir("\\projects\\airstate")
require File.dirname(__FILE__) + '/../config/environment.rb'
require 'lib/cisco_phone'
require 'lib/managepc'
require 'lib/switchgraph'
@mywork = AirstateDiscovery.new
mylog = Syslog.new
mylog.whom = "AirStateDiscovery"
mylog.logmessage = "Discovery Starting"
mylog.save
while running?
@mywork.doscan()
sleep 3
end
end
end
# Test Code
# require 'ipaddr'
# Dir.chdir("\\projects\\airstate")
# require File.dirname(__FILE__) + '/../config/environment.rb'
# require 'lib/managepc'
# require 'lib/cisco'
# require 'lib/switchgraph'
# require 'lib/cisco_phone'
# @mywork = AirstateDiscovery.new
# while 1
# @mywork.doscan()
# end
# exit
Daemon.mainloop
Sunday, April 05, 2009
Rails - Useful Plugins
In updating myself I found some really useful Rals plugins for my next project.
Hoptoad Notifier
A Rails app is a very dynamic thing. Finding out when your users have errors, and what the pattern of errors is a must have. The traditional approach is to view the log files from time to time, or use a notifier that sends the errors to your email. Then go thru it as they come in. First, it not uncommon for a error to repeat. So you get log of junk in your email. A better approach is to use the hoptoad service/plugin. This way, you get nice reporting and analysis, and for a single project, its free. It consists of a hosted site web gui, and a rails plugin, and a optional local mac gui to show your croak's, ie the errors in your apps.
http://www.hoptoadapp.com/welcome
http://blog.railsrumble.com/2008/10/6/hoptoad
http://github.com/thoughtbot/hoptoad_notifier/tree/master
http://github.com/bricooke/croak-app/tree/master
Simple Ruby Rest Client
REST is a lovely way of doing Remote Procedure Calls. It the rails way of creating web services. Sometime its handy to have a simple REST-client.
There is a simple to Rest client done by Adam Wiggins
http://github.com/adamwiggins/rest-client/tree/master
http://rest-client.heroku.com/rdoc/
Now you can call a REST api even from IRC or rails console
Saturday, March 21, 2009
C Code - Bucket Based Allocator
In realtime systems, malloc/free/garbage collection can take a very long time.
For a real-time system I'm doing, I want to have very fast response time, I call malloc/free often.
Traditional malloc/free will fragment memory quickly, and garbage collection will just take
forever.
The solution I've used many times is a bucket allocator, setting pre-defined sizes of memory, and putting them in a queue, or in this case a "bucket". The idea being find the bucket, pull out a entry and return it.
Also something I've found useful is to keep track of busy memory elements as well. That way if I suspect that memory is getting corrupted, I can easily add a tag at the end of the allocation, and run thru the busy memory looking for the corruption. Very useful in debug.
For a real-time system I'm doing, I want to have very fast response time, I call malloc/free often.
Traditional malloc/free will fragment memory quickly, and garbage collection will just take
forever.
The solution I've used many times is a bucket allocator, setting pre-defined sizes of memory, and putting them in a queue, or in this case a "bucket". The idea being find the bucket, pull out a entry and return it.
Also something I've found useful is to keep track of busy memory elements as well. That way if I suspect that memory is getting corrupted, I can easily add a tag at the end of the allocation, and run thru the busy memory looking for the corruption. Very useful in debug.
#include <stdio.h>
#include <stdlib.h>
#include "include/queue.h"
struct bucket_struct {
struct entry_struct entry;
struct queue_struct free_q;
struct queue_struct busy_q;
int size;
};
struct alloc_struct {
struct entry_struct entry;
struct bucket_struct *bucket;
};
struct queue_struct buckets;
int gkmalloc_init_needed = 1;
void gkmalloc_newbucket(thesize)
int thesize;
{
struct bucket_struct *mybucket;
mybucket = malloc(sizeof(struct bucket_struct));
mybucket->free_q.head = NULL;
mybucket->free_q.tail = NULL;
mybucket->busy_q.head = NULL;
mybucket->busy_q.tail = NULL;
mybucket->entry.next = NULL;
mybucket->entry.prev = NULL;
mybucket->size = thesize;
queue(&buckets, &mybucket->entry);
return;
}
void gkmalloc_init(void)
{
gkmalloc_init_needed = 0;
buckets.head = NULL;
buckets.tail = NULL;
gkmalloc_newbucket(32);
gkmalloc_newbucket(64);
gkmalloc_newbucket(128);
gkmalloc_newbucket(256);
gkmalloc_newbucket(512);
gkmalloc_newbucket(1024);
gkmalloc_newbucket(2048);
gkmalloc_newbucket(4096);
gkmalloc_newbucket(8192);
gkmalloc_newbucket(32768);
gkmalloc_newbucket(12000000); // Big Buf for jpegs
return;
}
char *gkmalloc(thesize)
int thesize;
{
struct bucket_struct *mybucket;
struct alloc_struct *myalloc;
void *ptr;
if (thesize == 0)
return(NULL);
if (gkmalloc_init_needed){
gkmalloc_init();
gkmalloc_init_needed = 0;
}
mybucket = (struct bucket_struct *)buckets.head;
while(mybucket->size < thesize){
if (mybucket == NULL){ // Should never happen
gkfatal("gkmalloc: Allocation bigger than max bucket");
return(NULL);
}
mybucket = (struct bucket_struct *)mybucket->entry.next;
}
myalloc = (struct alloc_struct *)unqueue(&mybucket->free_q);
if (myalloc == NULL){ // The allocation queue is empty
// Dynamically allocate from the system
myalloc = malloc(sizeof(struct alloc_struct) + mybucket->size);
if (myalloc == NULL){
gkfatal("gkmalloc: Cannot malloc new buffer");
return(NULL);
}
myalloc->bucket = mybucket;
}
queue(&mybucket->busy_q, &myalloc->entry);
ptr = (void *)(myalloc + 1); // Add the size of the header to get Memory After
return(ptr);
}
void gkfree(ptr)
void *ptr;
{
struct bucket_struct *mybucket;
struct alloc_struct *myalloc;
if (ptr == NULL){
gkfatal("gkfree: Bad Free - Null Ptr Passed");
return;
}
myalloc = (struct alloc_struct *)ptr;
myalloc = myalloc - 1; // Go back to the header info
mybucket = myalloc->bucket;
if (mybucket == NULL){
gkfatal("gkfree: Bad Bucket");
return;
}
queue(&mybucket->free_q,&myalloc->entry);
return;
}
Wednesday, March 18, 2009
C Code - Link List or Queues
One of my favorite real-time system tricks is to use queues, or linked-list
to handle message passing. Then the whole system can be lots of small co-routines, without the overhead of task-switching. Small and tight code really can run fast. Usually the problem you get with this is malloc/free overhead.
A pre-allocated chunk based system solves that. I'll post it to a future blog.
to handle message passing. Then the whole system can be lots of small co-routines, without the overhead of task-switching. Small and tight code really can run fast. Usually the problem you get with this is malloc/free overhead.
A pre-allocated chunk based system solves that. I'll post it to a future blog.
## queue.c
#include <stdio.h>
#include <stdlib.h>
#include "include/queue.h"
/*
** Queue.c
*/
struct entry_struct *unqueue(q)
struct queue_struct *q;
{
struct entry_struct *entry;
if (q==NULL) return(NULL);
if (q->head == NULL) return(NULL);
entry = q->head;
q->head = entry->next;
if (q->head == NULL) q->tail = NULL;
entry->next = NULL;
entry->prev = NULL;
return(entry);
}
void queue(q,entry)
struct queue_struct *q;
struct entry_struct *entry;
{
if (q==NULL) return;
if (entry == NULL) return;
entry->next = NULL;
entry->prev = NULL;
if (q->head == NULL){
q->head = entry;
q->tail = entry;
return;
}
entry->prev = q->tail;
q->tail->next = entry;
q->tail = entry;
return;
}
struct queue_struct *init_queue(void)
{
struct queue_struct *q;
q = (struct queue_struct *)malloc(sizeof(struct queue_struct *));
q->head = NULL;
q->tail = NULL;
return(q);
}
## queue.h
/*
** Queue.h
*/
struct entry_struct {
struct entry_struct *next;
struct entry_struct *prev;
};
struct queue_struct {
struct entry_struct *head;
struct entry_struct *tail;
};
struct entry_struct *unqueue(struct queue_struct *);
void queue(struct queue_struct *, struct entry_struct *);
struct queue_struct *init_queue(void);
Wednesday, February 25, 2009
Scraping Digikey using Ruby and ScRUBYt
Sometimes the information you need in your application is already out there on a web site. In a BOM (Bill-of-Materials) app I'm doing, I want to be able
to lookup distributor part numbers from a manufacture part number.
In this example I use Digikey, as they have the vast majority of parts I use in projects.
I use Scrubyt, a ruby library, by Peter Szinek, and Glenn Gillen.
It does the heavy lifting.
The data returned in this example, is in the form of a table. ScRUBYt makes
parsing the table straight forward. You can use firebug in firebird to find your
data if your doing a different website.
Here is the base code
require 'rubygems'
require 'scrubyt'
require 'pp'
digikey_data = Scrubyt::Extractor.define do
thepartnumber ='C0805C104K3RACTU'
thesearch = 'http://search.digikey.com/scripts/DkSearch/dksus.dll?
lang=en&site=US&x=0&y=0&keywords=' + thepartnumber
fetch thesearch
table "/html/body/div[2]/table" do
row "//tr" do
dist_pn "/td[1]"
manuf_pn "/td[2]"
pn_description "/td[3]"
pn_oem "/td[5]"
end
end
#
end
part_data = digikey_data.to_flat_hash
pp part_data
The result:
sin-gwest-laptop:digikey gwest$ ruby digikey.rb
[{:pn_oem=>"Kemet",
:dist_pn=>"399-1168-2-ND",
:manuf_pn=>"C0805C104K3RACTU",
:pn_description=>"CAP .10UF 25V CERAMIC X7R 0805"},
{:pn_oem=>"Kemet",
:dist_pn=>"399-1168-1-ND",
:manuf_pn=>"C0805C104K3RACTU",
:pn_description=>"CAP .10UF 25V CERAMIC X7R 0805"},
{:pn_oem=>"Kemet",
:dist_pn=>"399-1168-6-ND",
:manuf_pn=>"C0805C104K3RACTU",
:pn_description=>"CAP .10UF 25V CERAMIC X7R 0805"}]
sin-gwest-laptop:digikey gwest$
Thursday, February 19, 2009
ActiveScaffold - Example of using Roles for security
Recently I was ask for examples of using rolerequirement with activescaffod,
so here is the examples.
For security, two areas to consider.
1. Manage what menus appear.
2. Verify access in controllers.
You can also control what columns appear, and what actions are available as well. While I have done this all in one controller, to follow DRY, I find that having a adminspace controller, and a userspace controller a bit cleaner.
Note if you want to configure activescaffold dynamically, AS does some caching, and you need to specifically reconfigure AS each time, to make sure
of cache consistency.
For menus, I use widgets/tabnav
<%
# this partial renders a tabnav, you can call it in your views with:
# <%= tabnav :admin % > (just the tabnav)
# or, if you want a boxed tabnav:
# <% tabnav :admin do % >
# your html here
# <% end % >
# (remove the space between % and >, we don't want to mess up your brand new tabnav :-))
#
# you can pass render_tabnav a few options:
# :generate_css => true|false #=> generates a default inline css for the tabnav, defaults to false
# :html => aHash #=> sets html options for the tabnav's div (es :html => {:class=> 'myCssClass', :id=>'myCssId'})
#
render_tabnav :admin,
:generate_css => true do
if @current_user.has_role?('admin')
add_tab do |t|
t.named 'Admin'
t.titled 'Administration'
t.links_to :controller => 'adminspace/admin'
end
add_tab do |t|
t.named 'Users'
t.titled 'Users'
t.links_to :controller => 'adminspace/user'
end
add_tab do |t|
t.named 'Roles'
t.titled 'Roles'
t.links_to :controller => 'adminspace/role'
end
end
end
2. For the controllers:
class Adminspace::AdminController < ApplicationController
layout 'tier2admin'
before_filter :login_required
require_role "admin"
#
# Generated by GenRails
#
active_scaffold :Logmsg do |config|
config.label = 'AdminMessages'
config.actions = [:search, :show, :list]
end
end
Sunday, February 08, 2009
Converting a rails app to db2
DB2 has a long history, having ran on IBM mainframes, and mid-range for decades. Now it easy to move your rails app using it.
The lovely thing about the latest version 9.5 DB2 is that you can use
it for free, if your using the DB2-ExpressC version. It has most all the features you will need, and the database size is not limited. You can add support, or add-ons easily.
The issue you will have, if you have a existing rails app, is moving your data from mysql, or other rails database into db2.
The idea solution for this is a rails plugin, called YamlDb.
In Rails 2.1 its as easy as:
script/plugin install git://github.com/adamwiggins/yaml_db.git
rake db:data:dump
rake db:schema:dump
then update your database config file to point to db2:
development:
adapter: ibm_db
database: baseapp
username: gwest
password: xxxxxxxx
The rake db:create does not work in db2, so:
db2 create database baseapp
Then:
rake db:schema:load
rake db:data:load
And your ready to go.
Not that database names can only be 8 characters, and underlines and spaces may cause you problems. Otherwise, its painless.
Sunday, February 01, 2009
Mac DBI - dbi binary will not list gem drivers but will work
Thanks to the creator of DBI, I found out that if you install gem based dbi drivers, that the dbi command will not list them, but they will work properly in your ruby code.
So to list tables in our sqlite3 development database.
require 'rubygems'
gem 'dbi'
require 'dbi'
require 'pp'
# Assume dbd-sqlite3 is installed
# sudo gem install dbd-sqlite3
# Assumes we are in our project directory
fromdb = DBI.connect('dbi:SQLite3:db/development.sqlite3', nil, nil)
tables = fromdb.tables
tables.each { |tablename|
pp tablename
}
I tend to use DBI in migrating from foreign environments into a rails app.
I also use DBI to write automatic generators for controllers, models, and
menus. It really makes getting going on a legacy app very fast.
Sunday, January 25, 2009
Ruby DBI does find DBD drivers on Mac when using Gem
I found a very nice problem when trying to use DBI on a mac, with the sqlite3 driver
I've been a avid your of dbi, which is a direct database API in Ruby, that allow
low level access to databases.
I've used it on Linux, and Windows against Oracle extensively, and several other databases.
So expected it to work the same. Seems like I hit a small problem in that oh.
My script was telling me no driver. When I ran the dbi command, it showed there was no drivers.
If I used irb, and did a DBI.drivers_available, it told me none.
First, a quick review of the installed gems:
sh-3.2# gem list
*** LOCAL GEMS ***
actionmailer (2.2.2)
actionpack (2.2.2)
activerecord (2.2.2)
activeresource (2.2.2)
activesupport (2.2.2)
dbd-sqlite3 (1.2.4)
dbi (0.4.1)
deprecated (2.0.1)
gem_plugin (0.2.3)
hpricot (0.6.164)
mechanize (0.9.0)
nokogiri (1.1.1)
rails (2.2.2)
rake (0.8.3)
scrubyt (0.4.06)
sqlite3-ruby (1.2.4)
wxruby (1.9.9)
Yep, Its there, everything looks ok.
Now lets track down the source to find out what is going on.
# Returns a list (of String) of the currently available drivers on your system in
# 'dbi:driver:' format.
#
# This currently does not work for rubygems installations, please see
# DBI.collect_drivers for reasons.
def available_drivers
drivers = []
collect_drivers.each do |key, value|
drivers.push("dbi:#{key}:")
end
return drivers
end
That looks ok.
Lets go and see what collect_drivers()
#
# Return a list (of String) of the available drivers.
#
# NOTE:: This is non-functional for gem installations, due to the
# nature of how it currently works. A better solution for
# this will be provided in DBI 0.6.0.
def collect_drivers
drivers = { }
# FIXME rewrite this to leverage require and be more intelligent
path = File.join(File.dirname(__FILE__), "dbd", "*.rb")
Dir[path].each do |f|
if File.file?(f)
driver = File.basename(f, ".rb")
drivers[driver] = f
end
end
Ok, the FIXME gives us the clue.
Looks like it does not bother to search thru rubygems.
You must manually copy the dbd drivers into dbi.
The Solution:
Copy the drivers into the dbi gem.
cd /opt/local/lib/ruby/gems/1.8/gems/dbi-0.4.1/lib
cp -r ../../dbd-sqlite3-1.2.4/lib/dbd dbd
I've been a avid your of dbi, which is a direct database API in Ruby, that allow
low level access to databases.
I've used it on Linux, and Windows against Oracle extensively, and several other databases.
So expected it to work the same. Seems like I hit a small problem in that oh.
My script was telling me no driver. When I ran the dbi command, it showed there was no drivers.
If I used irb, and did a DBI.drivers_available, it told me none.
First, a quick review of the installed gems:
sh-3.2# gem list
*** LOCAL GEMS ***
actionmailer (2.2.2)
actionpack (2.2.2)
activerecord (2.2.2)
activeresource (2.2.2)
activesupport (2.2.2)
dbd-sqlite3 (1.2.4)
dbi (0.4.1)
deprecated (2.0.1)
gem_plugin (0.2.3)
hpricot (0.6.164)
mechanize (0.9.0)
nokogiri (1.1.1)
rails (2.2.2)
rake (0.8.3)
scrubyt (0.4.06)
sqlite3-ruby (1.2.4)
wxruby (1.9.9)
Yep, Its there, everything looks ok.
Now lets track down the source to find out what is going on.
# Returns a list (of String) of the currently available drivers on your system in
# 'dbi:driver:' format.
#
# This currently does not work for rubygems installations, please see
# DBI.collect_drivers for reasons.
def available_drivers
drivers = []
collect_drivers.each do |key, value|
drivers.push("dbi:#{key}:")
end
return drivers
end
That looks ok.
Lets go and see what collect_drivers()
#
# Return a list (of String) of the available drivers.
#
# NOTE:: This is non-functional for gem installations, due to the
# nature of how it currently works. A better solution for
# this will be provided in DBI 0.6.0.
def collect_drivers
drivers = { }
# FIXME rewrite this to leverage require and be more intelligent
path = File.join(File.dirname(__FILE__), "dbd", "*.rb")
Dir[path].each do |f|
if File.file?(f)
driver = File.basename(f, ".rb")
drivers[driver] = f
end
end
Ok, the FIXME gives us the clue.
Looks like it does not bother to search thru rubygems.
You must manually copy the dbd drivers into dbi.
The Solution:
Copy the drivers into the dbi gem.
cd /opt/local/lib/ruby/gems/1.8/gems/dbi-0.4.1/lib
cp -r ../../dbd-sqlite3-1.2.4/lib/dbd dbd
Saturday, January 24, 2009
A Review of Authentication and Access Control for Ruby On Rails
A review of Authentication and Access Control for Ruby On Rails
Previous Articles That Are Related
http://mentalpagingspace.blogspot.com/2007/11/getting-started-faster-in-rails.html
http://mentalpagingspace.blogspot.com/2008/12/rails-to-windows-integration-single.html
First, to give some context, when building Rails apps, its common practice to use
a common set of plugins. In "EDGE" rails, there is even a template system so that
when you create a rails apps, it can load your standard plugins.
For my basic set, I use the following:
My base set is:
ActsAsAuthenticated (Login)
http://www.railslodge.com/plugins/1-acts-as-authenticated
RestfulAuthentication (The Replacement for the above)
http://github.com/technoweenie/restful-authentication/tree/master
RoleRequirement (Access Control in controllers and menus)
http://code.google.com/p/rolerequirement/
Tabnav/Widgets (Menus)
http://wiki.github.com/paolodona/rails-widgets
ActiveScaffold (Controller/View for CRUD)
http://activescaffold.com/
In rails there are "two" different problems, first there is authentication, then there is access control.
Authentication is:
1. How do we login to the app
2. User Model
a. User Name
b. Email Address
3. What Access do we have? (Roles)
Do do the heavy lifting for this, we use a plugin
There are several different plugs for authentication:
1. ActsAsAuthenticated (The original)
3. RestfulAuthentication (The new "standard")
2. OpenId
3. NTLM (Use this when your app is only in a ActiveDirectory environment)
So once your logged in, using one of the above, we ad the rolerequirement plugin.
I find RoleRequirement handles my access control very nicely. It enhances your user model, with the ability to have
multiple role. You can then use the "role" to verify in your menus and controllers if a user can perform or use
certain actions.
If you look at rolerequirement, in your code, you can do something like the following:
class Adminspace::AdminController < ApplicationController
before_filter :login_required
require_role "admin"
layout 'basic'
def index
end
end
Notice the require_role above.
When you setup rolerequirement, You can login to your app, and add additional roles.
You can also add a migration to add these roles. (Useful in a empty database setting.
Also notice the before filter. This makes sure the user is logged in.
The next issue you will get is, updating menus on the fly.
I use tabnav typically for my menus.
So in order to make menus come and go based on roles, you can check roles in tabnav.
<%
# this partial renders a tabnav
render_tabnav :aircenter,
:generate_css => true do
add_tab do |t|
t.named 'ReqCenter'
t.titled 'ReqCenter'
t.links_to :controller => '/reqcenter'
end
if @current_user.has_role?('admin')
add_tab do |t|
t.named 'Admin'
t.titled 'admin'
t.links_to :controller => 'adminspace/admin'
end
end
add_tab do |t|
t.named 'Requests'
t.titled 'Requests'
t.links_to :controller => 'requestspace/requesthome'
end
add_tab do |t|
t.named 'Logout'
t.titled 'Logout'
t.links_to :controller => 'adminspace/account', :action => 'logout'
end
end
%>
A example of a Admin CRUD, using ActiveScaffold:
class Adminspace::UserController < ApplicationController
layout 'tier1admin'
before_filter :login_required
require_role "admin"
active_scaffold :users do |config|
config.list.columns = [:id,:display_name,:businessunit,:department,:location]
config.columns[:businessunit].ui_type = :select
config.columns[:location].ui_type = :select
config.columns[:department].ui_type = :select
config.columns[:roles].ui_type = :select
end
end
So now, using namespaces, we can have a admin controller, with a role of admin for access, that allows the admin
to change anything in all users, and another controller, that allows only the user to change his own fields.
Previous Articles That Are Related
http://mentalpagingspace.blogspot.com/2007/11/getting-started-faster-in-rails.html
http://mentalpagingspace.blogspot.com/2008/12/rails-to-windows-integration-single.html
First, to give some context, when building Rails apps, its common practice to use
a common set of plugins. In "EDGE" rails, there is even a template system so that
when you create a rails apps, it can load your standard plugins.
For my basic set, I use the following:
My base set is:
ActsAsAuthenticated (Login)
http://www.railslodge.com/plugins/1-acts-as-authenticated
RestfulAuthentication (The Replacement for the above)
http://github.com/technoweenie/restful-authentication/tree/master
RoleRequirement (Access Control in controllers and menus)
http://code.google.com/p/rolerequirement/
Tabnav/Widgets (Menus)
http://wiki.github.com/paolodona/rails-widgets
ActiveScaffold (Controller/View for CRUD)
http://activescaffold.com/
In rails there are "two" different problems, first there is authentication, then there is access control.
Authentication is:
1. How do we login to the app
2. User Model
a. User Name
b. Email Address
3. What Access do we have? (Roles)
Do do the heavy lifting for this, we use a plugin
There are several different plugs for authentication:
1. ActsAsAuthenticated (The original)
3. RestfulAuthentication (The new "standard")
2. OpenId
3. NTLM (Use this when your app is only in a ActiveDirectory environment)
So once your logged in, using one of the above, we ad the rolerequirement plugin.
I find RoleRequirement handles my access control very nicely. It enhances your user model, with the ability to have
multiple role. You can then use the "role" to verify in your menus and controllers if a user can perform or use
certain actions.
If you look at rolerequirement, in your code, you can do something like the following:
class Adminspace::AdminController < ApplicationController
before_filter :login_required
require_role "admin"
layout 'basic'
def index
end
end
Notice the require_role above.
When you setup rolerequirement, You can login to your app, and add additional roles.
You can also add a migration to add these roles. (Useful in a empty database setting.
Also notice the before filter. This makes sure the user is logged in.
The next issue you will get is, updating menus on the fly.
I use tabnav typically for my menus.
So in order to make menus come and go based on roles, you can check roles in tabnav.
<%
# this partial renders a tabnav
render_tabnav :aircenter,
:generate_css => true do
add_tab do |t|
t.named 'ReqCenter'
t.titled 'ReqCenter'
t.links_to :controller => '/reqcenter'
end
if @current_user.has_role?('admin')
add_tab do |t|
t.named 'Admin'
t.titled 'admin'
t.links_to :controller => 'adminspace/admin'
end
end
add_tab do |t|
t.named 'Requests'
t.titled 'Requests'
t.links_to :controller => 'requestspace/requesthome'
end
add_tab do |t|
t.named 'Logout'
t.titled 'Logout'
t.links_to :controller => 'adminspace/account', :action => 'logout'
end
end
%>
A example of a Admin CRUD, using ActiveScaffold:
class Adminspace::UserController < ApplicationController
layout 'tier1admin'
before_filter :login_required
require_role "admin"
active_scaffold :users do |config|
config.list.columns = [:id,:display_name,:businessunit,:department,:location]
config.columns[:businessunit].ui_type = :select
config.columns[:location].ui_type = :select
config.columns[:department].ui_type = :select
config.columns[:roles].ui_type = :select
end
end
So now, using namespaces, we can have a admin controller, with a role of admin for access, that allows the admin
to change anything in all users, and another controller, that allows only the user to change his own fields.
Monday, January 19, 2009
Rails verse Microsoft Access
Like everyone with significant Microsoft experience I've done some Access Applications.
The reality is, I'd prefer to do it in Rails. Ruby on Rails let me gain the power of using
any ruby gem or rails plug-in. I have access to practically every database in Rails.
And I have access to nice Menus as well. Plus the ability to define a web api,
using REST.
For me, for a small app, I can host it as a windows service directly on Windows with Mongrel. I can
use single sign-on, so if your in the domain, your automatically authenticated. Using role-requirement,
I can define user roles, and even have dynamic menus.
Even better I can use PRAWN or Pdf::Writer to directly generate reports. Or Scruby to scrape web pages.
And Add to the Data on the fly.
The other thing I've seen with Access applications, it too easy to get the file locked by one user, and
no one else able to access it. Scaling on Rails is Easy compared to Access. And on top of all of this is
easy cross platform functionality.
The nice thing is that the language of Rails is Ruby. And Ruby is significantly easier and faster to develop on than VBA.
The reality is, I'd prefer to do it in Rails. Ruby on Rails let me gain the power of using
any ruby gem or rails plug-in. I have access to practically every database in Rails.
And I have access to nice Menus as well. Plus the ability to define a web api,
using REST.
For me, for a small app, I can host it as a windows service directly on Windows with Mongrel. I can
use single sign-on, so if your in the domain, your automatically authenticated. Using role-requirement,
I can define user roles, and even have dynamic menus.
Even better I can use PRAWN or Pdf::Writer to directly generate reports. Or Scruby to scrape web pages.
And Add to the Data on the fly.
The other thing I've seen with Access applications, it too easy to get the file locked by one user, and
no one else able to access it. Scaling on Rails is Easy compared to Access. And on top of all of this is
easy cross platform functionality.
The nice thing is that the language of Rails is Ruby. And Ruby is significantly easier and faster to develop on than VBA.
Thursday, January 15, 2009
Windows 7 - The Bleeding Edge
Well I finally bit the bullet. While I had tried vista on my work laptop, and after sweeting it for 3 months, went back to xp.
Microsoft Windows 7 seemed to be the answer to what I was looking for.
So I finally decided to use the Beta on my main gaming machine. My main work machine is a MAC.
First, Lets see how well the beta behaves. So I created a new VM on my Mac, and did the install from the ISO.
Wow, it went smooth and easy. Very fast install compared to XP. And so much better than Vista.
So after playing a bit with that, I decided that my gaming machine needs a cleanup. So its a great time to
give Windows 7 a try. In my house I currently support 2 desktop for my kids under Vista, another of the same desktop for my wife, and a desktop for myself, plus a laptop. I've migrated most things to my MAC.
For me on the Gaming Machine, WOW is very important. Vista generally left something to be desired on Gaming performance.
So Windows 7, I thought there was hope.
So first making sure that I had copies of everything, I decided to do a "new" install without formating the disk.
(Not something I generally like doing by the way). Windows 7, renamed the windows folder, and moved the Document and Users, as well as the Program files under the Windows.old. I like that implementation. I then went into the Windows.old, and dragged the documents folder back to the Windows 7 location. Wow, it noticed and put the photos in the right place.
I then turned on my Pixma 5000 USB printer, and Windows 7, went out and grabbed the right printer driver and installed that as well. Nice.
Now to do a first go/no-go test. I sent a shortcut from my WOW.exe to the desktop. And ran it. It worked right out of the box.
WOW client is actually very good at copy and go. And with Windows 7 it was no different. Good job Microsoft. I'll do some detailed testing tonight, but so far the performance and drivers look good
Now lets move on and try a older version of office. Installed was clean.
And to my shock, the old version of Office was FAST. Now that is a improvement.
So I'll update the post over the next few days. I really want to try some of the new VM offerings from VMware,
and Microsoft.
Microsoft Windows 7 seemed to be the answer to what I was looking for.
So I finally decided to use the Beta on my main gaming machine. My main work machine is a MAC.
First, Lets see how well the beta behaves. So I created a new VM on my Mac, and did the install from the ISO.
Wow, it went smooth and easy. Very fast install compared to XP. And so much better than Vista.
So after playing a bit with that, I decided that my gaming machine needs a cleanup. So its a great time to
give Windows 7 a try. In my house I currently support 2 desktop for my kids under Vista, another of the same desktop for my wife, and a desktop for myself, plus a laptop. I've migrated most things to my MAC.
For me on the Gaming Machine, WOW is very important. Vista generally left something to be desired on Gaming performance.
So Windows 7, I thought there was hope.
So first making sure that I had copies of everything, I decided to do a "new" install without formating the disk.
(Not something I generally like doing by the way). Windows 7, renamed the windows folder, and moved the Document and Users, as well as the Program files under the Windows.old. I like that implementation. I then went into the Windows.old, and dragged the documents folder back to the Windows 7 location. Wow, it noticed and put the photos in the right place.
I then turned on my Pixma 5000 USB printer, and Windows 7, went out and grabbed the right printer driver and installed that as well. Nice.
Now to do a first go/no-go test. I sent a shortcut from my WOW.exe to the desktop. And ran it. It worked right out of the box.
WOW client is actually very good at copy and go. And with Windows 7 it was no different. Good job Microsoft. I'll do some detailed testing tonight, but so far the performance and drivers look good
Now lets move on and try a older version of office. Installed was clean.
And to my shock, the old version of Office was FAST. Now that is a improvement.
So I'll update the post over the next few days. I really want to try some of the new VM offerings from VMware,
and Microsoft.
Thursday, January 08, 2009
And Max ask how to find the user name? (Automatic Logon)
This answer varies depending on if you doing linux or windows.
The key is NTLM. If the client is coming from the local net, and part of a domain,
you can use NTLM based login, this is also know as Integrated Windows Authentication.
If you using NTLM to automatically logon, there is a nice plugin you can using with
mongrel on windows.
See my post
http://mentalpagingspace.blogspot.com/2008/12/rails-to-windows-integration-single.html
This is idea for light-duty applications hosted on a windows machine or vm. I use this when my
application needs to make WMI or other windows specific calls. Your application login
is automatic if the user is coming from the local network and part of a domain. The mongrel
can run as a service that auto starts at boot time.
In Linux you can also do the same trick, there is a apache plugin to allow you to use NTLM.
Since I've seen apache articles on using NTLM in a domain, then I'll leave those to the reader.
And as a note, I prefer LINUX for applications that do not need significant usage of Window API.
Its faster and more stable than running it on XP. My fav at the moment is to use Passenger to manage
linux hosting of Rails
On the other hand, If you have a app a handful of people are going to use sporadically, which
is common in the enterprise. A XP based Implementation has been problem free for me.
The key is NTLM. If the client is coming from the local net, and part of a domain,
you can use NTLM based login, this is also know as Integrated Windows Authentication.
If you using NTLM to automatically logon, there is a nice plugin you can using with
mongrel on windows.
See my post
http://mentalpagingspace.blogspot.com/2008/12/rails-to-windows-integration-single.html
This is idea for light-duty applications hosted on a windows machine or vm. I use this when my
application needs to make WMI or other windows specific calls. Your application login
is automatic if the user is coming from the local network and part of a domain. The mongrel
can run as a service that auto starts at boot time.
In Linux you can also do the same trick, there is a apache plugin to allow you to use NTLM.
Since I've seen apache articles on using NTLM in a domain, then I'll leave those to the reader.
And as a note, I prefer LINUX for applications that do not need significant usage of Window API.
Its faster and more stable than running it on XP. My fav at the moment is to use Passenger to manage
linux hosting of Rails
On the other hand, If you have a app a handful of people are going to use sporadically, which
is common in the enterprise. A XP based Implementation has been problem free for me.
Tuesday, December 30, 2008
Virtual Manufacturing Control - VmWare To the Factory Floor
In the industry, VmWare has taken over the Data Center. Server Consolidation has been going on for a while. Multiple servers
that used to run on dedicated hardware, are now Virtual Machines, multiple setting on one rack or blade server. Its not uncommon to see 10's to 100's of virtual server running on one physical server. This has resulted in great cost saving, easy upgrades, easier provisioning, and a reduction of power consumption. This technology is proven, and works.
To give you a example, before VmWare, it would take a week to get a quote, get the approvals for purchase, then another week for delivery, and then a day to set it up, install the OS, and then your ready to start thinking about putting a application on the server. You also had to worry about backup, and KVM (Keyboard and monitor switching).
With VmWare, I log into VmCenter, create a new Virtual machine, selecting how much memory and CPU I need, and pointing it to a ISO image of the OS. And Say Go. The new machine "boots" up, and starts installing the OS. Since a Dell Blade, or 1U server could support upwards of 32 machine, the physical act of buying hardware is reduced significantly. The reality is, most servers are under-utilized, and VmWare lets us use these otherwise wasted resources better.
Another layer of this puzzle is high-availability. We really dont want our applications to go down. It doent matter if its a DHCP server, or a Oracle Database, being up means people are working, and having a hardware issue can really spoil your day when you manage 100's of servers, and almost as many applications. The answer to this is has a few components.
First, we break the Storage component using iSCSI into a separate device. The RAID is now a external device we talk to over TCP/IP using SCSI over our ethernet connections. With 10Gig Ethernet, and hardware based switching, IP is the fastest SAN. Most blade give us 4 1-Gig links which allows 512Megabytes a second of capacity for getting data in and out. This concept means our blades or rack based servers now have no disk locally.
This separation means we can now migrate any virtual machine into any physical cpu, using what VmWare calls vMotion. So if you want to pull a blade server, and replace memory, or reprogram its bios, you can use VmCenter to move a running virtual machine to another physical cpu, without the application even knowing it. In the event of a failure the VM can be run on any physical CPU available. Also Snapshots can be taken, and the allows a virtual ghost type functionality where we can take a virtual photo of what is running, and go back to it at any time.
In the above example, it could have been even easier. In that you load your server-os into a VM, do your patching and updates, then convert it into a template. After you have a template, you can then create more servers using it by just cloning it, and giving it a new name.
None of this so far is new. I've been doing this for a whle. The next step is called VmView.
In manufacturing for example, its not uncommon that you need PC near or in the production areas. So for example lets say we need 320 PC to handle our ERP or SAP type application. These would be scattered over a large area of factory floor. The condition in a factory are not office environment. Heat, Temp, and Humidity are not comfortable for the workers, and worse for the equipment. There are other problems with this approach. 1) Updates - While WSUS and Microsoft tries to automate this, all to often you end up walking to every pc. 2) Security - USB Drives, Thumb Drives, CD-ROM, Phones, and Ipod are all avenues for data leakage, as well as virus to come into the system. The solution is a thin-terminal.
Traditionally with thin-terminals, you need a terminal-server. A terminal server gives you a shared environment to run your applications. This works reasonably well, but not every application can install in such a shared environment. Forcing us back to the common pc approach. With VmView, we can now apply the same technology from the DataCenter to the desktop.
We can create one or more template machines, that are automatically created, re-used, removed or restored that run in the data-center. We can choose what USB devices are connected to the VM on the fly. This allows us to support scanners, barcode guns, rfid readers, and printers that are connected via thin-terminals to the VM in the data-center.
To give a example. For 320 Users we could put those on six M905 Dell Blades. Each user would be allocated a 20Gig Virtual hard disk. Using templates, the OS and common apps could be shared. We would customize AD to do small roaming profiles to avoide any corruption issue, and re-direct MyDocuments etc to proper file server. Now we can "install" a new pc for a new staff in a minute instead of hours. We can snapshot, audit, and reboot all centrally. We can update and touch every pc in the shop with a script easily. And we can deploy new patches, even change from XP to Vista in minutes.
Even changing the desktop hardware is easy. We buy new servers for VmView. Mount them in the data center, and vMotion them over to the new servers.
Using proper network design, we can have a high-availability redundant solution from server to desktop onto the factory floor that is secure and easy to maintain.
that used to run on dedicated hardware, are now Virtual Machines, multiple setting on one rack or blade server. Its not uncommon to see 10's to 100's of virtual server running on one physical server. This has resulted in great cost saving, easy upgrades, easier provisioning, and a reduction of power consumption. This technology is proven, and works.
To give you a example, before VmWare, it would take a week to get a quote, get the approvals for purchase, then another week for delivery, and then a day to set it up, install the OS, and then your ready to start thinking about putting a application on the server. You also had to worry about backup, and KVM (Keyboard and monitor switching).
With VmWare, I log into VmCenter, create a new Virtual machine, selecting how much memory and CPU I need, and pointing it to a ISO image of the OS. And Say Go. The new machine "boots" up, and starts installing the OS. Since a Dell Blade, or 1U server could support upwards of 32 machine, the physical act of buying hardware is reduced significantly. The reality is, most servers are under-utilized, and VmWare lets us use these otherwise wasted resources better.
Another layer of this puzzle is high-availability. We really dont want our applications to go down. It doent matter if its a DHCP server, or a Oracle Database, being up means people are working, and having a hardware issue can really spoil your day when you manage 100's of servers, and almost as many applications. The answer to this is has a few components.
First, we break the Storage component using iSCSI into a separate device. The RAID is now a external device we talk to over TCP/IP using SCSI over our ethernet connections. With 10Gig Ethernet, and hardware based switching, IP is the fastest SAN. Most blade give us 4 1-Gig links which allows 512Megabytes a second of capacity for getting data in and out. This concept means our blades or rack based servers now have no disk locally.
This separation means we can now migrate any virtual machine into any physical cpu, using what VmWare calls vMotion. So if you want to pull a blade server, and replace memory, or reprogram its bios, you can use VmCenter to move a running virtual machine to another physical cpu, without the application even knowing it. In the event of a failure the VM can be run on any physical CPU available. Also Snapshots can be taken, and the allows a virtual ghost type functionality where we can take a virtual photo of what is running, and go back to it at any time.
In the above example, it could have been even easier. In that you load your server-os into a VM, do your patching and updates, then convert it into a template. After you have a template, you can then create more servers using it by just cloning it, and giving it a new name.
None of this so far is new. I've been doing this for a whle. The next step is called VmView.
In manufacturing for example, its not uncommon that you need PC near or in the production areas. So for example lets say we need 320 PC to handle our ERP or SAP type application. These would be scattered over a large area of factory floor. The condition in a factory are not office environment. Heat, Temp, and Humidity are not comfortable for the workers, and worse for the equipment. There are other problems with this approach. 1) Updates - While WSUS and Microsoft tries to automate this, all to often you end up walking to every pc. 2) Security - USB Drives, Thumb Drives, CD-ROM, Phones, and Ipod are all avenues for data leakage, as well as virus to come into the system. The solution is a thin-terminal.
Traditionally with thin-terminals, you need a terminal-server. A terminal server gives you a shared environment to run your applications. This works reasonably well, but not every application can install in such a shared environment. Forcing us back to the common pc approach. With VmView, we can now apply the same technology from the DataCenter to the desktop.
We can create one or more template machines, that are automatically created, re-used, removed or restored that run in the data-center. We can choose what USB devices are connected to the VM on the fly. This allows us to support scanners, barcode guns, rfid readers, and printers that are connected via thin-terminals to the VM in the data-center.
To give a example. For 320 Users we could put those on six M905 Dell Blades. Each user would be allocated a 20Gig Virtual hard disk. Using templates, the OS and common apps could be shared. We would customize AD to do small roaming profiles to avoide any corruption issue, and re-direct MyDocuments etc to proper file server. Now we can "install" a new pc for a new staff in a minute instead of hours. We can snapshot, audit, and reboot all centrally. We can update and touch every pc in the shop with a script easily. And we can deploy new patches, even change from XP to Vista in minutes.
Even changing the desktop hardware is easy. We buy new servers for VmView. Mount them in the data center, and vMotion them over to the new servers.
Using proper network design, we can have a high-availability redundant solution from server to desktop onto the factory floor that is secure and easy to maintain.
Wednesday, December 10, 2008
Ruby on Rails - Are you a Member?
Large Enterprises that use Windows are typically in a Domain environment.
Often application access is controlled by group membership in ActiveDirectory.
The easy and fast way of finding this out, is via ruby-net-ldap, a native ruby implmenation.
So first install ruby-net-ldap
gem install ruby-net-ldap
You will get:
C:\>gem install ruby-net-ldap
Successfully installed ruby-net-ldap-0.0.4
1 gem installed
Installing ri documentation for ruby-net-ldap-0.0.4...
Installing RDoc documentation for ruby-net-ldap-0.0.4...
Example:
ad = ActiveDirectory.new('user','password','server','dc=company,dc=com')
pp ad.GetMembers('user1')
pp ad.GetMembers('user2')
if ad.MemberOf?("user1","myappgroup")
puts "YES We can do work"
end
You will need the following library:
#lib\active_directory.rb
require 'rubygems'
require 'net/ldap'
require 'pp'
class ActiveDirectory
#server has to be full name.domain.com
#treebase is dc=domain,dc=com
def initialize(username,password,server,treebase)
@username = username
@password = password
@server = server
@treebase = treebase
@ldap_con = Net::LDAP.new( {:host => @server, :port => 389, :auth =>
{ :method => :simple, :username => @username, :password => @password}})
end
def cleanup_members(members)
mymembers = Array.new
mymembers << '-none'
members.each do |member|
cmc = member[3,64]
cmcs = cmc.split(',')
mymembers << String.new(cmcs[0].delete(' '))
end
return(mymembers)
end
def GetEmail(username)
op_filter = Net::LDAP::Filter.eq( "samaccountname", username)
entries = @ldap_con.search( :base => @treebase, :filter => op_filter,:attributes=>
['samaccountname','mail'])
entry = entries[0]
emailaddress = entry.mail
return(emailaddress)
end
def GetMembers(username)
op_filter = Net::LDAP::Filter.eq( "samaccountname", username)
entries = @ldap_con.search( :base => @treebase, :filter => op_filter,:attributes=>
['samaccountname','memberof'])
entry = entries[0]
begin
themembers = entry.memberof
rescue
themembers = Array.new
end
membership = cleanup_members(themembers)
return(membership)
end
def MemberOf?(username,thegroup)
groups = GetMembers(username)
return groups.include?(thegroup)
end
end #ActiveDirectory
Often application access is controlled by group membership in ActiveDirectory.
The easy and fast way of finding this out, is via ruby-net-ldap, a native ruby implmenation.
So first install ruby-net-ldap
gem install ruby-net-ldap
You will get:
C:\>gem install ruby-net-ldap
Successfully installed ruby-net-ldap-0.0.4
1 gem installed
Installing ri documentation for ruby-net-ldap-0.0.4...
Installing RDoc documentation for ruby-net-ldap-0.0.4...
Example:
ad = ActiveDirectory.new('user','password','server','dc=company,dc=com')
pp ad.GetMembers('user1')
pp ad.GetMembers('user2')
if ad.MemberOf?("user1","myappgroup")
puts "YES We can do work"
end
You will need the following library:
#lib\active_directory.rb
require 'rubygems'
require 'net/ldap'
require 'pp'
class ActiveDirectory
#server has to be full name.domain.com
#treebase is dc=domain,dc=com
def initialize(username,password,server,treebase)
@username = username
@password = password
@server = server
@treebase = treebase
@ldap_con = Net::LDAP.new( {:host => @server, :port => 389, :auth =>
{ :method => :simple, :username => @username, :password => @password}})
end
def cleanup_members(members)
mymembers = Array.new
mymembers << '-none'
members.each do |member|
cmc = member[3,64]
cmcs = cmc.split(',')
mymembers << String.new(cmcs[0].delete(' '))
end
return(mymembers)
end
def GetEmail(username)
op_filter = Net::LDAP::Filter.eq( "samaccountname", username)
entries = @ldap_con.search( :base => @treebase, :filter => op_filter,:attributes=>
['samaccountname','mail'])
entry = entries[0]
emailaddress = entry.mail
return(emailaddress)
end
def GetMembers(username)
op_filter = Net::LDAP::Filter.eq( "samaccountname", username)
entries = @ldap_con.search( :base => @treebase, :filter => op_filter,:attributes=>
['samaccountname','memberof'])
entry = entries[0]
begin
themembers = entry.memberof
rescue
themembers = Array.new
end
membership = cleanup_members(themembers)
return(membership)
end
def MemberOf?(username,thegroup)
groups = GetMembers(username)
return groups.include?(thegroup)
end
end #ActiveDirectory
Rails to Windows Integration - Single Signon with NTLM and Mongrel
I was moving a application for network management in Rails into a production environment.
It had originally started out as something for my own use.
Of course I did not do security for a single user app.
In previous apps I'd used authenticated system with ldap to give a sign-on.
I wanted easier and tighter integration. NTLM comes to mind, as you can just clink a link,
and your windows login is re-used for your web application.
The problem has always been that mongrel does not support NTLM.
Well that solved now. Thanks to the work of Seggy Umboh, there is a new
NTLM filter for mongrel.
http://github.com/secobarbital/mongrel-ntlm/tree/master
This actually passes you the windows domain user name
directly into your controller.
To get mongrel to use the filter, you must pass a config file, as
documented in the readme.
Then you need a few things in your app:
First in your routes:
map.connect 'ntlm/login', :controller => 'account', :action => 'login'
I added this so the ntlm mongrel filter will chain to my account controller. That way
your "user" controller can remain unchanged. I just prefer to keep them separate.
Easier to add into existing systems that way.
The in the account controller you need the following:
class AccountController < ApplicationController
layout 'tier1'
def login
my_user_name = request.env['REMOTE_USER']
if my_user_name.nil?
session[:logged_in] = :false
return
end
myuser = User.find(:first, :conditions => ["name = ?", my_user_name])
if myuser.nil?
myuser = User.new
myuser.name = my_user_name
myuser.save
end
session[:current_user] = my_user_name
session[:user_id] = myuser.id
session[:logged_in] = :true
redirect_back_or_default('/myapp')
end
end
Then you need a little helper lib. This one is cut down from authenticated system.
This should be in you lib directory.
#ntlm_system.rb
module NtlmSystem
protected
# Returns true or false if the user is logged in.
# Preloads @current_user with the user model if they're logged in.
def logged_in?
if session[:logged_in].nil?
return false
end
if session[:logged_in] == :false
return false
end
return true
end
def login_required
if logged_in?()
return
end
pp "Access Denied"
access_denied()
end
# Redirect as appropriate when an access request fails.
# Our application only uses ntlm based login - you must be part of the domain
def access_denied
redirect_to('/ntlm/login')
end
# Store the URI of the current request in the session.
#
# We can return to this location by calling #redirect_back_or_default.
def store_location
session[:return_to] = request.request_uri
end
# Redirect to the URI stored by the most recent store_location call or
# to the passed default.
def redirect_back_or_default(default)
session[:return_to] ? redirect_to(session[:return_to]) : redirect_to(default)
session[:return_to] = nil
end
end
Finally in your application.rb
# Filters added to this controller apply to all controllers in the application.
# Likewise, all the methods added will be available for all controllers.
class ApplicationController < ActionController::Base
helper :all # include all helpers, all the time
include NtlmSystem # Add this for NTLM to work
# See ActionController::RequestForgeryProtection for details
# Uncomment the :secret if you're not using the cookie session store
protect_from_forgery :secret => '6e2f000403020a0d9041a7bfb069239d'
# See ActionController::Base for details
# Uncomment this to filter the contents of submitted sensitive data parameters
# from your application log (in this case, all fields with names like "password").
filter_parameter_logging :password
end
When you start your application, make sure you do it like this:
mongrel_rails start -S config\mongrel_ntlm.conf
After reading, on NTLM, and playing with it, I found the URL are a bit picky, if you
want to use the app, without needing to key your username and password
http://servername:3000/ work perfectly fine.
If you put the whole name, it tends to ask you the user name and password
It had originally started out as something for my own use.
Of course I did not do security for a single user app.
In previous apps I'd used authenticated system with ldap to give a sign-on.
I wanted easier and tighter integration. NTLM comes to mind, as you can just clink a link,
and your windows login is re-used for your web application.
The problem has always been that mongrel does not support NTLM.
Well that solved now. Thanks to the work of Seggy Umboh, there is a new
NTLM filter for mongrel.
http://github.com/secobarbital/mongrel-ntlm/tree/master
This actually passes you the windows domain user name
directly into your controller.
To get mongrel to use the filter, you must pass a config file, as
documented in the readme.
Then you need a few things in your app:
First in your routes:
map.connect 'ntlm/login', :controller => 'account', :action => 'login'
I added this so the ntlm mongrel filter will chain to my account controller. That way
your "user" controller can remain unchanged. I just prefer to keep them separate.
Easier to add into existing systems that way.
The in the account controller you need the following:
class AccountController < ApplicationController
layout 'tier1'
def login
my_user_name = request.env['REMOTE_USER']
if my_user_name.nil?
session[:logged_in] = :false
return
end
myuser = User.find(:first, :conditions => ["name = ?", my_user_name])
if myuser.nil?
myuser = User.new
myuser.name = my_user_name
myuser.save
end
session[:current_user] = my_user_name
session[:user_id] = myuser.id
session[:logged_in] = :true
redirect_back_or_default('/myapp')
end
end
Then you need a little helper lib. This one is cut down from authenticated system.
This should be in you lib directory.
#ntlm_system.rb
module NtlmSystem
protected
# Returns true or false if the user is logged in.
# Preloads @current_user with the user model if they're logged in.
def logged_in?
if session[:logged_in].nil?
return false
end
if session[:logged_in] == :false
return false
end
return true
end
def login_required
if logged_in?()
return
end
pp "Access Denied"
access_denied()
end
# Redirect as appropriate when an access request fails.
# Our application only uses ntlm based login - you must be part of the domain
def access_denied
redirect_to('/ntlm/login')
end
# Store the URI of the current request in the session.
#
# We can return to this location by calling #redirect_back_or_default.
def store_location
session[:return_to] = request.request_uri
end
# Redirect to the URI stored by the most recent store_location call or
# to the passed default.
def redirect_back_or_default(default)
session[:return_to] ? redirect_to(session[:return_to]) : redirect_to(default)
session[:return_to] = nil
end
end
Finally in your application.rb
# Filters added to this controller apply to all controllers in the application.
# Likewise, all the methods added will be available for all controllers.
class ApplicationController < ActionController::Base
helper :all # include all helpers, all the time
include NtlmSystem # Add this for NTLM to work
# See ActionController::RequestForgeryProtection for details
# Uncomment the :secret if you're not using the cookie session store
protect_from_forgery :secret => '6e2f000403020a0d9041a7bfb069239d'
# See ActionController::Base for details
# Uncomment this to filter the contents of submitted sensitive data parameters
# from your application log (in this case, all fields with names like "password").
filter_parameter_logging :password
end
When you start your application, make sure you do it like this:
mongrel_rails start -S config\mongrel_ntlm.conf
After reading, on NTLM, and playing with it, I found the URL are a bit picky, if you
want to use the app, without needing to key your username and password
http://servername:3000/ work perfectly fine.
If you put the whole name, it tends to ask you the user name and password
Monday, December 08, 2008
Simple Rails Goodness - Annotate_Model
Ever have trouble remembering the fields in your rails models?
Now you dont have too.
This is really great time saver. Used to I'd keep up a SQL query windows
just to be able to look at fields. Now with this plugin I dont have too.
gem install annotate-models
Will get you the gem version.
Usage:
cd [your project]
annotate
And you will get something like:
# == Schema Information
#
# id :integer(11) not null
# quantity :integer(11)
# product_id :integer(11)
# unit_price :float
# order_id :integer(11)
#
class LineItem < ActiveRecord::Base
belongs_to :product
end
This is so Useful when your writing model code
http://github.com/ctran/annotate_models/tree/master
Now you dont have too.
This is really great time saver. Used to I'd keep up a SQL query windows
just to be able to look at fields. Now with this plugin I dont have too.
gem install annotate-models
Will get you the gem version.
Usage:
cd [your project]
annotate
And you will get something like:
# == Schema Information
#
# id :integer(11) not null
# quantity :integer(11)
# product_id :integer(11)
# unit_price :float
# order_id :integer(11)
#
class LineItem < ActiveRecord::Base
belongs_to :product
end
This is so Useful when your writing model code
http://github.com/ctran/annotate_models/tree/master
Sunday, November 23, 2008
RPG really still exists
I was looking thru some surveys and surprised that in hometown of Cleveland/Chattanoga TN, one of the more common environments is AS/400
or iSeries midrange machines. And the language of choice is RPG.
I've done a few projects in my early years with RPG, find it incredible that
its still alive and kicking.
The nice thing is now, there is ruby support on AS/400(iSeries) as well as
database support.
Using this, new applications can be developed rapidly. They can be viewed from standard pc, or mac, inside or if you choose outside the company. One of the
the truely powerful elements is creating Iphone applications, that can be access in a mobile environment. Using the DB2 adapter, exiting applications can be enhanced
or new functionality added, without changing the core infrastruture.
http://www.ibmsystemsmag.com/i5/july07/tipstechniques/16330p1.aspx
or iSeries midrange machines. And the language of choice is RPG.
I've done a few projects in my early years with RPG, find it incredible that
its still alive and kicking.
The nice thing is now, there is ruby support on AS/400(iSeries) as well as
database support.
Using this, new applications can be developed rapidly. They can be viewed from standard pc, or mac, inside or if you choose outside the company. One of the
the truely powerful elements is creating Iphone applications, that can be access in a mobile environment. Using the DB2 adapter, exiting applications can be enhanced
or new functionality added, without changing the core infrastruture.
http://www.ibmsystemsmag.com/i5/july07/tipstechniques/16330p1.aspx
Monday, November 03, 2008
Why Use Ruby? Why Not Java? Why Not C?
Its a common question, that is always ask when you start a project. What
tool to use. Having long experience, with all languages practically, I've programmed
in multiple assemblers, Basic, Pascal, Cobol, Fortran, RPG, VB, C, C++, Java Python, VHDL, Veriflog, and Ruby.
I agree with the comment that CPU get cheaper, and people get more expensive.
Also timelines are decreasing. Market timeframes get smaller. Also for me, as a
senior manager of technology, I have no time for the likes of C. (Yes I need
to go back and do a project just to keep the skill, that is coming soon)
But if I want a functioning prototype system, and I need it in a day, even
at my best I cannot do it in C. I can do it in Ruby ( on Rails).
Then I can go in the meeting with my programming staff, and tell them, ok this
is how I expect it to work. At that point, little room for discussion. With Ruby
you can do a system in the time it takes to do the powerpoint to describe it. Thats
a major statement as I eat and breath powerpoint.
The other thing that people know ruby for is perfomance, but to a large degree that is just startup time. With the new implementations using VM based technology there is no reason ruby cannot run on a 8 bit processor, and now we have 64bit ones almost for free.
So next time you need to fix 20 year old code, consider using Ruby, some nice plugin and gems, and a few days of your time. Its a lot better than spending 5-10 man years in java.
Reference:
http://engineroom.mixx.com/2008/08/10/ruby-on-rails-vs-java-vs-c-vs-assembler/
tool to use. Having long experience, with all languages practically, I've programmed
in multiple assemblers, Basic, Pascal, Cobol, Fortran, RPG, VB, C, C++, Java Python, VHDL, Veriflog, and Ruby.
I agree with the comment that CPU get cheaper, and people get more expensive.
Also timelines are decreasing. Market timeframes get smaller. Also for me, as a
senior manager of technology, I have no time for the likes of C. (Yes I need
to go back and do a project just to keep the skill, that is coming soon)
But if I want a functioning prototype system, and I need it in a day, even
at my best I cannot do it in C. I can do it in Ruby ( on Rails).
Then I can go in the meeting with my programming staff, and tell them, ok this
is how I expect it to work. At that point, little room for discussion. With Ruby
you can do a system in the time it takes to do the powerpoint to describe it. Thats
a major statement as I eat and breath powerpoint.
The other thing that people know ruby for is perfomance, but to a large degree that is just startup time. With the new implementations using VM based technology there is no reason ruby cannot run on a 8 bit processor, and now we have 64bit ones almost for free.
So next time you need to fix 20 year old code, consider using Ruby, some nice plugin and gems, and a few days of your time. Its a lot better than spending 5-10 man years in java.
Reference:
http://engineroom.mixx.com/2008/08/10/ruby-on-rails-vs-java-vs-c-vs-assembler/
Sunday, November 02, 2008
Solving Large Scale Company Software Issues
What to do when your dealing with complex enterprise issues,
with multiple legacy systems?
If you look at most large companies today, you find a long organic history
of IT software and hardware.
The hardware side of things is easy enough to fix over time. I find that
DELL has fastastic warranty, and great service, and moderate price. Get a
five year warranty, and forget about hardware problems for your servers and
pc's. Likewise on your network gear, Cisco does a great job.
That solves infrasturure, but not the complexity of software.
Some have current investment in java, some even in cobol still.
I could write a 100 page essay on the wide variety of systems in
large companies.
Going forward, what you want is the ability to integrate and have a standard
across the enterpise. As CTO and Technology managers, we need something that
will tie everything togeather. Something that will last, and has good performance
and that we will not have to spend 100Million on just to get thing consistent.
For this "glue" consider using Ruby on Rails. Its mature, It fast, and its efficient.
It can scale from 1 User to Millions of users. And it can be developmed in small
to large teams.
Add to this a development center in the phillipines for even more cost effective solution. The cost of english speaking staff in Manilla is 1/3 that of the US.
The people are smart, and hard workers. With good video conferencing, and
VOIP telephony they can be virtually down the hall from you. And my experience
they can work on US timezone allowing for no time lag.
with multiple legacy systems?
If you look at most large companies today, you find a long organic history
of IT software and hardware.
The hardware side of things is easy enough to fix over time. I find that
DELL has fastastic warranty, and great service, and moderate price. Get a
five year warranty, and forget about hardware problems for your servers and
pc's. Likewise on your network gear, Cisco does a great job.
That solves infrasturure, but not the complexity of software.
Some have current investment in java, some even in cobol still.
I could write a 100 page essay on the wide variety of systems in
large companies.
Going forward, what you want is the ability to integrate and have a standard
across the enterpise. As CTO and Technology managers, we need something that
will tie everything togeather. Something that will last, and has good performance
and that we will not have to spend 100Million on just to get thing consistent.
For this "glue" consider using Ruby on Rails. Its mature, It fast, and its efficient.
It can scale from 1 User to Millions of users. And it can be developmed in small
to large teams.
Add to this a development center in the phillipines for even more cost effective solution. The cost of english speaking staff in Manilla is 1/3 that of the US.
The people are smart, and hard workers. With good video conferencing, and
VOIP telephony they can be virtually down the hall from you. And my experience
they can work on US timezone allowing for no time lag.
Monday, September 22, 2008
Be Gentle Admins(Gods) - Be slow to judge
I recently tried the site getacoder.com The site offers contractors/programmer/engineers for work-for-hire, at good rates.
Thats the good news. The bad news, is there administration is heavy
handed to say the least.
I posted a request, and having done lots of outsourcing gave as much
information as I could, even siting the a open-source reference board.
Seems like the terms of service specify that you cannot provide contact info,
which I'm perfectly fine with. But the Open Source project has contact info for there project. And within a hour of posting, my account was suspended, for violation of terms-of-service.
I've done no violation of terms of service. As my contact details were not included.
-----------------------------------------------
Dear Glenn,
Your account was suspended due to the fact that you have provided contact information in a project page; Please be reminded that this is not allowed by GetACoder.
Statements like: \"For any questions, concerns, or issues submit them to gerald@BeagleBoard.org\"- Will not be tolerated.
If you wish to reactivate your account send us an email, informing that you understand our Terms of Use agreeing to abide by them at all times.
--------------------------------------------------
Funny dont you think that the person knows my name is Glenn, yet she suspended my
account for a unrelated person called gerald.
I've withheld her name to protect the guilty.
Its now been more than 16 hours, account suspended, with no response.
So just to make sure.
I Am Informing Getacoder.com that I AGREE TO THE TERMS OF SERVICE, HAVE NOT, NOR WILL EVER VIOLATE THEM IN THE FUTURE.
Finally I get a response:
---------------------------------------------------------------------------------
Dear Glenn,
Glad to know that you are aware of the situation and plan to follow a professional conduct with GetACoder. We have reopened your account.
Feel free to contact us if you have further questions.
NOTE: If you wish to send additional information regarding this ticket, please do not create a new email. Instead, reply to this one and do not change subject line.
Sincerely,
Anna Kesel
---------------------------------------------------------------------------------
I really dont know about these people. The quality of responses is 4 so far, 2 of
which were proposing web designs for a hardware design, ie they dont have a clue
of what is being talked about. Waiting on the other two to "bid" and see what the price is. Since this is a hobby project, cannot be so high.
Thats the good news. The bad news, is there administration is heavy
handed to say the least.
I posted a request, and having done lots of outsourcing gave as much
information as I could, even siting the a open-source reference board.
Seems like the terms of service specify that you cannot provide contact info,
which I'm perfectly fine with. But the Open Source project has contact info for there project. And within a hour of posting, my account was suspended, for violation of terms-of-service.
I've done no violation of terms of service. As my contact details were not included.
-----------------------------------------------
Dear Glenn,
Your account was suspended due to the fact that you have provided contact information in a project page; Please be reminded that this is not allowed by GetACoder.
Statements like: \"For any questions, concerns, or issues submit them to gerald@BeagleBoard.org\"- Will not be tolerated.
If you wish to reactivate your account send us an email, informing that you understand our Terms of Use agreeing to abide by them at all times.
--------------------------------------------------
Funny dont you think that the person knows my name is Glenn, yet she suspended my
account for a unrelated person called gerald.
I've withheld her name to protect the guilty.
Its now been more than 16 hours, account suspended, with no response.
So just to make sure.
I Am Informing Getacoder.com that I AGREE TO THE TERMS OF SERVICE, HAVE NOT, NOR WILL EVER VIOLATE THEM IN THE FUTURE.
Finally I get a response:
---------------------------------------------------------------------------------
Dear Glenn,
Glad to know that you are aware of the situation and plan to follow a professional conduct with GetACoder. We have reopened your account.
Feel free to contact us if you have further questions.
NOTE: If you wish to send additional information regarding this ticket, please do not create a new email. Instead, reply to this one and do not change subject line.
Sincerely,
Anna Kesel
---------------------------------------------------------------------------------
I really dont know about these people. The quality of responses is 4 so far, 2 of
which were proposing web designs for a hardware design, ie they dont have a clue
of what is being talked about. Waiting on the other two to "bid" and see what the price is. Since this is a hobby project, cannot be so high.
Sunday, September 14, 2008
Making Polywell Magnets Using Ceramics and Liquid Metals
One of my areas of interest is fusion. Since tradition tokamak fusion devices dont seem to have a future. I tend torwards the Inertial Confinement Fusion Camp, with
Polywell based technology being the area I believe would work the best with the lowest cost.
One of the cost issues of polywell that needs to be solved from engineering point of view is the magnet design. In polywell you use a sperical array of magnets to create a virtual grid.
The magnets needless to say need to create a strong magnetic field. So for continious operation you really need cooling. Water does not work because its going to flash to steam way to easily.
The concept I had was to use EoPlex Technologies (http://www.eoplex.com/index.html) to create a ceramic layered magnetic structure combining the magnet and a integrated channel for ibm's liquid metal as well as the copper magnetic wire. The magnets mounted in the reator chanber would connect to a liquid metal circulation system, carrying the liquid metal to a large heat exchanger capable of 1600 degree's C down to 85 degree C, as stated the IBM article.
Since the liquid metal has magnetic properties we may even be able to get rid of the copper conducter and have a liquid conductor. Need a bit more research, but believe this may offer the ability to produce 100Megawatt plants without the need to have cryo cooling.
http://www.nextenergynews.com/news1/next-energy-news5.16.08d.html
http://www.eoplex.com/index.html
http://www.talk-polywell.org/bb/index.php
http://en.wikipedia.org/wiki/Polywell
Polywell based technology being the area I believe would work the best with the lowest cost.
One of the cost issues of polywell that needs to be solved from engineering point of view is the magnet design. In polywell you use a sperical array of magnets to create a virtual grid.
The magnets needless to say need to create a strong magnetic field. So for continious operation you really need cooling. Water does not work because its going to flash to steam way to easily.
The concept I had was to use EoPlex Technologies (http://www.eoplex.com/index.html) to create a ceramic layered magnetic structure combining the magnet and a integrated channel for ibm's liquid metal as well as the copper magnetic wire. The magnets mounted in the reator chanber would connect to a liquid metal circulation system, carrying the liquid metal to a large heat exchanger capable of 1600 degree's C down to 85 degree C, as stated the IBM article.
Since the liquid metal has magnetic properties we may even be able to get rid of the copper conducter and have a liquid conductor. Need a bit more research, but believe this may offer the ability to produce 100Megawatt plants without the need to have cryo cooling.
http://www.nextenergynews.com/news1/next-energy-news5.16.08d.html
http://www.eoplex.com/index.html
http://www.talk-polywell.org/bb/index.php
http://en.wikipedia.org/wiki/Polywell
Tuesday, September 09, 2008
Questons Questions Questions - A meme inspired by David Mohundro
How old were you when you started programming?
15 Years old.
How did you get started in programming?
After following many articles in Byte Magazine. (Where I learned electronics
from the wondering writing of Steve Ciarcia (http://www.circellar.com/)
I saved my pennies and got me the first TRS80 PC from Radio Shack.
I did lots of small apps, like programs to teach me primary school spelling. Speach Output. Light Pen Input.
Then started doing commercial projects around 16 years old
What was your first language?
BASIC (Microsoft Basic)
Z80 Assembler
What was the first real program that you wrote?
Real Program, means I got paid for it.
First paying job was a mail-merge program for the TRS80 Word Processor
Followed by a automation system for a beer disti.
What languages have you used since you started programming?
Cobol
Fortran
RPG
C
TurboPascal
8086 Assembler
68K Assembler
8031/8051 Assembler
VB
C++
Java
Python
Ruby
If you knew what you know now, would you have started programming?
Absolutely. But I'd skip down to ruby.
What's the most fun you've ever had... programming?
Several that come to mind.
1. LightPen App for the TRS80.
2. Banking Analysis APP - Took it from a month of run-time to a weekend.
3. PDFtoRuby - A Ruby Gem to create ruby code from pdf files
4. Network Monitor APP I did recently in Ruby/Rails
I can’t think of any one specific instance, but I think one of the best feelings is, after having spent literally hours trying to debug some problem and then giving up and going home, waking up the next day and having the light bulb come on with the solution to this problem. I love solving problems with software.
Tag, you’re it!
15 Years old.
How did you get started in programming?
After following many articles in Byte Magazine. (Where I learned electronics
from the wondering writing of Steve Ciarcia (http://www.circellar.com/)
I saved my pennies and got me the first TRS80 PC from Radio Shack.
I did lots of small apps, like programs to teach me primary school spelling. Speach Output. Light Pen Input.
Then started doing commercial projects around 16 years old
What was your first language?
BASIC (Microsoft Basic)
Z80 Assembler
What was the first real program that you wrote?
Real Program, means I got paid for it.
First paying job was a mail-merge program for the TRS80 Word Processor
Followed by a automation system for a beer disti.
What languages have you used since you started programming?
Cobol
Fortran
RPG
C
TurboPascal
8086 Assembler
68K Assembler
8031/8051 Assembler
VB
C++
Java
Python
Ruby
If you knew what you know now, would you have started programming?
Absolutely. But I'd skip down to ruby.
What's the most fun you've ever had... programming?
Several that come to mind.
1. LightPen App for the TRS80.
2. Banking Analysis APP - Took it from a month of run-time to a weekend.
3. PDFtoRuby - A Ruby Gem to create ruby code from pdf files
4. Network Monitor APP I did recently in Ruby/Rails
I can’t think of any one specific instance, but I think one of the best feelings is, after having spent literally hours trying to debug some problem and then giving up and going home, waking up the next day and having the light bulb come on with the solution to this problem. I love solving problems with software.
Tag, you’re it!
Thursday, August 21, 2008
Ruby and Rails can do almost anything
I'd like to expose a few things that Rails is not just for Web Applications.
By definitions of traditional web applications I mean, your traditional shopping cart, or social networking sites.
Applications I've done with Rails
Network Management - This maps all nodes in a network, grabs there macs, find what ports they are on switches, gets there service tag, and even for phones find outs there extensions. Yes you use a IE or Firefox to access it, but its not a social networking site
Regional Aviation Maint. Mangament Applicaton
This manages the overhaul process for a company that repair aircraft components and does the regulatory paperwork. Its a very controlled environment. Again not your normal shopping cart
A Proof-of-concept Data Deduplication Backup system
This uses ActiveRecord and sqlite3 to do the tracking part of the backup. And Rails to be a GUI for the server.
Embedded Device Managment.
In my hobby, I'm building a 360 degree camera.
All the device management, and the gui code is done using ruby and
rails, running on a tiny avr32 cpu.
Database Conversion(s) - More than I care to count
Thing I could do with Rails
Any Commecial Program that you can think of. From Cash Registers in food courts, to Factory Management applications
So you see, Rails is not just about yet-another-web-site. Its a GUI for any application you want to write.
WIth Ruby maning the heavy-lifting, and Rails handling the GUI, there very few things that cannot be done with the combination.
By definitions of traditional web applications I mean, your traditional shopping cart, or social networking sites.
Applications I've done with Rails
Network Management - This maps all nodes in a network, grabs there macs, find what ports they are on switches, gets there service tag, and even for phones find outs there extensions. Yes you use a IE or Firefox to access it, but its not a social networking site
Regional Aviation Maint. Mangament Applicaton
This manages the overhaul process for a company that repair aircraft components and does the regulatory paperwork. Its a very controlled environment. Again not your normal shopping cart
A Proof-of-concept Data Deduplication Backup system
This uses ActiveRecord and sqlite3 to do the tracking part of the backup. And Rails to be a GUI for the server.
Embedded Device Managment.
In my hobby, I'm building a 360 degree camera.
All the device management, and the gui code is done using ruby and
rails, running on a tiny avr32 cpu.
Database Conversion(s) - More than I care to count
Thing I could do with Rails
Any Commecial Program that you can think of. From Cash Registers in food courts, to Factory Management applications
So you see, Rails is not just about yet-another-web-site. Its a GUI for any application you want to write.
WIth Ruby maning the heavy-lifting, and Rails handling the GUI, there very few things that cannot be done with the combination.
dotr - drawing network maps
Ever want to draw a network map.
Seems like there is a "great" way to do this.
The ruby 'dotr" library. It can be used for diagraming code, to diagramming
the whole internet.
DotR provides a simple Ruby interface for contructing directed graphs in a natural syntax and plotting them via the 'dot' utility to PNG, PS and other graphic formats.
the ruby dotr drives Graphviz
application done by ATT. Its open source, and you can "write" dot files using
the dotr gem.
I call the SwitchGraph in the "background" of a "rails" app, generating "maps" of all my network switches. In my rails app, they appear to "update" by themselves.
I use Rmagick to create smaller thumbnails, that can be "clicked" to a large version.
Example of driving dotr/graphviz
#switchgraph.rb
require 'tempfile'
require 'dotr'
require 'RMagick'
include Magick
class SwitchGraph
def initialize(theswitch)
@me = theswitch
end #init
def graph()
d = DotR::Digraph.new(@me.name,:rankdir=>"LR",:ranksep=>"2.75") do |g|
switchname = @me.name
switchlabel = @me.site.name + "/" + @me.name
g.node(switchname, :label => switchlabel)
@me.switch_ports.each do |port|
port.macs.each do |mac|
if Device.exists?(mac.device_id)
begin
nodename = mac.device.name
nodelabel = mac.device.name + "(" + mac.ip + ")"
rescue
nodename = "-Unknown-"
nodelabel = "(" + mac.ip + ")"
end
g.node(nodename,:label => nodelabel) do |n|
portlabel = port.name
n.connection(switchname, :label => portlabel)
if mac.device.user
username = mac.device.user.name
userlabel = username
g.node(username, :label => userlabel) do |u|
u.connection(nodename)
end # do u
end #mac.device.user
end #g.node mac
end #if mac.device
end #port.macs.each
end #@me.switch_ports
end #DotR
#C:\projects\airstate\public\switch\graph\4
thebase = 'public/switch/graph/' + @me.id.to_s + "/"
thefilename = thebase + @me.name + ".png"
thumbdir = thebase + "thumb"
FileUtils.mkdir_p thumbdir
File.open(thefilename, 'wb') { |f| f.write(d.diagram(:png)) }
img = Magick::Image::read(thefilename).first
thumb = img.resize(250,250)
thumb.write thumbdir + '/' + @me.name + ".png"
end
end #class switchgraph
Seems like there is a "great" way to do this.
The ruby 'dotr" library. It can be used for diagraming code, to diagramming
the whole internet.
DotR provides a simple Ruby interface for contructing directed graphs in a natural syntax and plotting them via the 'dot' utility to PNG, PS and other graphic formats.
the ruby dotr drives Graphviz
application done by ATT. Its open source, and you can "write" dot files using
the dotr gem.
I call the SwitchGraph in the "background" of a "rails" app, generating "maps" of all my network switches. In my rails app, they appear to "update" by themselves.
I use Rmagick to create smaller thumbnails, that can be "clicked" to a large version.
Example of driving dotr/graphviz
#switchgraph.rb
require 'tempfile'
require 'dotr'
require 'RMagick'
include Magick
class SwitchGraph
def initialize(theswitch)
@me = theswitch
end #init
def graph()
d = DotR::Digraph.new(@me.name,:rankdir=>"LR",:ranksep=>"2.75") do |g|
switchname = @me.name
switchlabel = @me.site.name + "/" + @me.name
g.node(switchname, :label => switchlabel)
@me.switch_ports.each do |port|
port.macs.each do |mac|
if Device.exists?(mac.device_id)
begin
nodename = mac.device.name
nodelabel = mac.device.name + "(" + mac.ip + ")"
rescue
nodename = "-Unknown-"
nodelabel = "(" + mac.ip + ")"
end
g.node(nodename,:label => nodelabel) do |n|
portlabel = port.name
n.connection(switchname, :label => portlabel)
if mac.device.user
username = mac.device.user.name
userlabel = username
g.node(username, :label => userlabel) do |u|
u.connection(nodename)
end # do u
end #mac.device.user
end #g.node mac
end #if mac.device
end #port.macs.each
end #@me.switch_ports
end #DotR
#C:\projects\airstate\public\switch\graph\4
thebase = 'public/switch/graph/' + @me.id.to_s + "/"
thefilename = thebase + @me.name + ".png"
thumbdir = thebase + "thumb"
FileUtils.mkdir_p thumbdir
File.open(thefilename, 'wb') { |f| f.write(d.diagram(:png)) }
img = Magick::Image::read(thefilename).first
thumb = img.resize(250,250)
thumb.write thumbdir + '/' + @me.name + ".png"
end
end #class switchgraph
Scraping Web Pages with Soup
I wanted to "grab" the contents of a "management" web page off a cisco 7941 phone.
After reading up on the variety of ruby gems to use for scrapping.
My choice was "soup".
http://www.crummy.com/software/RubyfulSoup/
This seems to work nice. I've managed to pull serial numbers, extensions, and
phone names as well as models from close to 100 phones this way.
This is a "class" for getting the phone information.
#cisco_phone.rb - Class to "Scrap" Cisco Phone
class CiscoPhone
require 'rubygems'
require 'open-uri'
require 'rubyful_soup'
require 'cgi'
require "pp"
def initialize(theip)
valuestack = Array.new
@values = Hash.new
url = "http://#{theip}"
begin
open(url) {
|page| page_content = page.read()
soup = BeautifulSoup.new(page_content)
result = soup.find_all('td')
valuename = nil
result.each do |item|
content= item.b
if content
value = content.string
if value
value.strip!
if !valuestack.include?(value)
valuestack.push(value)
end
end
end
end
}
rescue
@values = nil
return nil
end
thename = nil
thevalue = nil
valuestack.each do |content|
if thename.nil?
thename = content
else
thevalue = content
@values[thename] = thevalue
thename = nil
end
end
# Validations
if !@values.has_key?('Model Number')
@values = nil
return nil
end
if !@values.has_key?('Phone DN')
@values = nil
return nil
end
return self
end
def isphone?()
if @values.nil?
return false
else
return true
end
end
def getvalue(thename)
if !@values.include?(thename)
return nil
end
return @values[thename].downcase
end
end
#Test Cases
#thetest = CiscoPhone.new("192.168.1.20")
#pp thetest
After reading up on the variety of ruby gems to use for scrapping.
My choice was "soup".
http://www.crummy.com/software/RubyfulSoup/
This seems to work nice. I've managed to pull serial numbers, extensions, and
phone names as well as models from close to 100 phones this way.
This is a "class" for getting the phone information.
#cisco_phone.rb - Class to "Scrap" Cisco Phone
class CiscoPhone
require 'rubygems'
require 'open-uri'
require 'rubyful_soup'
require 'cgi'
require "pp"
def initialize(theip)
valuestack = Array.new
@values = Hash.new
url = "http://#{theip}"
begin
open(url) {
|page| page_content = page.read()
soup = BeautifulSoup.new(page_content)
result = soup.find_all('td')
valuename = nil
result.each do |item|
content= item.b
if content
value = content.string
if value
value.strip!
if !valuestack.include?(value)
valuestack.push(value)
end
end
end
end
}
rescue
@values = nil
return nil
end
thename = nil
thevalue = nil
valuestack.each do |content|
if thename.nil?
thename = content
else
thevalue = content
@values[thename] = thevalue
thename = nil
end
end
# Validations
if !@values.has_key?('Model Number')
@values = nil
return nil
end
if !@values.has_key?('Phone DN')
@values = nil
return nil
end
return self
end
def isphone?()
if @values.nil?
return false
else
return true
end
end
def getvalue(thename)
if !@values.include?(thename)
return nil
end
return @values[thename].downcase
end
end
#Test Cases
#thetest = CiscoPhone.new("192.168.1.20")
#pp thetest
Data Deplucation In Ruby - Backup Terabytes In Minutes
Recently I went looking at the "perpetual" problem of "backups".
I wanted to "know" how some of the commerical products, do there "magic".
So I wrote a "simulation", in ruby, with activerecord for my data storage.
The theory of this, is your "average" server/workstation has "lots" of duplicated files. They may have "different" names, and be in "different" directories, but the
contents is the same.
This is even more the case spread over multiple machines, or for machines that have been around for a while.
I created the following script to "explore" dedup backup.
First, terminology explained:
1. Chunk is a "unique" file, as indentified by a "hash" - In my case a SHA-512 hash.
2. Backup is a "typical" full "backup" of everything on my workstation.
Logic Flow:
1. Loop Thru All Available Files.
2. Compute a SHA-512 for the current file
3. Look up the hash, do we have a "chunk" that exists.
4. If we do, "connect" the file record to the chunk.
5. else create a new chunk, and link the file record
The first run, without "compression" we get a 15% saving
If you run this a second time, the second backup will go be reduced from 37+ gig
to 200Meg or less. Resulting in a deduplication ration of 99%+
The nice "tricks" that can be done with this, is to reduce your "backups" from days to "minutes", since the "normal" rotation of "removable" backups already have "data" that we can delta.
In addition we could keep 2 years of backup at a cheap price.
And a slew of other features.
Stats of the "backup" actually done on my workstation.
Chunks 382899
Backups 1
Backup Files 510960
Total Backup Size 40031553419 = 37.28G
Total Chunk Size 33727083825 = 31.41G
Savings 6304469594 = 5.87G = 15% for first backup
Second Backup would have a savings of 99%
Now the code:
First we need a "place" to track our progress.
#migration
class CreateChunks < ActiveRecord::Migration
def self.up
create_table "backupchunks", :force => true do |t|
t.string :filehash
t.integer :filesize
t.string :ctype
t.datetime :created_at
end
create_table "backups", :force => true do |t|
t.string :host
t.string :description
t.datetime :created_at
end
create_table "backupfiles", :force => true do |t|
t.integer :backup_id
t.string :filename
t.integer :filesize
t.integer :backupchunk_id
end
end
def self.down
drop_table :backupchunks
drop_table :backups
drop_table :backupfiles
end
end
Now our "agent"
#scandisk.rb
#---------------
require 'digest'
require 'find'
require 'pp'
require "rubygems"
require "activerecord"
ActiveRecord::Base.establish_connection(
:adapter => "sqlite3",
:database => "db/development.sqlite3"
)
class Backupchunk < ActiveRecord::Base
has_many :Files
end
class Backup < ActiveRecord::Base
has_many :Files
end
class Backupfile < ActiveRecord::Base
belongs_to :Backupchunk
end
def hash_file(file, hash_kind)
digest, chunk = hash_kind.new, ''
open(file, 'rb') do |f|
digest << chunk while f.read(4096, chunk)
end
digest
end
def handlebackup(f)
begin
thehash = hash_file(f, Digest::SHA512).to_s
rescue
puts "Cannot read #{f}"
return
end
thechunk = Backupchunk.find(:first,:conditions => {:filehash => thehash})
if thechunk.nil?
thechunk = Backupchunk.new
thechunk.filehash = thehash
thechunk.filesize = File.size(f)
thechunk.save
else
pp f
end
thefile = Backupfile.new
thefile.filename = f
thefile.filesize = File.size(f)
thefile.backupchunk_id = thechunk.id
thefile.save
end
target_dir = "h:/dedup/store/"
mybackup = Backup.new
mybackup.host = "sinub-glenn2"
mybackup.save
Find.find('/') do |f|
if File.file?(f)
handlebackup(f)
end
end
I wanted to "know" how some of the commerical products, do there "magic".
So I wrote a "simulation", in ruby, with activerecord for my data storage.
The theory of this, is your "average" server/workstation has "lots" of duplicated files. They may have "different" names, and be in "different" directories, but the
contents is the same.
This is even more the case spread over multiple machines, or for machines that have been around for a while.
I created the following script to "explore" dedup backup.
First, terminology explained:
1. Chunk is a "unique" file, as indentified by a "hash" - In my case a SHA-512 hash.
2. Backup is a "typical" full "backup" of everything on my workstation.
Logic Flow:
1. Loop Thru All Available Files.
2. Compute a SHA-512 for the current file
3. Look up the hash, do we have a "chunk" that exists.
4. If we do, "connect" the file record to the chunk.
5. else create a new chunk, and link the file record
The first run, without "compression" we get a 15% saving
If you run this a second time, the second backup will go be reduced from 37+ gig
to 200Meg or less. Resulting in a deduplication ration of 99%+
The nice "tricks" that can be done with this, is to reduce your "backups" from days to "minutes", since the "normal" rotation of "removable" backups already have "data" that we can delta.
In addition we could keep 2 years of backup at a cheap price.
And a slew of other features.
Stats of the "backup" actually done on my workstation.
Chunks 382899
Backups 1
Backup Files 510960
Total Backup Size 40031553419 = 37.28G
Total Chunk Size 33727083825 = 31.41G
Savings 6304469594 = 5.87G = 15% for first backup
Second Backup would have a savings of 99%
Now the code:
First we need a "place" to track our progress.
#migration
class CreateChunks < ActiveRecord::Migration
def self.up
create_table "backupchunks", :force => true do |t|
t.string :filehash
t.integer :filesize
t.string :ctype
t.datetime :created_at
end
create_table "backups", :force => true do |t|
t.string :host
t.string :description
t.datetime :created_at
end
create_table "backupfiles", :force => true do |t|
t.integer :backup_id
t.string :filename
t.integer :filesize
t.integer :backupchunk_id
end
end
def self.down
drop_table :backupchunks
drop_table :backups
drop_table :backupfiles
end
end
Now our "agent"
#scandisk.rb
#---------------
require 'digest'
require 'find'
require 'pp'
require "rubygems"
require "activerecord"
ActiveRecord::Base.establish_connection(
:adapter => "sqlite3",
:database => "db/development.sqlite3"
)
class Backupchunk < ActiveRecord::Base
has_many :Files
end
class Backup < ActiveRecord::Base
has_many :Files
end
class Backupfile < ActiveRecord::Base
belongs_to :Backupchunk
end
def hash_file(file, hash_kind)
digest, chunk = hash_kind.new, ''
open(file, 'rb') do |f|
digest << chunk while f.read(4096, chunk)
end
digest
end
def handlebackup(f)
begin
thehash = hash_file(f, Digest::SHA512).to_s
rescue
puts "Cannot read #{f}"
return
end
thechunk = Backupchunk.find(:first,:conditions => {:filehash => thehash})
if thechunk.nil?
thechunk = Backupchunk.new
thechunk.filehash = thehash
thechunk.filesize = File.size(f)
thechunk.save
else
pp f
end
thefile = Backupfile.new
thefile.filename = f
thefile.filesize = File.size(f)
thefile.backupchunk_id = thechunk.id
thefile.save
end
target_dir = "h:/dedup/store/"
mybackup = Backup.new
mybackup.host = "sinub-glenn2"
mybackup.save
Find.find('/') do |f|
if File.file?(f)
handlebackup(f)
end
end
Sunday, July 27, 2008
Corrupted Flash Card - Recovery of FAT32 Files
Recently, I was getting ready to copy some images from a "full" memory card, to my pc for archive. Somewhere during the process, the fat32 file system on the card, got confused, and the directory where the images were simply disappeared.
What to do now? I tried chkdsk? Nothing
I tried the "Trash" folder, Again nothing.
Over 540+ irreplaceable photos from my Nikon D70 were
gone.
So doing some research, I found many a "pay" solutions.
But not really wanting to pay, I wanted to find a "free"
solution, without running into a virus, spyware, etc.
After looking at three or four, I found my solution.
PhotoRec.
The interface is a bit primative, but it worked perfectly.
The first time. Every photo was recovered.
So if you hit that problem, give it a try.
Cost is $0.00. Very good perfomance for the price.
http://www.cgsecurity.org/wiki/PhotoRec
What to do now? I tried chkdsk? Nothing
I tried the "Trash" folder, Again nothing.
Over 540+ irreplaceable photos from my Nikon D70 were
gone.
So doing some research, I found many a "pay" solutions.
But not really wanting to pay, I wanted to find a "free"
solution, without running into a virus, spyware, etc.
After looking at three or four, I found my solution.
PhotoRec.
The interface is a bit primative, but it worked perfectly.
The first time. Every photo was recovered.
So if you hit that problem, give it a try.
Cost is $0.00. Very good perfomance for the price.
http://www.cgsecurity.org/wiki/PhotoRec
Thursday, July 10, 2008
Dynamic Tabnav Widget menus
I wanted to do a "dynamic" menu that is filled from data in my Rails Application.
I have a "table" (Model in rails speak) that has the values. Then I can "overload"
the controller to add more functionality.
The "gotcha" in coding this is, there is "no" each method for activerecord.
The other "gotcha" is a "site" can have mutliple networks, so since we are selecting my "site", and network does not have it, we do a lookup in the conditions_for_collections code. Note that active_scaffold can also "view" the devices via the site controller. But I find having "buttons" on the device controller a nice interface.
Using this technique, I can "dry" up the controller, and reuse it for several "purposes". No need to repeat the "controller".
Note that we have to "pass" the level3 status up thru the controller, and then back down thru the partials.
First the widgets for tabnav
File: app\views\widgets\_device_tier2_tabnav.rhtml
<%
level3 = nil
render_tabnav :device_tier2,
:generate_css => true do
add_tab do |t|
t.named 'All'
t.titled 'All'
t.links_to :controller => 'device'
end
add_tab do |t|
t.named 'Sites'
t.named 'Sites'
t.links_to :controller => 'device?level3=device_sites_tier3'
end
end
%>
File: app/views/widgets/_device_sites_tier3_tabnav.rhtml
<%
render_tabnav :device_sites_tier3,
:generate_css => true do
Site.find(:all).each do |thesite|
add_tab do |t|
t.named thesite.name
t.links_to :controller => "device?site=#{thesite.id}&level3=device_sites_tier3"
end
end
end
%>
File app\views\layouts\tier2device.rhtml
<html>
<%= javascript_include_tag :defaults %>
<%= active_scaffold_includes %>
<%= stylesheet_link_tag "/javascripts/jscalendar-1.0/calendar-win2k-cold-1.css" %>
<%= javascript_include_tag "jscalendar-1.0/calendar.js" %>
<%= javascript_include_tag "jscalendar-1.0/lang/calendar-en.js" %>
<%= javascript_include_tag "jscalendar-1.0/calendar-setup.js" %>
<head>
<title>AppName</title>
</head>
<body>
<br>
<%= tabnav :airstate %>
<%= tabnav :device_tier2 %>
<% if @level3 %>
<%= tabnav @level3 %>
<% end %>
<%= @content_for_layout %>
</body>
</html>
The controller, which is using activescaffold looks like this:
class DeviceController < ApplicationController
layout 'tier2device'
active_scaffold :device do |config|
config.actions = [:delete,:show, :list, :nested, :search]
config.list.columns = [:network, :name, :serialnum, :model, :macs, :purchase_date, :warranty_end]
config.columns[:macs].ui_type = :select
config.columns[:network].ui_type = :select
config.columns[:user].ui_type = :select
end
def conditions_for_collection
@level3 = nil
if params.include?("level3")
@level3 = params.delete("level3")
end
if params.include?("site")
mysite = params.delete("site")
mynets = Network.find(:all,:conditions => "site_id = #{mysite}")
mywhere = ['network_id IN (?)', mynets]
end
return mywhere
end
end
References - No .each in active record
http://weblog.jamisbuck.org/2007/4/6/faking-cursors-in-activerecord
I have a "table" (Model in rails speak) that has the values. Then I can "overload"
the controller to add more functionality.
The "gotcha" in coding this is, there is "no" each method for activerecord.
The other "gotcha" is a "site" can have mutliple networks, so since we are selecting my "site", and network does not have it, we do a lookup in the conditions_for_collections code. Note that active_scaffold can also "view" the devices via the site controller. But I find having "buttons" on the device controller a nice interface.
Using this technique, I can "dry" up the controller, and reuse it for several "purposes". No need to repeat the "controller".
Note that we have to "pass" the level3 status up thru the controller, and then back down thru the partials.
First the widgets for tabnav
File: app\views\widgets\_device_tier2_tabnav.rhtml
<%
level3 = nil
render_tabnav :device_tier2,
:generate_css => true do
add_tab do |t|
t.named 'All'
t.titled 'All'
t.links_to :controller => 'device'
end
add_tab do |t|
t.named 'Sites'
t.named 'Sites'
t.links_to :controller => 'device?level3=device_sites_tier3'
end
end
%>
File: app/views/widgets/_device_sites_tier3_tabnav.rhtml
<%
render_tabnav :device_sites_tier3,
:generate_css => true do
Site.find(:all).each do |thesite|
add_tab do |t|
t.named thesite.name
t.links_to :controller => "device?site=#{thesite.id}&level3=device_sites_tier3"
end
end
end
%>
File app\views\layouts\tier2device.rhtml
<html>
<%= javascript_include_tag :defaults %>
<%= active_scaffold_includes %>
<%= stylesheet_link_tag "/javascripts/jscalendar-1.0/calendar-win2k-cold-1.css" %>
<%= javascript_include_tag "jscalendar-1.0/calendar.js" %>
<%= javascript_include_tag "jscalendar-1.0/lang/calendar-en.js" %>
<%= javascript_include_tag "jscalendar-1.0/calendar-setup.js" %>
<head>
<title>AppName</title>
</head>
<body>
<br>
<%= tabnav :airstate %>
<%= tabnav :device_tier2 %>
<% if @level3 %>
<%= tabnav @level3 %>
<% end %>
<%= @content_for_layout %>
</body>
</html>
The controller, which is using activescaffold looks like this:
class DeviceController < ApplicationController
layout 'tier2device'
active_scaffold :device do |config|
config.actions = [:delete,:show, :list, :nested, :search]
config.list.columns = [:network, :name, :serialnum, :model, :macs, :purchase_date, :warranty_end]
config.columns[:macs].ui_type = :select
config.columns[:network].ui_type = :select
config.columns[:user].ui_type = :select
end
def conditions_for_collection
@level3 = nil
if params.include?("level3")
@level3 = params.delete("level3")
end
if params.include?("site")
mysite = params.delete("site")
mynets = Network.find(:all,:conditions => "site_id = #{mysite}")
mywhere = ['network_id IN (?)', mynets]
end
return mywhere
end
end
References - No .each in active record
http://weblog.jamisbuck.org/2007/4/6/faking-cursors-in-activerecord
Sunday, July 06, 2008
Reading the Service Tag or Serial Number in Ruby
I want to "do" a inventory of "all" my equipment. (Yes It will eventually be a Rails app), but the first issue is gettig all the "service" tags, and updating them on the fly. If you search, you'll find plenty of examples in vb or python, but not very many. (Read none) in ruby. So I decided to see if I could do it.
First I liked the ruby-wmi package, that gives a activerecord flavor to wmi. WMI is windows standard "Instrumentation" library, where you can find out all too many facts about a windows system.
Next a little "code" of my own to show "how" to do it. Implemented as a class on top of ruby-wmi, with some basic "caching" to make the overhead smaller.
require 'ruby-wmi'
require 'pp'
# Reference URL's
# http://www.vbforums.com/showthread.php?t=326425
# http://ruby-wmi.rubyforge.org/doc/classes/WMI/Base.html
# http://mentalpagingspace.blogspot.com/2008/01/ruby-class-tutorial-using-unify-dbi.html
class ManagePC
def initialize(ipaddress)
@myip = ipaddress
@biosv = WMI::Win32_BIOS.find(:all,:host=>@myip)
@csysv = WMI::Win32_ComputerSystem.find(:all,:host=>@myip)
@netv = WMI::Win32_NetworkAdapterConfiguration.find(:all,
:conditions => 'IPEnabled = True')
return
end #initialize
def prtnetvalues()
@netv.each do |netx|
netx.properties_.each do |p|
value = netx[p.name]
if value
if value.class == Fixnum
value = value.to_s
end
if value.class == Array
value = value.to_s
end
if value.class == TrueClass
if value
value = "True"
else
value = "False"
end
end
theprop = p.name + "="
theprop << value
puts theprop
end
end
end
end # prtnetvalues()
def prtcsysvalues()
@csysv.each do |csys|
csys.properties_.each do |p|
printf "#{p.name} = #{csys[p.name]}\n"
end
end
end # prtcsysvalues()
def netvalue(name)
@netv.each do |netx|
netx.properties_.each do |p|
if p.name == name
return netx[name]
end
end
end
return nil
end
def csysvalue(name)
@csysv.each do |csys|
csys.properties_.each do |p|
if p.name == name
return csys[name]
end
end
end
return nil
end
def macs()
mymacs = Array.new
current_mac = nil
@netv.each do |netx|
netx.properties_.each do |p|
if p.name == "IPAddress"
current_mac << netx[p.name].to_s
end
if p.name == "MACAddress"
current_mac << netx[p.name]
end
if p.name == "Caption" # A new network interface
if current_mac
mymacs << current_mac
end
current_mac = Array.new
end
end
end
mymacs << current_mac
return(mymacs)
end
def domain()
return csysvalue("Domain")
end
def model()
return csysvalue("Model")
end
def name()
return csysvalue("Name")
end
def domain?()
return csysvalue("PartOfDomain")
end
def username()
return csysvalue("UserName")
end
def memory()
return csysvalue("TotalPhysicalMemory")
end
def servicetag()
@biosv.each do |bios|
bios.properties_.each do |p|
if p.name == "SerialNumber"
return(bios[p.name])
end
end
end
return(nill)
end #GetServiceTag
end #ManagePC
mypc = ManagePC.new("127.0.0.1")
puts mypc.servicetag
puts mypc.domain
puts mypc.model
puts mypc.name
puts mypc.username
puts mypc.memory
pp mypc.macs
First I liked the ruby-wmi package, that gives a activerecord flavor to wmi. WMI is windows standard "Instrumentation" library, where you can find out all too many facts about a windows system.
Next a little "code" of my own to show "how" to do it. Implemented as a class on top of ruby-wmi, with some basic "caching" to make the overhead smaller.
require 'ruby-wmi'
require 'pp'
# Reference URL's
# http://www.vbforums.com/showthread.php?t=326425
# http://ruby-wmi.rubyforge.org/doc/classes/WMI/Base.html
# http://mentalpagingspace.blogspot.com/2008/01/ruby-class-tutorial-using-unify-dbi.html
class ManagePC
def initialize(ipaddress)
@myip = ipaddress
@biosv = WMI::Win32_BIOS.find(:all,:host=>@myip)
@csysv = WMI::Win32_ComputerSystem.find(:all,:host=>@myip)
@netv = WMI::Win32_NetworkAdapterConfiguration.find(:all,
:conditions => 'IPEnabled = True')
return
end #initialize
def prtnetvalues()
@netv.each do |netx|
netx.properties_.each do |p|
value = netx[p.name]
if value
if value.class == Fixnum
value = value.to_s
end
if value.class == Array
value = value.to_s
end
if value.class == TrueClass
if value
value = "True"
else
value = "False"
end
end
theprop = p.name + "="
theprop << value
puts theprop
end
end
end
end # prtnetvalues()
def prtcsysvalues()
@csysv.each do |csys|
csys.properties_.each do |p|
printf "#{p.name} = #{csys[p.name]}\n"
end
end
end # prtcsysvalues()
def netvalue(name)
@netv.each do |netx|
netx.properties_.each do |p|
if p.name == name
return netx[name]
end
end
end
return nil
end
def csysvalue(name)
@csysv.each do |csys|
csys.properties_.each do |p|
if p.name == name
return csys[name]
end
end
end
return nil
end
def macs()
mymacs = Array.new
current_mac = nil
@netv.each do |netx|
netx.properties_.each do |p|
if p.name == "IPAddress"
current_mac << netx[p.name].to_s
end
if p.name == "MACAddress"
current_mac << netx[p.name]
end
if p.name == "Caption" # A new network interface
if current_mac
mymacs << current_mac
end
current_mac = Array.new
end
end
end
mymacs << current_mac
return(mymacs)
end
def domain()
return csysvalue("Domain")
end
def model()
return csysvalue("Model")
end
def name()
return csysvalue("Name")
end
def domain?()
return csysvalue("PartOfDomain")
end
def username()
return csysvalue("UserName")
end
def memory()
return csysvalue("TotalPhysicalMemory")
end
def servicetag()
@biosv.each do |bios|
bios.properties_.each do |p|
if p.name == "SerialNumber"
return(bios[p.name])
end
end
end
return(nill)
end #GetServiceTag
end #ManagePC
mypc = ManagePC.new("127.0.0.1")
puts mypc.servicetag
puts mypc.domain
puts mypc.model
puts mypc.name
puts mypc.username
puts mypc.memory
pp mypc.macs
Tuesday, July 01, 2008
Annoucing PDFTORUBY Support Forum
PDFtoRuby now has a support "forum" hosted via Google Groups.
If your having issues or problems with PDFtoRuby, or its "just" working
please feel free to drop a message on the board.
I'll be glad to help "resolve" any issues you have
http://groups.google.com/group/pdftoruby?hl=en
If your having issues or problems with PDFtoRuby, or its "just" working
please feel free to drop a message on the board.
I'll be glad to help "resolve" any issues you have
http://groups.google.com/group/pdftoruby?hl=en
Wednesday, June 25, 2008
Rapid easy loading of CSV data in a Rails migration
Well, it had to happen sometimes.
I was sent some nice "huge" database tables, that
had been exported as "csv" files. So new there
were several "ways" of handling this in ruby,
so I wanted to do it in a migration, and since the
tables have "lots" of columns, didnt want to have to "write" them
out.
I use both "fastercsv" in combination with ar_extensions
I could not find a "blog" article that had a "simple" migration using
the combination.
Not that I load "ar_extensions" in my enviornment.rb
Also note that the table is "created" in another migration.
# 099_load_myrecipes.rb
require 'fastercsv'
class LoadRecipes < ActiveRecord::Migration
def self.up
mycnames = Recipes.columns.map{ |column| column.name }
mycnames.delete("id")
value_sets = FasterCSV.read("#{RAILS_ROOT}/mdata/Recipes.csv", :col_sep => ",",
:row_sep => "\n")
options = { :validate => false }
Recipes.import( mycnames, value_sets, options)
end
def self.down
Recipes.delete_all()
end
end
Resources:
FasterCSV Doc - http://fastercsv.rubyforge.org/
FasterCSV and arext article - http://www.jobwd.com/article/show/31
AR Extention - http://www.continuousthinking.com/tags/arext
I was sent some nice "huge" database tables, that
had been exported as "csv" files. So new there
were several "ways" of handling this in ruby,
so I wanted to do it in a migration, and since the
tables have "lots" of columns, didnt want to have to "write" them
out.
I use both "fastercsv" in combination with ar_extensions
I could not find a "blog" article that had a "simple" migration using
the combination.
Not that I load "ar_extensions" in my enviornment.rb
Also note that the table is "created" in another migration.
# 099_load_myrecipes.rb
require 'fastercsv'
class LoadRecipes < ActiveRecord::Migration
def self.up
mycnames = Recipes.columns.map{ |column| column.name }
mycnames.delete("id")
value_sets = FasterCSV.read("#{RAILS_ROOT}/mdata/Recipes.csv", :col_sep => ",",
:row_sep => "\n")
options = { :validate => false }
Recipes.import( mycnames, value_sets, options)
end
def self.down
Recipes.delete_all()
end
end
Resources:
FasterCSV Doc - http://fastercsv.rubyforge.org/
FasterCSV and arext article - http://www.jobwd.com/article/show/31
AR Extention - http://www.continuousthinking.com/tags/arext
Tuesday, June 24, 2008
Great new ruby "tool" - Free Message queue
Wow, now this is something that is easy to use, and works out of the "can".
gem install fmq
Gets you started. This is a "message" queue system. It lets you implement a "fire" and forget type messaging system. (In my old Stratus or VOS days, we called them "server" queues.)
So you set up a "server", which in this can is a "mongrel" rails application, that manages multiple "server" queues. The communications method is "http", which means it will go thru firewalls easy. Your "client", which could be a pc, or a "ruby" capable device then can "fire" work to be done at the server, such as request a report, or perform some "action".
For example, in "device" management, it could be to "get" outstanding commands, or "update" status. The work is "queued" and processed as quick as possible.
The nice thing about fmq that I can see is the "dependencies" are "tiny" to "none". There's a cute little demo/management system that lets you create "queues", and fire messages, and receive messages.
In doing a lot of "missing" critical applications I find a good message system is critical. I've even coded my own "system" to extend "server/message" queues to a pc.
This looks "simple" and "good".
So I'll give this a spin in a "real" project, and see how it flies.
http://fmq.rubyforge.org
gem install fmq
Gets you started. This is a "message" queue system. It lets you implement a "fire" and forget type messaging system. (In my old Stratus or VOS days, we called them "server" queues.)
So you set up a "server", which in this can is a "mongrel" rails application, that manages multiple "server" queues. The communications method is "http", which means it will go thru firewalls easy. Your "client", which could be a pc, or a "ruby" capable device then can "fire" work to be done at the server, such as request a report, or perform some "action".
For example, in "device" management, it could be to "get" outstanding commands, or "update" status. The work is "queued" and processed as quick as possible.
The nice thing about fmq that I can see is the "dependencies" are "tiny" to "none". There's a cute little demo/management system that lets you create "queues", and fire messages, and receive messages.
In doing a lot of "missing" critical applications I find a good message system is critical. I've even coded my own "system" to extend "server/message" queues to a pc.
This looks "simple" and "good".
So I'll give this a spin in a "real" project, and see how it flies.
http://fmq.rubyforge.org
Tuesday, May 20, 2008
Annoucing pdftoruby
Did you ever need to "modify" pdf files, or "recreate" them with
fields changed or added. Do you ever want to generate nice "invoices"
or other "forms" documents, but dont want to break out the ruler to
"recreate" the form.
Now there's a tool that can take a pdf form, and convert it to ruby code.
This "unifies" pdf::reader and pdf::writer libraries.
pdftoruby generates ruby code from a supplied pdf.
You can then "edit" the ruby code to add/delete/modfiy and run the code,
it will generate the pdf file using pdf::writer.
I've used the result code in a rails production application for months now, without issue.
http://code.google.com/p/pdftoruby/
fields changed or added. Do you ever want to generate nice "invoices"
or other "forms" documents, but dont want to break out the ruler to
"recreate" the form.
Now there's a tool that can take a pdf form, and convert it to ruby code.
This "unifies" pdf::reader and pdf::writer libraries.
pdftoruby generates ruby code from a supplied pdf.
You can then "edit" the ruby code to add/delete/modfiy and run the code,
it will generate the pdf file using pdf::writer.
I've used the result code in a rails production application for months now, without issue.
http://code.google.com/p/pdftoruby/
Creating PDF::Writer Compatible PNG Files
I've fought PNG for a while. Scanning google countless times to
figure out how to get a PNG to write, in as few as lines of possible.
So my final choice is to use a "clone" of png gem, with a few changes.
One, make sure to write in "binary", so your file opens in the library must
be in a "b" option. The default install of PNG, does not have the "b" option,
which means it will work perfectly fine on linux/unix, and not at all in windows.
The second problem is the "color" space type. The default is 6, which means there
is a transparency byte per pixel. This of course does not work with PDF::writer.
So My version uses "2", which drops the transparency byte, and writes 3 bytes.
The later versions of png gem, switch to using inline, which require a compiler to work. Since it makes installing pdftoruby a pain, I wanted something simple.
class String # :nodoc:
##
# Calculates a CRC using the algorithm in the PNG specification.
def png_crc
unless defined? @@crc then
@@crc = Array.new(256)
256.times do |n|
c = n
8.times do
c = (c & 1 == 1) ? 0xedb88320 ^ (c >> 1) : c >> 1
end
@@crc[n] = c
end
end
c = 0xffffffff
each_byte do |b|
c = @@crc[(c^b) & 0xff] ^ (c >> 8)
end
return c ^ 0xffffffff
end
end
## gsw - Make sure we have version 1 - as version 2 will require major havoc - also fix the "binary" issue
# A pure Ruby Portable Network Graphics (PNG) writer.
#
# http://www.libpng.org/pub/png/spec/1.2/
#
# PNG supports:
# + 8 bit truecolor PNGs
#
# PNG does not support:
# + any other color depth
# + extra data chunks
# + filters
#
# = Example
#
# require 'png'
#
# canvas = PNG::Canvas.new 200, 200
# canvas[100, 100] = PNG::Color::Black
# canvas.line 50, 50, 100, 50, PNG::Color::Blue
# png = PNG.new canvas
# png.save 'blah.png'
class PNG
##
# Creates a PNG chunk of type +type+ that contains +data+.
def self.chunk(type, data="")
[data.size, type, data, (type + data).png_crc].pack("Na*a*N")
end
##
# Creates a new PNG object using +canvas+
def initialize(canvas)
@height = canvas.height
@width = canvas.width
@bits = 8
@data = canvas.data
end
##
# Writes the PNG to +path+.
def save(path)
File.open(path, "wb") do |f|
f.write [137, 80, 78, 71, 13, 10, 26, 10].pack("C*") # PNG signature
f.write PNG.chunk('IHDR',
[ @height, @width, @bits, 2, 0, 0, 0 ].pack("N2C5"))
# 0 == filter type code "none"
data = @data.map { |row| [0] + row.map { |p| [p.values[0], p.values[1], p.values[2]] } }.flatten
#data = @data.map { |row| [0] + row.map { |p| p.values } }.flatten
f.write PNG.chunk('IDAT', Zlib::Deflate.deflate(data.pack("C*"), 9))
f.write PNG.chunk('IEND', '')
end
end
##
# RGBA colors
class Color
attr_reader :values
##
# Creates a new color with values +red+, +green+, +blue+, and +alpha+.
def initialize(red, green, blue, alpha)
@values = [red, green, blue, alpha]
end
##
# Transparent white
Background = Color.new 0xFF, 0xFF, 0xFF, 0x00
White = Color.new 0xFF, 0xFF, 0xFF, 0xFF
Black = Color.new 0x00, 0x00, 0x00, 0xFF
Gray = Color.new 0x7F, 0x7F, 0x7F, 0xFF
Red = Color.new 0xFF, 0x00, 0x00, 0xFF
Orange = Color.new 0xFF, 0xA5, 0x00, 0xFF
Yellow = Color.new 0xFF, 0xFF, 0x00, 0xFF
Green = Color.new 0x00, 0xFF, 0x00, 0xFF
Blue = Color.new 0x00, 0x00, 0xFF, 0xFF
Purple = Color.new 0XFF, 0x00, 0xFF, 0xFF
##
# Red component
def r; @values[0]; end
##
# Green component
def g; @values[1]; end
##
# Blue component
def b; @values[2]; end
##
# Alpha transparency component
def a; @values[3]; end
##
# Blends +color+ into this color returning a new blended color.
def blend(color)
return Color.new((r * (0xFF - color.a) + color.r * color.a) >> 8,
(g * (0xFF - color.a) + color.g * color.a) >> 8,
(b * (0xFF - color.a) + color.b * color.a) >> 8,
(a * (0xFF - color.a) + color.a * color.a) >> 8)
end
##
# Returns a new color with an alpha value adjusted by +i+.
def intensity(i)
return Color.new(r,b,g,(a*i) >> 8)
end
def inspect # :nodoc:
"#<%s %02x %02x %02x %02x>" % [self.class, *@values]
end
end
##
# PNG canvas
class Canvas
##
# Height of the canvas
attr_reader :height
##
# Width of the canvas
attr_reader :width
##
# Raw data
attr_reader :data
def initialize(height, width, background = Color::White)
@height = height
@width = width
@data = Array.new(@width) { |x| Array.new(@height) { background } }
end
##
# Retrieves the color of the pixel at (+x+, +y+).
def [](x, y)
raise "bad x value #{x} >= #{@height}" if x >= @height
raise "bad y value #{y} >= #{@width}" if y >= @width
@data[y][x]
end
##
# Sets the color of the pixel at (+x+, +y+) to +color+.
def []=(x, y, color)
raise "bad x value #{x} >= #{@height}" if x >= @height
raise "bad y value #{y} >= #{@width}" if y >= @width
@data[y][x] = color
end
##
# Iterates over each pixel in the canvas.
def each
@data.each_with_index do |row, y|
row.each_with_index do |pixel, x|
yield x, y, color
end
end
end
##
# Blends +color+ onto the color at point (+x+, +y+).
def point(x, y, color)
self[x,y] = self[x,y].blend(color)
end
##
# Draws a line using Xiaolin Wu's antialiasing technique.
#
# http://en.wikipedia.org/wiki/Xiaolin_Wu's_line_algorithm
def line(x0, y0, x1, y1, color)
dx = x1 - x0
sx = dx < 0 ? -1 : 1
dx *= sx # TODO: abs?
dy = y1 - y0
# 'easy' cases
if dy == 0 then
Range.new(*[x0,x1].sort).each do |x|
point(x, y0, color)
end
return
end
if dx == 0 then
(y0..y1).each do |y|
point(x0, y, color)
end
return
end
if dx == dy then
Range.new(*[x0,x1].sort).each do |x|
point(x, y0, color)
y0 += 1
end
return
end
# main loop
point(x0, y0, color)
e_acc = 0
if dy > dx then # vertical displacement
e = (dx << 16) / dy
(y0...y1-1).each do |i|
e_acc_temp, e_acc = e_acc, (e_acc + e) & 0xFFFF
x0 = x0 + sx if (e_acc <= e_acc_temp)
w = 0xFF-(e_acc >> 8)
point(x0, y0, color.intensity(w))
y0 = y0 + 1
point(x0 + sx, y0, color.intensity(0xFF-w))
end
point(x1, y1, color)
return
end
# horizontal displacement
e = (dy << 16) / dx
(x0...(x1-sx)).each do |i|
e_acc_temp, e_acc = e_acc, (e_acc + e) & 0xFFFF
y0 += 1 if (e_acc <= e_acc_temp)
w = 0xFF-(e_acc >> 8)
point(x0, y0, color.intensity(w))
x0 += sx
point(x0, y0 + 1, color.intensity(0xFF-w))
end
point(x1, y1, color)
end
end
end
figure out how to get a PNG to write, in as few as lines of possible.
So my final choice is to use a "clone" of png gem, with a few changes.
One, make sure to write in "binary", so your file opens in the library must
be in a "b" option. The default install of PNG, does not have the "b" option,
which means it will work perfectly fine on linux/unix, and not at all in windows.
The second problem is the "color" space type. The default is 6, which means there
is a transparency byte per pixel. This of course does not work with PDF::writer.
So My version uses "2", which drops the transparency byte, and writes 3 bytes.
The later versions of png gem, switch to using inline, which require a compiler to work. Since it makes installing pdftoruby a pain, I wanted something simple.
class String # :nodoc:
##
# Calculates a CRC using the algorithm in the PNG specification.
def png_crc
unless defined? @@crc then
@@crc = Array.new(256)
256.times do |n|
c = n
8.times do
c = (c & 1 == 1) ? 0xedb88320 ^ (c >> 1) : c >> 1
end
@@crc[n] = c
end
end
c = 0xffffffff
each_byte do |b|
c = @@crc[(c^b) & 0xff] ^ (c >> 8)
end
return c ^ 0xffffffff
end
end
## gsw - Make sure we have version 1 - as version 2 will require major havoc - also fix the "binary" issue
# A pure Ruby Portable Network Graphics (PNG) writer.
#
# http://www.libpng.org/pub/png/spec/1.2/
#
# PNG supports:
# + 8 bit truecolor PNGs
#
# PNG does not support:
# + any other color depth
# + extra data chunks
# + filters
#
# = Example
#
# require 'png'
#
# canvas = PNG::Canvas.new 200, 200
# canvas[100, 100] = PNG::Color::Black
# canvas.line 50, 50, 100, 50, PNG::Color::Blue
# png = PNG.new canvas
# png.save 'blah.png'
class PNG
##
# Creates a PNG chunk of type +type+ that contains +data+.
def self.chunk(type, data="")
[data.size, type, data, (type + data).png_crc].pack("Na*a*N")
end
##
# Creates a new PNG object using +canvas+
def initialize(canvas)
@height = canvas.height
@width = canvas.width
@bits = 8
@data = canvas.data
end
##
# Writes the PNG to +path+.
def save(path)
File.open(path, "wb") do |f|
f.write [137, 80, 78, 71, 13, 10, 26, 10].pack("C*") # PNG signature
f.write PNG.chunk('IHDR',
[ @height, @width, @bits, 2, 0, 0, 0 ].pack("N2C5"))
# 0 == filter type code "none"
data = @data.map { |row| [0] + row.map { |p| [p.values[0], p.values[1], p.values[2]] } }.flatten
#data = @data.map { |row| [0] + row.map { |p| p.values } }.flatten
f.write PNG.chunk('IDAT', Zlib::Deflate.deflate(data.pack("C*"), 9))
f.write PNG.chunk('IEND', '')
end
end
##
# RGBA colors
class Color
attr_reader :values
##
# Creates a new color with values +red+, +green+, +blue+, and +alpha+.
def initialize(red, green, blue, alpha)
@values = [red, green, blue, alpha]
end
##
# Transparent white
Background = Color.new 0xFF, 0xFF, 0xFF, 0x00
White = Color.new 0xFF, 0xFF, 0xFF, 0xFF
Black = Color.new 0x00, 0x00, 0x00, 0xFF
Gray = Color.new 0x7F, 0x7F, 0x7F, 0xFF
Red = Color.new 0xFF, 0x00, 0x00, 0xFF
Orange = Color.new 0xFF, 0xA5, 0x00, 0xFF
Yellow = Color.new 0xFF, 0xFF, 0x00, 0xFF
Green = Color.new 0x00, 0xFF, 0x00, 0xFF
Blue = Color.new 0x00, 0x00, 0xFF, 0xFF
Purple = Color.new 0XFF, 0x00, 0xFF, 0xFF
##
# Red component
def r; @values[0]; end
##
# Green component
def g; @values[1]; end
##
# Blue component
def b; @values[2]; end
##
# Alpha transparency component
def a; @values[3]; end
##
# Blends +color+ into this color returning a new blended color.
def blend(color)
return Color.new((r * (0xFF - color.a) + color.r * color.a) >> 8,
(g * (0xFF - color.a) + color.g * color.a) >> 8,
(b * (0xFF - color.a) + color.b * color.a) >> 8,
(a * (0xFF - color.a) + color.a * color.a) >> 8)
end
##
# Returns a new color with an alpha value adjusted by +i+.
def intensity(i)
return Color.new(r,b,g,(a*i) >> 8)
end
def inspect # :nodoc:
"#<%s %02x %02x %02x %02x>" % [self.class, *@values]
end
end
##
# PNG canvas
class Canvas
##
# Height of the canvas
attr_reader :height
##
# Width of the canvas
attr_reader :width
##
# Raw data
attr_reader :data
def initialize(height, width, background = Color::White)
@height = height
@width = width
@data = Array.new(@width) { |x| Array.new(@height) { background } }
end
##
# Retrieves the color of the pixel at (+x+, +y+).
def [](x, y)
raise "bad x value #{x} >= #{@height}" if x >= @height
raise "bad y value #{y} >= #{@width}" if y >= @width
@data[y][x]
end
##
# Sets the color of the pixel at (+x+, +y+) to +color+.
def []=(x, y, color)
raise "bad x value #{x} >= #{@height}" if x >= @height
raise "bad y value #{y} >= #{@width}" if y >= @width
@data[y][x] = color
end
##
# Iterates over each pixel in the canvas.
def each
@data.each_with_index do |row, y|
row.each_with_index do |pixel, x|
yield x, y, color
end
end
end
##
# Blends +color+ onto the color at point (+x+, +y+).
def point(x, y, color)
self[x,y] = self[x,y].blend(color)
end
##
# Draws a line using Xiaolin Wu's antialiasing technique.
#
# http://en.wikipedia.org/wiki/Xiaolin_Wu's_line_algorithm
def line(x0, y0, x1, y1, color)
dx = x1 - x0
sx = dx < 0 ? -1 : 1
dx *= sx # TODO: abs?
dy = y1 - y0
# 'easy' cases
if dy == 0 then
Range.new(*[x0,x1].sort).each do |x|
point(x, y0, color)
end
return
end
if dx == 0 then
(y0..y1).each do |y|
point(x0, y, color)
end
return
end
if dx == dy then
Range.new(*[x0,x1].sort).each do |x|
point(x, y0, color)
y0 += 1
end
return
end
# main loop
point(x0, y0, color)
e_acc = 0
if dy > dx then # vertical displacement
e = (dx << 16) / dy
(y0...y1-1).each do |i|
e_acc_temp, e_acc = e_acc, (e_acc + e) & 0xFFFF
x0 = x0 + sx if (e_acc <= e_acc_temp)
w = 0xFF-(e_acc >> 8)
point(x0, y0, color.intensity(w))
y0 = y0 + 1
point(x0 + sx, y0, color.intensity(0xFF-w))
end
point(x1, y1, color)
return
end
# horizontal displacement
e = (dy << 16) / dx
(x0...(x1-sx)).each do |i|
e_acc_temp, e_acc = e_acc, (e_acc + e) & 0xFFFF
y0 += 1 if (e_acc <= e_acc_temp)
w = 0xFF-(e_acc >> 8)
point(x0, y0, color.intensity(w))
x0 += sx
point(x0, y0 + 1, color.intensity(0xFF-w))
end
point(x1, y1, color)
end
end
end
Wednesday, April 23, 2008
Ruby Png Gem - Getting Rid of Inline
Hmm, I know I grew up cutting teeth on C, but the "overhead" of getting
decent development environment, and "syncing" your compiler version
to that of who-ever wrote the code originally gets to be fun.
Its even worse when your going cross platform. (Yes I now prefer to
do my coding in ruby whenever possible).
I was reading thru ruby group on goole, about how to speed up the "png"
gem without using inline. I'm going to give that a try. Then I can use
the resolve in my pdftoruby tool.
Should be interesting, will update the article after I finish.
decent development environment, and "syncing" your compiler version
to that of who-ever wrote the code originally gets to be fun.
Its even worse when your going cross platform. (Yes I now prefer to
do my coding in ruby whenever possible).
I was reading thru ruby group on goole, about how to speed up the "png"
gem without using inline. I'm going to give that a try. Then I can use
the resolve in my pdftoruby tool.
Should be interesting, will update the article after I finish.
Geting Ruby png1.0.0 to work in windows
I need to be able to save "png" files out of my pdftoruby application.
My big problem was to take a "rgb" file, and write it, without using something
heave like ImageMagik. (Which seems like everyone complains about)
So my first choice was "png", but the latest version of png is not "pure" ruby.
Version 1.0.0 is "pure". The reason this is important, is that with the 1.1.0
release of png gem, you have to use "inline" and that then requires you to
use the "latest" compiler.
I've installed, and reinstalled png several times, but it seems to never work,
then finally I found a "mention" in a old ruby-talk message thread that on windows
you need the "b" open on the open. (A following comment said it had been patched).
Well decicded to go look at the source, and sure enough, its not got the "b" option
on the open.
So do the following:
gem install --version 1.0.0 png
Then go to the png directory
cd \ruby\lib\ruby\gems\1.8\gems\png-1.0.0\lib
And edit png.rb
vi png.rb # Use the editor of your choice
def save(path)
File.open(path, "wb") do |f| #Must add the "b" next to the "w"
#here for windows to work
f.write [137, 80, 78, 71, 13, 10, 26, 10].pack("C*") # PNG signature
f.write PNG.chunk('IHDR',
[ @height, @width, @bits, 6, 0, 0, 0 ].pack("N2C5"))
# 0 == filter type code "none"
data = @data.map { |row| [0] + row.map { |p| p.values } }.flatten
f.write PNG.chunk('IDAT', Zlib::Deflate.deflate(data.pack("C*"), 9))
f.write PNG.chunk('IEND', '')
end
end
Now png examples will work, and I can get back to business.
(I've seen some nice pure ruby optizations in the threads, will implemented those in pdfto ruby)
My big problem was to take a "rgb" file, and write it, without using something
heave like ImageMagik. (Which seems like everyone complains about)
So my first choice was "png", but the latest version of png is not "pure" ruby.
Version 1.0.0 is "pure". The reason this is important, is that with the 1.1.0
release of png gem, you have to use "inline" and that then requires you to
use the "latest" compiler.
I've installed, and reinstalled png several times, but it seems to never work,
then finally I found a "mention" in a old ruby-talk message thread that on windows
you need the "b" open on the open. (A following comment said it had been patched).
Well decicded to go look at the source, and sure enough, its not got the "b" option
on the open.
So do the following:
gem install --version 1.0.0 png
Then go to the png directory
cd \ruby\lib\ruby\gems\1.8\gems\png-1.0.0\lib
And edit png.rb
vi png.rb # Use the editor of your choice
def save(path)
File.open(path, "wb") do |f| #Must add the "b" next to the "w"
#here for windows to work
f.write [137, 80, 78, 71, 13, 10, 26, 10].pack("C*") # PNG signature
f.write PNG.chunk('IHDR',
[ @height, @width, @bits, 6, 0, 0, 0 ].pack("N2C5"))
# 0 == filter type code "none"
data = @data.map { |row| [0] + row.map { |p| p.values } }.flatten
f.write PNG.chunk('IDAT', Zlib::Deflate.deflate(data.pack("C*"), 9))
f.write PNG.chunk('IEND', '')
end
end
Now png examples will work, and I can get back to business.
(I've seen some nice pure ruby optizations in the threads, will implemented those in pdfto ruby)
Sunday, March 30, 2008
The Evils of EBAY - Never never never use them
Well you know from the title I had a bad experience from ebay.]
Two years ago, I decided to "try" ebay by buying a lens.
The "seller" had a great reputation. (looks like there are tricks
to give you a false "reputation", if anyone know how, feel free to post them to a comment.)
So after reading the ad, the seller wanted a "money order" or a "cashier" check. Nevermind, I can do that. So I went to the bank, paid the fee, and sent off my check. I "have" a copy of the check laying in my library to prove it. I mailed it off to the individuals address, and waited and waited and after a month got nothing. THe person saying "he" never got the check. Needless to say I complained to ebay. I gave him "negative" reputation. He returned the favor.
So I havent tried to use ebay from that bad experience for two years. I did write in and ask for help. I always got "wonderfully" worded emails, but no action. Worse come to find out, after two years I want to "try" it again, and guess what, I still have the "-1" reputation from the "bad" seller. And cannot do a transaction.
So for me, I'me giving up ebay for life, and will tell everyone I know its a untrustworty company, that there not to be trusted. That you will lose your money, that you will get "sweet" email, when you complain from them, and nothing more.
Well looks like they (Ebay) will not change the post, but if you see the comment,
they gave a nice invitation.
So any lawyer in california interested in class action lawsuit against ebay?
I'm intersted in joining a class action suit for people abused, and mentally abused by them.
Two years ago, I decided to "try" ebay by buying a lens.
The "seller" had a great reputation. (looks like there are tricks
to give you a false "reputation", if anyone know how, feel free to post them to a comment.)
So after reading the ad, the seller wanted a "money order" or a "cashier" check. Nevermind, I can do that. So I went to the bank, paid the fee, and sent off my check. I "have" a copy of the check laying in my library to prove it. I mailed it off to the individuals address, and waited and waited and after a month got nothing. THe person saying "he" never got the check. Needless to say I complained to ebay. I gave him "negative" reputation. He returned the favor.
So I havent tried to use ebay from that bad experience for two years. I did write in and ask for help. I always got "wonderfully" worded emails, but no action. Worse come to find out, after two years I want to "try" it again, and guess what, I still have the "-1" reputation from the "bad" seller. And cannot do a transaction.
So for me, I'me giving up ebay for life, and will tell everyone I know its a untrustworty company, that there not to be trusted. That you will lose your money, that you will get "sweet" email, when you complain from them, and nothing more.
Well looks like they (Ebay) will not change the post, but if you see the comment,
they gave a nice invitation.
So any lawyer in california interested in class action lawsuit against ebay?
I'm intersted in joining a class action suit for people abused, and mentally abused by them.
Monday, March 24, 2008
ICC Profiles - Decoding Using BitStruct And Ruby
I wanted to be able to explore ICC Color Profiles.
Since the data is binary, I wanted to be able to define nice "Structures",
and see the data.
So I disovered "Bitstruct". Its a library that lets you define
binary data easily.
Example o fdecoding ICC Profiles:
require 'bit-struct'
require 'pp'
class IccHeader < BitStruct
unsigned :size, 32, "Profile Size"
char :type, 32, "CMM Type Signature"
octets :version, 32, "Profile Version Number"
char :pdclass, 32, "Profile/Device Class"
text :colorspace, 32, "Color Space of Data"
text :pconnspace, 32, "Profile Connection Space"
char :date, 96, "Creation Date"
text :sign, 32, "Profile Signature"
text :platform, 32, "Primary Platform"
text :flags, 32, "CMM Flags"
text :devmanuf, 32, "Device Manufacture"
text :devmodel, 32, "Device Model"
text :devattr, 64, "Device Attributes"
text :renderintent,32, "Rendering Intent"
char :xyz1, 32, "XYZ Number 1"
char :xyz2, 32, "XYZ Number 2"
char :xyz3, 32, "XYZ Number 3"
char :creator, 32, "Profile Creator"
text :id, 128, "Profile ID"
text :reserved, 224, "Reserved"
unsigned :tagcount, 32, "Tag Count"
end
class TagTableEntry < BitStruct
text :sign, 32, "Signature"
unsigned :offset, 32, "Offset"
unsigned :size, 32, "Size"
end
puts "Dump of ICC Profile"
pfile = File.open("profile","r")
icchdr = IccHeader.new(pfile.read(132))
pp icchdr
cnt = icchdr.tagcount
offset = 133
tags = Array.new
while cnt > 0
tags << TagTableEntry.new(pfile.read(12))
cnt -= 1
end
pp tags
Since the data is binary, I wanted to be able to define nice "Structures",
and see the data.
So I disovered "Bitstruct". Its a library that lets you define
binary data easily.
Example o fdecoding ICC Profiles:
require 'bit-struct'
require 'pp'
class IccHeader < BitStruct
unsigned :size, 32, "Profile Size"
char :type, 32, "CMM Type Signature"
octets :version, 32, "Profile Version Number"
char :pdclass, 32, "Profile/Device Class"
text :colorspace, 32, "Color Space of Data"
text :pconnspace, 32, "Profile Connection Space"
char :date, 96, "Creation Date"
text :sign, 32, "Profile Signature"
text :platform, 32, "Primary Platform"
text :flags, 32, "CMM Flags"
text :devmanuf, 32, "Device Manufacture"
text :devmodel, 32, "Device Model"
text :devattr, 64, "Device Attributes"
text :renderintent,32, "Rendering Intent"
char :xyz1, 32, "XYZ Number 1"
char :xyz2, 32, "XYZ Number 2"
char :xyz3, 32, "XYZ Number 3"
char :creator, 32, "Profile Creator"
text :id, 128, "Profile ID"
text :reserved, 224, "Reserved"
unsigned :tagcount, 32, "Tag Count"
end
class TagTableEntry < BitStruct
text :sign, 32, "Signature"
unsigned :offset, 32, "Offset"
unsigned :size, 32, "Size"
end
puts "Dump of ICC Profile"
pfile = File.open("profile","r")
icchdr = IccHeader.new(pfile.read(132))
pp icchdr
cnt = icchdr.tagcount
offset = 133
tags = Array.new
while cnt > 0
tags << TagTableEntry.new(pfile.read(12))
cnt -= 1
end
pp tags
Saturday, February 16, 2008
Rake Script to Load Seed Data or Update Existing Data
In the previous blog we descussed loading of seed data using rake/active record.
What was missing was examples.
So First a example of our "rake" file.
namespace :project do
desc 'parts_master_load'
task :parts_master_load => :environment do
PartsMaster.uoc :pn => '11-1111-11',
:description => 'Part description for part 11-111-11',
:application_code => ApplicationCode.foc('appcode1'),
:pn_type_code => PnTypeCode.foc('REF')
PartsMaster.uoc :pn => '22-2222-22',
:description => 'Part Description for part 22-222-22',
:application_code => ApplicationCode.foc('apcode2'),
:pn_type_code => PnTypeCode.foc('REF')
end
end
Rake has the advantage in that it runs with all of rails environment, and for us that means activerecord, so we can use our "models". Another advantage is if there is a error, if you have the rake trace option, you get exactly the line of "data" that caused the issue.
In the above example, there are "three" models that are directly mentioned.
One is the PartsMaster, Two is the ApplicationCode, and the Third is the PnTypeCode.
The "method" foc is one of my own, that means "find" or "create". Very handly for those "simple" tables that are used for "catagories".
So as a example of "foc" as used in the "application" code module.
class ApplicationCode < ActiveRecord::Base
set_table_name "APPLICATION_CODES"
defaults :application_code => "",
:cannot_modify => "F",
:converted => "F",
public
def self.foc(thevalue) #find_or_create
if thevalue.nil?
thevalue = "_"
end
if thevalue == ""
thevalue = "_"
end
therecord = find(:first,:conditions => "application_code = '#{thevalue}'")
if therecord
return therecord
end
therecord = self.new
therecord.application_code = thevalue
therecord.description = thevalue
therecord.save
return therecord
end
end
Here you can see the power of "foc". We actually use the "application" key to do a "lookup", on existing data. We can "also" cleanup any "issues" with the "incoming" data, to make it more "consistent". If the record "does" not exist, then the record is created. This also is good illustration of "default" values to make usage of the model "easier".
Now lets look at a "stippet" of the "update or create" or uoc method. For the main
table in the application, we need to update existing records, since the file has constraints, so we cannot "truncate" the table, and aways do a "create".
def self.uoc(options = {}) # update or create
thevalue = options[:pn] #fixme - Need to also look at manufactuer
record = find(:first,:conditions => "pn = '#{thevalue}'")
if record.nil?
record = new
end
record.attributes = options
record.save!
record
end
This illustrates a lovely "concept" in ruby of allowing us to use a hash, allowing us to take variable arguments in, and pass those variable arguments down. Code looks simple. I could code the same in "C" and it would be rather ugly.
In this code, the "new" will create a record, and establish defaults (The record has a couple hundred fields, so defaults is "not" small.) and then "update" it with what we pass. That allows the "create" rake script to be very small.
So there you have it. From now own, I'm going to use active record to do my data load and migration. It works very nice.
What was missing was examples.
So First a example of our "rake" file.
namespace :project do
desc 'parts_master_load'
task :parts_master_load => :environment do
PartsMaster.uoc :pn => '11-1111-11',
:description => 'Part description for part 11-111-11',
:application_code => ApplicationCode.foc('appcode1'),
:pn_type_code => PnTypeCode.foc('REF')
PartsMaster.uoc :pn => '22-2222-22',
:description => 'Part Description for part 22-222-22',
:application_code => ApplicationCode.foc('apcode2'),
:pn_type_code => PnTypeCode.foc('REF')
end
end
Rake has the advantage in that it runs with all of rails environment, and for us that means activerecord, so we can use our "models". Another advantage is if there is a error, if you have the rake trace option, you get exactly the line of "data" that caused the issue.
In the above example, there are "three" models that are directly mentioned.
One is the PartsMaster, Two is the ApplicationCode, and the Third is the PnTypeCode.
The "method" foc is one of my own, that means "find" or "create". Very handly for those "simple" tables that are used for "catagories".
So as a example of "foc" as used in the "application" code module.
class ApplicationCode < ActiveRecord::Base
set_table_name "APPLICATION_CODES"
defaults :application_code => "",
:cannot_modify => "F",
:converted => "F",
public
def self.foc(thevalue) #find_or_create
if thevalue.nil?
thevalue = "_"
end
if thevalue == ""
thevalue = "_"
end
therecord = find(:first,:conditions => "application_code = '#{thevalue}'")
if therecord
return therecord
end
therecord = self.new
therecord.application_code = thevalue
therecord.description = thevalue
therecord.save
return therecord
end
end
Here you can see the power of "foc". We actually use the "application" key to do a "lookup", on existing data. We can "also" cleanup any "issues" with the "incoming" data, to make it more "consistent". If the record "does" not exist, then the record is created. This also is good illustration of "default" values to make usage of the model "easier".
Now lets look at a "stippet" of the "update or create" or uoc method. For the main
table in the application, we need to update existing records, since the file has constraints, so we cannot "truncate" the table, and aways do a "create".
def self.uoc(options = {}) # update or create
thevalue = options[:pn] #fixme - Need to also look at manufactuer
record = find(:first,:conditions => "pn = '#{thevalue}'")
if record.nil?
record = new
end
record.attributes = options
record.save!
record
end
This illustrates a lovely "concept" in ruby of allowing us to use a hash, allowing us to take variable arguments in, and pass those variable arguments down. Code looks simple. I could code the same in "C" and it would be rather ugly.
In this code, the "new" will create a record, and establish defaults (The record has a couple hundred fields, so defaults is "not" small.) and then "update" it with what we pass. That allows the "create" rake script to be very small.
So there you have it. From now own, I'm going to use active record to do my data load and migration. It works very nice.
Friday, February 15, 2008
Integrating Legacy Applications with Rails
I'm current involved in a large software project which is taking a "legacy" applicaton to a "new" generation application. Of course for me, ruby is my "main" tool. So I'm doing the migration with ruby, and looking to use rails to "expand" the canned application with rails functionality.
So the issues are many, and countless.
1. Schema - While the database is oracle, the schema of the new system is "huge", 400+ tables is nothing to sneeze at. So first we need some good documentation of it. Does it come with the system? Nope, no schema documentation, so lets create some documentation for easy reference.
http://schemaspy.sourceforge.net/
Schemaspy will let us "create" a map of existing schema, so we can explore it, and get familiar with internal standards.
2. Walking around the data. Next we need to "explore" the data of the new system. For this in a oracle environment, Oracle Sql Developer is free, and lets us look at the data easily.
3. Write a "script" in ruby to create models automatically. Typing any normal of models, getting plurization right, and creating controllers is not for the faint of heart. I create a "gen" script that generates "missing" models, and controllers. (I use activescaffold plugin for handling the controller side, and models can be started without relationships if you wish. (If I do this a second time, I'll have the relationships also done by the script. Not much fun doing tables that have 20 relationships, where the keyname does not match the tablename.) (Now you know why you need schemaspy).
4. For importing your "existing data, assuming its not terabytes, I find that it works best in a "two" step process. In previous projects I would write a "dbi" ruby script taking one table to the "new" rails table. That works fine, but now I have destination tables that have "rich" relationships, writing dbi is no fun. ActiveRecord can save the day.
a. Step 1 - dbi process - Read existing table, and generate a rake script, calling active record "models"
Reference: http://railspikes.com/2008/2/1/loading-seed-data
b. Step 2 - Run your rake script
Reference:
http://www.slashdotdash.net/articles/2007/01/18/using-activerecord-outside-rails-part-ii
I find that this gives you the easy ability to handle the multi-tier adds
while keeping things simple.
When your creating your models, I find that often in "large" applicatons the tables from tradiitional applications can be "huge" number of fields. Of course the reality is that the majority of fields are always" default" values. To keep your create and update clean, use a "defalts" plugin. This will allow you to write you "create" or "create-and-update" call in your rake script to be "very" small, while all the flags and other values get set to the right values.
Reference:
http://agilewebdevelopment.com/plugins/activerecord_defaults
This makes my "rake" task incrediably small.
Finally consider in your models adding a method of either:
1. uoc - Update or Create - This will let you take a hash, and find a existing record and update it, or create a new one based on the "primary" key that is "application" specific. While my target application does have a numeric primary key, the "real" key is two other fields. using uoc allows me to do a find on the record and update it. (Yes I could "truncate" the table and always do creates, but remmeber its a complex app with a spiderweb of connections, updates work better)
2. foc - Find or create - This is great for those "simple" tables like unit of measure, or other tables that typically reasly are for "catorgories". It simply "finds" the right record or "create" a new one and returns it for the relationship your building.
This also makes the rake script re-usable. And the code in your modeles can also be used in "expansion" applications.
Next week I'll add a few examples.
So the issues are many, and countless.
1. Schema - While the database is oracle, the schema of the new system is "huge", 400+ tables is nothing to sneeze at. So first we need some good documentation of it. Does it come with the system? Nope, no schema documentation, so lets create some documentation for easy reference.
http://schemaspy.sourceforge.net/
Schemaspy will let us "create" a map of existing schema, so we can explore it, and get familiar with internal standards.
2. Walking around the data. Next we need to "explore" the data of the new system. For this in a oracle environment, Oracle Sql Developer is free, and lets us look at the data easily.
3. Write a "script" in ruby to create models automatically. Typing any normal of models, getting plurization right, and creating controllers is not for the faint of heart. I create a "gen" script that generates "missing" models, and controllers. (I use activescaffold plugin for handling the controller side, and models can be started without relationships if you wish. (If I do this a second time, I'll have the relationships also done by the script. Not much fun doing tables that have 20 relationships, where the keyname does not match the tablename.) (Now you know why you need schemaspy).
4. For importing your "existing data, assuming its not terabytes, I find that it works best in a "two" step process. In previous projects I would write a "dbi" ruby script taking one table to the "new" rails table. That works fine, but now I have destination tables that have "rich" relationships, writing dbi is no fun. ActiveRecord can save the day.
a. Step 1 - dbi process - Read existing table, and generate a rake script, calling active record "models"
Reference: http://railspikes.com/2008/2/1/loading-seed-data
b. Step 2 - Run your rake script
Reference:
http://www.slashdotdash.net/articles/2007/01/18/using-activerecord-outside-rails-part-ii
I find that this gives you the easy ability to handle the multi-tier adds
while keeping things simple.
When your creating your models, I find that often in "large" applicatons the tables from tradiitional applications can be "huge" number of fields. Of course the reality is that the majority of fields are always" default" values. To keep your create and update clean, use a "defalts" plugin. This will allow you to write you "create" or "create-and-update" call in your rake script to be "very" small, while all the flags and other values get set to the right values.
Reference:
http://agilewebdevelopment.com/plugins/activerecord_defaults
This makes my "rake" task incrediably small.
Finally consider in your models adding a method of either:
1. uoc - Update or Create - This will let you take a hash, and find a existing record and update it, or create a new one based on the "primary" key that is "application" specific. While my target application does have a numeric primary key, the "real" key is two other fields. using uoc allows me to do a find on the record and update it. (Yes I could "truncate" the table and always do creates, but remmeber its a complex app with a spiderweb of connections, updates work better)
2. foc - Find or create - This is great for those "simple" tables like unit of measure, or other tables that typically reasly are for "catorgories". It simply "finds" the right record or "create" a new one and returns it for the relationship your building.
This also makes the rake script re-usable. And the code in your modeles can also be used in "expansion" applications.
Next week I'll add a few examples.
Tuesday, February 05, 2008
Rails - The Menu Problem - Handling Large Scale Applications moving to Rails
It was interesting. I'm doing a large progject, and there are 500+ tables to deal with. This is my first "legacy" as it is project. And of course the "same" old tricks dont always work.
One of the problems, to check my models, of course I generate activescaffold controllers.
If your've read my previous post, my converts/imports usually do the following:
1. Move the table.
a. grab the schema
b. add missing fields
c. fix field names and table names for destination database
d. create the sequence number
e. create the table in the new schema
2. Create a model
3. Create a controller
4. Update the menu
This strategy has worked great. You get a reasobly native rails app,
and cleanup some of the database/schmema issues in the process.
New project is a bit different. Its all in oracle all ready. And its a "off-the-shelf" application.
So now I need to not "touch" the database schema, but man, I need a extra table, and
how to do menus etc when the functionality is so "rich".
So I decided to create a separate sqlite3 database just to do menus and functions.
(and maybe some other things along the way).
So first, need a do a migration (001_system_menu.rb). This will create our new
sqlite3 database, not touching our oracle database that is already working.
I did a trick to get this to work, I created a "empty" sqlite3 database file,
so the schema_version table would already exist, otherwise you get a "nice" error message saying you "cannot" create it.
Next wanted to get some data "into" the tables, so I created a second migration
(002_add_base_menus.rb) that creates some data so the menu system can be tweaked further.
So now how to "fill" up the tables in the operating "application"?
So in the models, I added the ability to "walk" the "controller" directory,
and any associated namespaces, and create entries. This will allow you then
to "play" them into the "menu" tree and "treak" the names, and the roles.
The "update_functions" method does a few things:
1. Verify "functions(controllers)" actually exist,
then enable or disable as appropriate
(If we delete we loose the custimization)
2. Find any new controllers and "add" them into the system
For now thats what we got, next will be to automatically generation of
menu (tabnav) partials for each level of the menu, and arrange for "hooks" in
the application to "know" what his menu is.
The system_function model has a very good example of "walking" directories.
#-----------------------001_system_menu.rb ------------------------
class SystemMenu < ActiveRecord::Migration
class SystemMenu < ActiveRecord::Base
ActiveRecord::Base.establish_connection(
:adapter => "sqlite3",
:database => "db/system.sqlite3"
)
acts_as_tree
has_many :SystemFunction
end
class SystemFunction < ActiveRecord::Base
ActiveRecord::Base.establish_connection(
:adapter => "sqlite3",
:database => "db/system.sqlite3"
)
belongs_to :SystemMenu
end
def self.up
create_table :system_functions do |t|
t.column :system_menu_id, :number
t.column :present, :boolean
t.column :enable, :boolean
t.column :name, :string, :limit =>32, :null => false
t.column :title, :string, :limit =>64, :null => false
t.column :controller, :string, :limit =>128
t.column :role, :string
t.column :created_at, :datetime
t.column :updated_at, :datetime
end
create_table :system_menus do |t|
t.column :name, :string, :limit =>32, :null => false
t.column :title, :string, :limit =>64, :null => false
t.column :role, :string, :limit =>32
t.column :enable, :boolean
t.column :parent_id, :number
t.column :created_at, :datetime
t.column :updated_at, :datetime
end
end
def self.down
drop_table :system_menus
drop_table :system_functions
end
end
class AddBaseMenus < ActiveRecord::Migration
class SystemMenu < ActiveRecord::Base
establish_connection :adapter => "sqlite3", :database => "db/system.sqlite3"
acts_as_tree
has_many :system_function
end
#-------------------- 002_add_base_menus.rb ------------------------
class SystemFunction < ActiveRecord::Base
establish_connection :adapter => "sqlite3", :database => "db/system.sqlite3"
belongs_to :system_menu
end
def self.up
homemenu = SystemMenu.create :name => "home", :title => "Home"
adminmenu = SystemMenu.create :name => "admin", :title => "Admin", :role => "Admin", :parent_id => homemenu.id
SystemFunction.create :system_menu_id => adminmenu.id, :name => "Menus", :title => "System Menus", :controller => "adminspace/system_menu", :role => "admin", :present => TRUE, :enable => TRUE
SystemFunction.create :system_menu_id => adminmenu.id, :name => "Functions", :title => "System Functions", :controller => "adminspace/system_function", :role => "admin", :present => TRUE, :enable => TRUE
end
def self.down
end
end
#----------------------- system_menu.rb ----------------------
class SystemMenu < ActiveRecord::Base
establish_connection :adapter => "sqlite3", :database => "db/system.sqlite3"
acts_as_tree
has_many :system_function
end
#---------------------- system_function.rb -------------------
class SystemFunction < ActiveRecord::Base
establish_connection :adapter => "sqlite3", :database => "db/system.sqlite3"
belongs_to :system_menu
public
def self.update_functions
app_directory = "./app/controllers"
verify_functions(app_directory)
system_function_process_dir(app_directory,"")
end
def self.verify_functions(thedir)
SystemFunction.find(:all).each do |record|
thepath = thedir + "/" + record.controller + "_controller.rb"
if File.exists?(thepath)
if record.present == FALSE
record.present = TRUE
record.save
end
else
if record.present == TRUE
record.present = FALSE
record.save
end
end
end
end
def self.system_function_process_entry(theparent,thedir, theentry)
if theentry == "."
return
end
if theentry == ".."
return
end
if theentry == "archive"
return
end
fullpath = thedir + "/" + theentry
if File.directory?(fullpath)
system_function_process_dir(fullpath,theentry)
return
end
if theentry.include? "_controller.rb"
thelen = theentry.index('_controller.rb')
thename = theentry[0,thelen]
if theparent == ""
mycontroller = thename
else
mycontroller = theparent + "/" + thename
end
myfunction = SystemFunction.find(:first, :conditions => "controller = '#{mycontroller}'")
if myfunction == nil # We dont exist
myfunction = SystemFunction.new
myfunction.name = thename
myfunction.system_menu_id = nil
myfunction.title = thename
myfunction.controller = mycontroller
myfunction.enable = FALSE
myfunction.present = TRUE
myfunction.role = ''
myfunction.save
return
end
end
end
def self.system_function_process_dir(thedir,theparent)
Dir.foreach(thedir){ |theentry| system_function_process_entry(theparent,thedir,theentry)}
end
end
One of the problems, to check my models, of course I generate activescaffold controllers.
If your've read my previous post, my converts/imports usually do the following:
1. Move the table.
a. grab the schema
b. add missing fields
c. fix field names and table names for destination database
d. create the sequence number
e. create the table in the new schema
2. Create a model
3. Create a controller
4. Update the menu
This strategy has worked great. You get a reasobly native rails app,
and cleanup some of the database/schmema issues in the process.
New project is a bit different. Its all in oracle all ready. And its a "off-the-shelf" application.
So now I need to not "touch" the database schema, but man, I need a extra table, and
how to do menus etc when the functionality is so "rich".
So I decided to create a separate sqlite3 database just to do menus and functions.
(and maybe some other things along the way).
So first, need a do a migration (001_system_menu.rb). This will create our new
sqlite3 database, not touching our oracle database that is already working.
I did a trick to get this to work, I created a "empty" sqlite3 database file,
so the schema_version table would already exist, otherwise you get a "nice" error message saying you "cannot" create it.
Next wanted to get some data "into" the tables, so I created a second migration
(002_add_base_menus.rb) that creates some data so the menu system can be tweaked further.
So now how to "fill" up the tables in the operating "application"?
So in the models, I added the ability to "walk" the "controller" directory,
and any associated namespaces, and create entries. This will allow you then
to "play" them into the "menu" tree and "treak" the names, and the roles.
The "update_functions" method does a few things:
1. Verify "functions(controllers)" actually exist,
then enable or disable as appropriate
(If we delete we loose the custimization)
2. Find any new controllers and "add" them into the system
For now thats what we got, next will be to automatically generation of
menu (tabnav) partials for each level of the menu, and arrange for "hooks" in
the application to "know" what his menu is.
The system_function model has a very good example of "walking" directories.
#-----------------------001_system_menu.rb ------------------------
class SystemMenu < ActiveRecord::Migration
class SystemMenu < ActiveRecord::Base
ActiveRecord::Base.establish_connection(
:adapter => "sqlite3",
:database => "db/system.sqlite3"
)
acts_as_tree
has_many :SystemFunction
end
class SystemFunction < ActiveRecord::Base
ActiveRecord::Base.establish_connection(
:adapter => "sqlite3",
:database => "db/system.sqlite3"
)
belongs_to :SystemMenu
end
def self.up
create_table :system_functions do |t|
t.column :system_menu_id, :number
t.column :present, :boolean
t.column :enable, :boolean
t.column :name, :string, :limit =>32, :null => false
t.column :title, :string, :limit =>64, :null => false
t.column :controller, :string, :limit =>128
t.column :role, :string
t.column :created_at, :datetime
t.column :updated_at, :datetime
end
create_table :system_menus do |t|
t.column :name, :string, :limit =>32, :null => false
t.column :title, :string, :limit =>64, :null => false
t.column :role, :string, :limit =>32
t.column :enable, :boolean
t.column :parent_id, :number
t.column :created_at, :datetime
t.column :updated_at, :datetime
end
end
def self.down
drop_table :system_menus
drop_table :system_functions
end
end
class AddBaseMenus < ActiveRecord::Migration
class SystemMenu < ActiveRecord::Base
establish_connection :adapter => "sqlite3", :database => "db/system.sqlite3"
acts_as_tree
has_many :system_function
end
#-------------------- 002_add_base_menus.rb ------------------------
class SystemFunction < ActiveRecord::Base
establish_connection :adapter => "sqlite3", :database => "db/system.sqlite3"
belongs_to :system_menu
end
def self.up
homemenu = SystemMenu.create :name => "home", :title => "Home"
adminmenu = SystemMenu.create :name => "admin", :title => "Admin", :role => "Admin", :parent_id => homemenu.id
SystemFunction.create :system_menu_id => adminmenu.id, :name => "Menus", :title => "System Menus", :controller => "adminspace/system_menu", :role => "admin", :present => TRUE, :enable => TRUE
SystemFunction.create :system_menu_id => adminmenu.id, :name => "Functions", :title => "System Functions", :controller => "adminspace/system_function", :role => "admin", :present => TRUE, :enable => TRUE
end
def self.down
end
end
#----------------------- system_menu.rb ----------------------
class SystemMenu < ActiveRecord::Base
establish_connection :adapter => "sqlite3", :database => "db/system.sqlite3"
acts_as_tree
has_many :system_function
end
#---------------------- system_function.rb -------------------
class SystemFunction < ActiveRecord::Base
establish_connection :adapter => "sqlite3", :database => "db/system.sqlite3"
belongs_to :system_menu
public
def self.update_functions
app_directory = "./app/controllers"
verify_functions(app_directory)
system_function_process_dir(app_directory,"")
end
def self.verify_functions(thedir)
SystemFunction.find(:all).each do |record|
thepath = thedir + "/" + record.controller + "_controller.rb"
if File.exists?(thepath)
if record.present == FALSE
record.present = TRUE
record.save
end
else
if record.present == TRUE
record.present = FALSE
record.save
end
end
end
end
def self.system_function_process_entry(theparent,thedir, theentry)
if theentry == "."
return
end
if theentry == ".."
return
end
if theentry == "archive"
return
end
fullpath = thedir + "/" + theentry
if File.directory?(fullpath)
system_function_process_dir(fullpath,theentry)
return
end
if theentry.include? "_controller.rb"
thelen = theentry.index('_controller.rb')
thename = theentry[0,thelen]
if theparent == ""
mycontroller = thename
else
mycontroller = theparent + "/" + thename
end
myfunction = SystemFunction.find(:first, :conditions => "controller = '#{mycontroller}'")
if myfunction == nil # We dont exist
myfunction = SystemFunction.new
myfunction.name = thename
myfunction.system_menu_id = nil
myfunction.title = thename
myfunction.controller = mycontroller
myfunction.enable = FALSE
myfunction.present = TRUE
myfunction.role = ''
myfunction.save
return
end
end
end
def self.system_function_process_dir(thedir,theparent)
Dir.foreach(thedir){ |theentry| system_function_process_entry(theparent,thedir,theentry)}
end
end
Wednesday, January 30, 2008
Rails Security - Is there passwords waiting for the world to read
Recently I was promoting a "Rails" app, that was going to be used by another division. And was asking my "boss" to login, and it got me thinking about log file "security". One of the things that has been bothering me for a while was when I was running a app, and looking at the log, either in production or development I would always "say" a few words when a user was loggin in while I was watching. One of my hero's has a favorite saying, "To much information".
To illustrate:
Processing AccountController#login (for 127.0.0.1 at 2008-01-31 13:10:58) [POST]
Session ID: 9a85051d00c5aee3dc20305ac0dd315e
Parameters: {"commit"=>"Log in", "action"=>"login", "controller"=>"adminspace/account", "login"=>"user", "password"=>"secret"}
And you wll see this in both production and development.
No there was a few solutions when I was googling:
1. Turn the logging to warning
in config/environments/production.rb
# Fix the logging so sensitive information is not logged
config.log_level = :warn
Cons: You loose all your request, now you log analyser and you have very little information on what is going on in your application.
2. "Filtering"
Now this is "nice". And is good even in development side. (ie I use ldap based authentication, so in development the account and password is REAL). So it
lets me still see whats going on, without printing out the password (My password)
1000 plus times.
The fix:
in app/controllers/application.rb
filter_parameter_logging "password"
This will get rid of any filtered parameter.
The result (in the development log):
Processing AccountController#login (for 127.0.0.1 at 2008-01-31 13:10:58) [POST]
Session ID: 9a85051d00c5aee3dc20305ac0dd315e
Parameters: {"commit"=>"Log in", "action"=>"login", "controller"=>"adminspace/account", "login"=>"user", "password"=>"[FILTERED]"}
Lovely.
You can also hide other things as well if you want to keep them secure, credit card numbers etc are good examples.
I tend to think this should be your "default" setting in any application.
It will be in mine from now on.
This works in 2.0.2 rails very nicely. (It was actually added quite a while back)
If you ask me it should be the "default" in the template when generating a rails applicaton.
To illustrate:
Processing AccountController#login (for 127.0.0.1 at 2008-01-31 13:10:58) [POST]
Session ID: 9a85051d00c5aee3dc20305ac0dd315e
Parameters: {"commit"=>"Log in", "action"=>"login", "controller"=>"adminspace/account", "login"=>"user", "password"=>"secret"}
And you wll see this in both production and development.
No there was a few solutions when I was googling:
1. Turn the logging to warning
in config/environments/production.rb
# Fix the logging so sensitive information is not logged
config.log_level = :warn
Cons: You loose all your request, now you log analyser and you have very little information on what is going on in your application.
2. "Filtering"
Now this is "nice". And is good even in development side. (ie I use ldap based authentication, so in development the account and password is REAL). So it
lets me still see whats going on, without printing out the password (My password)
1000 plus times.
The fix:
in app/controllers/application.rb
filter_parameter_logging "password"
This will get rid of any filtered parameter.
The result (in the development log):
Processing AccountController#login (for 127.0.0.1 at 2008-01-31 13:10:58) [POST]
Session ID: 9a85051d00c5aee3dc20305ac0dd315e
Parameters: {"commit"=>"Log in", "action"=>"login", "controller"=>"adminspace/account", "login"=>"user", "password"=>"[FILTERED]"}
Lovely.
You can also hide other things as well if you want to keep them secure, credit card numbers etc are good examples.
I tend to think this should be your "default" setting in any application.
It will be in mine from now on.
This works in 2.0.2 rails very nicely. (It was actually added quite a while back)
If you ask me it should be the "default" in the template when generating a rails applicaton.
Sunday, January 27, 2008
Oracle and Rails - Moving tables to new tablespaces
Hmmm. Things are growing in my oracle tables, and I really want to
move things around a bit. So moving a "application" or "user" to a new
tablespace is resonably common.
For me, language for first resort is ruby, so I have quite
a few "little" scripts to do oracle tasks.
Here's today's, move all of a users tables to a new tablespace.
require 'DBI'
require 'pp'
#movetablepace.rb
#***************************************
#make a Oracle Connection
fromdbconstr = ARGV[0] # For example - DBI:OCI8://myoracleserver/oracleinstance
fromdbconuser = ARGV[1] # MYRAILSDBUSER-DEVELOP
fromdbconpass = ARGV[2] # MYRAILSDBUSER-PASSWORD
dataspacename = ARGV[3] # APPLICATION-DATASPACE
fromdb = DBI.connect(fromdbconstr, fromdbconuser, fromdbconpass)
tables = fromdb.tables
tables.each do |tablename|
tablename = tablename.strip
if tablename.length == 0
next
end
if tablename.length > 4
if tablename[0,4] == "BIN$"
next
end
end
puts "Move Table [#{tablename}] to new tablespace"
sqlmove = "alter table " + '"' + tablename + '" move tablespace "' + dataspacename + '"'
puts sqlmove
begin
result = fromdb.execute(sqlmove)
rescue
puts "Table [#{tablename}] not moved"
end
end
And after you do that, you need to "rebuild" the indexes into the new tablespace
require 'DBI'
require 'pp'
#make a Oracle Connection
fromdbconstr = ARGV[0] # For example - DBI:OCI8://myoracleserver/oracleinstance
fromdbconuser = ARGV[1] # MYRAILSDBUSER-DEVELOP
fromdbconpass = ARGV[2] # MYRAILSDBUSER-PASSWORD
dataspacename = ARGV[3] # APPLICATION-DATASPACE
fromdb = DBI.connect(fromdbconstr, fromdbconuser, fromdbconpass)
tables = fromdb.tables
sqlget = "select index_name from user_indexes"
indexs = fromdb.execute(sqlget)
indexs.each do |theindex|
indexname = theindex[0]
printf "Rebuilding Index %s\n", indexname
sqlrebuild = 'alter index "' + indexname + '" rebuild tablespace "' + dataspacename + '"'
begin
result = fromdb.execute(sqlrebuild)
rescue
printf "[%s] Index not rebult\n", indexname
end
end
move things around a bit. So moving a "application" or "user" to a new
tablespace is resonably common.
For me, language for first resort is ruby, so I have quite
a few "little" scripts to do oracle tasks.
Here's today's, move all of a users tables to a new tablespace.
require 'DBI'
require 'pp'
#movetablepace.rb
#***************************************
#make a Oracle Connection
fromdbconstr = ARGV[0] # For example - DBI:OCI8://myoracleserver/oracleinstance
fromdbconuser = ARGV[1] # MYRAILSDBUSER-DEVELOP
fromdbconpass = ARGV[2] # MYRAILSDBUSER-PASSWORD
dataspacename = ARGV[3] # APPLICATION-DATASPACE
fromdb = DBI.connect(fromdbconstr, fromdbconuser, fromdbconpass)
tables = fromdb.tables
tables.each do |tablename|
tablename = tablename.strip
if tablename.length == 0
next
end
if tablename.length > 4
if tablename[0,4] == "BIN$"
next
end
end
puts "Move Table [#{tablename}] to new tablespace"
sqlmove = "alter table " + '"' + tablename + '" move tablespace "' + dataspacename + '"'
puts sqlmove
begin
result = fromdb.execute(sqlmove)
rescue
puts "Table [#{tablename}] not moved"
end
end
And after you do that, you need to "rebuild" the indexes into the new tablespace
require 'DBI'
require 'pp'
#make a Oracle Connection
fromdbconstr = ARGV[0] # For example - DBI:OCI8://myoracleserver/oracleinstance
fromdbconuser = ARGV[1] # MYRAILSDBUSER-DEVELOP
fromdbconpass = ARGV[2] # MYRAILSDBUSER-PASSWORD
dataspacename = ARGV[3] # APPLICATION-DATASPACE
fromdb = DBI.connect(fromdbconstr, fromdbconuser, fromdbconpass)
tables = fromdb.tables
sqlget = "select index_name from user_indexes"
indexs = fromdb.execute(sqlget)
indexs.each do |theindex|
indexname = theindex[0]
printf "Rebuilding Index %s\n", indexname
sqlrebuild = 'alter index "' + indexname + '" rebuild tablespace "' + dataspacename + '"'
begin
result = fromdb.execute(sqlrebuild)
rescue
printf "[%s] Index not rebult\n", indexname
end
end
Friday, January 25, 2008
When the C-LEVEL(CTO,CEO,CFO) ask so why should we use Rails
Ok Mr. Big CTO, why shuld we use RAILS?
A question that was recently asked in the Ruby forums.
So having done a few C-Level jobs, I thought I'd answer.
1. Is it going to be around a long time - Yes - Ruby itself has, and
Rails has hit 2. Plus recognition from
major software firms. (Oracle offered class on it on the Shanghai
Training Session,), oracle has support.
Major media companies are using it. Other MNC are using. Trading(As in
stock, on wallstreet) are using it.
Look at the "books" available. Need one every week. Look at the number
of web articles. By every metric its
everywhere.
2. Is training available? Yes, on multiple fronts. Also fresh java
grads tend to pick it up easier.
3. Does it marry with my "enterprise" systems. Now this one is
interesting. Ruby can marry to just
about anything, so thus "rails" can. I am just finishing a "marriage"
between Unify and Rails. The original
applicaiton is over 15 years old, and the data is still worth
millions. Of course I "moved" the data over to
oracle. (With ruby). I've also did "bridges" between applications.
3. Is it secure? If its running on "Linux" would be my answer. Look at
the security guide. Put a https proxy(Nginx)
in fromt. 100x more secure than windows.
4. Is it scalable. Big question. But in a short answer, yes yes yes.
For our users, 3000-4000 is max user count.
I even was helping a friend with handling million plus user bases.
Hardware is cheap, with nginx, mongrel clusters, and
good hardware its falling off a log. (If your public hosting, even a
single server can handle your average load").
5. Can I get consultants? Yes. Look at the tavnav guys for one. There
great.
6. But does it take "long" to do the job?
Its the "fastest" system I've found to get thing done fast, neat, and quick.
Look up "DRY" under Rails. It stands for "Dont Repeat Yourself".
7. Does it do "Windows"
Yes very well. How about a "Applicatoin-to-Application" Bridge done
in a week as a Windows "Service"
8. Does it do "Clusters"
Yes
9. Does it do "Linux"
Yes
10. Is it better than Java
Yes - very. So much more concise.
A question that was recently asked in the Ruby forums.
So having done a few C-Level jobs, I thought I'd answer.
1. Is it going to be around a long time - Yes - Ruby itself has, and
Rails has hit 2. Plus recognition from
major software firms. (Oracle offered class on it on the Shanghai
Training Session,), oracle has support.
Major media companies are using it. Other MNC are using. Trading(As in
stock, on wallstreet) are using it.
Look at the "books" available. Need one every week. Look at the number
of web articles. By every metric its
everywhere.
2. Is training available? Yes, on multiple fronts. Also fresh java
grads tend to pick it up easier.
3. Does it marry with my "enterprise" systems. Now this one is
interesting. Ruby can marry to just
about anything, so thus "rails" can. I am just finishing a "marriage"
between Unify and Rails. The original
applicaiton is over 15 years old, and the data is still worth
millions. Of course I "moved" the data over to
oracle. (With ruby). I've also did "bridges" between applications.
3. Is it secure? If its running on "Linux" would be my answer. Look at
the security guide. Put a https proxy(Nginx)
in fromt. 100x more secure than windows.
4. Is it scalable. Big question. But in a short answer, yes yes yes.
For our users, 3000-4000 is max user count.
I even was helping a friend with handling million plus user bases.
Hardware is cheap, with nginx, mongrel clusters, and
good hardware its falling off a log. (If your public hosting, even a
single server can handle your average load").
5. Can I get consultants? Yes. Look at the tavnav guys for one. There
great.
6. But does it take "long" to do the job?
Its the "fastest" system I've found to get thing done fast, neat, and quick.
Look up "DRY" under Rails. It stands for "Dont Repeat Yourself".
7. Does it do "Windows"
Yes very well. How about a "Applicatoin-to-Application" Bridge done
in a week as a Windows "Service"
8. Does it do "Clusters"
Yes
9. Does it do "Linux"
Yes
10. Is it better than Java
Yes - very. So much more concise.
Tuesday, January 22, 2008
Advanced Rails Migrations - Adding and Updating Related Table ID's
Ever have that "rails" problem.
You imported a butch of "existing" tables to create a rails application,
and you dont have appropriate index/id's to help with your belong_to, and
associated tables?
Well Migrations can do it for you.
Example migration using two tables that are in ActiveRecord.
This one builds a link between a inventory table, and a customer table.
class UpdateXco < ActiveRecord::Migration
def self.up
add_column :REG_INV_SERS, :REG_CUST_ID, :integer
RegInvSer.reset_column_information
RegInvSer.find(:all).each do |record|
regcustomer = RegCust.find(:first, :conditions => "CUST_NBR = '#{record.cust_nbr}'")
if not regcustomer.nil?
record.reg_cust_id = regcustomer.id
record.save
end
end
end
def self.down
remove_column "REG_INV_SERS", "REG_CUST_ID"
end
end
Another one, that illustrates doing a table import from a dbase/foxpro table, with a id update as well.
class Updatesalesorder < ActiveRecord::Migration
def self.up
thepathname = "h:xxxxx.DBF"
table = DBF::Table.new(thepathname)
table.records.each do |record|
mysalesorder = salesorder.create(record.attributes)
mycustomer = customer.find(:first, :conditions => "custno = '#{record.custno}'")
if not mycustomer.nil?
mysalesorder.customer_id = mycustomer.id
mysalesorder.save
end
end
end
def self.down
drop_table "salesorders"
end
end
You imported a butch of "existing" tables to create a rails application,
and you dont have appropriate index/id's to help with your belong_to, and
associated tables?
Well Migrations can do it for you.
Example migration using two tables that are in ActiveRecord.
This one builds a link between a inventory table, and a customer table.
class UpdateXco < ActiveRecord::Migration
def self.up
add_column :REG_INV_SERS, :REG_CUST_ID, :integer
RegInvSer.reset_column_information
RegInvSer.find(:all).each do |record|
regcustomer = RegCust.find(:first, :conditions => "CUST_NBR = '#{record.cust_nbr}'")
if not regcustomer.nil?
record.reg_cust_id = regcustomer.id
record.save
end
end
end
def self.down
remove_column "REG_INV_SERS", "REG_CUST_ID"
end
end
Another one, that illustrates doing a table import from a dbase/foxpro table, with a id update as well.
class Updatesalesorder < ActiveRecord::Migration
def self.up
thepathname = "h:xxxxx.DBF"
table = DBF::Table.new(thepathname)
table.records.each do |record|
mysalesorder = salesorder.create(record.attributes)
mycustomer = customer.find(:first, :conditions => "custno = '#{record.custno}'")
if not mycustomer.nil?
mysalesorder.customer_id = mycustomer.id
mysalesorder.save
end
end
end
def self.down
drop_table "salesorders"
end
end
Monday, January 21, 2008
Ruby - A "class" tutorial using a Unify DBI interface as a example
This is a "good" example of a "class".
It implemnets a simple "dbi" like interface to a unify IDS database.
(This is a embedded database used in linux, and embedded devices)
Recently needed to access data from a unify database in "rails".
So of course, needed a "dbi" like layer to work with one of my scripts to
pull the data. (See previous posts on data conversion and migrations).
So it was natural to create a "class". A class is a "object" consisting of code,
or methods, and "data". The data can come in two types "instance" and class data.
Here everything is "instance" data. So when we create a new "Unify" object, we
get another set of "data" for that instance.
In our example, we use two temp files per instance to communication to Unify sql command via a "shell" command. One is for the SQL statement, and one is for the "results".
Also the lovely thing about "instances" is it makes it easy to see all your "instance" data, simply by doing a "pp" (Pretty Print) on the "instance" variable.
So:
mydb = Unify.new(database,user,password)
pp mydb (Will let you see what going on with the instance)
Note that all the "@" instance varibles are recreated on each "new".
This allows this simple example to have multiple outstanding "queries", without
getting confused.
class Unify
def make_tmpname(type,instance)
return 'unify-' + type + '-' + instance + '.tmp'
end
def execute_unify_sql(thesql)
thequery = File.open(@uqueryfilename,"w")
thequery.puts "lines 0"
# thequery.puts "separator ','
thequery.puts thesql
thequery.close
thecommand = 'SQL ' + @uqueryfilename + '> ' + @udatafilename
result = system(thecommand)
if result == false
sleep(10)
result = system(thecommand)
end
theresult = File.open(@udatafilename)
@udatafile = theresult
return theresult
end
def initialize(database,user,password)
@udatabase = database
@uuser = user
@upassword = password
@udatafile = nil
@udatafilename = nil
@uqueryfilename = nil
queryinstance = rand(10000).to_s
@udatafilename = make_tmpname('result',queryinstance)
@uqueryfilename = make_tmpname('query',queryinstance)
end
def fields(tablename)
thesql = 'fields ' + tablename
thedata = execute_unify_sql(thesql)
header = thedata.gets
header = thedata.gets #throw away the two header lines
result = Array.new
thedata.readlines.each { |line|
theline = line.chomp!
# theline <= '|' # last element does not terminate with |
element = theline.split
result << element
}
self.finish()
return result
end
def tables
thedata = execute_unify_sql("tables")
thecontents = thedata.read
self.finish
mytables = thecontents.split
return mytables
end
def execute(sql)
#For execute create a new "instance"
unifyinstance = Unify.new(@udatabase,@uuser,@upassword)
if sql[-1,1] == ';'
sql[-1,1] = '/'
end
theresultfile = unifyinstance.execute_unify_sql(sql)
return unifyinstance
end
def fetch()
@udatafile.pos = 0 #Make sure we are at the beginning
thetable = Array.new
while thedata = @udatafile.gets
thedata.chomp!
therow = thedata.split('|',-1) # No supressed fields
thetable << therow
end
return thetable
end
def finish()
@udatafile.close
File.delete(@udatafilename)
File.delete(@uqueryfilename)
end
end
It implemnets a simple "dbi" like interface to a unify IDS database.
(This is a embedded database used in linux, and embedded devices)
Recently needed to access data from a unify database in "rails".
So of course, needed a "dbi" like layer to work with one of my scripts to
pull the data. (See previous posts on data conversion and migrations).
So it was natural to create a "class". A class is a "object" consisting of code,
or methods, and "data". The data can come in two types "instance" and class data.
Here everything is "instance" data. So when we create a new "Unify" object, we
get another set of "data" for that instance.
In our example, we use two temp files per instance to communication to Unify sql command via a "shell" command. One is for the SQL statement, and one is for the "results".
Also the lovely thing about "instances" is it makes it easy to see all your "instance" data, simply by doing a "pp" (Pretty Print) on the "instance" variable.
So:
mydb = Unify.new(database,user,password)
pp mydb (Will let you see what going on with the instance)
Note that all the "@" instance varibles are recreated on each "new".
This allows this simple example to have multiple outstanding "queries", without
getting confused.
class Unify
def make_tmpname(type,instance)
return 'unify-' + type + '-' + instance + '.tmp'
end
def execute_unify_sql(thesql)
thequery = File.open(@uqueryfilename,"w")
thequery.puts "lines 0"
# thequery.puts "separator ','
thequery.puts thesql
thequery.close
thecommand = 'SQL ' + @uqueryfilename + '> ' + @udatafilename
result = system(thecommand)
if result == false
sleep(10)
result = system(thecommand)
end
theresult = File.open(@udatafilename)
@udatafile = theresult
return theresult
end
def initialize(database,user,password)
@udatabase = database
@uuser = user
@upassword = password
@udatafile = nil
@udatafilename = nil
@uqueryfilename = nil
queryinstance = rand(10000).to_s
@udatafilename = make_tmpname('result',queryinstance)
@uqueryfilename = make_tmpname('query',queryinstance)
end
def fields(tablename)
thesql = 'fields ' + tablename
thedata = execute_unify_sql(thesql)
header = thedata.gets
header = thedata.gets #throw away the two header lines
result = Array.new
thedata.readlines.each { |line|
theline = line.chomp!
# theline <= '|' # last element does not terminate with |
element = theline.split
result << element
}
self.finish()
return result
end
def tables
thedata = execute_unify_sql("tables")
thecontents = thedata.read
self.finish
mytables = thecontents.split
return mytables
end
def execute(sql)
#For execute create a new "instance"
unifyinstance = Unify.new(@udatabase,@uuser,@upassword)
if sql[-1,1] == ';'
sql[-1,1] = '/'
end
theresultfile = unifyinstance.execute_unify_sql(sql)
return unifyinstance
end
def fetch()
@udatafile.pos = 0 #Make sure we are at the beginning
thetable = Array.new
while thedata = @udatafile.gets
thedata.chomp!
therow = thedata.split('|',-1) # No supressed fields
thetable << therow
end
return thetable
end
def finish()
@udatafile.close
File.delete(@udatafilename)
File.delete(@uqueryfilename)
end
end
Rails 2.0.2 Issues and Problems Resolved
Now this was fun, my main Rails box is now running Rails 2.0.2
First problem, when running Rake task, I get complaints about Active Recrd
C:\projects\ror\tempco>rake extract_fixtures
(in C:/projects/ror/tempco)
rake aborted!
can't activate activerecord (= 1.15.3), already activated activerecord-2.0.2]
This one stumped me for a moment, but then realized that the "old" gems may
be confusing issues. So a little cleanup is in order.
gem cleanup
This command cleans up all "the" old gems.
And solved my problems
The next problem I had, was:
C:\projects\ror\tempco>rake extract_fixtures
(in C:/projects/ror/tempco)
rake aborted!
Please install the oracle adapter: `gem install activerecord-oracle-adapter` (no
such file to load -- active_record/connection_adapters/oracle_adapter)
Rails 2 does not by default install ActiveRecord adapter for oracle.
Of course if you try the gem install command the error message recommends,
it still does not give you gratification.
The proper command that works is:
C:\projects\ror\tempco>gem install activerecord-oracle-adapter
--source http://gems.rubyonrails.org
Bulk updating Gem source index for: http://gems.rubyonrails.org
Successfully installed activerecord-oracle-adapter-1.0.0
1 gem installed
And now, I try to do a rake db:schema:dump
And the next problem occurs, seems like there is a "bug" in the
oracle adapter.
The fix seems to be here:
http://dev.rubyonrails.org/attachment/ticket/9062/fix_oracleadapter_problem_indexes.diff
But does not match the code in the 1.0 adapter. Hmmm.
Ok, lets try to setup a "session" migration.
Hmm same error
C:\projects\ror\tempco>
C:\projects\ror\tempco>rake db:migrate
(in C:/projects/ror/tempco)
== 1 CreateSessions: migrating ================================================
-- create_table(:sessions)
-> 38.5940s
-- add_index(:sessions, :session_id)
-> 0.0930s
-- add_index(:sessions, :updated_at)
-> 0.0630s
== 1 CreateSessions: migrated (38.7500s) ======================================
rake aborted!
select_rows is an abstract method
Ok, do a bit more google work, and what do we find:
http://dev.rubyonrails.org/ticket/10415
Insert the code into the oracle adapter, and now no error on our migration.
And our db:schema:dump works properly.
First problem, when running Rake task, I get complaints about Active Recrd
C:\projects\ror\tempco>rake extract_fixtures
(in C:/projects/ror/tempco)
rake aborted!
can't activate activerecord (= 1.15.3), already activated activerecord-2.0.2]
This one stumped me for a moment, but then realized that the "old" gems may
be confusing issues. So a little cleanup is in order.
gem cleanup
This command cleans up all "the" old gems.
And solved my problems
The next problem I had, was:
C:\projects\ror\tempco>rake extract_fixtures
(in C:/projects/ror/tempco)
rake aborted!
Please install the oracle adapter: `gem install activerecord-oracle-adapter` (no
such file to load -- active_record/connection_adapters/oracle_adapter)
Rails 2 does not by default install ActiveRecord adapter for oracle.
Of course if you try the gem install command the error message recommends,
it still does not give you gratification.
The proper command that works is:
C:\projects\ror\tempco>gem install activerecord-oracle-adapter
--source http://gems.rubyonrails.org
Bulk updating Gem source index for: http://gems.rubyonrails.org
Successfully installed activerecord-oracle-adapter-1.0.0
1 gem installed
And now, I try to do a rake db:schema:dump
And the next problem occurs, seems like there is a "bug" in the
oracle adapter.
The fix seems to be here:
http://dev.rubyonrails.org/attachment/ticket/9062/fix_oracleadapter_problem_indexes.diff
But does not match the code in the 1.0 adapter. Hmmm.
Ok, lets try to setup a "session" migration.
Hmm same error
C:\projects\ror\tempco>
C:\projects\ror\tempco>rake db:migrate
(in C:/projects/ror/tempco)
== 1 CreateSessions: migrating ================================================
-- create_table(:sessions)
-> 38.5940s
-- add_index(:sessions, :session_id)
-> 0.0930s
-- add_index(:sessions, :updated_at)
-> 0.0630s
== 1 CreateSessions: migrated (38.7500s) ======================================
rake aborted!
select_rows is an abstract method
Ok, do a bit more google work, and what do we find:
http://dev.rubyonrails.org/ticket/10415
Insert the code into the oracle adapter, and now no error on our migration.
And our db:schema:dump works properly.
Tuesday, November 27, 2007
RedhatOnRails or CentosOnRails
Wow, I finally did it.
A automatied script to get NGINX, Mongrel/MongrelCluster, and Rails up on Redhat/Centos.
The gotcha that was bad was I assumed the Centos/Redhat ruby was up to date. Its not.
And causes intersting problems. Even using yum to remove it does not get rid of the old binary.
So make sure you dont have ruby installed, and if you do you have to go and remove it both with yum and manually. Then you should be fine.
The script install ruby, ImageMagic, Nginx, and rails, and gem.
I found that gem install for rails was not working for linux. But installing with
the gem manually worked fine.
#install_rails - dont forget the chmod +x install_rails and run as root
yum install -y gcc
yum install -y buildsys-build
wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p110.tar.gz
tar -xzvf ruby-1.8.6-p110.tar.gz
cd ruby-1.8.6-p110
./configure && make && make install
cd ..
rm -r -f ruby-1.8.6-p110
rm ruby-1.8.6-p110.tar.gz
yum install -y openssl-devel
wget http://rubyforge.org/frs/download.php/28174/rubygems-0.9.5.tgz
mkdir rubygems
mv rubygems-0.9.5.tgz rubygems
cd rubygems
tar xzvf rubygems-0.9.5.tgz
cd rubygems-0.9.5
ruby setup.rb
cd ..
cd ..
rm -r -f rubygems
yum -y install libpng-devel libjpeg-devel libtiff-devel freetype-devel ghostscript-devel
wget ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz
tar -xzvf ImageMagick.tar.gz
cd ImageMagick-6.3.7
./configure
make && make install
cd ..
rm -rf ImageMagick-6.3.7
cp /etc/ld.so.conf /etc/ld.so.conf.backup
sed '$a\
/usr/local/lib' /etc/ld.so.conf > ld.so.conf.new
mv ld.so.conf.new /etc/ld.so.conf
/sbin/ldconfig -v
/sbin/ldconfig
gem install rmagick
yum -y install pcre pcre-devel
wget http://sysoev.ru/nginx/nginx-0.5.33.tar.gz
tar -xzvf nginx-0.5.33.tar.gz
cd nginx-0.5.33
./configure --with-http_ssl_module
make
make install
cd ..
rm -r -f nginx-0.5.33
rm nginx-0.5.33.tar.gz
wget http://rubyforge.org/frs/download.php/28337/rails-1.2.6.gem
gem install ./rails-1.2.6.gem
gem install mongrel mongrel_cluster
A automatied script to get NGINX, Mongrel/MongrelCluster, and Rails up on Redhat/Centos.
The gotcha that was bad was I assumed the Centos/Redhat ruby was up to date. Its not.
And causes intersting problems. Even using yum to remove it does not get rid of the old binary.
So make sure you dont have ruby installed, and if you do you have to go and remove it both with yum and manually. Then you should be fine.
The script install ruby, ImageMagic, Nginx, and rails, and gem.
I found that gem install for rails was not working for linux. But installing with
the gem manually worked fine.
#install_rails - dont forget the chmod +x install_rails and run as root
yum install -y gcc
yum install -y buildsys-build
wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p110.tar.gz
tar -xzvf ruby-1.8.6-p110.tar.gz
cd ruby-1.8.6-p110
./configure && make && make install
cd ..
rm -r -f ruby-1.8.6-p110
rm ruby-1.8.6-p110.tar.gz
yum install -y openssl-devel
wget http://rubyforge.org/frs/download.php/28174/rubygems-0.9.5.tgz
mkdir rubygems
mv rubygems-0.9.5.tgz rubygems
cd rubygems
tar xzvf rubygems-0.9.5.tgz
cd rubygems-0.9.5
ruby setup.rb
cd ..
cd ..
rm -r -f rubygems
yum -y install libpng-devel libjpeg-devel libtiff-devel freetype-devel ghostscript-devel
wget ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz
tar -xzvf ImageMagick.tar.gz
cd ImageMagick-6.3.7
./configure
make && make install
cd ..
rm -rf ImageMagick-6.3.7
cp /etc/ld.so.conf /etc/ld.so.conf.backup
sed '$a\
/usr/local/lib' /etc/ld.so.conf > ld.so.conf.new
mv ld.so.conf.new /etc/ld.so.conf
/sbin/ldconfig -v
/sbin/ldconfig
gem install rmagick
yum -y install pcre pcre-devel
wget http://sysoev.ru/nginx/nginx-0.5.33.tar.gz
tar -xzvf nginx-0.5.33.tar.gz
cd nginx-0.5.33
./configure --with-http_ssl_module
make
make install
cd ..
rm -r -f nginx-0.5.33
rm nginx-0.5.33.tar.gz
wget http://rubyforge.org/frs/download.php/28337/rails-1.2.6.gem
gem install ./rails-1.2.6.gem
gem install mongrel mongrel_cluster
Tuesday, November 06, 2007
Getting started faster in rails
Had a comment on my blog for "create xyz application" did not work.
I suspect the cut/paste must of went astray. Just to make sure
your start in rails is a bit faster, I wrote a quick ruby script.
So install ruby and rails per the previous instructions, then this script
will create your "rails" app tree, and automatically install the three plugins
that I recommend to get started in Rails.
1. ActiveScaffold
2. RoleRequirement
3. Widgets/Tabnav
To run the script
myrails appname
#myrails.rb
# Usage: myrails appname
myappname = ARGV[0]
puts "Creating Initial App - " + myappname
system ("rails " + myappname)
Dir.chdir(myappname)
#Each of these must be a single line, no returns/newline allowed - Must be 3 system lines
system ('ruby ./script/plugin install http://activescaffold.googlecode.com/svn/tags/active_scaffold')
system ('ruby ./script/plugin install http://rolerequirement.googlecode.com/svn/tags/role_requirement/')
system ('ruby ./script/plugin install svn://svn.seesaw.it/widgets/trunk')
I suspect the cut/paste must of went astray. Just to make sure
your start in rails is a bit faster, I wrote a quick ruby script.
So install ruby and rails per the previous instructions, then this script
will create your "rails" app tree, and automatically install the three plugins
that I recommend to get started in Rails.
1. ActiveScaffold
2. RoleRequirement
3. Widgets/Tabnav
To run the script
myrails appname
#myrails.rb
# Usage: myrails appname
myappname = ARGV[0]
puts "Creating Initial App - " + myappname
system ("rails " + myappname)
Dir.chdir(myappname)
#Each of these must be a single line, no returns/newline allowed - Must be 3 system lines
system ('ruby ./script/plugin install http://activescaffold.googlecode.com/svn/tags/active_scaffold')
system ('ruby ./script/plugin install http://rolerequirement.googlecode.com/svn/tags/role_requirement/')
system ('ruby ./script/plugin install svn://svn.seesaw.it/widgets/trunk')
Tuesday, October 30, 2007
Can you create xyz application in rails
Yes you can. And its easy.
First oh, Go to your favorate book seller and buy the following books:
1. The Ruby Way - Second Edition By Hal Fulton
2. Rails Recipes - Fowler
3. Ruby Cookbook - Lucas Carlson & Leonard Richardson
4. Agile Web Development with Rails - Thomas Heinemeler Hansson
Then your ready to get started:
Its very easy.
1. Install Rails
gem install rails --include-dependencies
2. Add ActiveScoffold PlugIn (http://activescaffold.com/)
./script/plugin install http://activescaffold.googlecode.com/svn/tags/active_scaffold
3. Add RoleRequirement (http://code.google.com/p/
rolerequirement/)
./script/plugin install http://rolerequirement.googlecode.com/svn/tags/role_requirement/
4. Add TabNav/Widgets (ruby script/plugin install
svn://svn.seesaw.it/widgets/trunk) (Information: http://www.seesaw.it/en/toolbox/widgets/)
ruby script/plugin install svn://svn.seesaw.it/widgets/trunk
5. Define your takes in a migration
6. Build your model files (3-4 lines each)
7. Buld your controllers (for each table)
8. Build your menus (using tabnav)
9. Add your relationships and tweak tabnav.
10. Add email confirmation, and notification.
First oh, Go to your favorate book seller and buy the following books:
1. The Ruby Way - Second Edition By Hal Fulton
2. Rails Recipes - Fowler
3. Ruby Cookbook - Lucas Carlson & Leonard Richardson
4. Agile Web Development with Rails - Thomas Heinemeler Hansson
Then your ready to get started:
Its very easy.
1. Install Rails
gem install rails --include-dependencies
2. Add ActiveScoffold PlugIn (http://activescaffold.com/)
./script/plugin install http://activescaffold.googlecode.com/svn/tags/active_scaffold
3. Add RoleRequirement (http://code.google.com/p/
rolerequirement/)
./script/plugin install http://rolerequirement.googlecode.com/svn/tags/role_requirement/
4. Add TabNav/Widgets (ruby script/plugin install
svn://svn.seesaw.it/widgets/trunk) (Information: http://www.seesaw.it/en/toolbox/widgets/)
ruby script/plugin install svn://svn.seesaw.it/widgets/trunk
5. Define your takes in a migration
6. Build your model files (3-4 lines each)
7. Buld your controllers (for each table)
8. Build your menus (using tabnav)
9. Add your relationships and tweak tabnav.
10. Add email confirmation, and notification.
Rails Tabnav - Fixing Menu Spacing
For my applications, I want a good "look" and feel, with standadized menus.
My "menu" of choice is Tabnav/widget done by seesaw.
In "Enterprise" applications, the number of menus sometimes can be "large".
For example if you want a "menu" per department, there could be 20 departments
in a large company.
While tabnav does not "limit" you in the number or levels of menus, there
is a minor overlap problem that can occur if the screen is swunken below
a certan point, or you have "to" many menus on one "level.
There's a "easy" fix for handling this.
In your project/vendor/plugins/widgets/lib/widgets/tabnav.css.erb file, you
can adjust the overlap.
If you look at the "top" of the tabnav.css.erb file, you will notice the "line-height", I found by setting this to 170% my overlap problems disappeared.
Modified tabnav.css.erb
color: #000;
line-height: 170%;
border-bottom: 2px solid black;
margin: 13px 0px 0px 0px;
padding: 0px;
z-index: 1;
padding-left: 10px
Original tabnav.css.erb (First few lines)
color: #000;
border-bottom: 2px solid black;
margin: 13px 0px 0px 0px;
padding: 0px;
z-index: 1;
padding-left: 10px
My "menu" of choice is Tabnav/widget done by seesaw.
In "Enterprise" applications, the number of menus sometimes can be "large".
For example if you want a "menu" per department, there could be 20 departments
in a large company.
While tabnav does not "limit" you in the number or levels of menus, there
is a minor overlap problem that can occur if the screen is swunken below
a certan point, or you have "to" many menus on one "level.
There's a "easy" fix for handling this.
In your project/vendor/plugins/widgets/lib/widgets/tabnav.css.erb file, you
can adjust the overlap.
If you look at the "top" of the tabnav.css.erb file, you will notice the "line-height", I found by setting this to 170% my overlap problems disappeared.
Modified tabnav.css.erb
color: #000;
line-height: 170%;
border-bottom: 2px solid black;
margin: 13px 0px 0px 0px;
padding: 0px;
z-index: 1;
padding-left: 10px
Original tabnav.css.erb (First few lines)
color: #000;
border-bottom: 2px solid black;
margin: 13px 0px 0px 0px;
padding: 0px;
z-index: 1;
padding-left: 10px
Ruby and ActiveDirectory - Finding Servers and Dead Computer Accounts
Lets expand our example to find servers, and dead computers.
One thing, the "servers" dont always appear when we search for "computer" accounts, in AD. Didnt find a known bug for this, but servers I know are "out" there just didnt come back.
So a few more "gotcha" when using AD from Ruby/LDAP
1. Servers dont "come-back" from a search of objectype = Computer
2. "Server" objectype does not give you the OS. Must search for server, then find it specifically to get the OS.
3. You might have duplicates if you add these togeather.
4. AD Date and Time is very different format than ruby's, so thanks to google, I found some nice routeines for converting between the two.
Example code:
require 'ldap'
require "pp"
def fetch_computers(domain_controller,the_username, the_password)
conn = LDAP::Conn.new( domain_controller, 389 )
conn.set_option( LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )
conn.bind( the_username, the_password ) do |conn|
base = 'DC=company,DC=com'
results = conn.search2(base, LDAP::LDAP_SCOPE_SUBTREE, "objectClass=Computer")
if results == LDAP::LDAP_SIZELIMIT_EXCEEDED
printf "To Much Data\n"
end
return results
end
end
def fetch_servers(domain_controller,the_username, the_password)
conn = LDAP::Conn.new( domain_controller, 389 )
conn.set_option( LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )
conn.bind( the_username, the_password ) do |conn|
base = 'DC=company,DC=com'
results = conn.search2(base, LDAP::LDAP_SCOPE_SUBTREE, "objectClass=Server")
if results == LDAP::LDAP_SIZELIMIT_EXCEEDED
printf "To Much Data\n"
end
return results
end
end
def fetch_computer(domain_controller,the_username, the_password,thecomputername)
conn = LDAP::Conn.new( domain_controller, 389 )
conn.set_option( LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )
conn.bind( the_username, the_password )
base = 'DC=company,DC=com'
results = conn.search2(base, LDAP::LDAP_SCOPE_SUBTREE, "cn=#{thecomputername}")
return results
end
#[["Windows 2000 Server"],
# ["Windows Server 2003"],
# ["Windows XP Professional"],
# ["Windows 2000 Professional"],
# ["Windows NT"],
# ["Windows Vista\342\204\242 Business"],
# ["Windows XP Tablet PC Edition"]]
AD_EPOCH = 116_444_736_000_000_000
AD_MULTIPLIER = 10_000_000
# convert a Time object to AD's epoch
def time2ad(time)
(time.to_i * AD_MULTIPLIER) + AD_EPOCH
end
# convert from AD's time string to a Time object
def ad2time(time)
begin
thetime = Time.at((time.to_i - AD_EPOCH) / AD_MULTIPLIER)
return thetime
rescue
return 0
end
end
thedc = "myserver" # The "nearest" domain controler name
theaccount = 'domain\username' # Change this to your domain and username
thepassword = 'password' # Gota have a password
operating_systems = Array.new
w2000servers = Array.new
w2003servers = Array.new
winservers = Array.new
oldaccounts = Array.new
result = fetch_computers(thedc, theaccount, thepassword)
today = Time.now
result.each {|entry|
#printf "%s - %s\n", entry["cn"], entry["operatingSystem"]
thePasswordTime = ad2time(entry["pwdLastSet"].to_s)
theAge = (today - thePasswordTime) / 86400 # Number of days
if theAge > 90
oldaccounts << entry["cn"]
end
if entry["operatingSystem"] != nil
operating_systems << entry["operatingSystem"]
operating_systems.uniq!
if entry["operatingSystem"] == ["Windows 2000 Server"]
w2000servers << entry["cn"]
winservers << entry["cn"]
end
if entry["operatingSystem"] == ["Windows Server 2003"]
w2003servers << entry["cn"]
winservers << entry["cn"]
end
end
}
result = fetch_servers(thedc, theaccount, thepassword)
today = Time.now
result.each {|entry|
winservers << entry["cn"]
# The server AD entry does not contain the OS, so fetch the Servers ObjectType = Computer entry
myservers = fetch_computer(thedc,theaccount,thepassword,entry["cn"].to_s)
myservers.each { |myserver|
if myserver["operatingSystem"] == ["Windows 2000 Server"]
w2000servers << entry["cn"]
end
if myserver["operatingSystem"] == ["Windows Server 2003"]
w2003servers << entry["cn"]
end
}
}
w2000servers.uniq!
w2003servers.uniq!
winservers.uniq!
puts
pp operating_systems
puts
puts "Win2000 Servers "
pp w2000servers.sort!
puts
puts "Win2003 Servers "
pp w2003servers.sort!
puts "Windows Servers"
pp winservers.sort!
puts
puts "Old/Unused Computer Accounts"
pp oldaccounts.sort!
One thing, the "servers" dont always appear when we search for "computer" accounts, in AD. Didnt find a known bug for this, but servers I know are "out" there just didnt come back.
So a few more "gotcha" when using AD from Ruby/LDAP
1. Servers dont "come-back" from a search of objectype = Computer
2. "Server" objectype does not give you the OS. Must search for server, then find it specifically to get the OS.
3. You might have duplicates if you add these togeather.
4. AD Date and Time is very different format than ruby's, so thanks to google, I found some nice routeines for converting between the two.
Example code:
require 'ldap'
require "pp"
def fetch_computers(domain_controller,the_username, the_password)
conn = LDAP::Conn.new( domain_controller, 389 )
conn.set_option( LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )
conn.bind( the_username, the_password ) do |conn|
base = 'DC=company,DC=com'
results = conn.search2(base, LDAP::LDAP_SCOPE_SUBTREE, "objectClass=Computer")
if results == LDAP::LDAP_SIZELIMIT_EXCEEDED
printf "To Much Data\n"
end
return results
end
end
def fetch_servers(domain_controller,the_username, the_password)
conn = LDAP::Conn.new( domain_controller, 389 )
conn.set_option( LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )
conn.bind( the_username, the_password ) do |conn|
base = 'DC=company,DC=com'
results = conn.search2(base, LDAP::LDAP_SCOPE_SUBTREE, "objectClass=Server")
if results == LDAP::LDAP_SIZELIMIT_EXCEEDED
printf "To Much Data\n"
end
return results
end
end
def fetch_computer(domain_controller,the_username, the_password,thecomputername)
conn = LDAP::Conn.new( domain_controller, 389 )
conn.set_option( LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )
conn.bind( the_username, the_password )
base = 'DC=company,DC=com'
results = conn.search2(base, LDAP::LDAP_SCOPE_SUBTREE, "cn=#{thecomputername}")
return results
end
#[["Windows 2000 Server"],
# ["Windows Server 2003"],
# ["Windows XP Professional"],
# ["Windows 2000 Professional"],
# ["Windows NT"],
# ["Windows Vista\342\204\242 Business"],
# ["Windows XP Tablet PC Edition"]]
AD_EPOCH = 116_444_736_000_000_000
AD_MULTIPLIER = 10_000_000
# convert a Time object to AD's epoch
def time2ad(time)
(time.to_i * AD_MULTIPLIER) + AD_EPOCH
end
# convert from AD's time string to a Time object
def ad2time(time)
begin
thetime = Time.at((time.to_i - AD_EPOCH) / AD_MULTIPLIER)
return thetime
rescue
return 0
end
end
thedc = "myserver" # The "nearest" domain controler name
theaccount = 'domain\username' # Change this to your domain and username
thepassword = 'password' # Gota have a password
operating_systems = Array.new
w2000servers = Array.new
w2003servers = Array.new
winservers = Array.new
oldaccounts = Array.new
result = fetch_computers(thedc, theaccount, thepassword)
today = Time.now
result.each {|entry|
#printf "%s - %s\n", entry["cn"], entry["operatingSystem"]
thePasswordTime = ad2time(entry["pwdLastSet"].to_s)
theAge = (today - thePasswordTime) / 86400 # Number of days
if theAge > 90
oldaccounts << entry["cn"]
end
if entry["operatingSystem"] != nil
operating_systems << entry["operatingSystem"]
operating_systems.uniq!
if entry["operatingSystem"] == ["Windows 2000 Server"]
w2000servers << entry["cn"]
winservers << entry["cn"]
end
if entry["operatingSystem"] == ["Windows Server 2003"]
w2003servers << entry["cn"]
winservers << entry["cn"]
end
end
}
result = fetch_servers(thedc, theaccount, thepassword)
today = Time.now
result.each {|entry|
winservers << entry["cn"]
# The server AD entry does not contain the OS, so fetch the Servers ObjectType = Computer entry
myservers = fetch_computer(thedc,theaccount,thepassword,entry["cn"].to_s)
myservers.each { |myserver|
if myserver["operatingSystem"] == ["Windows 2000 Server"]
w2000servers << entry["cn"]
end
if myserver["operatingSystem"] == ["Windows Server 2003"]
w2003servers << entry["cn"]
end
}
}
w2000servers.uniq!
w2003servers.uniq!
winservers.uniq!
puts
pp operating_systems
puts
puts "Win2000 Servers "
pp w2000servers.sort!
puts
puts "Win2003 Servers "
pp w2003servers.sort!
puts "Windows Servers"
pp winservers.sort!
puts
puts "Old/Unused Computer Accounts"
pp oldaccounts.sort!
Monday, October 29, 2007
Ruby LDAP Example - How to find Computers
Ever want to find "all" your computers in a large "domain".
Having trouble find examples?
Some gochas from using ruby ldap
1. Make sure the Active Directory Domain Controller is on the same "network" segment as you. I've seen this not work if your going over a WAN. And no error message is given. This happend when I was doing my first Active Directory login authentication. Once I used a local domain controller everything was fine.
2. On the "ObjectClass" string extra parens and quotes cause it not to work. and no error message is given.
Here's a working one
require 'ldap'
require "pp"
conn = LDAP::Conn.new( 'sinsiw2k1', 389 )
conn.set_option( LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )
conn.bind( 'domain\myname', 'mypass' ) do |conn|
base = 'DC=company,DC=com'
results = conn.search2(base, LDAP::LDAP_SCOPE_SUBTREE, "objectClass=Computer")
if results == LDAP::LDAP_SIZELIMIT_EXCEEDED
printf "To Much Data\n"
end
pp results
pp results.controls
pp results.referrals
results.each { |entry| puts entry }
end
Having trouble find examples?
Some gochas from using ruby ldap
1. Make sure the Active Directory Domain Controller is on the same "network" segment as you. I've seen this not work if your going over a WAN. And no error message is given. This happend when I was doing my first Active Directory login authentication. Once I used a local domain controller everything was fine.
2. On the "ObjectClass" string extra parens and quotes cause it not to work. and no error message is given.
Here's a working one
require 'ldap'
require "pp"
conn = LDAP::Conn.new( 'sinsiw2k1', 389 )
conn.set_option( LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )
conn.bind( 'domain\myname', 'mypass' ) do |conn|
base = 'DC=company,DC=com'
results = conn.search2(base, LDAP::LDAP_SCOPE_SUBTREE, "objectClass=Computer")
if results == LDAP::LDAP_SIZELIMIT_EXCEEDED
printf "To Much Data\n"
end
pp results
pp results.controls
pp results.referrals
results.each { |entry| puts entry }
end
Monday, October 22, 2007
Importing from ODBC sources into Oracle and Rails
Having lots of data in a "third" party database, I needed a way of moving it over into a "rails" friendly environment. ODBC provided the "interface". This should work for most odbc/dbi compliant databases.
Note that you might have to work a bit on "dates".
require 'DBI'
require 'pp'
require 'active_support'
require 'fileutils'
def numeric?(thenumber)
if thenumber =~ /^[0-9]+$/
return(TRUE)
else return(FALSE)
end
end
def tableexists?(wantedtablename)
tables = $existing_tables
tables.each do |tablename|
if wantedtablename == tablename
return TRUE
end
end
return FALSE
end
def sanitize(thevalue)
thevalue.gsub(/[^[:alnum:]]+/, '_')
end
def XlateColumnName(thename)
thename = thename.delete(" ")
thename = thename.upcase
if numeric?(thename)
thename = "X" + thename
end
thename = sanitize(thename) # Get rid of / and \ and other goodies
if thename == "LOCK"
thename = "XLOCK"
end
if thename == "SCOPE"
thename = "XSCOPE"
end
if thename == "TYPE"
thename = "XTYPE"
end
if thename == "CLASS"
thename = "XCLASS"
end
if thename == "COST"
thename = "XCOST"
end
if thename == "FIXED"
thename = "XFIXED"
end
if thename == "CURRENT"
thename = "XCURRENT"
end
if thename == "DATE"
thename = "XDATE"
end
if thename == "TIME"
thename = "XTIME"
end
if thename == "NUMBER"
thename = "XNUMBER"
end
if thename == "CONDITION"
thename = "XCONDITION"
end
if thename == "SEQ"
thename = "XSEQ"
end
if thename == "COMMENT"
thename = "XCOMMENT"
end
return(thename)
end
def dotable(fromdb, todb, tablename, prefix)
tablename = tablename.strip
if tablename.length == 0
return
end
stmt = "select * from #{tablename};"
begin
tabledata = fromdb.execute(stmt)
rescue
return
end
row = tabledata.fetch # Get one
if row == nil #maybe we dont have data:)
tabledata.finish #on empty data dont bother to create it
return
end
tablename = sanitize(tablename) #Get rid of nasties
railsname = prefix + "_" + tablename
railsname = Inflector.pluralize(railsname)
railsname = railsname.upcase
if tableexists?(railsname)
printf "Exists %s\n",railsname
tabledata.finish
return
end
puts "Created #{railsname}"
#sqldrop = "DROP TABLE " + '"' + railsname + '" PURGE'
#begin
# result = todb.execute(sqldrop)
#rescue
# puts "Table [#{railsname}] not dropped"
# end
sqlcreate = String.new()
sqlcreate = "CREATE TABLE "
sqlcreate << '"'
sqlcreate << railsname
sqlcreate << '" ('
sqlcreate << '"ID" NUMBER(38,0) NOT NULL'
# Good reference for data conversion to oracle types
# http://www.indiana.edu/~dss/Services/DataWarehouse/Oracle/Sybase/conversion.html
fieldnames = Array.new
tabledata.column_info.each_with_index do |info, i|
sqlcreate << ", "
sqlcreate << '"'
thename = info["name"]
thename = XlateColumnName(thename)
sqlcreate << thename
sqlcreate << '" '
thetype = info["type_name"]
theprecision = info["precision"]
thescale = info["scale"]
if thetype == "CHAR"
thetype = "VARCHAR2(" + theprecision.to_s() + " BYTE)"
end
if thetype == "DOUBLE"
thetype = "NUMBER(38,0)"
end
if thetype == "BIT"
thetype = "CHAR(1)"
end
if thetype == "TINYINT"
thetype = "NUMBER(5,0)"
end
sqlcreate << thetype
fieldnames << thename
end
sqlcreate << ', PRIMARY KEY ("ID") '
sqlcreate << ")"
begin
result = todb.execute(sqlcreate)
rescue
puts "convertdata: Cannot create #{railsname}"
puts sqlcreate
raise
end
#Drop any "old" sequence
sqldrop = "DROP SEQUENCE " + '"' + railsname + '_SEQ"'
begin
result = todb.execute(sqldrop)
rescue
#puts sqldrop
#puts "Sequence [#{railsname}] not dropped"
end
#Now create the RAILS Sequence number
#CREATE SEQUENCE "AIRSTATION-DEVELOP"."BUSINESSUNITS_SEQ"
# MINVALUE 1 MAXVALUE 999999999999999999999999999 INCREMENT BY 1 START WITH 10040 CACHE 20 NOORDER NOCYCLE ;
sqlseq = String.new()
sqlseq << "CREATE SEQUENCE "
sqlseq << '"'
sqlseq << railsname
sqlseq << '_SEQ" '
sqlseq << "MAXVALUE 999999999999999999999999999 INCREMENT BY 1 START WITH 10000 CACHE 20 NOORDER NOCYCLE"
begin
result = todb.execute(sqlseq)
rescue
puts sqlseq
raise
end
if row == nil #maybe we dont have data:)
tabledata.finish
return
end
doinsert(todb,railsname,fieldnames,row) # We already "Fetched" One
tabledata.fetch do |row|
doinsert(todb,railsname,fieldnames,row)
end
tabledata.finish
end
#
def doinsert(thedb, tablename, fieldnames, row)
if row == nil
return
end
sqlinsert = String.new
sqlinsert = "INSERT INTO "
sqlinsert << '"'
sqlinsert << tablename
sqlinsert << '" ('
sqlinsert << "ID,"
fieldnames.each do |myfield|
sqlinsert << myfield + ","
end
#Must get rid of trailing comma and space
sqlinsert = sqlinsert[0,sqlinsert.length - 1]
sqlinsert << ') VALUES ('
sqlinsert << tablename + '_SEQ.NEXTVAL, '
row.each_with_name do |val, name|
val = val.to_s.strip
if val.length == 0
sqlinsert << "NULL,"
else
sqlinsert << thedb.quote(val.to_s.strip)
sqlinsert << ","
end
end
#Must get rid of trailing comma and space
sqlinsert = sqlinsert[0,sqlinsert.length - 1]
sqlinsert << ')'
begin
result = thedb.execute(sqlinsert)
rescue
puts sqlinsert
raise
end
end
argcount = ARGV.size
if argcount < 7
puts "command line is convertdata.rb FromDbDbi Username Password ToDbDbi Username Password Prefix"
puts " ie: convertdata.rb DBI:ODBC:PRODDATA . . DBI:OCI8:XE . . MAG"
puts " . = a empty string"
exit
end
fromdbconstr = ARGV[0]
fromdbconuser = ARGV[1]
fromdbconpass = ARGV[2]
todbconstr = ARGV[3]
todbconuser = ARGV[4]
todbconpass = ARGV[5]
prefix = ARGV[6]
if fromdbconuser == '.'
fromdbconuser = ''
end
if fromdbconpass == '.'
fromdbconpass = ''
end
if todbconuser == '.'
todbconuser = ''
end
if todbconpass == '.'
todbconpass = ''
end
printf "FromDB: [%s] User [%s] Password [%s]\n",fromdbconstr,fromdbconuser,fromdbconpass
printf "ToDB: [%s] User [%s] Password [%s]\n",todbconstr, todbconuser, todbconpass
printf "Prefix: [%s]\n", prefix
#make a ODBC Connection
fromdb = DBI.connect(fromdbconstr,fromdbconuser,fromdbconpass);
todb = DBI.connect(todbconstr,todbconuser,todbconpass)
#fromdb = DBI.connect('DBI:ODBC:PRODDATA','','')
#todb = DBI.connect('DBI:OCI8:XE', 'AIRSTATION-DEVELOP', 'AIRSTATION-DEVELOP')
#prefix = "MAG"
tables = fromdb.tables
$existing_tables = todb.tables
tables.each do |tablename|
dotable(fromdb, todb, tablename, prefix)
end
Note that you might have to work a bit on "dates".
require 'DBI'
require 'pp'
require 'active_support'
require 'fileutils'
def numeric?(thenumber)
if thenumber =~ /^[0-9]+$/
return(TRUE)
else return(FALSE)
end
end
def tableexists?(wantedtablename)
tables = $existing_tables
tables.each do |tablename|
if wantedtablename == tablename
return TRUE
end
end
return FALSE
end
def sanitize(thevalue)
thevalue.gsub(/[^[:alnum:]]+/, '_')
end
def XlateColumnName(thename)
thename = thename.delete(" ")
thename = thename.upcase
if numeric?(thename)
thename = "X" + thename
end
thename = sanitize(thename) # Get rid of / and \ and other goodies
if thename == "LOCK"
thename = "XLOCK"
end
if thename == "SCOPE"
thename = "XSCOPE"
end
if thename == "TYPE"
thename = "XTYPE"
end
if thename == "CLASS"
thename = "XCLASS"
end
if thename == "COST"
thename = "XCOST"
end
if thename == "FIXED"
thename = "XFIXED"
end
if thename == "CURRENT"
thename = "XCURRENT"
end
if thename == "DATE"
thename = "XDATE"
end
if thename == "TIME"
thename = "XTIME"
end
if thename == "NUMBER"
thename = "XNUMBER"
end
if thename == "CONDITION"
thename = "XCONDITION"
end
if thename == "SEQ"
thename = "XSEQ"
end
if thename == "COMMENT"
thename = "XCOMMENT"
end
return(thename)
end
def dotable(fromdb, todb, tablename, prefix)
tablename = tablename.strip
if tablename.length == 0
return
end
stmt = "select * from #{tablename};"
begin
tabledata = fromdb.execute(stmt)
rescue
return
end
row = tabledata.fetch # Get one
if row == nil #maybe we dont have data:)
tabledata.finish #on empty data dont bother to create it
return
end
tablename = sanitize(tablename) #Get rid of nasties
railsname = prefix + "_" + tablename
railsname = Inflector.pluralize(railsname)
railsname = railsname.upcase
if tableexists?(railsname)
printf "Exists %s\n",railsname
tabledata.finish
return
end
puts "Created #{railsname}"
#sqldrop = "DROP TABLE " + '"' + railsname + '" PURGE'
#begin
# result = todb.execute(sqldrop)
#rescue
# puts "Table [#{railsname}] not dropped"
# end
sqlcreate = String.new()
sqlcreate = "CREATE TABLE "
sqlcreate << '"'
sqlcreate << railsname
sqlcreate << '" ('
sqlcreate << '"ID" NUMBER(38,0) NOT NULL'
# Good reference for data conversion to oracle types
# http://www.indiana.edu/~dss/Services/DataWarehouse/Oracle/Sybase/conversion.html
fieldnames = Array.new
tabledata.column_info.each_with_index do |info, i|
sqlcreate << ", "
sqlcreate << '"'
thename = info["name"]
thename = XlateColumnName(thename)
sqlcreate << thename
sqlcreate << '" '
thetype = info["type_name"]
theprecision = info["precision"]
thescale = info["scale"]
if thetype == "CHAR"
thetype = "VARCHAR2(" + theprecision.to_s() + " BYTE)"
end
if thetype == "DOUBLE"
thetype = "NUMBER(38,0)"
end
if thetype == "BIT"
thetype = "CHAR(1)"
end
if thetype == "TINYINT"
thetype = "NUMBER(5,0)"
end
sqlcreate << thetype
fieldnames << thename
end
sqlcreate << ', PRIMARY KEY ("ID") '
sqlcreate << ")"
begin
result = todb.execute(sqlcreate)
rescue
puts "convertdata: Cannot create #{railsname}"
puts sqlcreate
raise
end
#Drop any "old" sequence
sqldrop = "DROP SEQUENCE " + '"' + railsname + '_SEQ"'
begin
result = todb.execute(sqldrop)
rescue
#puts sqldrop
#puts "Sequence [#{railsname}] not dropped"
end
#Now create the RAILS Sequence number
#CREATE SEQUENCE "AIRSTATION-DEVELOP"."BUSINESSUNITS_SEQ"
# MINVALUE 1 MAXVALUE 999999999999999999999999999 INCREMENT BY 1 START WITH 10040 CACHE 20 NOORDER NOCYCLE ;
sqlseq = String.new()
sqlseq << "CREATE SEQUENCE "
sqlseq << '"'
sqlseq << railsname
sqlseq << '_SEQ" '
sqlseq << "MAXVALUE 999999999999999999999999999 INCREMENT BY 1 START WITH 10000 CACHE 20 NOORDER NOCYCLE"
begin
result = todb.execute(sqlseq)
rescue
puts sqlseq
raise
end
if row == nil #maybe we dont have data:)
tabledata.finish
return
end
doinsert(todb,railsname,fieldnames,row) # We already "Fetched" One
tabledata.fetch do |row|
doinsert(todb,railsname,fieldnames,row)
end
tabledata.finish
end
#
def doinsert(thedb, tablename, fieldnames, row)
if row == nil
return
end
sqlinsert = String.new
sqlinsert = "INSERT INTO "
sqlinsert << '"'
sqlinsert << tablename
sqlinsert << '" ('
sqlinsert << "ID,"
fieldnames.each do |myfield|
sqlinsert << myfield + ","
end
#Must get rid of trailing comma and space
sqlinsert = sqlinsert[0,sqlinsert.length - 1]
sqlinsert << ') VALUES ('
sqlinsert << tablename + '_SEQ.NEXTVAL, '
row.each_with_name do |val, name|
val = val.to_s.strip
if val.length == 0
sqlinsert << "NULL,"
else
sqlinsert << thedb.quote(val.to_s.strip)
sqlinsert << ","
end
end
#Must get rid of trailing comma and space
sqlinsert = sqlinsert[0,sqlinsert.length - 1]
sqlinsert << ')'
begin
result = thedb.execute(sqlinsert)
rescue
puts sqlinsert
raise
end
end
argcount = ARGV.size
if argcount < 7
puts "command line is convertdata.rb FromDbDbi Username Password ToDbDbi Username Password Prefix"
puts " ie: convertdata.rb DBI:ODBC:PRODDATA . . DBI:OCI8:XE . . MAG"
puts " . = a empty string"
exit
end
fromdbconstr = ARGV[0]
fromdbconuser = ARGV[1]
fromdbconpass = ARGV[2]
todbconstr = ARGV[3]
todbconuser = ARGV[4]
todbconpass = ARGV[5]
prefix = ARGV[6]
if fromdbconuser == '.'
fromdbconuser = ''
end
if fromdbconpass == '.'
fromdbconpass = ''
end
if todbconuser == '.'
todbconuser = ''
end
if todbconpass == '.'
todbconpass = ''
end
printf "FromDB: [%s] User [%s] Password [%s]\n",fromdbconstr,fromdbconuser,fromdbconpass
printf "ToDB: [%s] User [%s] Password [%s]\n",todbconstr, todbconuser, todbconpass
printf "Prefix: [%s]\n", prefix
#make a ODBC Connection
fromdb = DBI.connect(fromdbconstr,fromdbconuser,fromdbconpass);
todb = DBI.connect(todbconstr,todbconuser,todbconpass)
#fromdb = DBI.connect('DBI:ODBC:PRODDATA','','')
#todb = DBI.connect('DBI:OCI8:XE', 'AIRSTATION-DEVELOP', 'AIRSTATION-DEVELOP')
#prefix = "MAG"
tables = fromdb.tables
$existing_tables = todb.tables
tables.each do |tablename|
dotable(fromdb, todb, tablename, prefix)
end
Thursday, October 18, 2007
Import and Cleanup Dbase Files into Rails with Oracle Support
Wow, so many files, and how to clean them up.
I can do migrations (See my previous post) with DBF,
and do some nice cleanup as I go, but I really want to "get"
rid of the dbase files, and make them native as fast as possible.
So finally wrote a dbf-dbi-to-oracle converter.
The issues you get with dbase files into rails are numerous. Lack
of ID field, and names that are reserved words in SQL.
Even "splits" of tables into multiple files is resonbly common.
And worse the "schema" could change over the life of the dbf files,
with some "dbf" files having one schema, and newer ones having signficant
new fields.
So here is the solution to all of this.
A importdbf script
require 'DBI'
require 'pp'
require 'active_support'
require 'fileutils'
require 'dbf'
require 'date'
def sanitize(thevalue)
thevalue.gsub(/[^[:alnum:]]+/, '_')
end
def numeric?(thenumber)
if thenumber =~ /^[0-9]+$/
return(TRUE)
else return(FALSE)
end
end
def tableexists?(wantedtablename)
tables = $existing_tables
tables.each do |tablename|
if wantedtablename == tablename
return TRUE
end
end
return FALSE
end
def FileExtension(thepath)
thedot = thepath.rindex('.')
thelen = thepath.length - thedot
theextension = thepath[thedot,thelen]
theextension = theextension.downcase
return theextension
end
#
def doinsert(fromdb, thedb, tablename, fieldnames, row)
if row == nil
return
end
sqlinsert = String.new
sqlinsert = "INSERT INTO "
sqlinsert << '"'
sqlinsert << tablename
sqlinsert << '" ('
sqlinsert << "ID,"
fieldnames.each do |myfield|
sqlinsert << myfield + ",\n"
end
#Must get rid of trailing comma and space
sqlinsert = sqlinsert[0,sqlinsert.length - 2]
sqlinsert << ') VALUES ('
sqlinsert << tablename + '_SEQ.NEXTVAL, '
fromdb.columns.each_with_index do |info, i|
val = row.attributes[info.name].to_s
if info.type == "D" #DATE
# "Mon Jan 01 00:00:00 UTC 2001" - dbf date
dbfformat = "%a %b %d %H:%M:%S %Z %Y"
#to_date('12-31-2007 12:15','MM-DD-YYYY HH:MI)
if val != ""
thedate = Date.parse(val)
val = "to_date('" + thedate.strftime("%m-%d-%Y %H:%M") + "', 'MM-DD-YYYY HH24:MI')"
sqlinsert << val
else
sqlinsert << "NULL"
end
elsif info.type == "M"
sqlinsert << thedb.quote(val.to_s.strip)
elsif info.type == "L"
if val == "true"
val = "Y"
end
if val == "false"
val = "N"
end
sqlinsert << thedb.quote(val.to_s.strip)
else
sqlinsert << thedb.quote(val.to_s.strip)
end
# printf "Name = %s Type = %s Value = %s\n", info.name, info.type, val
sqlinsert << ",\n"
end
#Must get rid of trailing comma and space
sqlinsert = sqlinsert[0,sqlinsert.length - 2]
sqlinsert << ')'
begin
# puts sqlinsert
tabledata = thedb.execute(sqlinsert)
rescue
puts sqlinsert
thesql = File.open("convertdbf.sql","w")
thesql.puts sqlinsert + ";"
thesql.close
raise
end
tabledata.finish
end
def ImportTable(fromdb,todb,railsname,fieldnames)
fromdb.records.each do |record|
doinsert(fromdb, todb,railsname,fieldnames,record)
end
end
def XlateColumnName(thename,thetype)
thename = thename.delete(" ")
thename = thename.upcase
if thetype == "D" #DATE Should have it in the name
if !thename.include?("DATE")
thename << "DATE"
end
end
if numeric?(thename)
thename = "X" + thename
end
if thename == "FILE"
thename = "XFILE"
end
if thename == "LOCK"
thename = "XLOCK"
end
if thename == "SCOPE"
thename = "XSCOPE"
end
if thename == "TYPE"
thename = "XTYPE"
end
if thename == "CLASS"
thename = "XCLASS"
end
if thename == "COST"
thename = "XCOST"
end
if thename == "FIXED"
thename = "XFIXED"
end
if thename == "CURRENT"
thename = "XCURRENT"
end
if thename == "DATE"
thename = "XDATE"
end
if thename == "TIME"
thename = "XTIME"
end
if thename == "NUMBER"
thename = "XNUMBER"
end
if thename == "CONDITION"
thename = "XCONDITION"
end
if thename == "SEQ"
thename = "XSEQ"
end
if thename == "COMMENT"
thename = "XCOMMENT"
end
return(thename)
end
def XlateColumntype(thetype, theprecision, thescale)
if thetype == "C"
thetype = "VARCHAR2(" + theprecision.to_s() + " BYTE)"
end
if thetype == "L" #Logical - false or true
thetype = "CHAR(1)"
end
if thetype == "N"
thetype = "NUMBER(38,0)"
end
if thetype == "D"
thetype = "DATE"
end
if thetype == "B"
thetype = "CHAR(1)"
end
if thetype == "M"
thetype = "CLOB"
end
return(thetype)
end
def CreateTable(fromdb,todb,railsname)
sqlcreate = String.new()
sqlcreate = "CREATE TABLE "
sqlcreate << '"'
sqlcreate << railsname
sqlcreate << '" ('
sqlcreate << '"ID" NUMBER(38,0) NOT NULL'
# Good reference for data conversion to oracle types
# http://www.indiana.edu/~dss/Services/DataWarehouse/Oracle/Sybase/conversion.html
fieldnames = Array.new
fromdb.columns.each_with_index do |info, i|
sqlcreate << ", "
sqlcreate << '"'
thename = XlateColumnName(info.name,info.type)
sqlcreate << thename
sqlcreate << '" '
thetype = info.type
theprecision = info.length
thescale = info.decimal
thetype = XlateColumntype(thetype,theprecision, thescale)
sqlcreate << thetype
fieldnames << thename
end
sqlcreate << ', PRIMARY KEY ("ID") '
sqlcreate << ")"
begin
tabledata = todb.execute(sqlcreate)
rescue
puts "convertdata: Cannot create #{railsname}"
thesql = File.open("convertdbf.sql","w")
thesql.puts sqlcreate + ";"
thesql.close
puts sqlcreate
raise
end
tabledata.finish
#Drop any "old" sequence
sqldrop = "DROP SEQUENCE " + '"' + railsname + '_SEQ"'
begin
tabledata = todb.execute(sqldrop)
tabledata.finish
rescue
#puts sqldrop
#puts "Sequence [#{railsname}] not dropped"
end
#Now create the RAILS Sequence number
#CREATE SEQUENCE "AIRSTATION-DEVELOP"."BUSINESSUNITS_SEQ"
# MINVALUE 1 MAXVALUE 999999999999999999999999999 INCREMENT BY 1 START WITH 10040 CACHE 20 NOORDER NOCYCLE ;
sqlseq = String.new()
sqlseq << "CREATE SEQUENCE "
sqlseq << '"'
sqlseq << railsname
sqlseq << '_SEQ" '
sqlseq << "MAXVALUE 999999999999999999999999999 INCREMENT BY 1 START WITH 10000 CACHE 20 NOORDER NOCYCLE"
begin
tabledata = todb.execute(sqlseq)
tabledata.finish
rescue
puts sqlseq
raise
end
$existing_tables << railsname
return(fieldnames)
end
def create_master_content_table(thedb,mastercontent_table)
sqlcreate = "CREATE TABLE " + mastercontent_table + "( SUBFILENAME VARCHAR2(32) )"
begin
tabledata = thedb.execute(sqlcreate)
rescue
puts sqlcreate
raise
end
tabledata.finish
$existing_tables << mastercontent_table
end
def is_in_mastertable?(thedb, mastertable,thename)
mastercontent_table = mastertable + "_FILES"
if tableexists?(mastercontent_table) == FALSE
create_master_content_table(thedb,mastercontent_table)
return(FALSE)
end
sqlquery = "SELECT * FROM " + mastercontent_table +
" WHERE SUBFILENAME = " +
thedb.quote(thename.to_s.strip)
tabledata = thedb.execute(sqlquery)
row = tabledata.fetch
if row == nil
tabledata.finish
return(FALSE)
end
tabledata.finish
return TRUE
end
def add_to_file_table(thedb, mastertable, thefilename )
sqlinsert = "INSERT INTO " + mastertable + "_FILES " + " (SUBFILENAME) VALUES ("
sqlinsert << thedb.quote(thefilename) + ")"
begin
tabledata = thedb.execute(sqlinsert)
rescue
puts sqlinsert
raise
end
tabledata.finish
end
def DescribeTable(todb,railsname)
fieldnames = Array.new
sqlquery = "SELECT * FROM " + railsname
tabledata = todb.execute(sqlquery)
for column in tabledata.column_info()
if column['name'] == "ID" # We "assume" that ID is present
next
end
fieldnames << column['name']
end
tabledata.finish
return(fieldnames)
end
def UpdateTableDef(fieldnames,fromdb,todb,railsname)
#alter table author add (author_last_published date);
sqlmodify = "ALTER TABLE "
sqlmodify << '"'
sqlmodify << railsname
sqlmodify << '" ADD ('
thistablefields = Array.new
newfields = Array.new
fromdb.columns.each_with_index do |info, index|
thename = XlateColumnName(info.name,info.type)
thistablefields << thename
if fieldnames.include?(thename)
next
end
newfields << thename
sqlmodify << '"'
sqlmodify << thename
sqlmodify << '" '
thetype = info.type
theprecision = info.length
thescale = info.decimal
thetype = XlateColumntype(thetype,theprecision, thescale)
sqlmodify << thetype
sqlmodify << ", "
fieldnames << thename
end
if newfields.empty? # Table definition is identical
return(thistablefields)
end
# pp newfields
#Must get rid of trailing comma and space
sqlmodify = sqlmodify[0,sqlmodify.length - 2]
sqlmodify << ")"
begin
tabledata = todb.execute(sqlmodify)
rescue
puts "convertdata: Cannot modify #{railsname}"
thesql = File.open("convertdbf.sql","w")
thesql.puts sqlmodify + ";"
thesql.close
puts sqlmodify
raise
end
tabledata.finish
return(thistablefields)
end
def ProcessFile(thename, thepath,todb,prefix,mastertable)
filetype = FileExtension(thepath)
if filetype != ".dbf"
return
end
tablename = thename.slice(0,thename.length - 4)
tablename = sanitize(tablename)
if numeric?(tablename)
tablename = mastertable
railsname = Inflector.pluralize(tablename)
railsname = railsname.upcase
if is_in_mastertable?(todb,mastertable,thename)
return
end
add_to_file_table(todb,mastertable,thename)
puts "Adding #{thename} to #{railsname}"
else
tablename = prefix + '_' + tablename
tablename = tablename.upcase
railsname = Inflector.pluralize(tablename)
railsname = railsname.upcase
if tableexists?(railsname)
printf "Exists %s\n",railsname
return
end
end
fromdb = DBF::Table.new(thepath,:in_memory => false)
if tableexists?(railsname) == FALSE
fieldnames = CreateTable(fromdb,todb,railsname)
puts "Created #{railsname}"
else
fieldnames = DescribeTable(todb,railsname)
fieldnames = UpdateTableDef(fieldnames,fromdb,todb,railsname)
end
ImportTable(fromdb,todb,railsname,fieldnames)
fromdb = nil
end
def DoDir(mydirectory,todb,prefix,mastertable)
Dir.foreach(mydirectory) do |entry|
# Dont bother with . and ..
next if [".",".."].include? entry
fullname = mydirectory + "\\" + entry
if FileTest::directory?(fullname)
DoDir(fullname,todb,prefix)
elsif FileTest::file?(fullname)
ProcessFile(entry,fullname,todb,prefix,mastertable)
else
puts "Unknown file type - #{fullname}"
return;
end
end
end
argcount = ARGV.size
if argcount < 6
puts "command line is convertdbf.rb FromPath ToDbDbi Username Password Prefix"
exit
end
fromdir = ARGV[0]
todbconstr = ARGV[1]
todbconuser = ARGV[2]
todbconpass = ARGV[3]
prefix = ARGV[4]
mastertable = ARGV[5]
if todbconuser == '*'
todbconuser = ''
end
if todbconpass == '*'
todbconpass = ''
end
#make a ODBC Connection
todb = DBI.connect(todbconstr,todbconuser,todbconpass)
$existing_tables = todb.tables
DoDir(fromdir,todb,prefix,mastertable)
I can do migrations (See my previous post) with DBF,
and do some nice cleanup as I go, but I really want to "get"
rid of the dbase files, and make them native as fast as possible.
So finally wrote a dbf-dbi-to-oracle converter.
The issues you get with dbase files into rails are numerous. Lack
of ID field, and names that are reserved words in SQL.
Even "splits" of tables into multiple files is resonbly common.
And worse the "schema" could change over the life of the dbf files,
with some "dbf" files having one schema, and newer ones having signficant
new fields.
So here is the solution to all of this.
A importdbf script
require 'DBI'
require 'pp'
require 'active_support'
require 'fileutils'
require 'dbf'
require 'date'
def sanitize(thevalue)
thevalue.gsub(/[^[:alnum:]]+/, '_')
end
def numeric?(thenumber)
if thenumber =~ /^[0-9]+$/
return(TRUE)
else return(FALSE)
end
end
def tableexists?(wantedtablename)
tables = $existing_tables
tables.each do |tablename|
if wantedtablename == tablename
return TRUE
end
end
return FALSE
end
def FileExtension(thepath)
thedot = thepath.rindex('.')
thelen = thepath.length - thedot
theextension = thepath[thedot,thelen]
theextension = theextension.downcase
return theextension
end
#
def doinsert(fromdb, thedb, tablename, fieldnames, row)
if row == nil
return
end
sqlinsert = String.new
sqlinsert = "INSERT INTO "
sqlinsert << '"'
sqlinsert << tablename
sqlinsert << '" ('
sqlinsert << "ID,"
fieldnames.each do |myfield|
sqlinsert << myfield + ",\n"
end
#Must get rid of trailing comma and space
sqlinsert = sqlinsert[0,sqlinsert.length - 2]
sqlinsert << ') VALUES ('
sqlinsert << tablename + '_SEQ.NEXTVAL, '
fromdb.columns.each_with_index do |info, i|
val = row.attributes[info.name].to_s
if info.type == "D" #DATE
# "Mon Jan 01 00:00:00 UTC 2001" - dbf date
dbfformat = "%a %b %d %H:%M:%S %Z %Y"
#to_date('12-31-2007 12:15','MM-DD-YYYY HH:MI)
if val != ""
thedate = Date.parse(val)
val = "to_date('" + thedate.strftime("%m-%d-%Y %H:%M") + "', 'MM-DD-YYYY HH24:MI')"
sqlinsert << val
else
sqlinsert << "NULL"
end
elsif info.type == "M"
sqlinsert << thedb.quote(val.to_s.strip)
elsif info.type == "L"
if val == "true"
val = "Y"
end
if val == "false"
val = "N"
end
sqlinsert << thedb.quote(val.to_s.strip)
else
sqlinsert << thedb.quote(val.to_s.strip)
end
# printf "Name = %s Type = %s Value = %s\n", info.name, info.type, val
sqlinsert << ",\n"
end
#Must get rid of trailing comma and space
sqlinsert = sqlinsert[0,sqlinsert.length - 2]
sqlinsert << ')'
begin
# puts sqlinsert
tabledata = thedb.execute(sqlinsert)
rescue
puts sqlinsert
thesql = File.open("convertdbf.sql","w")
thesql.puts sqlinsert + ";"
thesql.close
raise
end
tabledata.finish
end
def ImportTable(fromdb,todb,railsname,fieldnames)
fromdb.records.each do |record|
doinsert(fromdb, todb,railsname,fieldnames,record)
end
end
def XlateColumnName(thename,thetype)
thename = thename.delete(" ")
thename = thename.upcase
if thetype == "D" #DATE Should have it in the name
if !thename.include?("DATE")
thename << "DATE"
end
end
if numeric?(thename)
thename = "X" + thename
end
if thename == "FILE"
thename = "XFILE"
end
if thename == "LOCK"
thename = "XLOCK"
end
if thename == "SCOPE"
thename = "XSCOPE"
end
if thename == "TYPE"
thename = "XTYPE"
end
if thename == "CLASS"
thename = "XCLASS"
end
if thename == "COST"
thename = "XCOST"
end
if thename == "FIXED"
thename = "XFIXED"
end
if thename == "CURRENT"
thename = "XCURRENT"
end
if thename == "DATE"
thename = "XDATE"
end
if thename == "TIME"
thename = "XTIME"
end
if thename == "NUMBER"
thename = "XNUMBER"
end
if thename == "CONDITION"
thename = "XCONDITION"
end
if thename == "SEQ"
thename = "XSEQ"
end
if thename == "COMMENT"
thename = "XCOMMENT"
end
return(thename)
end
def XlateColumntype(thetype, theprecision, thescale)
if thetype == "C"
thetype = "VARCHAR2(" + theprecision.to_s() + " BYTE)"
end
if thetype == "L" #Logical - false or true
thetype = "CHAR(1)"
end
if thetype == "N"
thetype = "NUMBER(38,0)"
end
if thetype == "D"
thetype = "DATE"
end
if thetype == "B"
thetype = "CHAR(1)"
end
if thetype == "M"
thetype = "CLOB"
end
return(thetype)
end
def CreateTable(fromdb,todb,railsname)
sqlcreate = String.new()
sqlcreate = "CREATE TABLE "
sqlcreate << '"'
sqlcreate << railsname
sqlcreate << '" ('
sqlcreate << '"ID" NUMBER(38,0) NOT NULL'
# Good reference for data conversion to oracle types
# http://www.indiana.edu/~dss/Services/DataWarehouse/Oracle/Sybase/conversion.html
fieldnames = Array.new
fromdb.columns.each_with_index do |info, i|
sqlcreate << ", "
sqlcreate << '"'
thename = XlateColumnName(info.name,info.type)
sqlcreate << thename
sqlcreate << '" '
thetype = info.type
theprecision = info.length
thescale = info.decimal
thetype = XlateColumntype(thetype,theprecision, thescale)
sqlcreate << thetype
fieldnames << thename
end
sqlcreate << ', PRIMARY KEY ("ID") '
sqlcreate << ")"
begin
tabledata = todb.execute(sqlcreate)
rescue
puts "convertdata: Cannot create #{railsname}"
thesql = File.open("convertdbf.sql","w")
thesql.puts sqlcreate + ";"
thesql.close
puts sqlcreate
raise
end
tabledata.finish
#Drop any "old" sequence
sqldrop = "DROP SEQUENCE " + '"' + railsname + '_SEQ"'
begin
tabledata = todb.execute(sqldrop)
tabledata.finish
rescue
#puts sqldrop
#puts "Sequence [#{railsname}] not dropped"
end
#Now create the RAILS Sequence number
#CREATE SEQUENCE "AIRSTATION-DEVELOP"."BUSINESSUNITS_SEQ"
# MINVALUE 1 MAXVALUE 999999999999999999999999999 INCREMENT BY 1 START WITH 10040 CACHE 20 NOORDER NOCYCLE ;
sqlseq = String.new()
sqlseq << "CREATE SEQUENCE "
sqlseq << '"'
sqlseq << railsname
sqlseq << '_SEQ" '
sqlseq << "MAXVALUE 999999999999999999999999999 INCREMENT BY 1 START WITH 10000 CACHE 20 NOORDER NOCYCLE"
begin
tabledata = todb.execute(sqlseq)
tabledata.finish
rescue
puts sqlseq
raise
end
$existing_tables << railsname
return(fieldnames)
end
def create_master_content_table(thedb,mastercontent_table)
sqlcreate = "CREATE TABLE " + mastercontent_table + "( SUBFILENAME VARCHAR2(32) )"
begin
tabledata = thedb.execute(sqlcreate)
rescue
puts sqlcreate
raise
end
tabledata.finish
$existing_tables << mastercontent_table
end
def is_in_mastertable?(thedb, mastertable,thename)
mastercontent_table = mastertable + "_FILES"
if tableexists?(mastercontent_table) == FALSE
create_master_content_table(thedb,mastercontent_table)
return(FALSE)
end
sqlquery = "SELECT * FROM " + mastercontent_table +
" WHERE SUBFILENAME = " +
thedb.quote(thename.to_s.strip)
tabledata = thedb.execute(sqlquery)
row = tabledata.fetch
if row == nil
tabledata.finish
return(FALSE)
end
tabledata.finish
return TRUE
end
def add_to_file_table(thedb, mastertable, thefilename )
sqlinsert = "INSERT INTO " + mastertable + "_FILES " + " (SUBFILENAME) VALUES ("
sqlinsert << thedb.quote(thefilename) + ")"
begin
tabledata = thedb.execute(sqlinsert)
rescue
puts sqlinsert
raise
end
tabledata.finish
end
def DescribeTable(todb,railsname)
fieldnames = Array.new
sqlquery = "SELECT * FROM " + railsname
tabledata = todb.execute(sqlquery)
for column in tabledata.column_info()
if column['name'] == "ID" # We "assume" that ID is present
next
end
fieldnames << column['name']
end
tabledata.finish
return(fieldnames)
end
def UpdateTableDef(fieldnames,fromdb,todb,railsname)
#alter table author add (author_last_published date);
sqlmodify = "ALTER TABLE "
sqlmodify << '"'
sqlmodify << railsname
sqlmodify << '" ADD ('
thistablefields = Array.new
newfields = Array.new
fromdb.columns.each_with_index do |info, index|
thename = XlateColumnName(info.name,info.type)
thistablefields << thename
if fieldnames.include?(thename)
next
end
newfields << thename
sqlmodify << '"'
sqlmodify << thename
sqlmodify << '" '
thetype = info.type
theprecision = info.length
thescale = info.decimal
thetype = XlateColumntype(thetype,theprecision, thescale)
sqlmodify << thetype
sqlmodify << ", "
fieldnames << thename
end
if newfields.empty? # Table definition is identical
return(thistablefields)
end
# pp newfields
#Must get rid of trailing comma and space
sqlmodify = sqlmodify[0,sqlmodify.length - 2]
sqlmodify << ")"
begin
tabledata = todb.execute(sqlmodify)
rescue
puts "convertdata: Cannot modify #{railsname}"
thesql = File.open("convertdbf.sql","w")
thesql.puts sqlmodify + ";"
thesql.close
puts sqlmodify
raise
end
tabledata.finish
return(thistablefields)
end
def ProcessFile(thename, thepath,todb,prefix,mastertable)
filetype = FileExtension(thepath)
if filetype != ".dbf"
return
end
tablename = thename.slice(0,thename.length - 4)
tablename = sanitize(tablename)
if numeric?(tablename)
tablename = mastertable
railsname = Inflector.pluralize(tablename)
railsname = railsname.upcase
if is_in_mastertable?(todb,mastertable,thename)
return
end
add_to_file_table(todb,mastertable,thename)
puts "Adding #{thename} to #{railsname}"
else
tablename = prefix + '_' + tablename
tablename = tablename.upcase
railsname = Inflector.pluralize(tablename)
railsname = railsname.upcase
if tableexists?(railsname)
printf "Exists %s\n",railsname
return
end
end
fromdb = DBF::Table.new(thepath,:in_memory => false)
if tableexists?(railsname) == FALSE
fieldnames = CreateTable(fromdb,todb,railsname)
puts "Created #{railsname}"
else
fieldnames = DescribeTable(todb,railsname)
fieldnames = UpdateTableDef(fieldnames,fromdb,todb,railsname)
end
ImportTable(fromdb,todb,railsname,fieldnames)
fromdb = nil
end
def DoDir(mydirectory,todb,prefix,mastertable)
Dir.foreach(mydirectory) do |entry|
# Dont bother with . and ..
next if [".",".."].include? entry
fullname = mydirectory + "\\" + entry
if FileTest::directory?(fullname)
DoDir(fullname,todb,prefix)
elsif FileTest::file?(fullname)
ProcessFile(entry,fullname,todb,prefix,mastertable)
else
puts "Unknown file type - #{fullname}"
return;
end
end
end
argcount = ARGV.size
if argcount < 6
puts "command line is convertdbf.rb FromPath ToDbDbi Username Password Prefix"
exit
end
fromdir = ARGV[0]
todbconstr = ARGV[1]
todbconuser = ARGV[2]
todbconpass = ARGV[3]
prefix = ARGV[4]
mastertable = ARGV[5]
if todbconuser == '*'
todbconuser = ''
end
if todbconpass == '*'
todbconpass = ''
end
#make a ODBC Connection
todb = DBI.connect(todbconstr,todbconuser,todbconpass)
$existing_tables = todb.tables
DoDir(fromdir,todb,prefix,mastertable)
A Production generator for Rails
As I get into Rails, I'm generating "large" applications, and I really like the RAILS DRY principle. So for production I'm using a variety of "plug-ins"
ActiveScaffold
RoleRequirement
Widgets/Tabnav
In addition I'm using a Oracle Database.
So I wrote a generator that "reads" the Oracle Database, and generates any
"missing" controllers and models. The lovely part is, when I change the basic
controller, I just "erase" the ones I want updated, and fire the script again.
This also handles the pluraziation, making sure the controllers, models are named the same. Also tries to do some cleanup so the menu names are a little more humanized. In addition good example of using name spaces to help organize a large project better.
require 'DBI'
require 'pp'
require 'active_support'
require 'fileutils'
def domodel(tablename)
modelname = tablename.downcase
modelname = modelname.singularize
modeldir = "app\\models\\"
FileUtils.makedirs(modeldir)
modelpath = modeldir + modelname + ".rb"
if FileTest::exists?(modelpath)
printf "exists %s\n", modelpath
return
end
themodel = File.open(modelpath,"w")
themodel.puts "class " + modelname.camelize + " < ActiveRecord::Base"
themodel.puts "# "
themodel.puts "# Created By GenRails"
themodel.puts "# "
themodel.puts "end"
themodel.close
printf "create %s\n", modelpath
end
def docontroller(thenamespace,tablename, layout, prefix)
controllername = tablename.downcase
controllername = controllername.singularize
controllerfilename = controllername + "_controller.rb"
controllerdir = "app\\controllers\\" + thenamespace + "\\"
FileUtils.makedirs(controllerdir)
controllerpath = controllerdir + controllerfilename
if FileTest::exists?(controllerpath)
printf "exists %s\n", controllerpath
return
end
thecontroller = File.open(controllerpath,"w")
thecontroller.puts "class " + thenamespace.camelize + '::' + controllername.camelize + 'Controller < ApplicationController'
thecontroller.puts " layout '" + layout + "'"
thecontroller.puts " before_filter :login_required"
thecontroller.puts "# "
thecontroller.puts "# Generated by GenRails"
thecontroller.puts "# "
thecontroller.puts " "
thecontroller.puts "active_scaffold :" + controllername.camelize + " do |config|"
menuname = tablename.slice(prefix.length+1,tablename.length - prefix.length + 1)
menuname = menuname.camelize
thecontroller.puts " config.label = '" + menuname + "'"
thecontroller.puts " config.actions = [:search, :show, :list, :create, :delete]"
thecontroller.puts " end"
thecontroller.puts "end"
thecontroller.close
printf "create %s\n", controllerpath
end
def dotable(fromdb, namespace, tablename, layout, prefix)
tablename = tablename.strip
if tablename.length == 0
return
end
# begin
# row = fromdb.select_one("select count(*) from #{tablename}")
# rescue
# return
# end
# record_count = row[0]
# if record_count == 0
# return
# end
domodel(tablename)
docontroller(namespace,tablename, layout, prefix)
end
#make a ODBC Connection
fromdb = DBI.connect('DBI:OCI8:XE', 'AIRSTATION-DEVELOP', 'AIRSTATION-DEVELOP')
prefix = "MAG"
appname = 'overhaul'
namespace = appname + "space"
layout = "tier2" + appname
tables = fromdb.tables
tabnavname = '_' + appname + '_tabnav.rhtml'
tabnavdir = 'app\\views\\widgets\\'
FileUtils.makedirs(tabnavdir)
tabnavpath = tabnavdir + tabnavname
thetabnav = File.open(tabnavpath,"w")
thetabnav.puts '<%'
thetabnav.puts '# GenRails - Tabnav based menu'
thetabnav.puts 'render_tabnav :' + appname + ","
thetabnav.puts ' :generate_css => true do'
thetabnav.puts ' '
thetabnav.puts ' if @current_user.has_role?(' + '\'' + appname + '\')'
thetabnav.puts ' add_tab do |t|'
thetabnav.puts ' t.named ' + '\'' + appname.capitalize + '\''
thetabnav.puts ' t.titled ' + '\'' + appname.capitalize + ' Home\''
thetabnav.puts ' t.links_to :controller => ' + '\'' + namespace + '/' + appname + 'home' + '\''
thetabnav.puts ' end'
tables.each do |tablename|
thetableprefix = tablename.slice(0,prefix.length)
if thetableprefix == prefix
dotable(fromdb, namespace, tablename, layout,prefix)
menuname = tablename.slice(prefix.length+1,tablename.length - prefix.length + 1)
menuname = menuname.downcase
menuname = menuname.singularize
controllername = tablename.downcase
controllername = controllername.singularize
thetabnav.puts ' add_tab do |t|'
thetabnav.puts ' t.named ' + '\'' + menuname.capitalize + '\''
thetabnav.puts ' t.titled ' + '\'' + menuname.capitalize + '\''
thetabnav.puts ' t.links_to :controller => ' + '\'' + namespace + '/' + controllername + '\''
thetabnav.puts ' end'
end
end
thetabnav.puts ' end'
thetabnav.puts 'end'
thetabnav.puts '%>'
thetabnav.close
printf "updated %s", tabnavpath
ActiveScaffold
RoleRequirement
Widgets/Tabnav
In addition I'm using a Oracle Database.
So I wrote a generator that "reads" the Oracle Database, and generates any
"missing" controllers and models. The lovely part is, when I change the basic
controller, I just "erase" the ones I want updated, and fire the script again.
This also handles the pluraziation, making sure the controllers, models are named the same. Also tries to do some cleanup so the menu names are a little more humanized. In addition good example of using name spaces to help organize a large project better.
require 'DBI'
require 'pp'
require 'active_support'
require 'fileutils'
def domodel(tablename)
modelname = tablename.downcase
modelname = modelname.singularize
modeldir = "app\\models\\"
FileUtils.makedirs(modeldir)
modelpath = modeldir + modelname + ".rb"
if FileTest::exists?(modelpath)
printf "exists %s\n", modelpath
return
end
themodel = File.open(modelpath,"w")
themodel.puts "class " + modelname.camelize + " < ActiveRecord::Base"
themodel.puts "# "
themodel.puts "# Created By GenRails"
themodel.puts "# "
themodel.puts "end"
themodel.close
printf "create %s\n", modelpath
end
def docontroller(thenamespace,tablename, layout, prefix)
controllername = tablename.downcase
controllername = controllername.singularize
controllerfilename = controllername + "_controller.rb"
controllerdir = "app\\controllers\\" + thenamespace + "\\"
FileUtils.makedirs(controllerdir)
controllerpath = controllerdir + controllerfilename
if FileTest::exists?(controllerpath)
printf "exists %s\n", controllerpath
return
end
thecontroller = File.open(controllerpath,"w")
thecontroller.puts "class " + thenamespace.camelize + '::' + controllername.camelize + 'Controller < ApplicationController'
thecontroller.puts " layout '" + layout + "'"
thecontroller.puts " before_filter :login_required"
thecontroller.puts "# "
thecontroller.puts "# Generated by GenRails"
thecontroller.puts "# "
thecontroller.puts " "
thecontroller.puts "active_scaffold :" + controllername.camelize + " do |config|"
menuname = tablename.slice(prefix.length+1,tablename.length - prefix.length + 1)
menuname = menuname.camelize
thecontroller.puts " config.label = '" + menuname + "'"
thecontroller.puts " config.actions = [:search, :show, :list, :create, :delete]"
thecontroller.puts " end"
thecontroller.puts "end"
thecontroller.close
printf "create %s\n", controllerpath
end
def dotable(fromdb, namespace, tablename, layout, prefix)
tablename = tablename.strip
if tablename.length == 0
return
end
# begin
# row = fromdb.select_one("select count(*) from #{tablename}")
# rescue
# return
# end
# record_count = row[0]
# if record_count == 0
# return
# end
domodel(tablename)
docontroller(namespace,tablename, layout, prefix)
end
#make a ODBC Connection
fromdb = DBI.connect('DBI:OCI8:XE', 'AIRSTATION-DEVELOP', 'AIRSTATION-DEVELOP')
prefix = "MAG"
appname = 'overhaul'
namespace = appname + "space"
layout = "tier2" + appname
tables = fromdb.tables
tabnavname = '_' + appname + '_tabnav.rhtml'
tabnavdir = 'app\\views\\widgets\\'
FileUtils.makedirs(tabnavdir)
tabnavpath = tabnavdir + tabnavname
thetabnav = File.open(tabnavpath,"w")
thetabnav.puts '<%'
thetabnav.puts '# GenRails - Tabnav based menu'
thetabnav.puts 'render_tabnav :' + appname + ","
thetabnav.puts ' :generate_css => true do'
thetabnav.puts ' '
thetabnav.puts ' if @current_user.has_role?(' + '\'' + appname + '\')'
thetabnav.puts ' add_tab do |t|'
thetabnav.puts ' t.named ' + '\'' + appname.capitalize + '\''
thetabnav.puts ' t.titled ' + '\'' + appname.capitalize + ' Home\''
thetabnav.puts ' t.links_to :controller => ' + '\'' + namespace + '/' + appname + 'home' + '\''
thetabnav.puts ' end'
tables.each do |tablename|
thetableprefix = tablename.slice(0,prefix.length)
if thetableprefix == prefix
dotable(fromdb, namespace, tablename, layout,prefix)
menuname = tablename.slice(prefix.length+1,tablename.length - prefix.length + 1)
menuname = menuname.downcase
menuname = menuname.singularize
controllername = tablename.downcase
controllername = controllername.singularize
thetabnav.puts ' add_tab do |t|'
thetabnav.puts ' t.named ' + '\'' + menuname.capitalize + '\''
thetabnav.puts ' t.titled ' + '\'' + menuname.capitalize + '\''
thetabnav.puts ' t.links_to :controller => ' + '\'' + namespace + '/' + controllername + '\''
thetabnav.puts ' end'
end
end
thetabnav.puts ' end'
thetabnav.puts 'end'
thetabnav.puts '%>'
thetabnav.close
printf "updated %s", tabnavpath
Thursday, October 04, 2007
Ruby Rails - Importorting Dbase and Foxpro Files
Did you have foxprofiles that you just want to move into a rails environment.
Then dbf is your friend.
It lets you do "mirgrations" that move dbase into the rails database of your choice.
So for example, one I just did:
class CreateMrosalesorder < ActiveRecord::Migration
def self.up
create_table "mrosalesorders", :force => true do |t|
t.column "mrocustomer_id", :integer
t.column "sono", :string, :limit => 8
....
t.column "tot_mcmosr", :float
t.column "tot_labosr", :float
end
thepathname = "...\SOMAST01.DBF"
table = DBF::Table.new(thepathname)
table.records.each do |record|
mysalesorder = Mrosalesorder.create(record.attributes)
mymrocustomer = Mrocustomer.find(:first, :conditions => "custno = '#{record.custno}'")
if not mymrocustomer.nil?
mysalesorder.mrocustomer_id = mymrocustomer.id
mysalesorder.save
end
end
end
def self.down
drop_table "mrosalesorders"
end
end
Now the issue will occur that some "field" names in dbase/foxpro may be invalid in your database of choice.
So its easy to resolve that in your model.
When "fields" are set they call a "method" of varname= in your model.
So take care of your "renames" there. If your a "cobol" programmer this is
like a "move" corresponding, with a "rename" in the middle, without needeing
to handle all the fields that are not renamed.
class Mrosalesorder < ActiveRecord::Base
belongs_to :customer
belongs_to :mrocustomer
def comment=(value)
self.socomment=value
end
def current=(value)
self.curr=value
end
def to_label
"#{sono}"
end
end
Then dbf is your friend.
It lets you do "mirgrations" that move dbase into the rails database of your choice.
So for example, one I just did:
class CreateMrosalesorder < ActiveRecord::Migration
def self.up
create_table "mrosalesorders", :force => true do |t|
t.column "mrocustomer_id", :integer
t.column "sono", :string, :limit => 8
....
t.column "tot_mcmosr", :float
t.column "tot_labosr", :float
end
thepathname = "...\SOMAST01.DBF"
table = DBF::Table.new(thepathname)
table.records.each do |record|
mysalesorder = Mrosalesorder.create(record.attributes)
mymrocustomer = Mrocustomer.find(:first, :conditions => "custno = '#{record.custno}'")
if not mymrocustomer.nil?
mysalesorder.mrocustomer_id = mymrocustomer.id
mysalesorder.save
end
end
end
def self.down
drop_table "mrosalesorders"
end
end
Now the issue will occur that some "field" names in dbase/foxpro may be invalid in your database of choice.
So its easy to resolve that in your model.
When "fields" are set they call a "method" of varname= in your model.
So take care of your "renames" there. If your a "cobol" programmer this is
like a "move" corresponding, with a "rename" in the middle, without needeing
to handle all the fields that are not renamed.
class Mrosalesorder < ActiveRecord::Base
belongs_to :customer
belongs_to :mrocustomer
def comment=(value)
self.socomment=value
end
def current=(value)
self.curr=value
end
def to_label
"#{sono}"
end
end
Tuesday, October 02, 2007
Fetcher Typos Resolved
Well, somehow it happened. A few typo's entered my perfect program. Why did it even work? :) The story of code. The new version starts faster, and more reliable as well.
Fetch your Exchange Mail, and call recievemail function directy in your ActiveRecord Models.
require 'rubygems'
require 'win32/service'
require 'logger'
require 'net/imap'
require 'net/pop'
include Win32
Dir.chdir("\\projects\\ror\\AirCenter")
require File.dirname(__FILE__) + '/../config/environment.rb'
SERVICE_NAME = 'AirFetchService'
SERVICE_DISPLAYNAME = 'AirCenterEmailFetcherService'
$logger = Logger.new("c:\\projects\\ror\\AirCenter\\log\\AirFetchService.log")
class Mailman < ActionMailer::Base
def receive(email)
sender = email.from.to_s()
subject = email.subject.to_s()
body = email.body.to_s()
subjectparts = subject.split(/ /)
classname = subjectparts[0]
$logger.info("Mailman: Sender = #{sender} Class = #{classname} Subject = #{subject}")
myclass = Object.const_get(classname)
if myclass
myclass.ReceiveEmail(sender,subject,subjectparts,body)
else $logger.warn("Mailman: Invalid class #{classname}")
#Request.ReceiveEmail(sender,subject,body)
end
end
end
if ARGV[0] == "register"
#Start The Service
svc = Service.new
svc.create_service do |s|
s.service_name = SERVICE_NAME
s.display_name = SERVICE_DISPLAYNAME
s.binary_path_name = "c:\\ruby\\bin\\ruby.exe c:\\projects\\ror\\AirCenter\\script\\AirFetchService.rb"
# + File.expand_path($0)
s.dependencies = []
end
svc.close
puts "Registered Service - " + SERVICE_DISPLAYNAME
elsif ARGV[0] == "test"
$logger.info "Service Starting"
$logger.info "Loading EMAIL Config"
@config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
$logger.info "Setting up config"
@config = @config[RAILS_ENV].to_options
$logger.info "Setting up sleep time"
@sleep_time = @config.delete(:sleep_time) || 60
$logger.info "Service Init Complete"
$logger.info "AirFetchService Establish mail processor"
# Add your own receiver object below
@fetcher = Fetcher.create({:receiver => Mailman}.merge(@config))
@fetcher.fetch
exit
elsif ARGV[0] == "delete"
if Service.status(SERVICE_NAME).current_state == "running"
Service.stop(SERVICE_NAME)
end
Service.delete(SERVICE_NAME)
else
if ENV["HOMEDRIVE"] !=nil
puts "Usage: ruby airfetchservice.rb [options]"
puts " where options are: register or delete"
exit
end
class MyDaemon < Win32::Daemon
def service_init
sleep 1
$logger.info "Service Starting"
end
def service_stop
$logger.info "AirFetchService Stopped"
end
def service_main
$logger.info "Loading EMAIL Config"
@config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
$logger.info "Setting up config"
@config = @config[RAILS_ENV].to_options
$logger.info "Setting up sleep time"
@sleep_time = @config.delete(:sleep_time) || 60
$logger.info "Service Init Complete"
$logger.info "AirFetchService Establish mail processor"
# Add your own receiver object below
@fetcher = Fetcher.create({:receiver => Mailman}.merge(@config))
$logger.info "AirFetchService Mail Processor Ready"
$logger.info "AirFetchService: Service in Running State"
while state == RUNNING
#begin
@fetcher.fetch
#rescue
#end
sleep(@sleep_time)
end
end
end
d = MyDaemon.new
d.mainloop
end
Fetch your Exchange Mail, and call recievemail function directy in your ActiveRecord Models.
require 'rubygems'
require 'win32/service'
require 'logger'
require 'net/imap'
require 'net/pop'
include Win32
Dir.chdir("\\projects\\ror\\AirCenter")
require File.dirname(__FILE__) + '/../config/environment.rb'
SERVICE_NAME = 'AirFetchService'
SERVICE_DISPLAYNAME = 'AirCenterEmailFetcherService'
$logger = Logger.new("c:\\projects\\ror\\AirCenter\\log\\AirFetchService.log")
class Mailman < ActionMailer::Base
def receive(email)
sender = email.from.to_s()
subject = email.subject.to_s()
body = email.body.to_s()
subjectparts = subject.split(/ /)
classname = subjectparts[0]
$logger.info("Mailman: Sender = #{sender} Class = #{classname} Subject = #{subject}")
myclass = Object.const_get(classname)
if myclass
myclass.ReceiveEmail(sender,subject,subjectparts,body)
else $logger.warn("Mailman: Invalid class #{classname}")
#Request.ReceiveEmail(sender,subject,body)
end
end
end
if ARGV[0] == "register"
#Start The Service
svc = Service.new
svc.create_service do |s|
s.service_name = SERVICE_NAME
s.display_name = SERVICE_DISPLAYNAME
s.binary_path_name = "c:\\ruby\\bin\\ruby.exe c:\\projects\\ror\\AirCenter\\script\\AirFetchService.rb"
# + File.expand_path($0)
s.dependencies = []
end
svc.close
puts "Registered Service - " + SERVICE_DISPLAYNAME
elsif ARGV[0] == "test"
$logger.info "Service Starting"
$logger.info "Loading EMAIL Config"
@config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
$logger.info "Setting up config"
@config = @config[RAILS_ENV].to_options
$logger.info "Setting up sleep time"
@sleep_time = @config.delete(:sleep_time) || 60
$logger.info "Service Init Complete"
$logger.info "AirFetchService Establish mail processor"
# Add your own receiver object below
@fetcher = Fetcher.create({:receiver => Mailman}.merge(@config))
@fetcher.fetch
exit
elsif ARGV[0] == "delete"
if Service.status(SERVICE_NAME).current_state == "running"
Service.stop(SERVICE_NAME)
end
Service.delete(SERVICE_NAME)
else
if ENV["HOMEDRIVE"] !=nil
puts "Usage: ruby airfetchservice.rb [options]"
puts " where options are: register or delete"
exit
end
class MyDaemon < Win32::Daemon
def service_init
sleep 1
$logger.info "Service Starting"
end
def service_stop
$logger.info "AirFetchService Stopped"
end
def service_main
$logger.info "Loading EMAIL Config"
@config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
$logger.info "Setting up config"
@config = @config[RAILS_ENV].to_options
$logger.info "Setting up sleep time"
@sleep_time = @config.delete(:sleep_time) || 60
$logger.info "Service Init Complete"
$logger.info "AirFetchService Establish mail processor"
# Add your own receiver object below
@fetcher = Fetcher.create({:receiver => Mailman}.merge(@config))
$logger.info "AirFetchService Mail Processor Ready"
$logger.info "AirFetchService: Service in Running State"
while state == RUNNING
#begin
@fetcher.fetch
#rescue
#end
sleep(@sleep_time)
end
end
end
d = MyDaemon.new
d.mainloop
end
Friday, September 28, 2007
Getting Email from Exchange Into Rails
So here I was, lost on a windows box, looking for a way to get supplies
that were badly needed. And what do you know, but Dan Weinand came to the rescue. Dan had written a plug-in for rails that "does" fetching of a IMAP server.
Dan Weinand site: http://www.danweinand.com/
In my application, I wanted my "clients" existing email infrastructure which consisted of Exchange 2003 soon to be exchange 2007. Just wanted to fetch a "email" address from the existing server.
So after asking permision we enable IMAP on exchange. And I pulled in the "fetcher"
into the application. Of course I wanted it to run as a "service" on windows. The original was designed for Linux. So I converted to a windows service. In addition
I found that the "standard" authentications in the fetcher for IMAP needed a "tweak" to do a "login" at the IMAP level, rather than "authorize".
So I added "dumb" authentication to the code.
The imap.rb in vendors/plugins/fetcher/lib needed the two patches
1. Added dumb authentication
2. Fixed the "recovery" so it would not cause a exception on a empty mailbox
--- imap.rb ---
module Fetcher
class Imap < Base
protected
# Additional Options:
# * :authentication - authentication type to use, defaults to DUMB
def initialize(options={})
@authentication = options.delete(:authentication) || 'DUMB'
super(options)
end
# Open connection and login to server
def establish_connection
@connection = Net::IMAP.new(@server)
if (@authentication == "DUMB")
@connection.login(@username,@password)
elsif
@connection.authenticate(@authentication, @username, @password)
end
end
# Retrieve messages from server
def get_messages
@connection.select('INBOX')
@connection.search(['ALL']).each do |message_id|
msg = @connection.fetch(message_id,'RFC822')[0].attr['RFC822']
begin
process_message(msg)
rescue
# handle_bogus_message(msg)
end
# Mark message as deleted
@connection.store(message_id, "+FLAGS", [:Deleted])
end
end
The Fetcher is designed with a bit of "meta" programming so it can "submit" the email to the activerecord model.
-- script/AirFetchservice.rb
require 'rubygems'
require 'win32/service'
require 'logger'
require 'net/imap'
require 'net/pop'
include Win32
Dir.chdir("\\projects\\ror\\AirCenter")
require File.dirname(__FILE__) + '/../config/environment.rb'
SERVICE_NAME = 'AirFetchService'
SERVICE_DISPLAYNAME = 'AirCenterEmailFetcherService'
$logger = Logger.new("c:\\projects\\ror\\AirCenter\\log\\AirFetchService.log")
class Mailman < ActionMailer::Base
def receive(email)
sender = email.from.to_s()
subject = email.subject.to_s()
body = email.body.to_s()
subjectparts = subject.split(/ /)
classname = subjectparts[0]
$logger.info("Mailman: Sender = #{sender} Class = #{classname} Subject = #{subject}")
myclass = Object.const_get(classname)
if myclass
myclass.ReceiveEmail(sender,subject,subjectparts,body)
else $logger.warn("Mailman: Invalid class #{classname}")
end
end
end
if ARGV[0] == "register"
#Start The Service
svc = Service.new
svc.create_service do |s|
s.service_name = SERVICE_NAME
s.display_name = SERVICE_DISPLAYNAME
s.binary_path_name = "c:\\ruby\\bin\\ruby.exe c:\\projects\\ror\\AirCenter\\script\\AirFetchService.rb"
# + File.expand_path($0)
s.dependencies = []
end
svc.close
puts "Registered Service - " + SERVICE_DISPLAYNAME
elsif ARGV[0] == "test"
$logger.info "Service Starting"
$logger.info "Loading EMAIL Config"
@config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
$logger.info "Setting up config"
@config = @config[RAILS_ENV].to_options
$logger.info "Setting up sleep time"
@sleep_time = @config.delete(:sleep_time) || 60
$logger.info "Service Init Complete"
$logger.info "AirFetchService Establish mail processor"
# Add your own receiver object below
@fetcher = Fetcher.create({:receiver => Mailman}.merge(@config))
@fetcher.fetch
exit
elsif ARGV[0] == "delete"
if Service.status(SERVICE_NAME).current_state == "running"
Service.stop(SERVICE_NAME)
end
Service.delete(SERVICE_NAME)
else
if ENV["HOMEDRIVE"] !=nil
puts "Usage: ruby airfetchservice.rb [options]"
puts " where options are: register or delete"
exit
end
$stderr = $logger
$stdout = $logger
class MyDaemon < Win32::Daemon
def service_init
sleep 1
$logger.info "Service Starting"
$logger.info "Loading EMAIL Config"
@config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
$logger.info "Setting up config"
@config = @config[RAILS_ENV].to_options
$logger.info "Setting up sleep time"
@sleep_time = @config.delete(:sleep_time) || 60
$logger.info "Service Init Complete"
$logger.info "AirFetchService Establish mail processor"
# Add your own receiver object below
@fetcher = Fetcher.create({:receiver => Mailman.receive}.merge(@config))
end
def service_stop
$logger.info "AirFetchService Stopped"
end
def service_main
$logger.info "AirFetchService: Service in Running State"
while state == RUNNING
@fetcher.fetch
sleep(@sleep_time)
end
end
end
d = MyDaemon.new
d.mainloop
end
that were badly needed. And what do you know, but Dan Weinand came to the rescue. Dan had written a plug-in for rails that "does" fetching of a IMAP server.
Dan Weinand site: http://www.danweinand.com/
In my application, I wanted my "clients" existing email infrastructure which consisted of Exchange 2003 soon to be exchange 2007. Just wanted to fetch a "email" address from the existing server.
So after asking permision we enable IMAP on exchange. And I pulled in the "fetcher"
into the application. Of course I wanted it to run as a "service" on windows. The original was designed for Linux. So I converted to a windows service. In addition
I found that the "standard" authentications in the fetcher for IMAP needed a "tweak" to do a "login" at the IMAP level, rather than "authorize".
So I added "dumb" authentication to the code.
The imap.rb in vendors/plugins/fetcher/lib needed the two patches
1. Added dumb authentication
2. Fixed the "recovery" so it would not cause a exception on a empty mailbox
--- imap.rb ---
module Fetcher
class Imap < Base
protected
# Additional Options:
# * :authentication - authentication type to use, defaults to DUMB
def initialize(options={})
@authentication = options.delete(:authentication) || 'DUMB'
super(options)
end
# Open connection and login to server
def establish_connection
@connection = Net::IMAP.new(@server)
if (@authentication == "DUMB")
@connection.login(@username,@password)
elsif
@connection.authenticate(@authentication, @username, @password)
end
end
# Retrieve messages from server
def get_messages
@connection.select('INBOX')
@connection.search(['ALL']).each do |message_id|
msg = @connection.fetch(message_id,'RFC822')[0].attr['RFC822']
begin
process_message(msg)
rescue
# handle_bogus_message(msg)
end
# Mark message as deleted
@connection.store(message_id, "+FLAGS", [:Deleted])
end
end
The Fetcher is designed with a bit of "meta" programming so it can "submit" the email to the activerecord model.
-- script/AirFetchservice.rb
require 'rubygems'
require 'win32/service'
require 'logger'
require 'net/imap'
require 'net/pop'
include Win32
Dir.chdir("\\projects\\ror\\AirCenter")
require File.dirname(__FILE__) + '/../config/environment.rb'
SERVICE_NAME = 'AirFetchService'
SERVICE_DISPLAYNAME = 'AirCenterEmailFetcherService'
$logger = Logger.new("c:\\projects\\ror\\AirCenter\\log\\AirFetchService.log")
class Mailman < ActionMailer::Base
def receive(email)
sender = email.from.to_s()
subject = email.subject.to_s()
body = email.body.to_s()
subjectparts = subject.split(/ /)
classname = subjectparts[0]
$logger.info("Mailman: Sender = #{sender} Class = #{classname} Subject = #{subject}")
myclass = Object.const_get(classname)
if myclass
myclass.ReceiveEmail(sender,subject,subjectparts,body)
else $logger.warn("Mailman: Invalid class #{classname}")
end
end
end
if ARGV[0] == "register"
#Start The Service
svc = Service.new
svc.create_service do |s|
s.service_name = SERVICE_NAME
s.display_name = SERVICE_DISPLAYNAME
s.binary_path_name = "c:\\ruby\\bin\\ruby.exe c:\\projects\\ror\\AirCenter\\script\\AirFetchService.rb"
# + File.expand_path($0)
s.dependencies = []
end
svc.close
puts "Registered Service - " + SERVICE_DISPLAYNAME
elsif ARGV[0] == "test"
$logger.info "Service Starting"
$logger.info "Loading EMAIL Config"
@config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
$logger.info "Setting up config"
@config = @config[RAILS_ENV].to_options
$logger.info "Setting up sleep time"
@sleep_time = @config.delete(:sleep_time) || 60
$logger.info "Service Init Complete"
$logger.info "AirFetchService Establish mail processor"
# Add your own receiver object below
@fetcher = Fetcher.create({:receiver => Mailman}.merge(@config))
@fetcher.fetch
exit
elsif ARGV[0] == "delete"
if Service.status(SERVICE_NAME).current_state == "running"
Service.stop(SERVICE_NAME)
end
Service.delete(SERVICE_NAME)
else
if ENV["HOMEDRIVE"] !=nil
puts "Usage: ruby airfetchservice.rb [options]"
puts " where options are: register or delete"
exit
end
$stderr = $logger
$stdout = $logger
class MyDaemon < Win32::Daemon
def service_init
sleep 1
$logger.info "Service Starting"
$logger.info "Loading EMAIL Config"
@config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
$logger.info "Setting up config"
@config = @config[RAILS_ENV].to_options
$logger.info "Setting up sleep time"
@sleep_time = @config.delete(:sleep_time) || 60
$logger.info "Service Init Complete"
$logger.info "AirFetchService Establish mail processor"
# Add your own receiver object below
@fetcher = Fetcher.create({:receiver => Mailman.receive}.merge(@config))
end
def service_stop
$logger.info "AirFetchService Stopped"
end
def service_main
$logger.info "AirFetchService: Service in Running State"
while state == RUNNING
@fetcher.fetch
sleep(@sleep_time)
end
end
end
d = MyDaemon.new
d.mainloop
end
My Life On Rails
I've been looking for a "idea" environment to do "applications".
I used to do alot of VB6, and felt that Microsoft "messed" things up
after that. My apps didnt move, and my mind was even less willing to
face the "explosion" of technology and "false" starts that Microsoft
offered in the way of programming.
My background covers C,assembler,Cobol,VB, PL1 and a variety of others.
I found Python about a year and a half ago, and did lots of small backend
projects with it for "backend" processes. Everything from colecting router
logs to database conversion, and realtime "interconnect" between legacy
applications. Python could not be beat. My problem was doing a "gui". The
gui must connect to a "database" and our database of choice is Oracle. In addition it should connect to a variety of others. As well as have libraries for everything else.
Enter "Stage-left" ruby and its framework ruby-on-rails.
Now I can develop a variety of applications with lovely "web" based gui in no time flat.
My technology stack:
active-scaffold
This take care of all the java-script for me. :)
tabnav/widgets
ruby-on-rails
activerecord - Manages the database
activemailer - Handles outgoing and incoming mail
ruby
Supporting cast:
Oracle (10g, Oracle XE)
win32-service - Lets me make my background rails functions run
as windows services
I used to do alot of VB6, and felt that Microsoft "messed" things up
after that. My apps didnt move, and my mind was even less willing to
face the "explosion" of technology and "false" starts that Microsoft
offered in the way of programming.
My background covers C,assembler,Cobol,VB, PL1 and a variety of others.
I found Python about a year and a half ago, and did lots of small backend
projects with it for "backend" processes. Everything from colecting router
logs to database conversion, and realtime "interconnect" between legacy
applications. Python could not be beat. My problem was doing a "gui". The
gui must connect to a "database" and our database of choice is Oracle. In addition it should connect to a variety of others. As well as have libraries for everything else.
Enter "Stage-left" ruby and its framework ruby-on-rails.
Now I can develop a variety of applications with lovely "web" based gui in no time flat.
My technology stack:
active-scaffold
This take care of all the java-script for me. :)
tabnav/widgets
ruby-on-rails
activerecord - Manages the database
activemailer - Handles outgoing and incoming mail
ruby
Supporting cast:
Oracle (10g, Oracle XE)
win32-service - Lets me make my background rails functions run
as windows services
Sunday, August 12, 2007
Malaria Killer
Hmm. Here's a interesting weekend project. (Wish I had time, but got two others ahead of this one).
Use one of the NEW high performance IR(Infrared LEDS) they now can handle 1-2 watts sustained, maybe more in flash mode. Use a "microcontroller", such as a Cypress PSOC to do a "simulation" of human breething using the IR LED. Using the PSOC capacitive sensing detect with a Mosquito has "entered" the kill zone, and then bump up the power to max power (Can add a few LEDS to increase the "flash" to 100watts) to
"fry" the insect, then return to "breathing" mode.
This should reduce the population of mosquitos very nicely and reduce the chance of malaria and other mosquito born infections.
I know mosquitos are "drawn" to CO2 and IR, the question is a "IR" simulation of breathing may be enough to get them going. The other question is the kill ration.
Its a simple and inexpensive design. Considering the "breathing" can be realtive low level and kill is "pulse" mode, you may even be able to solar power it.
If your a mosquito researcher, drop me a line.
References:
LED Example - Hmm What Frequency will have best effect?
http://catalog.osram-os.com/catalogue/catalogue.do?favOid=000000030002856201230023&act=showBookmark
Processor with Cap Sense
http://www.cypress.com/portal/server.pt?space=CommunityPage&control=SetCommunity&CommunityID=209&PageID=215&gid=13&fid=24&category=All&showall=false&CID=ILC-shortlinks&shk=psoc
Use one of the NEW high performance IR(Infrared LEDS) they now can handle 1-2 watts sustained, maybe more in flash mode. Use a "microcontroller", such as a Cypress PSOC to do a "simulation" of human breething using the IR LED. Using the PSOC capacitive sensing detect with a Mosquito has "entered" the kill zone, and then bump up the power to max power (Can add a few LEDS to increase the "flash" to 100watts) to
"fry" the insect, then return to "breathing" mode.
This should reduce the population of mosquitos very nicely and reduce the chance of malaria and other mosquito born infections.
I know mosquitos are "drawn" to CO2 and IR, the question is a "IR" simulation of breathing may be enough to get them going. The other question is the kill ration.
Its a simple and inexpensive design. Considering the "breathing" can be realtive low level and kill is "pulse" mode, you may even be able to solar power it.
If your a mosquito researcher, drop me a line.
References:
LED Example - Hmm What Frequency will have best effect?
http://catalog.osram-os.com/catalogue/catalogue.do?favOid=000000030002856201230023&act=showBookmark
Processor with Cap Sense
http://www.cypress.com/portal/server.pt?space=CommunityPage&control=SetCommunity&CommunityID=209&PageID=215&gid=13&fid=24&category=All&showall=false&CID=ILC-shortlinks&shk=psoc
Tuesday, March 14, 2006
New "Water" Economy
Everyone is talking about the new "Hydrogen" Economy, and what changes is will make.
In reality this is a misnomer, its actually the new "Water" economy. When you burn "hydrogen" and when I say burn that applies to "fuel" cells, or just converting a internal combustion engine (read "Car" engine) to burn hydrogen you get water. The flip side is to get hydrogen the best source is water.
There are several ways of doing this, "electrolysis" being the first that comes to mind, but the far better is algae. Electrolysis is sticking "two" wires, (Actually electrodes) in water and separating the two. Here, unless you want to build lots of new Nuclear plants, is rather costly. (And even if you are into pebble bed n-plants, its still not cheap, just cheaper than tradional plants). The far better approach, where rather progress is algae. There is already algae that does this on a small scale, and with genetic engineering, its gotten about 100,000 times better already. Another 100x improvement will make it "production" ready.
So projecting the future, you will have "algae" farms producing hydrogen, at affordable rates.
So what is needed, and how do we get there.
Things that are needed:
1. Water - Its our raw material, high-quality, pure in large amounts
Solution - Think desalination or better yet "membranes", check out http://www.hyflux.com While there are others, these guys are smaller in size and hungry. Idea "site" is near a ocean if possible
2. Land - Lots and lots of it. Hmm. It would be handy and generally favorable to use deserts for this. The middle-east has plenty, as well as dont mind a large scale investment to make things happen. Of course that means the oil-producing nations have three of the things needed to do this, land, water, and lots of sunshine.
3. Investment - Need to start looking at "all" the needs now, building pilot plants, building "foundation" for the whole eco-system
4. Farming "System" - A system of encapsulating the algae, circulating, and growing them on huge scale with exposure to sunshine. This is the "hardware" of the whole system. Since we are looking at desert locale, doing this in safety glass may be idea. A "plant" producing the panels is one of the pieces of infrastrucutor that is needed. The control software, prodcess monitoring, etc also need to be developed. All of this is "sellable" to all other parties wanting to do the farming.
5. Storage/Transport - Lots of work has been done on the foundation of "storage" of hydrogen, further applications need to be developed. ie. containers ("Think the standard size container used in the "container" industry), do user replaceble fuel modules, to large scale tankers. A plan needs to be done to address all the products for the ecosystem.
6. Algae - Lots of groups are already doing this, operational knowledge and license needs to be built. This will allow the customization of #4 to increase production. Not that generally the algae can be changed and readly updated to allow the farm production to increase over time. So you build a "plant" that covers 100 square miles, and every year you will get better production as you improve the algae.
Hydrogen can be safely transported today, but we have to start on the products that will make it useful everywhere. Corporate, utilities and the public are interested in "clean" energy today. Large scale hydogen production can start, its just a matter of "doing" now, and making it happen.
In reality this is a misnomer, its actually the new "Water" economy. When you burn "hydrogen" and when I say burn that applies to "fuel" cells, or just converting a internal combustion engine (read "Car" engine) to burn hydrogen you get water. The flip side is to get hydrogen the best source is water.
There are several ways of doing this, "electrolysis" being the first that comes to mind, but the far better is algae. Electrolysis is sticking "two" wires, (Actually electrodes) in water and separating the two. Here, unless you want to build lots of new Nuclear plants, is rather costly. (And even if you are into pebble bed n-plants, its still not cheap, just cheaper than tradional plants). The far better approach, where rather progress is algae. There is already algae that does this on a small scale, and with genetic engineering, its gotten about 100,000 times better already. Another 100x improvement will make it "production" ready.
So projecting the future, you will have "algae" farms producing hydrogen, at affordable rates.
So what is needed, and how do we get there.
Things that are needed:
1. Water - Its our raw material, high-quality, pure in large amounts
Solution - Think desalination or better yet "membranes", check out http://www.hyflux.com While there are others, these guys are smaller in size and hungry. Idea "site" is near a ocean if possible
2. Land - Lots and lots of it. Hmm. It would be handy and generally favorable to use deserts for this. The middle-east has plenty, as well as dont mind a large scale investment to make things happen. Of course that means the oil-producing nations have three of the things needed to do this, land, water, and lots of sunshine.
3. Investment - Need to start looking at "all" the needs now, building pilot plants, building "foundation" for the whole eco-system
4. Farming "System" - A system of encapsulating the algae, circulating, and growing them on huge scale with exposure to sunshine. This is the "hardware" of the whole system. Since we are looking at desert locale, doing this in safety glass may be idea. A "plant" producing the panels is one of the pieces of infrastrucutor that is needed. The control software, prodcess monitoring, etc also need to be developed. All of this is "sellable" to all other parties wanting to do the farming.
5. Storage/Transport - Lots of work has been done on the foundation of "storage" of hydrogen, further applications need to be developed. ie. containers ("Think the standard size container used in the "container" industry), do user replaceble fuel modules, to large scale tankers. A plan needs to be done to address all the products for the ecosystem.
6. Algae - Lots of groups are already doing this, operational knowledge and license needs to be built. This will allow the customization of #4 to increase production. Not that generally the algae can be changed and readly updated to allow the farm production to increase over time. So you build a "plant" that covers 100 square miles, and every year you will get better production as you improve the algae.
Hydrogen can be safely transported today, but we have to start on the products that will make it useful everywhere. Corporate, utilities and the public are interested in "clean" energy today. Large scale hydogen production can start, its just a matter of "doing" now, and making it happen.
Monday, January 02, 2006
Python - Microsoft Excel And Telnet = MRTG
Hmm. Got a unreliable T1 line in one of my sites, so I wanted to monitor it. Of course I could use MRTG, but then I want to run it on windows xp machine. (And yes I know I can do that with MRTG as well)
Ok, lets use Excel for our data repository, and formating. For gatering the data, lets use telnet. (Yes, I know I could also use ssh, but the router is a old one) And python to put it all togeather.
This "example" uses .split as the basis of parsing the telnet output, and updates a excel spreadsheet with the results. The spreadsheet is saved. The "whole" script is run once a hour to collect the data. For simplicity, I use the microsoft task scheduler.
Of course, this can easily be expanded to support multiple routers and multiple serial interfaces.
If anyone wants to "see" the final spreadsheet, drop me a email.
## check-the-line.py
port sys
import telnetlib
from win32com.client import Dispatch
import pywintypes
import datetime
import time
import pythoncom
prevupdate = 6
lastupdate = 5
deltaslot = 7
tn = telnetlib.Telnet("1.2.3.4") # Change your ip address
tn.read_until("Password: ")
tn.write("your.password.here.for.telnet\n")
tn.read_until(">")
tn.write("show interface serial 0\n") # The interface
router_status = tn.read_until(">")
tn.write("exit\n")
thelines = router_status.split("\n")
for theline in thelines:
thelen = len(theline) - 1
theline = theline[:thelen]
recvcnt = thelines[16].split()
recverr = thelines[17].split()
sentcnt = thelines[18].split()
senterr = thelines[19].split()
x1 = Dispatch("Excel.Application")
x1.Visible = 0
x1.Workbooks.Open("mn-line.xls")
myc = x1.ActiveSheet.Cells
# Received 11045 broadcasts, 0 runts, 0 giants, 0 throttles
# 3 input errors, 3 CRC, 0 frame, 0 overrun, 0 ignored, 3 abort
# 332207 packets output, 54998720 bytes, 0 underruns
# 0 output errors, 0 collisions, 15 interface resets
now = datetime.datetime.now()
#save the old
for thecol in range(1,18):
myc(prevupdate,thecol).Value = myc(lastupdate,thecol).Value
myc(lastupdate,1).Value = pythoncom.MakeTime(time.time())
#recvcnt
myc(lastupdate,2).Value = recvcnt[1]
myc(lastupdate,3).Value = recvcnt[3]
myc(lastupdate,4).Value = recvcnt[5]
myc(lastupdate,5).Value = recvcnt[7]
#recverr
myc(lastupdate,6).Value = recverr[0]
myc(lastupdate,7).Value = recverr[3]
myc(lastupdate,8).Value = recverr[5]
myc(lastupdate,9).Value = recverr[7]
myc(lastupdate,10).Value = recverr[9]
myc(lastupdate,11).Value = recverr[11]
#sentcnt
myc(lastupdate,12).Value = sentcnt[0]
myc(lastupdate,13).Value = sentcnt[3]
myc(lastupdate,14).Value = sentcnt[5]
myc(lastupdate,15).Value = senterr[0]
myc(lastupdate,16).Value = senterr[3]
myc(lastupdate,17).Value = senterr[5]
#put the deltas in the hour slot
thehour = now.hour
hourslot = 10 + thehour
for thecol in range(2,18):
myc(hourslot,thecol).Value = myc(deltaslot,thecol).Value
myc(hourslot,1).Value = myc(lastupdate,1).Value
x1.ActiveWorkbook.Save()
if thehour == 23: # Witching hour - Save a copy for reference
filename = "ml-line-" + now.strftime("%y-%m-%d")
x1.ActiveWorkbook.SaveAs(filename)
if thehour == 0: # Reset the Spreadsheet Date
myc(1,2).Value = myc(lastupdate,1).Value
x1.ActiveWorkbook.Close(0)
x1.Quit()
del x1
Ok, lets use Excel for our data repository, and formating. For gatering the data, lets use telnet. (Yes, I know I could also use ssh, but the router is a old one) And python to put it all togeather.
This "example" uses .split as the basis of parsing the telnet output, and updates a excel spreadsheet with the results. The spreadsheet is saved. The "whole" script is run once a hour to collect the data. For simplicity, I use the microsoft task scheduler.
Of course, this can easily be expanded to support multiple routers and multiple serial interfaces.
If anyone wants to "see" the final spreadsheet, drop me a email.
## check-the-line.py
port sys
import telnetlib
from win32com.client import Dispatch
import pywintypes
import datetime
import time
import pythoncom
prevupdate = 6
lastupdate = 5
deltaslot = 7
tn = telnetlib.Telnet("1.2.3.4") # Change your ip address
tn.read_until("Password: ")
tn.write("your.password.here.for.telnet\n")
tn.read_until(">")
tn.write("show interface serial 0\n") # The interface
router_status = tn.read_until(">")
tn.write("exit\n")
thelines = router_status.split("\n")
for theline in thelines:
thelen = len(theline) - 1
theline = theline[:thelen]
recvcnt = thelines[16].split()
recverr = thelines[17].split()
sentcnt = thelines[18].split()
senterr = thelines[19].split()
x1 = Dispatch("Excel.Application")
x1.Visible = 0
x1.Workbooks.Open("mn-line.xls")
myc = x1.ActiveSheet.Cells
# Received 11045 broadcasts, 0 runts, 0 giants, 0 throttles
# 3 input errors, 3 CRC, 0 frame, 0 overrun, 0 ignored, 3 abort
# 332207 packets output, 54998720 bytes, 0 underruns
# 0 output errors, 0 collisions, 15 interface resets
now = datetime.datetime.now()
#save the old
for thecol in range(1,18):
myc(prevupdate,thecol).Value = myc(lastupdate,thecol).Value
myc(lastupdate,1).Value = pythoncom.MakeTime(time.time())
#recvcnt
myc(lastupdate,2).Value = recvcnt[1]
myc(lastupdate,3).Value = recvcnt[3]
myc(lastupdate,4).Value = recvcnt[5]
myc(lastupdate,5).Value = recvcnt[7]
#recverr
myc(lastupdate,6).Value = recverr[0]
myc(lastupdate,7).Value = recverr[3]
myc(lastupdate,8).Value = recverr[5]
myc(lastupdate,9).Value = recverr[7]
myc(lastupdate,10).Value = recverr[9]
myc(lastupdate,11).Value = recverr[11]
#sentcnt
myc(lastupdate,12).Value = sentcnt[0]
myc(lastupdate,13).Value = sentcnt[3]
myc(lastupdate,14).Value = sentcnt[5]
myc(lastupdate,15).Value = senterr[0]
myc(lastupdate,16).Value = senterr[3]
myc(lastupdate,17).Value = senterr[5]
#put the deltas in the hour slot
thehour = now.hour
hourslot = 10 + thehour
for thecol in range(2,18):
myc(hourslot,thecol).Value = myc(deltaslot,thecol).Value
myc(hourslot,1).Value = myc(lastupdate,1).Value
x1.ActiveWorkbook.Save()
if thehour == 23: # Witching hour - Save a copy for reference
filename = "ml-line-" + now.strftime("%y-%m-%d")
x1.ActiveWorkbook.SaveAs(filename)
if thehour == 0: # Reset the Spreadsheet Date
myc(1,2).Value = myc(lastupdate,1).Value
x1.ActiveWorkbook.Close(0)
x1.Quit()
del x1
Thursday, November 17, 2005
PIX Wars - CISCO Pix Bug
Unrelated Comment: If your going to leave me a comment, and want a answer, please add your email address. The normal form does not give me the address.
In helping out a friend of mine, I was debugging a CISCO firewall. Of course the normal thing in the process is to setup a SYSLOG server to capture the bulk of what is going on. CISCO now recommends the TCP syslog. Well, of course prefer to follow there recomendation, so I set it up. Used there Windows Based SysLog server. Of course when I rebooted I didnt notice the logger process did not restart. Now my firewall was not taking any new connections.
"What did I do?" was what I was thinking. Check every setting, and come to fine out
there is a bug, if the destination is not available, it keeps spawning logging connections till all available connections are used up. After switching to UDP based logging, everything is fine.
Comming soon, Python scripts to automatically parse PIX logs.
In helping out a friend of mine, I was debugging a CISCO firewall. Of course the normal thing in the process is to setup a SYSLOG server to capture the bulk of what is going on. CISCO now recommends the TCP syslog. Well, of course prefer to follow there recomendation, so I set it up. Used there Windows Based SysLog server. Of course when I rebooted I didnt notice the logger process did not restart. Now my firewall was not taking any new connections.
"What did I do?" was what I was thinking. Check every setting, and come to fine out
there is a bug, if the destination is not available, it keeps spawning logging connections till all available connections are used up. After switching to UDP based logging, everything is fine.
Comming soon, Python scripts to automatically parse PIX logs.
Wednesday, October 26, 2005
Thermal Radiance - A way to cool your PC using Quantum Mechanics
Your PC has lots of "glow", that really increases the bling factor, but does nothing to keep it actually cooler. There may be a way to "convert" the heat to light, giving that "nice" glow, while actually helping with the cooling.
In a recent invention, "Quantum" dots, which have 25 +/- electrons per dot, converted a blue LED, into a "white" led. No big deal to most people, but the physics of it, opens up several possibilities.
The first is a modification of your heat-sink, with tuning of the Quantum Dots to allow the conversion of heat to light. Since the light will stream nicely out of the case, it allows another way of getting heat out of the case quicker.
The technique should be relatively low-cost, and massed produced.
Note that another technique is to excite the quantum dots with electrical standing waves. (Think tesla coil). This should allow the production of large amounts of light, at high efficiently.
In a recent invention, "Quantum" dots, which have 25 +/- electrons per dot, converted a blue LED, into a "white" led. No big deal to most people, but the physics of it, opens up several possibilities.
The first is a modification of your heat-sink, with tuning of the Quantum Dots to allow the conversion of heat to light. Since the light will stream nicely out of the case, it allows another way of getting heat out of the case quicker.
The technique should be relatively low-cost, and massed produced.
Note that another technique is to excite the quantum dots with electrical standing waves. (Think tesla coil). This should allow the production of large amounts of light, at high efficiently.
Monday, October 17, 2005
Assisted Reality
Assisted Reality
Intel has announced that they can reduce leakage current by 1000 to 1 in coming generation of processors. Since a majority of power consumption is power consumption, we are talking about being able to run PC on a whole lot less power, and heat. With this it opens up new usage models, and new applications.
Assisted Reality – Belt PC
First, we reduce the pc to the size of a blackberry, or modern hand-phone/cell-phone. With the reduction in power, this will become a whole lot easier. Next we need the killer application, from the software for such a PC. But first, we surfer from one issue, we need to be able to “see” the world, and “see” the computer screen, or better yet, both at the same time.
Intel, as well as others are busy creating Silicon based Display Devices. Used commonly for projection style TV’s, these can also be engineered into a “new” pair of glasses. Each eye is given a “personal” LCD, built into “normal” frames. A high-resolution, thin CMOS sensor is also mounted into the frames. The whole package is designed to be thin, and look like “normal” sun glasses. Of course, there very abnormal in how they work. First the CMOS sensor captures images at video rates, and there displayed, (with optional correction for vision), on the in-build LCD’s. But now we can get interesting results. First with a bit of sensor work, we can detect where the eye is looking. And with facial recognition we can build in the ability for the “glasses” to remember people and give us clues to there identity.
In addition, the system can also include GPS, and compass information, allowing us to identify location, and buildings, as well as give us information of what is in the building.
All of this is going to get “really” interesting from the point-of-view of GUI development. Layering all the information in “Heads-Up” format will be a challenge. Of course we can also go totally to PC “mode” where the displays are “normal” large screen monitors, or even, add the ability such that if we are in “one” location, and looking in “one” direction, the “monitor” appears. Now you see the “meaning” of “assisted” reality coming into play. The modern PC has the ability to complement, and add to our day to day lives in a direct and real-time fashion.
The good news is the hardware, and a variety of the software already exist, it’s a matter of market force and some integration, along with the “next” generation of PC hardware.
Intel has announced that they can reduce leakage current by 1000 to 1 in coming generation of processors. Since a majority of power consumption is power consumption, we are talking about being able to run PC on a whole lot less power, and heat. With this it opens up new usage models, and new applications.
Assisted Reality – Belt PC
First, we reduce the pc to the size of a blackberry, or modern hand-phone/cell-phone. With the reduction in power, this will become a whole lot easier. Next we need the killer application, from the software for such a PC. But first, we surfer from one issue, we need to be able to “see” the world, and “see” the computer screen, or better yet, both at the same time.
Intel, as well as others are busy creating Silicon based Display Devices. Used commonly for projection style TV’s, these can also be engineered into a “new” pair of glasses. Each eye is given a “personal” LCD, built into “normal” frames. A high-resolution, thin CMOS sensor is also mounted into the frames. The whole package is designed to be thin, and look like “normal” sun glasses. Of course, there very abnormal in how they work. First the CMOS sensor captures images at video rates, and there displayed, (with optional correction for vision), on the in-build LCD’s. But now we can get interesting results. First with a bit of sensor work, we can detect where the eye is looking. And with facial recognition we can build in the ability for the “glasses” to remember people and give us clues to there identity.
In addition, the system can also include GPS, and compass information, allowing us to identify location, and buildings, as well as give us information of what is in the building.
All of this is going to get “really” interesting from the point-of-view of GUI development. Layering all the information in “Heads-Up” format will be a challenge. Of course we can also go totally to PC “mode” where the displays are “normal” large screen monitors, or even, add the ability such that if we are in “one” location, and looking in “one” direction, the “monitor” appears. Now you see the “meaning” of “assisted” reality coming into play. The modern PC has the ability to complement, and add to our day to day lives in a direct and real-time fashion.
The good news is the hardware, and a variety of the software already exist, it’s a matter of market force and some integration, along with the “next” generation of PC hardware.
Wednesday, October 12, 2005
Keeping PowerPoint Running
Keeping PowerPoint Running
Very commonly we want to keep PowerPoint running, and updating it once in a while. Plasma and LCD’s displays are cheap. But the question is, how do I do it, without keyboard and mouse.
Well, what I found was a simple python script. It uses several techniques that are useful.
The target is to use PowerPointViewer 2003. PPTVIEW2003 does not support COM, which would been a good way of implementing this type of approach. So instead the viewer Is killed when the file changes.
How its used.
The PC is configured “auto-login”. The autologin user is configured to run the script. The user should be the local machine administrator. The script will automatically restart the viewer if the file is modified. So a “share” of the directory will need to be done. Also of the viewer “dies” it will be restarted.
Future Enhancements.
1. Scheduling “new” PowerPoints to be displayed on certain dates.
2. Scheduled reboots of machine to solve “PowerPoint” Memory leaks and software updates
Useful Resources:
Text To Speach Examples
Python And WMI
Python Library Reference
The script.
import os, time, win32api, winsound, os.path
import pyTTS
import wmi
def kill(handle):
"""kill function for Win32"""
return (0 != win32api.TerminateProcess(handle, 0))
application = "\\Program Files\\Microsoft Office\\PowerPoint Viewer\\pptview.exe"
slides = "\\present\\default.ppt"
timemodified = os.path.getmtime(slides)
x1App = os.spawnl(os.P_NOWAIT,application,slides,slides)
tts = pyTTS.Create()
tts.Speak("Starting Presentation")
while 1:
# Check to see if we have a running viewer
c = wmi.WMI ()
pptProcessId = 0
for process in c.Win32_Process (name="pptview.exe"):
pptProcessId = process.ProcessId, process.Name
if pptProcessId == 0:
tts.Speak("Presentation not running, Restarting")
x1App = os.spawnl(os.P_NOWAIT,application,slides,slides)
# Check to see if we have a modified file
newtime = os.path.getmtime(slides)
if newtime > timemodified:
timemodified = newtime
tts.Speak("New Presentation Available")
tts.Speak("Shutting Down Old Presentation")
try:
kill(x1App)
break
except:
pass
tts.Speak("New Presentation Starting")
x1App = os.spawnl(os.P_NOWAIT,application,slides,slides)
tts.Speak("Presentation Commensed")
Very commonly we want to keep PowerPoint running, and updating it once in a while. Plasma and LCD’s displays are cheap. But the question is, how do I do it, without keyboard and mouse.
Well, what I found was a simple python script. It uses several techniques that are useful.
The target is to use PowerPointViewer 2003. PPTVIEW2003 does not support COM, which would been a good way of implementing this type of approach. So instead the viewer Is killed when the file changes.
How its used.
The PC is configured “auto-login”. The autologin user is configured to run the script. The user should be the local machine administrator. The script will automatically restart the viewer if the file is modified. So a “share” of the directory will need to be done. Also of the viewer “dies” it will be restarted.
Future Enhancements.
1. Scheduling “new” PowerPoints to be displayed on certain dates.
2. Scheduled reboots of machine to solve “PowerPoint” Memory leaks and software updates
Useful Resources:
Text To Speach Examples
Python And WMI
Python Library Reference
The script.
import os, time, win32api, winsound, os.path
import pyTTS
import wmi
def kill(handle):
"""kill function for Win32"""
return (0 != win32api.TerminateProcess(handle, 0))
application = "\\Program Files\\Microsoft Office\\PowerPoint Viewer\\pptview.exe"
slides = "\\present\\default.ppt"
timemodified = os.path.getmtime(slides)
x1App = os.spawnl(os.P_NOWAIT,application,slides,slides)
tts = pyTTS.Create()
tts.Speak("Starting Presentation")
while 1:
# Check to see if we have a running viewer
c = wmi.WMI ()
pptProcessId = 0
for process in c.Win32_Process (name="pptview.exe"):
pptProcessId = process.ProcessId, process.Name
if pptProcessId == 0:
tts.Speak("Presentation not running, Restarting")
x1App = os.spawnl(os.P_NOWAIT,application,slides,slides)
# Check to see if we have a modified file
newtime = os.path.getmtime(slides)
if newtime > timemodified:
timemodified = newtime
tts.Speak("New Presentation Available")
tts.Speak("Shutting Down Old Presentation")
try:
kill(x1App)
break
except:
pass
tts.Speak("New Presentation Starting")
x1App = os.spawnl(os.P_NOWAIT,application,slides,slides)
tts.Speak("Presentation Commensed")
Thursday, August 25, 2005
Intel Wakes Up to Portable Devices
Is Intel seeing the Green Light on Large scale volume for portable devices? ARM(Advanced Risc Machines) is the the current leader in the handset areana. The reasson is simple ARM buiness focus is Mips-Per-Watt. The more perfomance for the least power is why so many of phones, pda's, and cameras use ARM chips. This may be changing. With Intel's ability to leverage advance fabs, and putting a dedicated design team, we may finally get a real "PDA sized PC". With good battery life, and with great expansion.
Think of it, with PCI-Express, and USB everything can easliy be jacked into a hand-held cpu package. We can always carry our PC, then lay it into a "dock" and get even better peformance.
Also all our pc applications can run, reasonably unchanged. With a pair of "glasses" with lcd/led's instead of lens we can have virtual 30 inch displays.
The issue comes down to "software". X86 has a legacy of software that goes back 20 years. Everyone is used to this legacy.
Good example, you want Microsoft Messanger, Yahoo Chat, and Skype? Can be done on a pda, but its easier on a laptop. Want Java on a PDA? Very painful. But a "tablet" pc the size of a PDA, with good power consumption(translates low) is far better. Even "enhanced" reality with this setup becomes reasonbly straight forward. Our true "Pocket" PC driving LED/LCD microdisplay, with a array of cameras weaved into our clothing, and dsp processed mikes would let our eyes roam. Add-in GPS and location awareness means that we would never get lost, and would know to the foot where the nearest pizza place is, and what google says about the quality or lack of it. It also means that Microsoft will not choose our fate, as we can also run other OS's on the x86 platform.
Think of it, with PCI-Express, and USB everything can easliy be jacked into a hand-held cpu package. We can always carry our PC, then lay it into a "dock" and get even better peformance.
Also all our pc applications can run, reasonably unchanged. With a pair of "glasses" with lcd/led's instead of lens we can have virtual 30 inch displays.
The issue comes down to "software". X86 has a legacy of software that goes back 20 years. Everyone is used to this legacy.
Good example, you want Microsoft Messanger, Yahoo Chat, and Skype? Can be done on a pda, but its easier on a laptop. Want Java on a PDA? Very painful. But a "tablet" pc the size of a PDA, with good power consumption(translates low) is far better. Even "enhanced" reality with this setup becomes reasonbly straight forward. Our true "Pocket" PC driving LED/LCD microdisplay, with a array of cameras weaved into our clothing, and dsp processed mikes would let our eyes roam. Add-in GPS and location awareness means that we would never get lost, and would know to the foot where the nearest pizza place is, and what google says about the quality or lack of it. It also means that Microsoft will not choose our fate, as we can also run other OS's on the x86 platform.
Friday, August 19, 2005
Fixing the MBR After Grub Install
What a long day. I am doing a talk at our local LUGS (Singapore Linux Users Group), and its a very straight forward one. Show new users how to install Linux. So of course want to walk thru, and try it out.
Tools:
Knoppix 3.9 Release - For Resizing
OSLoader - The special "Magic"
Ubuntu 5.10 "Brezzy Badger" Release Candidate
Presario 2500 PC Laptop
First, to resize, you want to use either qtparted, or ntfsresize.
The GUI version it the qtparted, and you can run it directly from the desktop.
Be careful, that free space is not to large, the 3.9 version has a older version of ntfsresize, that causes problems if you try to make the partition to large.
So on a 40 gigabyte drive, for dual boot, I'd recommend about 17Gigabytes.
If you want to see more "messages", I would recommend you using the ntfsresize command.
http://mlf.linux.rulez.org/mlf/ezaz/ntfsresize.html
Is a good FAQ on the status of ntfsresize, so you can see the issues and problems.
Resize, and then reboot with you distro of choice for easy install. Dont forget to select the option to use the freespace you created.
The biggist issue is if you decide to "return" to windows only. Most people would think of just "deleting" the new partitions, and rebooting. Got Cha. The mbr now is pointing to "GRUB" in the partition you just deleted. Machine no long will boot.
So how to "uninstall" GRUB, and return to a native windows machine. You can use the recovery console. Or, better yet, before you delete the Linux partition, you can install OSL2000. This lives nicely in the boot sector, now you can directly boot Win2k. Of course if you want the normal MBR record, with nothing fancy, after installing OSL2000, uninstall it. It will rewrite the MBR nicely.
No fuss, no muss. No WinXP disk needed, and no lost data.
Tools:
Knoppix 3.9 Release - For Resizing
OSLoader - The special "Magic"
Ubuntu 5.10 "Brezzy Badger" Release Candidate
Presario 2500 PC Laptop
First, to resize, you want to use either qtparted, or ntfsresize.
The GUI version it the qtparted, and you can run it directly from the desktop.
Be careful, that free space is not to large, the 3.9 version has a older version of ntfsresize, that causes problems if you try to make the partition to large.
So on a 40 gigabyte drive, for dual boot, I'd recommend about 17Gigabytes.
If you want to see more "messages", I would recommend you using the ntfsresize command.
http://mlf.linux.rulez.org/mlf/ezaz/ntfsresize.html
Is a good FAQ on the status of ntfsresize, so you can see the issues and problems.
Resize, and then reboot with you distro of choice for easy install. Dont forget to select the option to use the freespace you created.
The biggist issue is if you decide to "return" to windows only. Most people would think of just "deleting" the new partitions, and rebooting. Got Cha. The mbr now is pointing to "GRUB" in the partition you just deleted. Machine no long will boot.
So how to "uninstall" GRUB, and return to a native windows machine. You can use the recovery console. Or, better yet, before you delete the Linux partition, you can install OSL2000. This lives nicely in the boot sector, now you can directly boot Win2k. Of course if you want the normal MBR record, with nothing fancy, after installing OSL2000, uninstall it. It will rewrite the MBR nicely.
No fuss, no muss. No WinXP disk needed, and no lost data.
Thursday, June 09, 2005
Clustering and Grid Computing
Despite Apple leaving PowerPC it appears that the latest part(CPU) is nice, the e700 has the following on chip,
Dual 1.5Ghz PowerPC Core
4 Gigibit Ethernets
Dual 64Bit DRAM Bus (Wow lots of thruput)
PCI-Express (x8)
And the most important part, 15Watts Power.
Now we can build a "mesh" of these in two configs:
One is in a PC Motherboard config:
With good design, (high-quality motherboard substrate), may be able to get as many as 20 of these beasties on the motherboard. This equates to 40 CPU's in a PC case. Cooling is no worse than 4-6 AMD processors, and the perfomance for a single is equivlent to the best AMD(but with less power.).
The design would allow a solid grid of processors on the top of the motherboard, with memory(DRAM) undersneath using the memory chips directly. (No DIMMS they take up to much space.) Blind and Buried via's would be used to interconnect memory and dram.
I need to explore the booting options, while a single Flash chip could provide bootloader(u-boot) per CPU, a more indirect approach of using PCI-Express Switch as a secondary communication and boot would be a nice approach. Then all the CPU's would be in the memory-map of the control CPU.
I'd use Liquid Metal Cooling(No fans on the heat-sinks, less space), with a nice vortex cooler.
cooler integrated in the case. Add a couple of SLI PCI-Express card slots for "Graphics" would be nice.
OS would be debian (or yellow dog).
Dual 1.5Ghz PowerPC Core
4 Gigibit Ethernets
Dual 64Bit DRAM Bus (Wow lots of thruput)
PCI-Express (x8)
And the most important part, 15Watts Power.
Now we can build a "mesh" of these in two configs:
One is in a PC Motherboard config:
With good design, (high-quality motherboard substrate), may be able to get as many as 20 of these beasties on the motherboard. This equates to 40 CPU's in a PC case. Cooling is no worse than 4-6 AMD processors, and the perfomance for a single is equivlent to the best AMD(but with less power.).
The design would allow a solid grid of processors on the top of the motherboard, with memory(DRAM) undersneath using the memory chips directly. (No DIMMS they take up to much space.) Blind and Buried via's would be used to interconnect memory and dram.
I need to explore the booting options, while a single Flash chip could provide bootloader(u-boot) per CPU, a more indirect approach of using PCI-Express Switch as a secondary communication and boot would be a nice approach. Then all the CPU's would be in the memory-map of the control CPU.
I'd use Liquid Metal Cooling(No fans on the heat-sinks, less space), with a nice vortex cooler.
cooler integrated in the case. Add a couple of SLI PCI-Express card slots for "Graphics" would be nice.
OS would be debian (or yellow dog).
Wednesday, June 01, 2005
Matrix Technology: Large Scale Fish to Oil Conversion
With the rising price of oil, and the peaking of oil production, there is new ways of making "money" in the world. While Matrix viewed humans as a energy source, I tend to think a fast growing, fast multiplying species is a bit more suitable.
First, a introduction. It appears in the next 3-10 years that the world oil supply production capacity will peak. This means that oil prices will continue to rise. The issue is how do we increase the supply of oil, considering there is no m0re "new" dinosaurs available.
So we have two technical problems for our business plan.
1. Production of large amounts of "Protein" in the most cost effective and space/time conservative way possible.
2. Conversion of the Protein into Gasoline.
Problem #1
The easiest way of producing protein I believe is Fish Farming in lay terms. If the Shallow raceway technique is used, with a recycling system, we can produce huge amounts of fish in limited space, which translates to we can rack and stack "trays" in a continuous system to allow for the large scale production of fish. We will want to tune the system to "increase" the production and yield, while minimizing the impact in the environment. Since the concept is a large scale production factory, with all of the waste input into the next "stage" the environment should be saved alot of issues.
http://www.blackwell-synergy.com/links/doi/10.1046/j.1365-2109.1999.00408.x/abs/
Problem #2
We need to convert the fish into oil. The leading technology for this is currently used to convert "turnkey" waste into oil.
To see how it works, look at http://www.changingworldtech.com/what/index.asp
Combining the technology with lower labor and construction cost, where there currently exists large scale fishfarming, and having a direct connect between a high-density automated fish-farm directly feeding a conversion plant should yield oil that is profitable in the world with rising oil prices.
First, a introduction. It appears in the next 3-10 years that the world oil supply production capacity will peak. This means that oil prices will continue to rise. The issue is how do we increase the supply of oil, considering there is no m0re "new" dinosaurs available.
So we have two technical problems for our business plan.
1. Production of large amounts of "Protein" in the most cost effective and space/time conservative way possible.
2. Conversion of the Protein into Gasoline.
Problem #1
The easiest way of producing protein I believe is Fish Farming in lay terms. If the Shallow raceway technique is used, with a recycling system, we can produce huge amounts of fish in limited space, which translates to we can rack and stack "trays" in a continuous system to allow for the large scale production of fish. We will want to tune the system to "increase" the production and yield, while minimizing the impact in the environment. Since the concept is a large scale production factory, with all of the waste input into the next "stage" the environment should be saved alot of issues.
http://www.blackwell-synergy.com/links/doi/10.1046/j.1365-2109.1999.00408.x/abs/
Problem #2
We need to convert the fish into oil. The leading technology for this is currently used to convert "turnkey" waste into oil.
To see how it works, look at http://www.changingworldtech.com/what/index.asp
Combining the technology with lower labor and construction cost, where there currently exists large scale fishfarming, and having a direct connect between a high-density automated fish-farm directly feeding a conversion plant should yield oil that is profitable in the world with rising oil prices.
Saturday, May 28, 2005
Star Wars Technology: Cybernetic Cooling
One of my areas of interest is AI. The transference of Human level intelligence into a "computer" brain. Once famous futurest is saying within 10 years it should be possible because PC's now are 1% of a human brain. Hate to tell him, we already have the technology. A hundred "blades", or even a "baby" supercomputer(There is one that has 100+ pc's in a normal pc size case) already can be purchased off the shelf. So least having a non-mobile version is easy. Actually packet processors are getting to the point they make the idea solution for grey-matter replacement. Built-in 10Gig Ethernet ports, and 16 64-bit processors means we can string a large number of these little guys togeather with not much glue. The issue we always get is "cooling".
In the body, one of the things that helps to distribute "heat" and keep everything in balance is the blood. Even PC's now can easily be cooled with Water Cooling, with a simple kit. The product is done by Cooler Master and is called AquaGate Mini. The heat is easily dumped outside. This idea for your PC you have in your bedroom, or in your home office and want low-noise and dont want the machine to overheat when the aircon is off.
http://www.hardwarezone.com/news/view.php?id=1313&cid=3
But the problem is, in a "brain" replacement, our array of 30 x 16 x 64bit packet processors and memory are really going to generate some heat. Water is just not enough, and we really dont want to heat up the blood that much. So a "aux" cooling system is mandantory. Also the "fan" is a major problem of reliability.
Enter the solution, the "new" blood to solve our problem, is "liquid" metal. With the ability to carry 16x the heat of water, and no worry of conversion to steam to about 2000 degrees. (New meaning to blowing your top if your using water cooling). The product is invented by a company called NanoCoolers. Non-toxic, no moving parts for the pump, it allows us to increase the amount of computing power in a small space. So the dirty little secret in all computing is heat is the biggest limiting factor to making things go faster. Now we can increase the density and power signficantly while keeping things below melting.
http://www.nanocoolers.com/products_cooling.php
In the body, one of the things that helps to distribute "heat" and keep everything in balance is the blood. Even PC's now can easily be cooled with Water Cooling, with a simple kit. The product is done by Cooler Master and is called AquaGate Mini. The heat is easily dumped outside. This idea for your PC you have in your bedroom, or in your home office and want low-noise and dont want the machine to overheat when the aircon is off.
http://www.hardwarezone.com/news/view.php?id=1313&cid=3
But the problem is, in a "brain" replacement, our array of 30 x 16 x 64bit packet processors and memory are really going to generate some heat. Water is just not enough, and we really dont want to heat up the blood that much. So a "aux" cooling system is mandantory. Also the "fan" is a major problem of reliability.
Enter the solution, the "new" blood to solve our problem, is "liquid" metal. With the ability to carry 16x the heat of water, and no worry of conversion to steam to about 2000 degrees. (New meaning to blowing your top if your using water cooling). The product is invented by a company called NanoCoolers. Non-toxic, no moving parts for the pump, it allows us to increase the amount of computing power in a small space. So the dirty little secret in all computing is heat is the biggest limiting factor to making things go faster. Now we can increase the density and power signficantly while keeping things below melting.
http://www.nanocoolers.com/products_cooling.php
Thursday, May 19, 2005
The "Extreme" Motherboard
Everyone has heard of the rapid progress that graphics and CPU's have made. Now we can have two of the highend graphics cards in our pc, and even let them help each other render the latest game schene.
One of the key ways that PC's are moving ahead is PCI-Express.
First a little bit of bus history
IBM Channels - The beginning of all things - Think Firehose size cables
ISA - The beginning of
Derived PCMCIA/Compact Flash
EISA - A upgrade to ISA, short term fix for making ISA faster
MCA - Dead End
PCI - One of the first complete redesign in PC history
AGP(x) - A mid-life fix for the slowness of PCI to make graphics card happy.
PCI-X - Faster Clock and Wider (64 Bit)
PCI-Express - Wow a totally new and radical design
PCI-Express is a "serial" bus. Most people have heard of SATA/SATA2. Well good old PCI has had a "rewrite" at the hardware level. The lovely thing is it keeps the software infrastructure of PCI.
PCI-Express comes in a few different sizes. The number of "lanes". So you have x1, x4, x8, x16, and eventually x32 and x64. One lane is "equal" in speed to a PCI bus, but only need two pairs of wires. So a x8 or x16 blows away PCI as well as PCI-X, as well as AGP.
So the perfect motherboard, first the CPU's, dual/dual is the base line, so we are talking 2 CPU Chips each with a dual processor. AMD or INtel is fine. So that gets us a quad CPU. Lots of memory is ok. But the best part is the "slots". Forget the number x1, x4 type slots.
The perfect motherboard would have 6 x8 slots, preferably with support for 3 SLI bridges. Yes we are talking the ability to have 6 of the biggest N-Vidia or ATI cards known to man.
But you say "how". Well the "secret" to doing this is already being made.
A company called "PLX Technology" www.plxtech.com makes a part that will allow us to take a x8 (Most MB have dual x8 that support SLI today, and expand it. The part is called a PEG8532. This is a complete "switch" for PCI-Express. Allowing us to talk to each card at full speed. So must faster than a bridge chip.
Of course I would recommend you use water cooling, since the total heat output is going to be the biggest issue. (Hint for this one, HeatPipe, or WaterCooling)
In addition, this allows for large ammounts of other types of high capacity cards. A X8 slot can serve 10G ethernet quite well. Or a high-end RAID card can use the capacity well. So many things to consume bandwidth.
One of the key ways that PC's are moving ahead is PCI-Express.
First a little bit of bus history
IBM Channels - The beginning of all things - Think Firehose size cables
ISA - The beginning of
Derived PCMCIA/Compact Flash
EISA - A upgrade to ISA, short term fix for making ISA faster
MCA - Dead End
PCI - One of the first complete redesign in PC history
AGP(x) - A mid-life fix for the slowness of PCI to make graphics card happy.
PCI-X - Faster Clock and Wider (64 Bit)
PCI-Express - Wow a totally new and radical design
PCI-Express is a "serial" bus. Most people have heard of SATA/SATA2. Well good old PCI has had a "rewrite" at the hardware level. The lovely thing is it keeps the software infrastructure of PCI.
PCI-Express comes in a few different sizes. The number of "lanes". So you have x1, x4, x8, x16, and eventually x32 and x64. One lane is "equal" in speed to a PCI bus, but only need two pairs of wires. So a x8 or x16 blows away PCI as well as PCI-X, as well as AGP.
So the perfect motherboard, first the CPU's, dual/dual is the base line, so we are talking 2 CPU Chips each with a dual processor. AMD or INtel is fine. So that gets us a quad CPU. Lots of memory is ok. But the best part is the "slots". Forget the number x1, x4 type slots.
The perfect motherboard would have 6 x8 slots, preferably with support for 3 SLI bridges. Yes we are talking the ability to have 6 of the biggest N-Vidia or ATI cards known to man.
But you say "how". Well the "secret" to doing this is already being made.
A company called "PLX Technology" www.plxtech.com makes a part that will allow us to take a x8 (Most MB have dual x8 that support SLI today, and expand it. The part is called a PEG8532. This is a complete "switch" for PCI-Express. Allowing us to talk to each card at full speed. So must faster than a bridge chip.
Of course I would recommend you use water cooling, since the total heat output is going to be the biggest issue. (Hint for this one, HeatPipe, or WaterCooling)
In addition, this allows for large ammounts of other types of high capacity cards. A X8 slot can serve 10G ethernet quite well. Or a high-end RAID card can use the capacity well. So many things to consume bandwidth.
Thursday, May 05, 2005
Matrix Technology - Low Cost Mass Market "Rapid" Production
The question is on in "DARPA" to make a "replicator". A desktop "lab" that would allow you to "make" anything you wanted, on demand.
Well, its going to take a long time to get to that point, but something that can really shake up production and manufacturing technology has arrived. Its called HVPF™ Print-Forming.
Its done by a company called EOPLEX. What's so special about it. It allows you to product complex "devices" that include physical, mechanical, electrical, and "plumbing" elements, either in ceramic, or polymer. http://www.eoplex.com
So what are the application examples:
1. A new "P4" socket - Based on EOPLEX ceramic ink -
Image getting the perfomance of "million" dollar "test" sockets - Your front-side bus could run at multi-gigahertz speeds, and the cost of the socket be reasonable.
2. Ceramic PCB for handphones - RF loves ceramic. Being able to do a very dense RF friendly PCB or substrate solves all types of packaging and RF issues.
3. Bio-Mems - In Singopre one of the universities has created a custom fish-tank that uses bateria to "scrub" the various waste products directly out of the water. The trick is making a filter that holds that bateria. With the polymer, a "bio-reactor" can be created that hold standard or custom bacteria, and reagents and perform useful work. From disease detection thru keeping your fishtank clean. You may think the fishtank idea is "low" value, but the price on the current product is in the multiple thousands. Having a "disposable" catridge system that can be mass producted is great for the consumer, and the "technology".
4. New generation printheads - HP and Canon all have issues in manufacturing enough "printheads". The technology gets even more complex, and needs very custom equipment and tooling to make the current printheads. This technology could lower the cost, and push up the qaulity even more.
There's many more earth shattering products that can be made with this process. Stay tuned, and I will write more.
Well, its going to take a long time to get to that point, but something that can really shake up production and manufacturing technology has arrived. Its called HVPF™ Print-Forming.
Its done by a company called EOPLEX. What's so special about it. It allows you to product complex "devices" that include physical, mechanical, electrical, and "plumbing" elements, either in ceramic, or polymer. http://www.eoplex.com
So what are the application examples:
1. A new "P4" socket - Based on EOPLEX ceramic ink -
Image getting the perfomance of "million" dollar "test" sockets - Your front-side bus could run at multi-gigahertz speeds, and the cost of the socket be reasonable.
2. Ceramic PCB for handphones - RF loves ceramic. Being able to do a very dense RF friendly PCB or substrate solves all types of packaging and RF issues.
3. Bio-Mems - In Singopre one of the universities has created a custom fish-tank that uses bateria to "scrub" the various waste products directly out of the water. The trick is making a filter that holds that bateria. With the polymer, a "bio-reactor" can be created that hold standard or custom bacteria, and reagents and perform useful work. From disease detection thru keeping your fishtank clean. You may think the fishtank idea is "low" value, but the price on the current product is in the multiple thousands. Having a "disposable" catridge system that can be mass producted is great for the consumer, and the "technology".
4. New generation printheads - HP and Canon all have issues in manufacturing enough "printheads". The technology gets even more complex, and needs very custom equipment and tooling to make the current printheads. This technology could lower the cost, and push up the qaulity even more.
There's many more earth shattering products that can be made with this process. Stay tuned, and I will write more.
Wednesday, May 04, 2005
The MAID Did it - Tape is Dead
We all "hate" backups, they take to long, and the media is problematic. After all it did start life as sticky tape. (Referece to the "Secret Life of Machines"). So what I generally do for backups is to use USB based hard drives. Since max capacity is 400Gigabytes now, and expanding it looks like to a terabyte, it makes a idea way of plug and go. If you havent priced the cost of high-perfomance tape drives, and tapes, you should. Its very expensive. A small autochanger and drive, plus some tapes exceed 15K US$. (30K S$). So you can buy alot of US$100+ drives and usb enclosures. Backup speed with USB is fast, and the drive can be plugged into anything from a server to a workstation. But for true automation and enterprise class storage, the answer has traditionally been a SILO. A SILO is the size of your typical agri silo, but its full of 1inch tapes, and 8 or more drives. Lot of weight, log of space, and still its tape. In my previous life, I did product definition and product marketing in this segment. Wonderful to play with even the small versions of the technology, but alas expensive. So what happens when we combine the latest in hard drive technology, with the need to store alot of data. The answer is a "MAID".
MAID stands for "Massive Array of Inactive or Independant Disk". Thing about it, with a SATA interface, we can now stack 896 drives in one normal rack. The power to the drives can be managed in software, keeping most powered down, and with "first" generation product you get 224Terabytes of storage. Of course with next generation anounced drives your looking at 896Terabytes, pontentially this year in the same or similar product. Cost per "terabyte" will go down even further. The other benefits are "fast" access. If the drive is powered down, takes no time to spin him up, and if he's up, then access is disk speed. Also we can put 4 to 8 racks in the place of one silo. Lower maintence, full automation, and automatic redundancy is all included.
So all of the above exists today, see COPAN Systems, http://www.copansys.com/products/
There packaging is unique, and innovative. You have to go thru a few photos to get to the "meat", but in the end you see it. Each "blade" of the system is mulitple drives. So in a nomal raid you could get 8 or 16 drives in 3 to 4U, now you can turn them on edge and get 8X the number. Very very good design.
Now, for my value add.
Lets look at the next generation of COVAN. Since I'm not a staff, I can write it up for all to see. (By the way I believe that some of the management that COVAN has done is great).
First SATA is good, so we stick with that.
So specs are:
896 1 Terabyte SATA-2 Drives
Each Tray has a "Octal" SATA controller with PCI-Express connection.
The backplane then connects to a large-scale PCI-Express Switch. Allowing us to rack and stack as many or more drives. This switch is composed of off-the-shelf high-perfomance PCI-Express chips.
All of the above is connected to a Cavium Network Processor.
16 64 bit MIPS on one die, plug 10Gigabit Ethernet out.
We run linux, plus a set of management software that further expands the
functions of the MAID to make it a complete archival storage server offering WEB/SMB and NFS suppport.
The lovely part of it, is all the silicon exists today, and the drives are already anounced.
MAID stands for "Massive Array of Inactive or Independant Disk". Thing about it, with a SATA interface, we can now stack 896 drives in one normal rack. The power to the drives can be managed in software, keeping most powered down, and with "first" generation product you get 224Terabytes of storage. Of course with next generation anounced drives your looking at 896Terabytes, pontentially this year in the same or similar product. Cost per "terabyte" will go down even further. The other benefits are "fast" access. If the drive is powered down, takes no time to spin him up, and if he's up, then access is disk speed. Also we can put 4 to 8 racks in the place of one silo. Lower maintence, full automation, and automatic redundancy is all included.
So all of the above exists today, see COPAN Systems, http://www.copansys.com/products/
There packaging is unique, and innovative. You have to go thru a few photos to get to the "meat", but in the end you see it. Each "blade" of the system is mulitple drives. So in a nomal raid you could get 8 or 16 drives in 3 to 4U, now you can turn them on edge and get 8X the number. Very very good design.
Now, for my value add.
Lets look at the next generation of COVAN. Since I'm not a staff, I can write it up for all to see. (By the way I believe that some of the management that COVAN has done is great).
First SATA is good, so we stick with that.
So specs are:
896 1 Terabyte SATA-2 Drives
Each Tray has a "Octal" SATA controller with PCI-Express connection.
The backplane then connects to a large-scale PCI-Express Switch. Allowing us to rack and stack as many or more drives. This switch is composed of off-the-shelf high-perfomance PCI-Express chips.
All of the above is connected to a Cavium Network Processor.
16 64 bit MIPS on one die, plug 10Gigabit Ethernet out.
We run linux, plus a set of management software that further expands the
functions of the MAID to make it a complete archival storage server offering WEB/SMB and NFS suppport.
The lovely part of it, is all the silicon exists today, and the drives are already anounced.
Tuesday, April 26, 2005
Embedded Systems Overivew
There are lots of "processors" in the world. The ones you dont tihnk about are called embedded systems. So a "non-embedded" system is your PC Desktop or Laptop.
Embedded systems are:
1. TV - Assuming it has a CPU, but most do for the last 10+ years
2. DVD Player
3. MP3
4. Gameboy
5. Xbox
6. Printer
7. Handphone
8. Digital Camera
9. Osim Blood Preasure Monitor
10. Your Wifi card in your pc - Most have 2 CPU onboard
11. Your AP(AccessPoint)
12. HighPerfomance RAID Card inside your PC
13. Router
14. Firewall
15. Washing Machine
In short anything that has a CPU that not a laptop or a pc.
A lot of embedded systems use linux inside, for hints of what these are see http://www.linuxdevices.com/
Embedded Systems Sizing:
1. Standard Remote controls - Typically 8 bit processors - See www.zilog.com or www.microchip.com
2. Toys with blinkey lights, Simple test equipment, and a variety of gadgets - See www.microchip.com
3. Better Games - Such as Gameboy SP - 16 Bit ARM 7 - www.arm.com
4. Settop boxes, DVD Players, Mpeg Recorders, Handphones - Arm 9 - 32 Bit - A variety of guys, see www.intel.com www.atmel.com In atmel the 9200 part is nice. USB support and NOR and NAND flash
5. X86 in embedded systems - Very common since software development is easy. Even originall pentium is often ok for embedded systems. Often these will run linux.
Linux Embedded
Typically running linux embedded means you may not have all the resources that are available on a PC. While DRAM is typically still used in most embedded linux systems, having a hard dirive is a ultra luxury. Since most embedded systems the ease of a hard drive breaking makes it not viable, and also it add signficant cost. So flash is typically used. So the system typically uses a boot looder(u-boot or redboot), that boots the kernel(linux 2.4 or 2.6) and the file system is typically in flash. Linux as always supports a variety of file systems. JFFS2 is the best in that it gives you a full-featured and robust disk like file system while solving all the funny issues related to flash. After that, it just a matter of what application your doing, and what software you want to configure on your embedded system.
Learning Embedded Systems
Good way to learn about embedded systems if you dont mind reading or scanning code is just download the kernel from www.kernel.org, and look at arch other than the pc. There also several good books on the subject that gives you a how-to.
Definitions of Embedded system on the Web:
A computer system that is a component of a larger machine or system. Embedded systems can respond to events in real time. Most digital appliances, such as watches or cars, utilize an embedded system.www.mbizsolutions.com/knowledge/guides/wirelessglossary.htm
Hardware and software that forms a component of some larger system and is expected to function without human intervention. Typically an embedded system consists of a single-board microcomputer with software in ROM, which starts running a dedicated application as soon as power is turned on and does not stop until power is turned off.
A specialized computer system which is dedicated to a specific task. Embedded systems range in size from a single processing board to systems with operating systems (ex, Linux, Windows® NT Embedded). Examples of embedded systems are medical equipment and manufacturing equipment.www.systemsoft.com/l-2/l-3/support-glossary.htm
A combination of computer hardware and software, and perhaps additional mechanical or other parts, designed to perform a dedicated function. In some cases, embedded systems are part of a larger system or product, as in the case of an antilock braking system in a car. Contrast with general-purpose computer.www.netsilicon.com/support/embeddedglossary.jsp
A device, usually with a singular function such as controlling a piece of machinery on an assembly line, that contains a microprocessor. Inability to handle information such as four-digit dates or leap days can cause these systems that depend on them to shut down in the year 2000.www.crm.mb.ca/guide/glossary.html
An embedded system is some combination of computer hardware and software, either fixed in capability or programmable, that is specifically designed for a particular kind of application device. Industrial machines, automobiles, medical equipment, cameras, household appliances, airplanes, vending machines, and toys (as well as the more obvious cellular phone and PDA) are among the myriad possible hosts of an embedded system. Embedded systems that are programmable are provided with a programming interface, and embedded systems programming is a specialized occupation.www.caps-entreprise.com/en/glossaire_contenu.html
A phrase that refers to a device that contains computer logic on a chip inside it. Such equipment is electrical or battery powered. The chip controls one or more functions of the equipment, such as remembering how long it has been since the device last received maintenance.www.ucsf.edu/y2k/toolkit/gloss.html
An embedded system is a special-purpose computer system, which is completely encapsulated by the device it controls.An embedded system has specific requirements and performs pre-defined tasks, unlike a general-purpose personal computer. en.wikipedia.org/wiki/Embedded_system
Embedded systems are:
1. TV - Assuming it has a CPU, but most do for the last 10+ years
2. DVD Player
3. MP3
4. Gameboy
5. Xbox
6. Printer
7. Handphone
8. Digital Camera
9. Osim Blood Preasure Monitor
10. Your Wifi card in your pc - Most have 2 CPU onboard
11. Your AP(AccessPoint)
12. HighPerfomance RAID Card inside your PC
13. Router
14. Firewall
15. Washing Machine
In short anything that has a CPU that not a laptop or a pc.
A lot of embedded systems use linux inside, for hints of what these are see http://www.linuxdevices.com/
Embedded Systems Sizing:
1. Standard Remote controls - Typically 8 bit processors - See www.zilog.com or www.microchip.com
2. Toys with blinkey lights, Simple test equipment, and a variety of gadgets - See www.microchip.com
3. Better Games - Such as Gameboy SP - 16 Bit ARM 7 - www.arm.com
4. Settop boxes, DVD Players, Mpeg Recorders, Handphones - Arm 9 - 32 Bit - A variety of guys, see www.intel.com www.atmel.com In atmel the 9200 part is nice. USB support and NOR and NAND flash
5. X86 in embedded systems - Very common since software development is easy. Even originall pentium is often ok for embedded systems. Often these will run linux.
Linux Embedded
Typically running linux embedded means you may not have all the resources that are available on a PC. While DRAM is typically still used in most embedded linux systems, having a hard dirive is a ultra luxury. Since most embedded systems the ease of a hard drive breaking makes it not viable, and also it add signficant cost. So flash is typically used. So the system typically uses a boot looder(u-boot or redboot), that boots the kernel(linux 2.4 or 2.6) and the file system is typically in flash. Linux as always supports a variety of file systems. JFFS2 is the best in that it gives you a full-featured and robust disk like file system while solving all the funny issues related to flash. After that, it just a matter of what application your doing, and what software you want to configure on your embedded system.
Learning Embedded Systems
Good way to learn about embedded systems if you dont mind reading or scanning code is just download the kernel from www.kernel.org, and look at arch other than the pc. There also several good books on the subject that gives you a how-to.
Definitions of Embedded system on the Web:
A computer system that is a component of a larger machine or system. Embedded systems can respond to events in real time. Most digital appliances, such as watches or cars, utilize an embedded system.www.mbizsolutions.com/knowledge/guides/wirelessglossary.htm
Hardware and software that forms a component of some larger system and is expected to function without human intervention. Typically an embedded system consists of a single-board microcomputer with software in ROM, which starts running a dedicated application as soon as power is turned on and does not stop until power is turned off.
A specialized computer system which is dedicated to a specific task. Embedded systems range in size from a single processing board to systems with operating systems (ex, Linux, Windows® NT Embedded). Examples of embedded systems are medical equipment and manufacturing equipment.www.systemsoft.com/l-2/l-3/support-glossary.htm
A combination of computer hardware and software, and perhaps additional mechanical or other parts, designed to perform a dedicated function. In some cases, embedded systems are part of a larger system or product, as in the case of an antilock braking system in a car. Contrast with general-purpose computer.www.netsilicon.com/support/embeddedglossary.jsp
A device, usually with a singular function such as controlling a piece of machinery on an assembly line, that contains a microprocessor. Inability to handle information such as four-digit dates or leap days can cause these systems that depend on them to shut down in the year 2000.www.crm.mb.ca/guide/glossary.html
An embedded system is some combination of computer hardware and software, either fixed in capability or programmable, that is specifically designed for a particular kind of application device. Industrial machines, automobiles, medical equipment, cameras, household appliances, airplanes, vending machines, and toys (as well as the more obvious cellular phone and PDA) are among the myriad possible hosts of an embedded system. Embedded systems that are programmable are provided with a programming interface, and embedded systems programming is a specialized occupation.www.caps-entreprise.com/en/glossaire_contenu.html
A phrase that refers to a device that contains computer logic on a chip inside it. Such equipment is electrical or battery powered. The chip controls one or more functions of the equipment, such as remembering how long it has been since the device last received maintenance.www.ucsf.edu/y2k/toolkit/gloss.html
An embedded system is a special-purpose computer system, which is completely encapsulated by the device it controls.An embedded system has specific requirements and performs pre-defined tasks, unlike a general-purpose personal computer. en.wikipedia.org/wiki/Embedded_system
Monday, April 25, 2005
BusyBox and Linux 2.6.11.6
Hmm. Finally got to do something fun. So I returned back to updating our distro to 2.6.11.6 on a ixp425(ixp4xx).
Had a problem for a while when using Busybox, that insmod would not work from busybox. Everyone seems to be using it just fine, but I could not use it. I would get a wonderful error message about "structures" not having the right version.
Well your kernel config seems to have alot to do with this coming out. Actually now its listed in bugtracking with busybox as a problem. (You'll see my comments as well). The good news if you turn off the genration of debug symbols in your kernel the problem disappears.
Now I can move on to the next problem. (Of course I get to do this fun stuff about 1 day every 2 weeks, so guess I cant complain to much).
Had a problem for a while when using Busybox, that insmod would not work from busybox. Everyone seems to be using it just fine, but I could not use it. I would get a wonderful error message about "structures" not having the right version.
Well your kernel config seems to have alot to do with this coming out. Actually now its listed in bugtracking with busybox as a problem. (You'll see my comments as well). The good news if you turn off the genration of debug symbols in your kernel the problem disappears.
Now I can move on to the next problem. (Of course I get to do this fun stuff about 1 day every 2 weeks, so guess I cant complain to much).
Thursday, April 21, 2005
Blade Servers
Interesting News
I have always liked Intel Advanced Communications Architecture. Of course I should since it was near one of the last projects I did in my job at Celerity systems. The theory is very simple. Build a chassis that uses ethernet as the "fabric". And Plug in "blades" that get there power/monitor/keyboard/network from the "backplane". This makes a lovely system, with easy maintence and expansion. Seems like PC are like potato chips, never can get enough. So rack and stack and slide in more. With the cooling and the cables all neetly handled.
Well, just found a revelation, seems like Intel and IBM worked togeather. IBM blade servers are based on the ACA specification. I love that. Come to find out the specification is open, so I can build my own blades. Even better. Now just need to figure out how to build a switch fabric that can switch 10 gigs per slot. smile
I have always liked Intel Advanced Communications Architecture. Of course I should since it was near one of the last projects I did in my job at Celerity systems. The theory is very simple. Build a chassis that uses ethernet as the "fabric". And Plug in "blades" that get there power/monitor/keyboard/network from the "backplane". This makes a lovely system, with easy maintence and expansion. Seems like PC are like potato chips, never can get enough. So rack and stack and slide in more. With the cooling and the cables all neetly handled.
Well, just found a revelation, seems like Intel and IBM worked togeather. IBM blade servers are based on the ACA specification. I love that. Come to find out the specification is open, so I can build my own blades. Even better. Now just need to figure out how to build a switch fabric that can switch 10 gigs per slot. smile
Wednesday, April 20, 2005
Microsoft Bill Gates Final Answer to Fight Linux
Interesting to follow the strategy of Microsoft and Bill Gates in the fight against Linux. Linux and its associated projects are starting to make major inroads into every industry and government. This is taking market share away from Microsoft. While overall the effect is small its gaining momentum.
Current Microsoft Strategy
1. FUD - Fear Uncertainty and Doubt
a. A bit of funding to the SCO law suit is one of the keys of this. Then having talks about
how Linux has stollen IPR from Microsoft. (Hmm look in the mirror lately Mr. Gates),
and that cusotmers could be sued for this IPR theft. Of course IBM countered by setting up a IPR opensource pool that countered this totally. Generally software patents are a bit shackey anyway, and on top of that is its alot easier to find "pre-existing" art. in the software domain.
b. No one knows who wrote linux. Excuse me. If you ever look at any "project" of linux, from the kernel to device drivers to open-office, the maintainers are all listed with email contacts. On top of this pracallity all projects have a complete history of who offered what patch and when it was integrated. In reality we know no one other than Mr. Gates that contributed code to Windows. Nor is the source code auditable, nor do we see evidence of audit. Look at any history related to the linux kernel. Every version is available, and every patch, and all the comments are available any time you wish to review it.
2. Hire Key Sales Staff away from Linux Partners
This one is a local "report" from a friend that there is a general "Compaign" by Microsoft that key staff at Red Hat, Sun and Other Linux/Unix Players are being targeted for hiring in mass. There reported job is to "dish" linux and sell windows in it place.
The Better Strategy
A better strategy is to "join" the movement. There is nothing preventing Microsoft doing a Linux version. Seems I remember in the early days that Microsoft even played with this. (15 years ago if memory serves). So why not "adapt" and do a version of Linux with Microsoft name. Support Microsoft Office and applications, and Microsoft applications on top, as well as any open-source ones needed. I'd prefer it to be a Debian one. The pricing may not be that different than that of Windows. This would let Microsoft and the Linux world win.
Current Microsoft Strategy
1. FUD - Fear Uncertainty and Doubt
a. A bit of funding to the SCO law suit is one of the keys of this. Then having talks about
how Linux has stollen IPR from Microsoft. (Hmm look in the mirror lately Mr. Gates),
and that cusotmers could be sued for this IPR theft. Of course IBM countered by setting up a IPR opensource pool that countered this totally. Generally software patents are a bit shackey anyway, and on top of that is its alot easier to find "pre-existing" art. in the software domain.
b. No one knows who wrote linux. Excuse me. If you ever look at any "project" of linux, from the kernel to device drivers to open-office, the maintainers are all listed with email contacts. On top of this pracallity all projects have a complete history of who offered what patch and when it was integrated. In reality we know no one other than Mr. Gates that contributed code to Windows. Nor is the source code auditable, nor do we see evidence of audit. Look at any history related to the linux kernel. Every version is available, and every patch, and all the comments are available any time you wish to review it.
2. Hire Key Sales Staff away from Linux Partners
This one is a local "report" from a friend that there is a general "Compaign" by Microsoft that key staff at Red Hat, Sun and Other Linux/Unix Players are being targeted for hiring in mass. There reported job is to "dish" linux and sell windows in it place.
The Better Strategy
A better strategy is to "join" the movement. There is nothing preventing Microsoft doing a Linux version. Seems I remember in the early days that Microsoft even played with this. (15 years ago if memory serves). So why not "adapt" and do a version of Linux with Microsoft name. Support Microsoft Office and applications, and Microsoft applications on top, as well as any open-source ones needed. I'd prefer it to be a Debian one. The pricing may not be that different than that of Windows. This would let Microsoft and the Linux world win.
Monday, April 18, 2005
Spying and Hacking 101 - Backing up a hard drive - Free Tools
Hmm. I have lot of respect for Norton and other tools, but my favorite way of doing backups now is using a version of Linux called Knoppix. Knoppix is a Linux ditribution designed to run totally from a CD-ROM. Everything you could possibly need is included on the CD. From hard disk tools thru office applications. It does not touch your hard drive at all. (Unless of course you tell it to).
Windowx(XP) has this problem that backup is not part of the OS. ntbackup is unreliable on the best day, and not exactly fast. Most people prefer to do image backup. Norton Ghost being a very popular one. Often we need to look around, recover files from a virus attack, or backup a pc from a employee that is not up to our ethics standard. I've had situation where a staff password locked the PC at the bios level, and changed all the passwords on the machine to hide his work, or lack of it. So how do you a) backup the data so it cannot be lost or altered, b) get arround all those levels of security. Well first the bios passwords are pretty easy, removing the motherboard battery fixes that problem.
The next issue is getting a backup.
So, how to do it. First step get you a copy of knoppix. The latest as of this writing is 3.8.1. The web site is www.knoppix.com. This gives you 2.6.11 linux kernel. Write support for ntfs, centrino 2 wirless support, Graphical Windows Environment KDE 3.3.2, as well as the basic tools needed to do your backup and recovery.
So to backup, my first recommendation is get you a USB2 400 Megabyte hard drive. This way you can plug it directly to the PC in question, boot knoppix on a CD, and copy away. I would recommend you put a REISER file system on the external hard drive. Knoppix will recognize it, and let you mount it very easily. You other choice is to use a wired(ethernet) or wireless network card. This of course is not as fast. Dont forget the typicall hard drive is 40 to 80 gigabytes in todays laptops.
So now you have selected either a "fileserver" which can be another pc running linux or windows, or a local usb2 hard drive. Knoppix will mount the external hard drive for you, and mount your internal as a read-only one by default.
Now you can do a image backup using the "dd" command.
To get a good overview of how this process works consult:
http://www.inference.phy.cam.ac.uk/saw27/notes/backup-hard-disk-partitions.html
Features of this method.
1. Can extract a exact image of the whole disk complete with MBR(Master Boot record) as well as all partitions.
2. Can be compressed if you prefer
3. On another machine the image can be mounted directly. (Assuming Linux) and any or all files may be accessed.
4. A 400Gigabyte hard drive can backup 9 machines(Assuming laptops with 40 Gigabyte drives.).
5. A script can be created that would allow the update of the image if its mounted on a file server(Another linux box). So think about it. You can archive a "base" image, and monthly images, then update/sync the current image to the server, so that everything stays in sync.
Now the wonderful thing is, on a disk crash we can restore everything without worry of needeing to re-install individual applications.
Windowx(XP) has this problem that backup is not part of the OS. ntbackup is unreliable on the best day, and not exactly fast. Most people prefer to do image backup. Norton Ghost being a very popular one. Often we need to look around, recover files from a virus attack, or backup a pc from a employee that is not up to our ethics standard. I've had situation where a staff password locked the PC at the bios level, and changed all the passwords on the machine to hide his work, or lack of it. So how do you a) backup the data so it cannot be lost or altered, b) get arround all those levels of security. Well first the bios passwords are pretty easy, removing the motherboard battery fixes that problem.
The next issue is getting a backup.
So, how to do it. First step get you a copy of knoppix. The latest as of this writing is 3.8.1. The web site is www.knoppix.com. This gives you 2.6.11 linux kernel. Write support for ntfs, centrino 2 wirless support, Graphical Windows Environment KDE 3.3.2, as well as the basic tools needed to do your backup and recovery.
So to backup, my first recommendation is get you a USB2 400 Megabyte hard drive. This way you can plug it directly to the PC in question, boot knoppix on a CD, and copy away. I would recommend you put a REISER file system on the external hard drive. Knoppix will recognize it, and let you mount it very easily. You other choice is to use a wired(ethernet) or wireless network card. This of course is not as fast. Dont forget the typicall hard drive is 40 to 80 gigabytes in todays laptops.
So now you have selected either a "fileserver" which can be another pc running linux or windows, or a local usb2 hard drive. Knoppix will mount the external hard drive for you, and mount your internal as a read-only one by default.
Now you can do a image backup using the "dd" command.
To get a good overview of how this process works consult:
http://www.inference.phy.cam.ac.uk/saw27/notes/backup-hard-disk-partitions.html
Features of this method.
1. Can extract a exact image of the whole disk complete with MBR(Master Boot record) as well as all partitions.
2. Can be compressed if you prefer
3. On another machine the image can be mounted directly. (Assuming Linux) and any or all files may be accessed.
4. A 400Gigabyte hard drive can backup 9 machines(Assuming laptops with 40 Gigabyte drives.).
5. A script can be created that would allow the update of the image if its mounted on a file server(Another linux box). So think about it. You can archive a "base" image, and monthly images, then update/sync the current image to the server, so that everything stays in sync.
Now the wonderful thing is, on a disk crash we can restore everything without worry of needeing to re-install individual applications.
Matrix Technology Review - Regrowing Limbs
I was surprised the other day that the a protein has been discovered that allows the regrowth of limbs in some animals. The surprising part of this is, it actually reverts cells back to the stem stage. Now, imagine. Take a sample of human cells from a patient, treat them with a soup of this compound. You now have stem cells compatible with the donor. Take the result, and treat them again to make the appropriate cell types needed. (We are starting to understand how to select some pathways in stem cells, and the body does a pretty good job of doing it as well.).
Now, add the next level. A research group has developed a prototype of "printing" organs three dimensionally. Surprise, it uses a ink-jet printer. So you load each of the needed cell-types into cartridge, and drive the setup with a PC. Then you "lay" down each layer of the organ.
One of the hard problems surgeons go thru is that each person is "different", meaning that organ size is different and the shape can be slightly different. So now with a good scan, all things can line up. From the shape, to having the blood vessels in the right place. Kidneys, Heart, Liver all should be replaceable. Longer term limbs may be replaced.
Of course the next problem will be the brain, but hey, science is working on it.
Now, add the next level. A research group has developed a prototype of "printing" organs three dimensionally. Surprise, it uses a ink-jet printer. So you load each of the needed cell-types into cartridge, and drive the setup with a PC. Then you "lay" down each layer of the organ.
One of the hard problems surgeons go thru is that each person is "different", meaning that organ size is different and the shape can be slightly different. So now with a good scan, all things can line up. From the shape, to having the blood vessels in the right place. Kidneys, Heart, Liver all should be replaceable. Longer term limbs may be replaced.
Of course the next problem will be the brain, but hey, science is working on it.
Wednesday, April 13, 2005
The Perfect Embedded Processor - The Beast of a Chip
The Beast is coming. The beast is the modern microprocessor. It has 300-400 legs, and the bad thing is most of these are not really needed. I had a great conversation recently, and found out that some basic beliefs on "cost" are not true. Seems like "MCM"s can be done cheeply now. You wonder what a MCM is. A "MCM" is a multi-chip-module. You ask, "so what". Most people, least the ones in asia, like small sexy little phones. Every phone has to have dram, flash, cpu, plus the radio, not counting the lcd, mike and speaker. Notice the "core" three are always the same. When you design any piece of gear you have to have at least these three. DRAM and Flash memory processes are radically different to that of a processor. But if you can do a MCM, you can take "three" die (Raw chips without a package) and put them into one. (Typically by stacking). So, we come to the "perfect" embedded processor. First a great processor (Xscale, ARM or MIPS is ok), add a large dram(128Megabytes is good), and a large flash(128Megabyte is great), stack them togeather, and give me only my "user" io out the bottom. All of a sudden a large number of pins are gone. Now how can we get rid of the rest? The perfect answer is in USB 2.0, give me 6 to 10 host ports and the typical one device port. There's already off-the-shelf chips that can covert USB2.0 into pci. So a pci bus is not needed. Of course if you want, you can always do x16 lanes of PCI-X (The new PCI is serial). So all of a sudden we can get chips down in the 100's of pins. (Actually I'd be happy with a little as possible, is 40 low enough for you). This will reduce the cost, and increase the flexability. Ethernet, PCI, VGA, etc can all be implemented with a large variety of I/O's.
|
BitKeeper Redux and Open Source
Well, if you read the linux kernel sites, seems like a major battle occured. With the hero switching sides. (Yes I'm talking about the founding farther of Linux). In the end he switched back, sometimes friendships can cause us not to see clearly.
The interesting part of this, over almost 2 years, the linux kernel has been using a "commercial" product for source countrol. This product, "Bitkeeper" did a great job, and was provided to kernel developers for free. I did a eval about a month ago and was getting interested in looking at converting our development away from "Subversion" to bitkeeper. Well needless to say, since its been "pulled" from open-source world, I will make sure none of my staff ever use it. The CEO of bitkeeper says they loose $500,000 a year to the free license they were giving open-source developers. What they dont understand is the amount of goodwill that is created. Of course they will understand the "ill-will" I suspect in about once year. (Assuming the quarterly number dont show it sooner.
Open-Source is reaching critical mass. Its usage is skyrocketing. Money can and will be make out of open-source, but it cannot be extorted. I tend to think that a huge amount of good-will was burned in the instance of bitkeeper.
I believe that contributing to open-source is a great thing to do. Hardware vendors tend to have huge problems in generating good quality code. Let the open-source world do it. With minimal dollars fatastic things can be done. Leverage, use it, make money off of it, but contribute back to it. Yes it cost money, but the proper open-source based approach can make your products richer, more features and functions at a lower-price than any other.
The interesting part of this, over almost 2 years, the linux kernel has been using a "commercial" product for source countrol. This product, "Bitkeeper" did a great job, and was provided to kernel developers for free. I did a eval about a month ago and was getting interested in looking at converting our development away from "Subversion" to bitkeeper. Well needless to say, since its been "pulled" from open-source world, I will make sure none of my staff ever use it. The CEO of bitkeeper says they loose $500,000 a year to the free license they were giving open-source developers. What they dont understand is the amount of goodwill that is created. Of course they will understand the "ill-will" I suspect in about once year. (Assuming the quarterly number dont show it sooner.
Open-Source is reaching critical mass. Its usage is skyrocketing. Money can and will be make out of open-source, but it cannot be extorted. I tend to think that a huge amount of good-will was burned in the instance of bitkeeper.
I believe that contributing to open-source is a great thing to do. Hardware vendors tend to have huge problems in generating good quality code. Let the open-source world do it. With minimal dollars fatastic things can be done. Leverage, use it, make money off of it, but contribute back to it. Yes it cost money, but the proper open-source based approach can make your products richer, more features and functions at a lower-price than any other.
Sunday, March 27, 2005
Debian / Samba Cold War
Wow, finally the light at the end of a tunnel, and come to find out, its all a
cold war ploy.
For a long time on my home server, I've had this "aggravating" problem, that I believe now I'm getting to the bottom of. My "file-server" is a "cast" off machine
running typically debian, and a big hard disk.
I've always had a problem "regardless" of the hardware. The current cast off is really good, its a Pentium 4 with RDRAM ex-gaming machine. So this is antastic hardware for "Debian" or any other linux. With a 120Gig drive, no problem with space.
If you follow Linux history, you will know there is various issues over time with "Large" file support. I've been trying to learn more. All of this swirls in the 4-bit cpu game to add to the entertainment value. Actually big-files (Individual ones) and 64-bit cpu have no relation. The root of the problem is in glibc (C-library) and was solved some time ago. Long as you have the right switch and you application has been compiled recently it will work fine. Just as a test to prove this was not a issue, did a quick dd from /dev/zero, sure enough 11Gigabytes in no time. So we know its not there.
Then went to the next phase, tried doing a dd of a hard disk in my laptop with knoppix to a smb mount. (Wow thats ugly). Should work fine, it does not. Creating a 40G file dies at 2 gig. The samba server dies a poor death. I even tried it with NFS and it dies in a similar fashion.
So now, the big hint comes, whats the problem. Its not the kernel, (See the dd test above) but both NFS and Samba have a issue.
Seems like the hint came from Samba. Debian does not update packages even in unstable or testing it appears very often. Since as well all know updates fix bugs, add features(hmm is it a bug or a feature) routinely. So the samba that I can get from any of the normal repository has samba that is very old. (4 years when the patches first came out). So now let me do the update from binaries provided by Samba, and I bet the problem will disappear. Now a problem I've seen for months will disappear.
cold war ploy.
For a long time on my home server, I've had this "aggravating" problem, that I believe now I'm getting to the bottom of. My "file-server" is a "cast" off machine
running typically debian, and a big hard disk.
I've always had a problem "regardless" of the hardware. The current cast off is really good, its a Pentium 4 with RDRAM ex-gaming machine. So this is antastic hardware for "Debian" or any other linux. With a 120Gig drive, no problem with space.
If you follow Linux history, you will know there is various issues over time with "Large" file support. I've been trying to learn more. All of this swirls in the 4-bit cpu game to add to the entertainment value. Actually big-files (Individual ones) and 64-bit cpu have no relation. The root of the problem is in glibc (C-library) and was solved some time ago. Long as you have the right switch and you application has been compiled recently it will work fine. Just as a test to prove this was not a issue, did a quick dd from /dev/zero, sure enough 11Gigabytes in no time. So we know its not there.
Then went to the next phase, tried doing a dd of a hard disk in my laptop with knoppix to a smb mount. (Wow thats ugly). Should work fine, it does not. Creating a 40G file dies at 2 gig. The samba server dies a poor death. I even tried it with NFS and it dies in a similar fashion.
So now, the big hint comes, whats the problem. Its not the kernel, (See the dd test above) but both NFS and Samba have a issue.
Seems like the hint came from Samba. Debian does not update packages even in unstable or testing it appears very often. Since as well all know updates fix bugs, add features(hmm is it a bug or a feature) routinely. So the samba that I can get from any of the normal repository has samba that is very old. (4 years when the patches first came out). So now let me do the update from binaries provided by Samba, and I bet the problem will disappear. Now a problem I've seen for months will disappear.
Sunday, February 27, 2005
IXP425 and Linux 2.6
Summary: ixp425 Linux 2.6.11-rc5 gcc 3.3.2 glibc 2.3.2 working finally
Well, my software guys said it could not be done. :) I said it could not be done, but then it just had to be done.
In our work we use the Intel IXP425 (IXP4XX) which is used in a variety of Wireless Products, NAS devices, Firewalls etc.
The most common distro for this type of work is SNAPGEAR (www.snapgear.org).
We have wanted to upgrade to 2.6, but have been prevented for some time. After my study, people have been running 2.6 since the 2.6.0-rc1 days, and have been pretty well tracking the releases.
The recipe was missing so to speak.
If you have had similar troubles, fear not the recipe is enclosed.
First, as of this date, the snapgear compiler is really out of date, arm-linux-tools-20031127.tar.gz. The reason behind this, is the kernel now likes being compiled in O2 Mode (Optimizing).
So first thing we need to do is upgrade our tools. The best way to do this is a nice tool called (Crosstool). This can be downloaded at http://kegel.com/crosstool/.
With crosstool, one of the hard lessons learned, that wasted by time was the gcc built by crosstool dont like to be moved. (or at least copied) around. So use them where you build them. (This advice was from kegel). And seems to hold out in reality.
Next thing, that seems to work the best, is in the build, put the full path in the right config file. So in snapgear/vendors/config/arm/config.arch the CROSS_COMPILE variable should contain the full path to the compiler chain.
CROSS_COMPILE=/home/data/armeb-unknown-linux-gnu/gcc-3.3.2-glibc-2.3.2/bin/armeb-unknown-linux-gnu-
The WORST problem, was the bootloader, Redboot 1.92, which so many people tend to use, does not pass the right ÄRCH/board id. This will result in the wonderful thing of getting onll the Uncompressing Linux line and nothing more.
So till I can get to the bottom, the following was added.
In /home/data/snapgear/linux-2.6.x/arch/arm/boot/compressed/head-xscale.S
Reason: Redboot does not pass the right machine ID.
@ Force Coyote -- GSW
mov r7, #(MACH_TYPE_ADI_COYOTE & 0xff)
orr r7, r7, #(MACH_TYPE_ADI_COYOTE & 0xff00)
In the file, you should put this right before the #ifdef for others that have the same problem.
Its a short term solution, but least things start working.
After all of this, you will get something like the log below. Personally I prefer u-boot for the bootloader, I find its easy to build, more features, and less buggy than redboot. The biggest issue the lack of NPE support in u-boot.
Shortly I will get a nice clean kernel build, and a decent file system and put it on a web server so any that want a know working config will have one.
Another issue, the default config for snapgear did not match the default config for 2.6. So we needed to copy the 2.6.11 config over to snapgear so that it would have a good starting point. Generally things dont work in 2.6 if you start with a 2.4 kernel config, or def-config.
The kernel source used with the effort was linux-2.6.11-rc5.tar.gz direct from kernel.org. For arm processors all updates are directly sent to linux, so its proper for arm to pull from here for 2.6. The ARM UK project that used to do patches to kernel.org kernels is no more doing so. All there patches are merged up-stream directly to kernel.org releases. While its a bad idea to pull from kernel.org for your pc, for embedded work, its the best place.
To be done:
A patch to upgrade 3.2.0 snapgear to 2.6.11-rc5, and fix all of the above coming soon.
If you have questions, feel free to email or drop a comment on the blog.
RedBoot> load -r -v -b 0x01600000 zImage
Using default protocol (TFTP)
-
Raw file loaded 0x01600000-0x01746aff, assumed entry at 0x01600000
RedBoot> go -n 0x01600000
Uncompressing Linux.............................................................
Linux version 2.6.11-rc5alo (gwest@ron1) (gcc version 3.3.2) #3 Sun Feb 27 16:05
CPU: XScale-IXP42x Family [690541c1] revision 1 (ARMv5TE)
CPU0: D VIVT undefined 5 cache
CPU0: I cache: 32768 bytes, associativity 32, 32 byte lines, 32 sets
CPU0: D cache: 32768 bytes, associativity 32, 32 byte lines, 32 sets
Machine: ADI Engineering Coyote
Warning: bad configuration page, trying to continue
Memory policy: ECC disabled, Data cache writeback
Built 1 zonelists
Kernel command line: console=ttyS0,115200 ip=bootp root=/dev/nfs
PID hash table entries: 128 (order: 7, 2048 bytes)
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 16MB = 16MB total
Memory: 13288KB available (2049K code, 504K data, 320K init)
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: Testing write buffer coherency: ok
NET: Registered protocol family 16
PCI: IXP4xx is host
PCI: IXP4xx Using direct access for memory space
PCI: bus0: Fast back to back transfers enabled
dmabounce: registered device 0000:00:0e.0 on pci bus
dmabounce: registered device 0000:00:0f.0 on pci bus
NetWinder Floating Point Emulator V0.97 (double precision)
JFFS2 version 2.2. (C) 2001-2003 Red Hat, Inc.
IXP4xx Watchdog Timer: heartbeat 60 sec
Serial: 8250/16550 driver $Revision: 1.90 $ 2 ports, IRQ sharing disabled
ttyS0 at MMIO 0xc8001000 (irq = 13) is a XScale
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 8192K size 1024 blocksize
loop: loaded (max 8 devices)
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
IXP4XX-Flash0: Found 1 x16 devices at 0x0 in 16-bit bank
Intel/Sharp Extended Query Table at 0x0031
Using buffer write method
cfi_cmdset_0001: Erase suspend on write enabled
Searching for RedBoot partition table in IXP4XX-Flash0 at offset 0xfe0000
3 RedBoot partitions found on MTD device IXP4XX-Flash0
Creating 3 MTD partitions on "IXP4XX-Flash0":
0x00000000-0x00060000 : "RedBoot"
0x00fc0000-0x00fc1000 : "RedBoot config"
mtd: partition "RedBoot config" doesn't end on an erase block -- force read-only
0x00fe0000-0x01000000 : "FIS directory"
mice: PS/2 mouse device common for all mice
i2c /dev entries driver
NET: Registered protocol family 2
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
NET: Registered protocol family 1
NET: Registered protocol family 8
NET: Registered protocol family 20
IP-Config: No network devices available.
Root-NFS: No NFS server available, giving up.
VFS: Unable to mount root fs via NFS, trying floppy.
VFS: Cannot open root device "nfs" or unknown-block(2,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
Well, my software guys said it could not be done. :) I said it could not be done, but then it just had to be done.
In our work we use the Intel IXP425 (IXP4XX) which is used in a variety of Wireless Products, NAS devices, Firewalls etc.
The most common distro for this type of work is SNAPGEAR (www.snapgear.org).
We have wanted to upgrade to 2.6, but have been prevented for some time. After my study, people have been running 2.6 since the 2.6.0-rc1 days, and have been pretty well tracking the releases.
The recipe was missing so to speak.
If you have had similar troubles, fear not the recipe is enclosed.
First, as of this date, the snapgear compiler is really out of date, arm-linux-tools-20031127.tar.gz. The reason behind this, is the kernel now likes being compiled in O2 Mode (Optimizing).
So first thing we need to do is upgrade our tools. The best way to do this is a nice tool called (Crosstool). This can be downloaded at http://kegel.com/crosstool/.
With crosstool, one of the hard lessons learned, that wasted by time was the gcc built by crosstool dont like to be moved. (or at least copied) around. So use them where you build them. (This advice was from kegel). And seems to hold out in reality.
Next thing, that seems to work the best, is in the build, put the full path in the right config file. So in snapgear/vendors/config/arm/config.arch the CROSS_COMPILE variable should contain the full path to the compiler chain.
CROSS_COMPILE=/home/data/armeb-unknown-linux-gnu/gcc-3.3.2-glibc-2.3.2/bin/armeb-unknown-linux-gnu-
The WORST problem, was the bootloader, Redboot 1.92, which so many people tend to use, does not pass the right ÄRCH/board id. This will result in the wonderful thing of getting onll the Uncompressing Linux line and nothing more.
So till I can get to the bottom, the following was added.
In /home/data/snapgear/linux-2.6.x/arch/arm/boot/compressed/head-xscale.S
Reason: Redboot does not pass the right machine ID.
@ Force Coyote -- GSW
mov r7, #(MACH_TYPE_ADI_COYOTE & 0xff)
orr r7, r7, #(MACH_TYPE_ADI_COYOTE & 0xff00)
In the file, you should put this right before the #ifdef for others that have the same problem.
Its a short term solution, but least things start working.
After all of this, you will get something like the log below. Personally I prefer u-boot for the bootloader, I find its easy to build, more features, and less buggy than redboot. The biggest issue the lack of NPE support in u-boot.
Shortly I will get a nice clean kernel build, and a decent file system and put it on a web server so any that want a know working config will have one.
Another issue, the default config for snapgear did not match the default config for 2.6. So we needed to copy the 2.6.11 config over to snapgear so that it would have a good starting point. Generally things dont work in 2.6 if you start with a 2.4 kernel config, or def-config.
The kernel source used with the effort was linux-2.6.11-rc5.tar.gz direct from kernel.org. For arm processors all updates are directly sent to linux, so its proper for arm to pull from here for 2.6. The ARM UK project that used to do patches to kernel.org kernels is no more doing so. All there patches are merged up-stream directly to kernel.org releases. While its a bad idea to pull from kernel.org for your pc, for embedded work, its the best place.
To be done:
A patch to upgrade 3.2.0 snapgear to 2.6.11-rc5, and fix all of the above coming soon.
If you have questions, feel free to email or drop a comment on the blog.
RedBoot> load -r -v -b 0x01600000 zImage
Using default protocol (TFTP)
-
Raw file loaded 0x01600000-0x01746aff, assumed entry at 0x01600000
RedBoot> go -n 0x01600000
Uncompressing Linux.............................................................
Linux version 2.6.11-rc5alo (gwest@ron1) (gcc version 3.3.2) #3 Sun Feb 27 16:05
CPU: XScale-IXP42x Family [690541c1] revision 1 (ARMv5TE)
CPU0: D VIVT undefined 5 cache
CPU0: I cache: 32768 bytes, associativity 32, 32 byte lines, 32 sets
CPU0: D cache: 32768 bytes, associativity 32, 32 byte lines, 32 sets
Machine: ADI Engineering Coyote
Warning: bad configuration page, trying to continue
Memory policy: ECC disabled, Data cache writeback
Built 1 zonelists
Kernel command line: console=ttyS0,115200 ip=bootp root=/dev/nfs
PID hash table entries: 128 (order: 7, 2048 bytes)
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 16MB = 16MB total
Memory: 13288KB available (2049K code, 504K data, 320K init)
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: Testing write buffer coherency: ok
NET: Registered protocol family 16
PCI: IXP4xx is host
PCI: IXP4xx Using direct access for memory space
PCI: bus0: Fast back to back transfers enabled
dmabounce: registered device 0000:00:0e.0 on pci bus
dmabounce: registered device 0000:00:0f.0 on pci bus
NetWinder Floating Point Emulator V0.97 (double precision)
JFFS2 version 2.2. (C) 2001-2003 Red Hat, Inc.
IXP4xx Watchdog Timer: heartbeat 60 sec
Serial: 8250/16550 driver $Revision: 1.90 $ 2 ports, IRQ sharing disabled
ttyS0 at MMIO 0xc8001000 (irq = 13) is a XScale
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 8192K size 1024 blocksize
loop: loaded (max 8 devices)
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
IXP4XX-Flash0: Found 1 x16 devices at 0x0 in 16-bit bank
Intel/Sharp Extended Query Table at 0x0031
Using buffer write method
cfi_cmdset_0001: Erase suspend on write enabled
Searching for RedBoot partition table in IXP4XX-Flash0 at offset 0xfe0000
3 RedBoot partitions found on MTD device IXP4XX-Flash0
Creating 3 MTD partitions on "IXP4XX-Flash0":
0x00000000-0x00060000 : "RedBoot"
0x00fc0000-0x00fc1000 : "RedBoot config"
mtd: partition "RedBoot config" doesn't end on an erase block -- force read-only
0x00fe0000-0x01000000 : "FIS directory"
mice: PS/2 mouse device common for all mice
i2c /dev entries driver
NET: Registered protocol family 2
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
NET: Registered protocol family 1
NET: Registered protocol family 8
NET: Registered protocol family 20
IP-Config: No network devices available.
Root-NFS: No NFS server available, giving up.
VFS: Unable to mount root fs via NFS, trying floppy.
VFS: Cannot open root device "nfs" or unknown-block(2,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
Monday, February 14, 2005
Matrix Revisited - Technology for Black Budget Dollars
The following are "Black" Budget Proposals. They are for things
that have immense worth to society, but need DARPA and/or
Government level funding in order to make them a reality.
While inspired by the fiction, they are based on cutting edge
research funded by DARPA today, and are being chased in
the USA and around the world. Details to follow.
Matrix Proposal 100-1-2000 Caloric Inductive Charger
A "In-Body" electrical charging system
to allow the operation of electronic and
computation system without other power
sources. Based on the use of Blood-Sugar,
the system is designed for implantation directly
in the main body cavity of a human. The electric
charge is conducted via induction to external or
internal devices. Proposed external devices
include, but are not limited to laptop's, pda's,
laser and night sights, and tasers.
Detailed project proposal to follow.
Matrix Proposal 110-1-2405 Inner Cranial Remote V/R
Using a combination of IT standards, VR, and
brain/computer interface, creates a stereo graphic
display in the brain, and a set of main motivators
(legs) and manipulators (arms). Using the brains
ability to "adjust" to limb like signalling intro-
duced in alternate locations (brain surface), the
human brain is taught to process signals from a
virtual set of arms and legs. This will be design
to simulate crt/keyboard interface in phase one
and remote presence in phase two.
The technology is designed to be compatible
with remote desktop, and wireless USB
technology to allow for interoperation with
existing computer technology.
Foundational signal processing technology
exists today. Various issues need resolution,
and proposed solutions will be addressed in
full write-up.
Matrix Proposal 129-3-4501 DRONE Silicon Control System
With the human nervous system signalling
system decode work being compete, it is now
possible to leverage on this work to create a
"drone" control system. A "drone" is defined
as a human adult body in a "vegatative" state.
The non-functional human brain is removed,
and a custom asic that will be developed under
the project will allow for the wireless control of
all nomal muscle groups. By using the Inner
Cranial Remote V/R project as a controller, a
drone may be used in dangerous or other
environments that a normal operator may not
be able to otherwise undertake. Full details of
the research, the asic design overview, and
future enhancements will be provided in the
full proposal.
that have immense worth to society, but need DARPA and/or
Government level funding in order to make them a reality.
While inspired by the fiction, they are based on cutting edge
research funded by DARPA today, and are being chased in
the USA and around the world. Details to follow.
Matrix Proposal 100-1-2000 Caloric Inductive Charger
A "In-Body" electrical charging system
to allow the operation of electronic and
computation system without other power
sources. Based on the use of Blood-Sugar,
the system is designed for implantation directly
in the main body cavity of a human. The electric
charge is conducted via induction to external or
internal devices. Proposed external devices
include, but are not limited to laptop's, pda's,
laser and night sights, and tasers.
Detailed project proposal to follow.
Matrix Proposal 110-1-2405 Inner Cranial Remote V/R
Using a combination of IT standards, VR, and
brain/computer interface, creates a stereo graphic
display in the brain, and a set of main motivators
(legs) and manipulators (arms). Using the brains
ability to "adjust" to limb like signalling intro-
duced in alternate locations (brain surface), the
human brain is taught to process signals from a
virtual set of arms and legs. This will be design
to simulate crt/keyboard interface in phase one
and remote presence in phase two.
The technology is designed to be compatible
with remote desktop, and wireless USB
technology to allow for interoperation with
existing computer technology.
Foundational signal processing technology
exists today. Various issues need resolution,
and proposed solutions will be addressed in
full write-up.
Matrix Proposal 129-3-4501 DRONE Silicon Control System
With the human nervous system signalling
system decode work being compete, it is now
possible to leverage on this work to create a
"drone" control system. A "drone" is defined
as a human adult body in a "vegatative" state.
The non-functional human brain is removed,
and a custom asic that will be developed under
the project will allow for the wireless control of
all nomal muscle groups. By using the Inner
Cranial Remote V/R project as a controller, a
drone may be used in dangerous or other
environments that a normal operator may not
be able to otherwise undertake. Full details of
the research, the asic design overview, and
future enhancements will be provided in the
full proposal.
Subscribe to:
Posts (Atom)
