General Crap

04 Mar 2013

PHP just does some things better. cloud backups, pecl-expect

 Backups, yes we do backup occasionally, and I've been looking for a better solutions than my historical office<->hosting replication, which although cheap, always made me wonder if I was still making copies. So after our recent office move, and minor server upgrade, I thought better double check on it all.. As usual, the thing had failed due to various reasons, and needed replacing. 

So since I've been thinking about a new solution for our clients, I decided to go ahead and try it out for our data. I've been googling affordable cloud based storage for a while and found a company onlinestoragesolutions.com, the pricing is very reasonable US$45 for 2 years currently, with unlimited everything. There are a few review sites that seem to throw some cold water over the offer, but I had one client sign up without any major issues (for another reason), so at that price I though let's give it a go....

One of the key features they advertise is rsync, which since we run all linux machines would be ideal. However after paying, and getting access, I realized it's not quite a simple as pointing your rsync at their server. You need to set up an ssh tunnel to route rsync through.

Can't be that difficult I thought.... Turns out that setting up a password based ssh tunnel, automatically on a cron job, is no small task, there are questions all over stackexchange and various forums, none that I saw managed to find a solution. Most of the suggestions are based around the 'expect' program, a usefull unix tool which can be used to script ssh access. The problem in this case was that setting up the tunnel, then doing the rsync, and then closing the tunnel is not something that a bash, expect or any other method I found could do easily if at all.

So almost at the point of giving up, I started looking around at php's popen (which would not work either), and fell over the pecl expect extension. 

Below is the result of a few minutes coding, which does exactly what is needed, and can be run directly from cron. Feel free to escape from overpriced backup solutions..


02 Nov 2012

Interesting Problems.

If there is one reason I would put down to why I still enjoy coding after all these years, it is the pleasure finding a really challenging problem and being able to solve it quickly and efficiently.

This week saw one of those problems, strange, obtuse, and to most non crazy people would seem to be an odd thing to get excited about. It all started with going through the requirements list on one of our projects.

As I've mentioned before, we are building quite heavily ontop of Xtuple, a Web based GUI using the RooJS Toolkit, most of that part has been just process orientated - add this feature, and it just works. But some of the implementation process has been a challenge.

The company we are working with has 2 main offices (and more comming on line soon), and we have successfully migrated them off of netsuite, which in reality made an absolute mess of their accounts. When migrating to Xtuple (which in essence is a postgresql database, with a seemingly endless number of table triggers, stored procedures and views), we concluded quite early on that setting up a seperate instance for each office would be the best way to go.

Core issues like difference currencies in each office, so the chart of accounts can be reported each year in the local currency to the tax department and auditors, along with the different end-of-year tax reporting made this a simple decission. The otehr issues was that the basic installation of Xtuple does not really support Multiple companies in the Desktop UI (you have to pay for that feature). 

So as we are now live, and just about finished with the rollout issues, we are now on to the more critical things for long term operations - Management reports (and a nice dashboard). 

The issue that raised it's head was that the company needs to see consolidated Balance sheet and Income statements. It did not take to much time to call the internal methods in Xtuple 'financialreport()' and in PHP convert this to JSON so that a master script can call the report in both systems, and merge them together. 

The snag came later when we realized that the resulting report was a little off.. Due to the magic of differeing accounting periods.

28 Apr 2012

Migrating off Netsuite - The hidden cost of Clouds..

One of the more interesting projects I have ongoing is a migration and deployment of an ERP system. For a bit of background, our customer was persuaded about 3-4 years ago that moving their accounting and Stock management into Netsuite would be a good idea.

For those who have not heard of Netsuite, it's a Cloud based ERP system, that from my understanding runs on an oracle backend and is totally web based. All the users log into the website, enter the various stock activity ( Purchase Orders, Item Receipt, Bills, Sales orders, Invoices, Item Fulfilments and Various Credit Memos), along with the standard accounting features like Journal entries. etc.... The system keeps track of your stock and should give you a value of stock on hand, along with doing your audit-able reports.

Well, I'm guessing anyone who has worked with ERP systems, probably knows that the devil is in the details for this. The company I have been working with have been using this for over 2 years now, and are beginning to see the ugly side of cloud based system (which is why we are trying to migrate off of it.), Let's have a look through the issues.

26 Jan 2011

Browser rending weirdness OSX and firefox

Sometimes rendering bugs can be very annoying, IE has it's share of misbehaving, but usually firefox, chrome are consistant and never need much testing. However on one site, prior to going live someone testing on OSX kept saying that the menu bar was broken in Firefox.

Testing on Windows and Linux it looked fine. however on their browser, the horizontal line of buttons for the menu broke into two lines. It was not until I finally did remote control on her PC was I able to see what was going on.

Basically, on first load it rendered fine, but as soon as you went back to the page, the buttons would break. Inspecting it in firebug indicated that the second time it rendered the DOM tree was actually different to the HTML source.

The button code looks a bit like this

<a href="somurl"><div class="btn"><div class="btn-right"><div class="btn-left"><div 
   class="btn-body">Title here</div></div></div></div></a>

The idea is that the div's provide the rounded button edges, and allow nice hover effects all in CSS.

however inspecting the reloaded page indicated the tree looked like
<a href="somurl"></a>
<div class="btn"><div class="btn-right"><div class="btn-left"><div
         class="btn-body">Title here</div></div></div></div>
Where the a tag had broken away from the div, strangely only on one button.

It was only after I ran it through the w3 validator and  got an error message about block elements inside and inline one, that I had the idea to rewrite that HTML to use span's, rather than div's. Along with adding display:block to all the CSS for the button elements.

The resulting HTML being
<a href="somurl"><span class="btn"><span class="btn-right"><span class="btn-left"><span 
        class="btn-body">Title here</span></span></span></span>
And amazingly enough it rendered perfectly on all browser.. strange but true....

10 Oct 2010

Leadership lessons, or are MBA courses any good?

And now for something different.. I have a friend doing a course at HKU on Leadership, and with the feedback I'm getting, the course might just be missing the plot a bit.

From my side of the fence, ever since I gave up on my Masters, I have been generally dismissive of post-graduate courses at Universities. There was a serious sense of 'If you can't do it, teach it..' (or now if you really can't do it, blog about it..). My sense is that there is a reasonably large body of acedemia that suffers from the fact that they are kept on board as a result of quantity of research publications. They however they do not appear to understand the principles behind teaching (which teacher training wonderfully encapsulates). I wonder if tutors at university should be required to undertake a teacher training course, rather than slowly migrating from research into tutoring.

Along with all this, I wonder if the type of students these course attract are the exact opposite to 'self-learners', and are more used to a spoon feed education, especially here in asia. 

Anyway back to the subject at hand. HKU was teaching a course based around a Harvard case study about a climb of mount Everest resulting in a number of deaths. The students are given the course material and are asked to produce a short piece explaining it and illustrating their ability to extract the leadership lessons that caused the failure.

This is where adult reading (not the picture kind) comes in handy. In Malcom Gladwell's book Outliers, a chapter is dedicated to an interesting study of plane crashes, and pilot assertiveness. If you read the book in the wrong way it appears to sensensionalize the idea that different countries and cultures have different levels of assertiveness which lead to greater crashes with pilot's of different countries. While this inference tends to lead to great press and I presume book sales. It is not really the message that come accross if you read it in detail.

In essense the book points out that individuals (and perhaps to some degree culturally based) are less assertive, in that they feel unable due to their role in society of workforce to assert their opinions. In the case of the pilot's it was the quiet mention that there might be a small issue with the weather. In the case of the climb it was more a case of Doctor's not asserting the seriousness of health issues, or the sherpa's inability to express that they where concern with various issues about the trip. All these would have likely been non-fatal if both communication had been clear and strong, and the subordinates had felt comfortable in the life threatening situations in overriding the leaders actions.

From the sense I got, the course was being taught on leadership, but unfortunatly learning that there are problems occuring with these type of situations has very little to do with leadership, and more to do with realizing that assertiveness training is an essential part of senior management, and more essential in situations which may be life threatening. 

What is worse is that the course dynamic's of putting groups into teams and tasking them with these studies looks like it is a perfect example of why failure happens. Rather than using the course to entrench this positive assertiveness it basically leaves the groups to their own devices to produce a result and does not monitor their interaction trying to illustrate the individual's in the groups own abilility to assert their opinions.  Would this type of couse not be better taught better with some kind of situation based role playing and instant feedback. 

25 Aug 2010

Company rename and RooJS sites now online

After what seems like forever in design and migration, I've turned on the roojs.com and roojs.org domains, I'm slowly in the process phasing out my old company name - AK BK Consulting, which is now rooJSolutions

Quite a few of the key pages are missing content at present, but the blog and it's features are mostly moved over.

After I fill out all the missing content, I'll get back to blogging about how most of the Javascript tools and associated backend libraries all work together. Exciting times for a quality platform neutral Javascript Rich User Interface, guaranteed $(wtf) free...


03 Jul 2009

nested trees in mysql - handy stored procedures.

Storing tree data in Mysql databases, is relatively common, however, all the existing documentation about doing this, makes the whole process rather complex.

If you google it, you will probably find the quite definitive answer at mysql.com, describing the classic  parent_id method, and the left/right numbering process. Both of these methods involve rather complex SQL to fetch and update the tree.

In seeking a better solutions for a tree that was infrequently updated, but frequently queried, I thought I'd try seeing if I could write a few stored procedures to simplify the process.

Our basic database structure looks like this:
CREATE TABLE _TREE_ (
id int(11) NOT NULL auto_increment,
parent_id int(11) NOT NULL DEFAULT 0,
seqid int(11) NOT NULL DEFAULT 0,
depth int(11) NOT NULL DEFAULT 0,
leaf int(1) NOT NULL DEFAULT 0,
name varchar(128) default '',
fullpath TEXT default '',
PRIMARY KEY (`id`),
INDEX qlookup( parent_id , seqid , depth)
);
our key components are
  • id (the nodes id)
  • parent_id (the nodes parent - pretty clasic)
  • name (the textual name of the node)
  • seqid - the generated order item for the whole tree
  • depth - how deep the node is (usefull for indenting)
  • leaf - is it a leaf node (eg. has no children) - usefull for icons
The trick of the stored procedures is to correctly generate the seqid, this consists of two parts, the top level wrapper and the recursive prodedure to update the table.



DROP PROCEDURE IF EXISTS _TREE__resequence;

DELIMITER $$
CREATE PROCEDURE _TREE__resequence(i_sep VARCHAR(4)) DETERMINISTIC
BEGIN
DECLARE v_p, v_d, v_s INT(11);
DECLARE v_fp TEXT;
SET v_fp = '';
SET v_p =0;
SET v_d =0;
SET v_s =0;
SET max_sp_recursion_depth=255;
CALL _TREE__resequence_sub(v_p, v_d, v_fp, i_sep, v_s);


END $$
DELIMITER ;





DROP PROCEDURE IF EXISTS _TREE__resequence_sub;

DELIMITER $$
CREATE PROCEDURE _TREE__resequence_sub(
i_parent INT(11),
i_depth INT(11),
i_fullpath TEXT,
i_sep VARCHAR(4),
INOUT i_seqid INT(11)
) DETERMINISTIC
BEGIN
DECLARE v_nid, v_ex_seqid INT(11);
DECLARE v_name VARCHAR(128);
DECLARE v_leaf INT(1);
DECLARE v_fullpath TEXT;
DECLARE done INT DEFAULT 0;

DECLARE qry CURSOR FOR SELECT id, seqid, name FROM _TREE_
WHERE parent_id = i_parent ORDER BY seqid;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN qry;

REPEAT
FETCH qry INTO v_nid, v_ex_seqid, v_name;

IF NOT done THEN
IF v_ex_seqid != i_seqid THEN
UPDATE _TREE_ SET seqid = i_seqid, depth=i_depth WHERE id=v_nid;
END IF;
IF i_depth > 0 THEN
SET v_fullpath = CONCAT(i_fullpath, i_sep, v_name);
ELSE
SET v_fullpath = v_name;
END IF;
SET v_leaf =0;
SELECT COUNT(id) INTO v_leaf FROM _TREE_ where parent_id = v_nid;

UPDATE _TREE_ SET
fullpath = v_fullpath,
leaf = IF (v_leaf > 0, 0 , 1)
WHERE id=v_nid;

SET i_seqid = i_seqid +1;
#// do the children..
CALL _TREE__resequence_sub(v_nid, i_depth+1, v_fullpath, i_sep, i_seqid);
END IF;
UNTIL done END REPEAT;

CLOSE qry;

END $$
DELIMITER ;

This does the hard work of iterating through the tree, and updating the sequence number, depth, leaf field and filling in the fullpath field

Now our simple add node code just adds the node in the correct place, bumps the seqid along, so that you can then regenerate the tree.



DROP FUNCTION IF EXISTS _TREE__add_node;

DELIMITER $$
CREATE FUNCTION _TREE__add_node(
i_parent INT(11),
i_after INT(11),
i_name VARCHAR(128)
) RETURNS INT(11) DETERMINISTIC
BEGIN

DECLARE v_depth INT(11);
DECLARE v_seqid INT(11);
DECLARE v_ret INT(11);
DECLARE v_tmp INT(11);

SET v_depth = 0;
SET v_seqid = 0;
SET v_ret = 0;
SET v_tmp = 0;


#// grap parent depth.
SELECT depth +1, seqid + 1 INTO v_depth, v_seqid FROM _TREE_ WHERE id= i_parent LIMIT 1;
#// grab previous id..
IF i_after > 0 THEN
SELECT seqid INTO v_tmp FROM _TREE_ WHERE id = i_after AND parent_id = i_parent LIMIT 1;
IF v_tmp > -1 THEN
SET v_seqid = v_tmp+1;
END IF;
END IF;



INSERT INTO _TREE_ SET
depth = v_depth ,
seqid = v_seqid ,
parent_id = i_parent,
name = i_name;


SELECT LAST_INSERT_ID() INTO v_ret;

#// fix the seqend id


UPDATE _TREE_ SET seqid = seqid +1 WHERE seqid >= v_seqid AND id != v_ret;

RETURN v_ret;

END $$
DELIMITER ;


You can now create a generic tree code by replacing my _TREE_ word with your table name:
eg.
#sed -e "s/_TREE_/yourtable/g" tree.js | mysql yourdb -f 
Once all that is done the SQL to create tree nodes is very simple:

 delete from nametree; alter table nametree AUTO_INCREMENT =1 ;

select nametree_add_node(0,0, 'one at top');
call nametree_resequence(':');

select nametree_add_node(1,0, 'two on one');
call nametree_resequence(':');

 select nametree_add_node(1,0, 'three on one (first)');
call nametree_resequence(':');
 
select nametree_add_node(1,2, 'four on one (last)');
call nametree_resequence(':');
 
select nametree_add_node(0,1, 'five on top (last)');
 call nametree_resequence(':');
 
select nametree_add_node(0,0, 'six is first');
call nametree_resequence(':');

SELECT * FROM nametree ORDER BY seqid;

Gives you a nice little table

+----+-----------+-------+-------+------+----------------------+---------------------------------+
| id | parent_id | seqid | depth | leaf | name | fullpath |
+----+-----------+-------+-------+------+----------------------+---------------------------------+
| 6 | 0 | 0 | 0 | 1 | six is first | six is first |
| 1 | 0 | 1 | 0 | 0 | one at top | one at top |
| 3 | 1 | 2 | 1 | 1 | three on one (first) | one at top:three on one (first) |
| 2 | 1 | 3 | 1 | 1 | two on one | one at top:two on one |
| 4 | 1 | 4 | 1 | 1 | four on one (last) | one at top:four on one (last) |
| 5 | 0 | 5 | 0 | 1 | five on top (last) | five on top (last) |
+----+-----------+-------+-------+------+----------------------+---------------------------------+

Enjoy tree.js (I name my mysql stored procs as js, just so my editor works better with them)


22 Dec 2008

2008 roundup - Recession kicking in?

As my thumbs are twiddling waiting to see if projects start coming in next year, I thought I'd do a roundup of this years projects (open and closed). I was pondering if everyone else was seeing the effects of the recession, so I'd be interested to hear if you've also seen work slacking off.

Let's hope next year we see some more life from the Sales and Marketing, as my current reliance on word of mouth, may not keep feeding my mouth at this rate.

Anyway Merry Christmas and Happy new year to all.

Read on for my Year of Projects

04 Dec 2005

Where have all the good mice gone?

On almost every PC I use, I still have a nice reliable ball mouse, usually wireless, and every few months, it get's a small clean and new batteries. But one of them is now at least 4 years old, and is showing it's age a little, so off I went hunting for a replacement.

To my horror, nobody is selling mice with balls these days (male mice are extinct!!! - female only versions available?). They only sell the optical crap. I've tried optical mouses quite a few times, you always get really bad feedback and the mouse behaviour is eratic at best of times. I usually end up bining them. But I'm still in shock today, that it's so increadibly difficult to buy them. So if you know of any good on-line resellers who sell real mice.. let me know..

On a side line to this, I ended up with a dud piece of hardware from Targus (Note Mac unfriendly supplier!) - their cute looking wireless presenter mouse, has no drivers for OSX, so looks like that one's going back to the shop.. (even linux had support for this!)
[update] I took the Wireless presenter mouse (Targus ARM01AP) back to the shop, after a bit of wiggling by the sales assistant, and pressing buttons on the USB bit, suddly it came to life on the OSX desktop, so after spending 2hours+ trying to find a solution last night, minutes after I plugged it in today, it worked out the box, without any extra software...agghhhh

10 Nov 2005

Microsoft, always delivers... crap...

You know it is going to be one of those days when the lift to the office starts going up, even when you pushed the down button in the lobby..

The morning started with a heated discussion about proposing a video conferencing system to one of our clients. I had come down to two options, a polycom plug and play (hopefully) system, or using 2 iMac's with iChat. But one of my collegues suggested that we recommend a Windows box, with MSN or similar. I was addement, I will never recommend windows for anything as complex as that, the implications for support, securing it (as it needed to have a fixed IP), and knowing that the thing would be guarenteed to fail a good proportion of the times the end user wanted to do a conference. While I sounded alot like a anti-microsoft zealot, I've been burnt far to many times in recommending and installing Windows to even consider it as a reliable option. And it get's difficult arguing with someone who actually think's that OS X has viruses ;)

Anyway, that was just the start of the Microsoft crap for the day. Developers, Developers, Developers I hear them cry, well the day their software actually works as expected, I think they might actually attract enthusiastic developers.

The task for the day was to fix some code I had written to export data to excel. I had practically copy and pasted some code from one project to this one, to implement this process. (I think I've blogged about the concept before), basically you render a HTML table with the data you want to export (a preview), then you use a javascript library, to read the table and generate a gnumeric xml file. This get's sent to the server, that then runs ssconvert to output a beautifully formated excel file, (doing all the colour, background, font effects etc.)

Of course, this library had only be tested on Firefox (as most of the projects that it was used with where XUL based). So I had just said to the tester, does it work with IE?.. of course the answer came back .. NO!

Given the fact that the javascript code in the application followed W3C DOM standards, and did not do anything that amazing, I guess I was a little supprised. So out came vmware, and my nice sandboxed Windows 98. In all it took over 4 hours to solve this, and I guess it's time I regard as a waste of my life (like most times I have to use windows ;)

  • First indication of a problem was a little yellow triangle on the browser, pressing on it, told me error at line 0: download()
    This it turned out (from what I remember), was that for some reason, IE did not like a href="javascript:download()" as a link, and only when it was replaced with onclick, did it start working ok.
  • Next up was some message about object does not support that operation on line 134 of xyz.html.
    Well, looking at the HTML file, there was no javascript on that line!, it took me quite a while to realize, that the debugger can not handle included javascript files (and uses the line number from the original file) duh!!!, I had to copy all the javascript from the included files, into the main one just to debug it.
  • That bug turned out to be something as simple as this
    x = this.createXMLHttpRequest();
    if (x != false) {
    Since x was an object, it could not be checked against a boolean (even though this works perfectly on Firefox)
  • After getting those bugs fixed, the next was a unknown method on an object. which was basically
    document.getElementsByTagNameNS('*'',"rows");
    used to grab sections of the Gnumeric template file, much to my amazement (well less so by now), I realized that IE doesnt actually implement DOM!!! (or DOM3 AFAIK) properly...
  • At this point I gave up thinking that IE would ever work (or was worth supporting) for my nice exporter. So I started thinking just doing a HTML table to CSV importer, using pretty much the same concept. I however ended up spending a good half hour puzzling over a "Can not download" message, as soon as I added the Content-type header to the output. The kludge workaround for which is actually mentioned on the PHP manual comments.
All in all not one of my better days....
   (Page 1 of 2, totalling 14 entries)    next page »

Follow us on