Latest
RSS and Categories
Migrating off Netsuite - The hidden cost of Clouds..
One of the more interesting projects I have ongoing is a migration and deployment of an ERP system. For a bit of background, our customer was persuaded about 3-4 years ago that moving their accounting and Stock management into Netsuite would be a good idea.
For those who have not heard of Netsuite, it's a Cloud based ERP system, that from my understanding runs on an oracle backend and is totally web based. All the users log into the website, enter the various stock activity ( Purchase Orders, Item Receipt, Bills, Sales orders, Invoices, Item Fulfilments and Various Credit Memos), along with the standard accounting features like Journal entries. etc.... The system keeps track of your stock and should give you a value of stock on hand, along with doing your audit-able reports.
Well, I'm guessing anyone who has worked with ERP systems, probably knows that the devil is in the details for this. The company I have been working with have been using this for over 2 years now, and are beginning to see the ugly side of cloud based system (which is why we are trying to migrate off of it.), Let's have a look through the issues.
Javascript Templating, AngularJS and Roo.XTemplate
Ok, as I said earlier, a nice Easter break, meant I could get back to coding for fun. The other issue I had to deal with over the weekend was how to do the templating for the Javascript application.
Having written a template engine for PHP, I can pretty much say that not using a template engine (which automatically escapes output) is essential for outputing HTML. It makes everything more maintainable, and reduces the risks that as a error prone human being, you are likely to open your application up to exploits.
Since most of the display logic on my applications is moving away from PHP generating HTML to PHP generating JSON, and the Javascript UI rendering this data and displaying it to the user. There has been one area I've slacked off on, that being sorting out a solution for using this JSON data from the PHP backend and rendering a elegant front end in HTML to show the end user.
Obviously this is not going to be indexed via google (Somehow JSON / Javascript interfaces are not very google friendly). But for applications that require authentication this not only helps scale the application under load, but also make it far easier to maintain.
My first effort to solve this issue was based on the original Template code in Roo (Roo.Template) however initially I decided to do the implementation in PHP. The concept was to take a group of 'HTML' template files, and use the PHP to convert those files into 'Javascript functions'. so calling the generated javascript function with 'data' as the argument, it would return the HTML to be rendered.
This basic implementation supported things like conditional inclusion, embedding Javascript, nested functions, looping and much more. It was however horribly crude and rather fragile if the template syntax was invalid. I used this for a few files relating to the ticket rendering in web.mtrack, however, I was never really happy with the solution.
Read on for my opinion / review of Anjular and the introduction to the updated Roo.XTemplate
Roo.XComponent introduction
http://www.roojs.com/roojs1/docs/symbols/Roo.XComponent.html
With a nice long Easter holiday, I finally got a chance to hack on some of those todo items that I'd been putting off for quite a while. One of the core mini tasks was to enable mtrack information inside of my main email/accounting do everything platform.
The Pman codebase, as I've mentioned before forms the core of pretty much all applications I work on now, mostly intranet or extranet focused web applications, that work like desktop applications. All of these applications are built up of various componenents, for example the Accouting module has Components like general ledger, managing invoices, timesheet and tracking editing and a pricing management component.
The whole premise of the Pman project codebase on the Javascript side, is to register all the components sometime after the page is loaded, then after the user has logged in, or authentication has been checked, it will create and render all these components in sequence, appending them to the container panels.
In the case of the Accounting module, there is a default top level module called 'Pman', which the Accounting 'Tab' is added to, then all the sub components are added to that. This whole design was done a couple of years ago, and has been working very well, it's reliable and adding extra componenets, is just a mater creating a file then saying what the parent is and in what sequence it will appear.
Read on to find out how this all works...
Roo J Solutions Limited is recruiting
Since we have been very busy already this year, I have now almost completed the process of migrating from a Sole Proprietor into a Limited Company. Roo J Solutions Limited is now a registered Hong Kong Company. We are now looking for full or part-time staff (based in Hong Kong).
Please read the full post for details.
Free your data... seed webkit browser mirror button
One of the great things about the internet is the availability of cheap or free services online, so many clients are using gmail, dropbox, github etc. for their business operations. But all to often they forget that these services are often playing the oldest game in the technology industry. "Vendor Lock-in".
While the ones I mentioned are not to bad, you can cheaply and easily rescue or backup your data to another location, or move to an alternative provider. Not all of them are like that.
We are in the middle of a migration project from Netsuite (It's a SAS Oracle based ERP system) to Xtuple, which is a open source ERP system, based around postgresql. This is a slow and painfull migration, as there is no standard for ERP data, and exporting is slow and clumsy over SOAP. Anyway, as a plesant distraction from this large migration, the same client also wanted us to look at migrating from backpack, a 37 signals product.
Backpack, unlike all the SAS systems I mentioned has deliberately made it hard, or practically impossible to migrate from their services. The primary offering of backpack is a online file storage service that you can permit clients or suppliers the ability to do share files and folders. It is only web based (unlike dropbox or box.net), and there is no desktop client that you can use to access the files other than the web interface.
When I started looking at how the company could extract the data, I tried out a few of the classic tools, like wget and httrack however the strong use of javascript, and the convoluted login system with login keys ensured that those kind of tools did not work. The other requirement was the ability to organise the files into folder, by just mirroring the site, you would just end up with thousands of folders called asset/123123/ where the number is probably the UID of the database record.
So how to rescue the data... Read on for the trick..
Deleting the View and Controller..
This is NOT a post for people who do not use MVC, Please delete your code, and write it properly.. Anyway, as anybody who has used or written a reasonable framework in PHP knows, MVC is pretty much the golden rule for implementation. There are a dozen frameworks out their based around the principles, with different levels of complexity.
My own framework was designed around those principles, and for many years worked perfectly for those classic display a crap load of HTML pages using information from a database. The Model (DB_DataObject's), View (HTML_Template_Flexy) and Controller (classes that extend HTML_FlexyFramework_Page) delivered pages. Designing sites basically involved gluing all these pieces together. As the sites grew over time, shared code usually ended up in the Models, and each page had a controller which might render the share templates. All was well, and code was reasonably easy to maintain and extend.
Now however almost all the projects I've worked on in the last few years use the Roo Javascript library (the ExtJS fork), and are built ontop of the Pman components (originally a project management tool, that grew into a whole kit of parts). One of the key changes in the way the code is written, is how little code is now done to get the information from the database to the end user.
Obviously the whole HTML templating is been thrown out the window, (other than the first primary HTML page), the whole user interface is built with Javascript, and generated by User interface builder tools. The interaction of the interface is handled by signals (listeners) on the Roo Javascript components. These in turn call the Back end (PHP code) and almost always retrieve JSON encoded data, and that is rendered using the UI toolkit.
When I first started moving to this development model, I tended to retain the previous idea of having multiple controllers to handle the Select/Create/Update/Delete actions, as time went rather than have multiple controllers for each of those actions, I would use a single controller to manage a single Model entity (like Product). POST would always update/modify the model, and GET would always just view and query the data.
Eventually I realized that since all these controllers where essentially doing the same thing, a single generic controller should be able to do everything that all these single controllers where doing. And so was born the Pman_Roo class.
So Basically {index.php}/Roo/{TableName} provides generic database access, for the whole application. Most code development on the PHP side is now contained within the DataObject Models. This greatly enhances code reuse as similar code ends up closer together, Unlike before where shared code was moved from the controllers to the model when necessary, now most of the code starts off in the model. This speed project development up considerably not to mention the huge savings of not having to try and manipulate data into HTML.
How does it work.
A GET or POST request is recieved by the server either from Roo's Form/Grid/Tree or directly by Pman.Request(), a handy wrapper arround Roo.Ajax, that handles error messages nicely.
The request {index.php}/Roo/{TableName} checks that the tablename is valid, then goes on to do the following actions depending on the params supplied. The documentation in the Pman_Roo class is the most up-to-date documentation. and details what calls are made (if available) on the relivant dataobject.
Using the class, it is now possible to handle pretty much any Database related query without implement any controller, and easily managing data permissions.
Snapshot of current documentation is in the extended view.. (latest will be in the source)
What was I doing last night... Seed querying xscreensaver
Like quite a few developers, I earn income by selling my time, either packaged on a project, or by the hour. For this to work, keeping track of time is essential. Unfortuntally, like most creative people, I really enjoy hacking, but filling in timesheets just really doesnt do it for me... So read on for my solutions using gnome seed...
Watch-out PHP 5.3.7+ is about.. and the is_a() / __autoload() mess.
Well, for the first time in a very long while I had to post to the PHP core developers list last week, unfortunately the result of which was not particulary usefull.
The key issue was that 5.3.7 accidentally broke is_a() for a reasonably large number of users. Unfortunately the fixup release 5.3.8 did not address this 'mistake', and after a rather fruitless exchange I gave up trying to persuade the group (most people on mailing list), that reverting the change was rather critical (at least pierre supported reverting it in the 5.3.* series).
Anyway, what's this all about, basically if you upgrade to any of these versions and
a) use __autoload()
or
b) any of your code calls is_a() on a string,
you will very likely get strange failures..
The change in detail.
in all versions of PHP since 4.2 the is_a signature looked like this
bool is_a ( object $object , string $class_name )
As a knock on effect from fixing a bug with is_subclass_of, somebody thought it was a good idea to make the two functions signature consistant, so in 5.3.7+ the signature is now
bool is_a ( mixed $object_or_string , string $class_name )
And to make matters worse, that change to the first 'object_or_string', will also call the autoloader if the class is not found.
How is_a() has been used in the past.
On the face of this, it would not seem like a significant change, however, you have to understand the history of is_a(), and why it was introduced. In the early days of PEAR (before PHP 4.2) there was a method called PEAR::isError($mixed), which contained quite a few tests to check if the $mixed was an object, and was an instance of 'PEAR_Error'. A while after PHP 4.2 was released, this was changed to use this new wonderfull feature, and basically became return is_a($mixed, 'PEAR_Error').
Since PEAR existed before exceptions (and is still a reasonable pattern to handle errors), It became quite common practice to have returns from methods which looked like this.
@return {String|PEAR_Error} $mixed return some data..So the callee would check the return using PEAR::isError(), or quite often just is_a($ret,'PEAR_Error'), if you knew that the PEAR class might not have been loaded.
So now comes the change and let's see what happens.
The __autoload() issue.
Personally I never use __autoload, it's the new magic_quotes for me, making code unpredicatable and difficult to follow (read the post about require_once is part of your documentation). But anyway, each to their own, and for the PEAR packages I support I will usually commit any reasonable change that helps out people who are using autoload.
So there are users out there using autoload with my PEAR packages, as I quickly found last week. Quite a few of these packages use the is_a() pattern, and the users who had implemented __autoload() had very smartly decided that calling autoload with a name of a class that could not or did not exist was a serious error condition, and they either died, or threw exceptions.
Unfortunatly, since is_a() was sending all of the string data it got straight into __autoload(), this happened rather a lot. Leading to a run around hunt for all calls to is_a(), and code changes being put it to ensure that it never puts a string in the first argument.
The is_a(string) issue
While I'm not likely to see the autoload issue on my code, I'm not sure I really appreciate having to fix it so quickly without a timetable to change it. The other change that may cause random, undetectable bugs is the accepting a string.
imagine this bit of code
function nextTokString() {
if (!is_string($this->tok[$this->pos])) {
return PEAR::raiseError('....')
}
return $this->tok[$this->pos++];
}... some code..
$tok =$this->nextTokString()
if (is_a($tok,'PEAR_Error')) {
return $tok;
}
... do stuff with string.
Now what happens if the token is 'PEAR_Error', is_a() will now return true. The big issue with this is that unless you know about the is_a() change, this bug is going to be next to impossible to find.. No warning is issued, is_a() just silently returns true, where before it just returned false.
I was hoping that PHP 5.3.9 would go out the door with this reverted, or at least a warning stuck on string usage of is_a(), but nope, none of my efforts of persuasion appear to have worked.
While I do not think the change is particularly necessary (as the use case for the new signature is very rare, and acheivable in other ways), I think reverting this change before PHP 5.3.7+ went into major deployment is rather critical. (yes it can take months before PHP releases start commonly arriving on servers). Then if it's deemed a necessary change (by vote) then go for it in 5.4... and add a warning in the next version in the 5.3 series..
Anyway the fixes / workaround:
The simplest fix is to prepend tests with is_object
eg.
if (is_a($tok,'PEAR_Error')) {becomes
if (is_object($tok) && is_a($tok,'PEAR_Error')) {
if ( $tok instanceof PEAR_Error)) {
While you could start looking at the code and determining if you really need to prefix it with is_object(), the reality is unfortuntaly it may be simpler to stick this extra code in, just in case you start delivering strings where objects where expected.
Update
This has been fixed in 5.3.9, however part of this derives from some confusion over instanceof
When PHP5 was released and added instanceof, doing this when the class did not exist caused a fatal error.
if ( $tok instanceof Unknown_Class ) {However, this was changed in 5.1 to not cause a fatal error, The documentation is not totally clear on this, especially for anyone who used PHP 5.0.*.
Unfortunately, since migration times are slow, supporting 5.0-5.1 is a reality of life for anyone writing libraries (actually Most of the libraries I write for still provide support for PHP4). So using any 'new' feature of the language basically prevents you from supporting older version of PHP with new code.
In this case, PHP5 usage has slipped below 0.3% so removing support for this should be fine.
Cli parsing in FlexyFramework, PEAR Console_GetArg
And another rare article get's published, I've been slacking off posting recently. As I've been busy getting some interesting sites online. The biggest being a rather fun viral advertising campaign on facebook www.facebook.com/deargoodboy. Which I ended up project managing, after originally only committing to do the facebook integration.
Anyway back to the open source stuff. One of the tasks I've had on my todo list for a while is revamping the CLI handling of my framework, Which probably has a tiny following, but is one of those increadably simple, yet powerfull backbones to all my projects.
While this article focuses on the changes to the framework, it should also be of interest to others developing frameworks, and anyone interested in using PEAR's Console_GetArg.
Gtk3 introspection updates and Unusable Unity..
Well, as Gnome 3 is out, it has to be tested. Luckily I've not got a huge deployment to sort out, but as I have a few applications that use Gtk, I thought it was about time I upgraded one of my machines to see what chaos I will have to deal with in the future.
So it was one of my Ubuntu boxes that got the pleasure of a Natty and Gnome3 PPA upgrade. (I use debian on my other development box, which actually got destroyed last week with a complete disk failure, although I suspect the motherboard may have problems... It's getting old like me...)
Upgrading to Natty is not to bad, from what I remember it only took a small amount of brain surgery to get it to boot correctly after the upgrade. But once up, you get the pleasure of the Unity desktop. My first impressions where not to hot on unity, my wife uses it on her netbook, it's great there, after the initial shock of me upgrading without her knowing, she actually said it was alot better than compiz. Although she missed the special effects.
But after using Unity on the big screens, it just became unbearable. Detached menus may seem like a cool idea, and are quite handy on a netbook, but they are an absolute nightmare when using things like gimp on dual head full HD monitors, my wrists hurt after a few minutes....
Along with the removal of the Applications/Places/System menu's which while klunky are still handy for quickly finding applications. A classic example of this Alleyoop Memory Checker, a very nice wrapper around valgrind. In the Unity world if you do not know the name of the application, then finding it is a huge mouse journey around big icons.
As for the left icon menu, all I can say is that I'm not the worlds best designer (although at least I did study it), but it's so graphically noisy that it unusable. It's basically a bad re-invention of Docky/Cairo Dock, which do far better jobs at providing a similar task role.
So after all that I did try and get gnome-shell going, but unfortunatly the Gnome3 PPA build is not currrently working, and also has a rather nasty habit removing all usable desktop enviroments. I ended up adding xterm to one of the /etc/Xorg/X.sessiond files and starting up gnome-panel, mutter and docky to produce a usable desktop for the time being, while I wait to test out the latest gnome-shell.
So on with the harder stuff.. - Gtk3 and introspection.
One of the key applications I use to develop is app.Builder.js , it's a drag/drop interface to build web applications, that also allows you to fill in all the code and associate it clearly with the element and event occuring. It's written in Javascript, and uses Gnome seed to run on the desktop. As I've mentioned before Seed is a bridge layer between the Webkit Javascript engine, and Gobject-introspection, the now standard way to interface Gnome/Gtk/Glib etc. projects to non-C languages, eg. Python, Javascript (and others...)
With the introduction of Gtk3, GObject introspection has also been updated, and the updated mix between the two had quite a few knock on effects to the builder I had written using Gtk2 and pre-0.9 versions of Gobject introspections. Heres a general summary of the changes.
TreeIter and TextIter
The latest version of GObject introspection has a feature called caller allocates, this basically means that previously with Seed we had to create an instance of a TreeIter, the the TreeIter call would be of type 'inout' (eg. the Iter would be sent into the method, and returned out)
eg.
var iter = new Gtk.TreeIter();
model.get_iter_from_string(iter, path);
// iter would now contain the tree iter for that path..
In newer versions, the iter is an 'out' value, which means you have to create an object for the iter to be added to. eg.
var iret = {};
model.get_iter_from_string(iret, path);
// iret.iter now contains the tree iter.TreeSelection
since the get_selected method for a GtkTreeSelection now has 2 out values, the call has change from
OLD:
var iter = new Gtk.TreeIter();
selection.get_selected(model, iter);
NEW
var sret = {};
selection.get_selected(sret);
// sret now contains { model: **THE MODEL**, iter: **THE ITER** }TreeModel get_value
Since get_value does not have a return value, seed with return the 'out' values as the return object.
OLD:
var value = new GObject.Value('');
model.get_value(iter, 2, value);
print(value.value);NEW
var str = model.get_value(iter, 2).value.get_string();
print(str);
Drag pixmap becomes surfaces
This is a pure Gtk3 API change (BC break)
OLD:
var pix = widget.create_row_drag_icon ( path);
Gtk.drag_set_icon_pixmap (ctx, pix.get_colormap(), pix, null, ..... )
NEW:
var pix = widget.create_row_drag_icon ( path);
Gtk.drag_set_icon_surface(ctx, pix);
Drag drop data passing..
The drag drop signals appear to work ok, however I've not managed to get the data to go back and forth,
a quick workaround is to just use some form of global variable to store the current dragged item (I doubt you will get more than one dragged item at once..)
Drag drop API
alot of these appear to have played musical chairs.
GtkWidget.prototype.drag_source_set -> Gtk.drag_source_set
Gtk.drag_source_set_target_list -> GtkWidget.prototype.drag_source_set_target_list
Gtk.drag_dest_set -> GtkWidget.prototype.drag_dest_set
Internal Seed changes
I've added a few more fixes to Seed in the last few weeks, mostly to handle compiling correctly and detecting the correct version of introspection. for the most part it's working fine, however I'm still a bit baffled by a Glib memory corruption bug, which occured after multiple model.set_value and model.get_value calls. After running valgrind, I managed to stop the corruption occuring by increasing the allocated size for a struct by 1 byte
Around line 546 and 640 of seed-engine.c the change goes something like this.
- out_args[n_out_args].v_pointer = g_malloc0 (size);
+ out_args[n_out_args].v_pointer = g_malloc0 (size+ 1);
Arround line 738 of seed-structs.c
- object = g_slice_alloc0 (size);
+ object = g_slice_alloc0 (size +1);
Follow us
-
- Migrating off Netsuite - The hidden cost of Clouds..
- Javascript Templating, AngularJS and Roo.XTemplate
- Roo.XComponent introduction
- Roo J Solutions Limited is recruiting
- Free your data... seed webkit browser mirror button
- Deleting the View and Controller..
- What was I doing last night... Seed querying xscreensaver
- Watch-out PHP 5.3.7+ is about.. and the is_a() / __autoload() mess.
Blog Latest
-
Twitter - @Roojs