Category Archives: programming

Really Simple Systems

Organisations large and small depend on packaged software solutions for their core processes, and this is how it should be, the idea of writing your own ERP system or word processor makes no economic sense even if your business model is an extreme outlier. Differentiation does exist, the ERP system that a mining company would need would be significantly different from a chain of retail stores but most of the major (and minor) niches have packaged offerings to suit.

(Unless you’re a certain SME manufacturing company in the South-West of England that couldn’t find a suitable and affordable ERP package for its type of business; well then, developing an open-source one to help you and your fellow SMEs is, it seems, a viable option, resulting in uzERP!)

Many organisations will commission changes to these packages to better fit their way of doing business . In fact, such modifications are a major source of revenue for package providers (and as a result a major cost to those customers who will not “bend” to the packages’ default process flows).

The advice for those who either don’t have the money to commission changes (or who wish to avoid such costs) is to fit in with the package’s way of doing things (which as most will be “industry best practices” can often be a good thing) or use “off-line” processes to handle the exceptions. In some such cases a bespoke professionally delivered software solution might be the result; but for many the more usual “solution” to to those out-of-scope processes is “Simple!, you could easily do that in Excel“. Yet another recruit for the Excel -driven world of Shadow IT.

Whether such “systems” are the casualties of scope warfare or the preferred means of getting things done by Excel-savvy end-users they are part of every organisation’s IT landscape. They’re not going away, yet they receive scant attention from professional IT who often regard them as “somebody else’s problem”.

The end result is sub-optimal; although the business logic encoded in these spreadsheets is often of the highest quality and extremely fit-for-purpose (as it’s the work of those with the greatest insight of the problems at hand) the processes of moving data in and out (and when inside, efficently processing such data) is usually highly time-consuming, manual and error prone. This is where I come in.

I can wrap such “Simple Systems” is a professional “envelope” that will automate the flow of data in, out and within the process while keeping the likely-to-change business logic in Excel. Even when the Excel formula-based business logic needs to be converted to “code” (perhaps for performance or automation reasons) most likely that code will be in VBA (a language familiar to many citizen programmers) which can also be made configurable via a simple Excel “table” rather than some obscure XML dialect.

To get started, send me your existing workbooks (with or without data) with a short note describing the process they support and what you would like to automate or change. I’ll likely respond with some further questions re the process and perhaps made some further suggestions for automation. When we’ve agreed a “deliverable” I’ll get back to you with a fixed price quote.

The idea is to turn “Simple Systems” into “Really Simple System” not works of software art. The main interface will continue to be Excel enhanced by Excel-based tools (mainly my microETL add-in and perhaps add-ins such as PowerPivot for added reporting goodness). Separating data-flow from business logic is usually the major change. It’s also important to be clear what parts of the process need to remain “user-configurable” and which parts are to be “professionally” automated.

The user-configurable parts will need to continue to be Excel-formula based or as code that’s at a level the user is happy to engage with, or as code controlled via configuration tables. The professional parts will be developed by a professional (me!) with the expectation that a professional (i.e somebody proficient in Excel, VBA and SQL) would be required to modify the (always supplied) source code.

Highly configurable solutions are sometimes more expensive than “black boxed” ones, with the most user-configurable option – leave as is – the cheapest (until you take into account the payroll costs of those staff repeating the same easily-automated task over and over again!).

Contact Tom …

Boy scratches Python…

I’ve written before about Scratch, a teaching platform developed by MIT to introduce kids to the art of programming. My son has been playing around with Scratch for over a year and although he still enjoys it, he’s showing signs of needing to move to the next level, a ‘real’ programming language.  I decided that Python, being one of my own favourite languages, would be an ideal next step, particularly when I discovered PyGame, a Python library based on SDL.

Using Pygame with its similar problem domain to that of Scratch would, I figured, make the transition to a grown-up platform easier, and so it has; concepts such as sprite, coordinates, animation etc. are common to both.  I took him through the “Pummel the Chimp” tutorial, expecting his young eyes to glaze over within 10 minutes, but no, a hour later he was still engaged and learning.  Why? He already has a deep understanding of programming, particularly object oriented programming, all thanks to Scratch.

Most of this knowledge he acquired without any help for me, I simply introduced him to Scratch and explained one or two concepts (variables and messages/method calls) which he initially had trouble with, the rest he picked up from looking at other Scratch projects and from writing his own.

So if your kids (or even you) have an itch to learn the essence of programming in a fun and effective way, then Scratch it.

xlAWS – 100,000 downloads?

Not sure, but this morning I received my monthly AWS bill, and it was double its usual amount! When I investigated the extra cost it was due to 133GBs of downloads from my www2.gobansaor.com bucket. This is the S3 bucket in which I store the xlAWS zip file, xlAWS being a “library-of-sorts” of VBA/VB6 helper code for accessing Amazon S3 and SimpleDB.

It’s linked to from this page on my blog (which has had 200 or so hits this month) and from this AWS Community Code page. The excessive hits on the bucket started on the 28th of Feb , the day the xlAWS code was published on Amazon and continued through most of March. Talking the size of the zip file, 133GB represents approximately 100,000 downloads. I don’t have server logging enabled on the bucket, so I can’t be sure how much is due to the other public files in the bucket (all belonging to the VBA/Proto SQLite xLite project), but as that project has been available for months and is accessible only through my website (who’s stats show a consistent 5-10 downloads per week) I’m guessing the downloads are for xlAWS.

Who would have though that there would be such interest in VBA/VB6 code for accessing AWS services! I wonder was it the Excel VBA side of the house or the dispossessed (and p*ssed off) VB6 developer hoards who downloaded it the most? Leave a comment if you downloaded and used the library, I’d love to know.

xlAWS – Excel VBA Code for accessing Amazon’s S3 and SimpleDB

I’ve been using Amazon’s S3 service from within Excel for sometime now and as there are no libraries or examples for calling AWS services from VBA (or VB6) I had to roll my own. As with most things Excel, getting the job done always triumphs over elegance and industrial strength implementations, in other words it was all a bit of a “dog’s dinner”. To remedy this and to share my experience of using S3 from within a VBA/VB6 environment, I decided to re-factor the code and to assemble it into a more re-usable form; the end result is xlAWS.

It was going to be called xlS3, but while doing the exercise SimpleDB appeared on the scene, so I decided to try accessing it from Excel, particularly as both products have a lot in common; both “simple”, both “schema-less” data stores. Like the S3Helper code, the simpleDBHelper module is less of comprehensive library, more a collection of useful functions which (hopefully) make working with AWS a bit easier.

To use this code library, you’ll need to have a good grasp of the S3 and SimpleDB APIs and be reasonably proficient with VBA. This is not an end-user tool, it’s for VBA (or VB6) developers. There’s a README and some basic examples within the Excel VBA project to help you get started. Code is released “in the spirit” of LGPL, you can use it how you wish, but if you add something new to the “library” (or find/fix a bug) do let the rest of us know.

As I’ve not been able to find a pure VBA implementation of the HMAC-SHA1 hash algorithm (and I couldn’t see an implementation within the standard “Microsoft Enhanced Cryptographic Provider” ) I’ve wrapped the open source XySSL SHA1 HMAC C code in a VBA friendly DLL. This DLL (and the source, under LGPL) is included in the zip file as AWS authentication requires SHA1 HMAC signatures.

You’ll also obviously require an AWS account. Credentials are stored within the workbook’s custom properties and can be encrypted via a “key file” if required. If you intend to use this code within VB6 (or Proto) you’ll need to provide your own implementation of the AWSKeyData class in order to use a non-Excel persistence store.

You can download the project ZIP file from here.

Have fun.

UPDATE

The simpleDB signature code no longer works as it used Signature Version 1 (which it turns out was insecure), you must now use Signature Version 2. It should be simple enough to change, but as I no longer use SimpleDB I’ve not done so. Also, see below for the many non-DLL methods for calculating HMACSHA1 hashes.

An alternative for calculating HMAC-SHA1 signatures in VBA/VB6 is a Google Checkout supplied COM DLL see http://bit.ly/9CIKtM

There’s the bones of a pure VBA HMAC-SHA1 implementation here http://www.eggheadcafe.com/software/aspnet/32187540/hmac-sha1-challenge.aspx

>>>>>>UPDATE: 17-April -1011

Here’s pure VB6 HMAC-SHA256 implementation it should be easily changed to provide and it now includes a HMAC-SHA1 facility (see comment below)  http://www.vbforums.com/showthread.php?t=635398

<<<<<<<<<

File this under: “Ya learn something new everyday”:

It seems you can access a significant subset of .NET libraries from VB6/VBA/VbScript (including System.Security.Cryptography.HMACSHA1) see this and dotNet in VBScript.

Zimki – the spirt lives on …

Although Zimki is to shut down on Christmas Eve, the ideas behind the service live on. Two new offerings, Horuku and AppJet, offer variations on the idea of hosted application development/deployment.

AppJet, funded by Paul Graham‘s Y-Combinator, is very similar to Zimki, being a server-side JavaScript platform. No details yet as to what sort of paid options will be offered (all accounts are free at the moment). Unlike Zimki there’s no plans to create an open-source version. I like the easy “build a Facebook app” feature; and I guess this is the sort of light-weight applications that they hope to attract.

Although Heroku uses Ruby-on-Rails technology, rather than JavaScript, it is closer to the original Zimki idea; but rather than take the hard (and ultimately unsuccessful in Zimki’s case) road of building an open-source platform from scratch, Heroku takes an already popular open-source project and offers it wrapped in a full on-line development and deployment environment. Again, being in beta, there’s no indication as to what pricing model it will operate under, but I would think that it will attract more “serious” projects than AppJet since anything developed under Heroku is pure Rails which means it can be migrated to any other Rails hosting environment; so no lock-in. The online editor is excellent and whatever about its merits as a hosting service it’s by far the easiest way to learn and explore Ruby and Rails, even easier than this…

If Facebook apps are your goal but you wish to use Ruby rather than AppJet’s JavaScript then not to panic, as being Ruby some bright young spark (no, not me I’m afraid) will already have done a lot of the hard graft for you…