Category Archives: olap

Star Schemas – to boldly go where no Excel spreadsheet has gone before

Posted on July 9, 2010 by gobansaor| 7 Comments

One of the many things that delights me about PowerPivot is the central role played by the Star Schema. Those of you reading with a data-warehousing background would shrug your shoulders and say: “So what, what else would you expect to find at the core of a BI tool?”.

Those from an Excel PivotTable background would ask: “What’s a Star Schema, why do we need one,what’s wrong with a the good old-fashioned single flattened table?”.

Those from a classic MOLAP background (Essbase, TM1, Palo) might also ask: “Why do we need this extra layer? Load the cube directly from the operational data model and get on with it!”.

A quick Q&A is perhaps the best way for me to explain why star schema design is a powerful skill in a datasmith’s toolset.

First off, what’s a Star Schema?

A Star Schema (also know as the dimensional model) is a denormalised (flattened) data model used to simplify an operational (OLTP) data model to better accommodate reporting and what-if analysis.

At its simplest, it consists of a central fact table with links back to a “surrounding” set of dimensional tables, hence the star name. A variation is the snow-flake schema, where the dimensional tables are not fully denormalised (e.g. Product Category->Product->Fact instead of  Product->Fact).

The role of the fact table (besides being the table that hosts most of the measure fields) is to create linkages between dimensions (such as Customer, Product, Date) usually based on an actual transactional event (e.g. Invoice) or a proposed event (such as a Budget or Forecasted Sale). In effect, simplifying  the often complex work-flow-driving connections of a typical operational system by using a single many-to-many relationship (modern ERP/CRM systems’ data models consist of scores of configurable many-to-many relationships).

Many wrongly believe the star-schema was adopted for performance reasons and now that in-memory OLAP is becoming the norm it’s no longer necessary to use dimensional modelling techniques. In fact, in the early days of data-warehousing, RDBMs had great difficulty efficiently handling star-queries (and some such as MySQL and SQLite, still do).

The original primary purpose of the star schema was to simplify the SQL required to access reporting data; to make the model more approachable to non-technical users. Of course, even simple SQL was beyond the knowledge or interest of most end-users but a sizeable proportion were happy to do so (often helped by SQL “generators” such as MicroStratery or Business Objects). But even in situations where SQL-wielding civilians were not to be found, the simplicity of the dimensional models  proved to be a valuable aid when establishing and developing the warehouse data requirements. PowerPivot requires no SQL knowledge to manipulate the dimensional model which brings the original concept full-circle but this time opening its possibilities to a much wider audience.

But surely, concentrating on the actual reports would be a more valuable requirements gathering exercise?

A so called “bottom-up approach” is often the best way to approach a reporting request particularly if the reports are simple one-off “traditional” reports. But for self-service BI, this needs to be combined with a top-down dimensional design. The idea is not to build out each and every report or indeed cube but to build a structure that’ll support likely queries. The process of building a star schema provides both a logical model and a physical implementation of that model against which potential queries can be tested. I’ve worked on several POCs destined for implementation in Essbase where the star-schema was built and potential cubes mocked up using Excel PivotTables that subsequently never went any further (except for the star-schema ETL process). The end-users derived sufficient value from the denormalised star-schema pivoted and reported in Excel.

In traditional ROLAP data-warehouses where the cubes were built directly against star-schemas, the pure logical approach to the data model often had to take a back-seat to the necessity of fine-tuning it to make response times (be that ETL or user-pivoting) acceptable. This is why I much preferred situations where the star acted as a logical model from which MOLAP cubes were built.

With PowerPivot, ROLAP has a new champion. The column-oriented high-compression in-memory architecture means that the compromises of the past are no longer necessary. The fact table reverts back to it primary role as a many-to-many connector. In a pure hypercube, measures are just another dimension (the approach that Palo takes), this is also now true of PowerPivot models; measures can be sourced from dimension tables and dimensions from fact tables as it logically should be, but without the performance hit of old.

But what’s the advantage of a star schema over a flattened table when using PowerPivot?

It is true that the same flattened table model as used to backend a PivotTable can be used within PowerPivot. But doing so would limit the potential of the DAX language to construct measures such as average sales spread over potential customers (rather than actual customers that would typically be represented on a flattened table). Also, by creating “conformed dimensions” (single cross-business views of Customer, Product etc.) and using such tables as dimensional sources for multiple fact tables, “virtual cubes” that combine values from multiple fact tables can be built.

If you’re new to dimensional modelling I’d recommend the books & articles of Ralph Kimbal as good starting point. You do have to be aware that some of the advice regarding efficiency trade-offs, surrogate keys etc. do not  apply in a PowerPivot scenario (even though other performance issues still apply) but the logical design tips still apply.

Star Schemas: to explore strange new conformed dimensions, to seek out new measures, to boldly go where no Excel spreadsheet has gone before.

→ 7 Comments

Posted in ETL, Palo, PowerPivot, excel, olap

Tagged

TAG Cubes – SQLite Star Query Part III

Posted on September 29, 2009 by gobansaor| 9 Comments

It’s no secret that I’m a huge fan of SQLite and Excel, particularly when used in combination. I also greatly admire the open source BI engines, Palo and Mondrian. Mondrian appeals because of its “ROLAP with a cache” architecture and its implementation of MS’s excellent MDX language. When I say MDX is excellent I’m talking with my professional programmer’s hat on, as an end-user tool it’s a non-runner. This is where Palo comes in, building on the hypercube concepts pioneered by the likes of  TM1 and ESSbase, it presents a designview that’s approachable by a vastly greater percentage of “civilians” than is the case with ROLAP-based solutions.

The trick behind TM1, Essbase, Palo etc. is the extension of the spreadsheet metaphor from two to multiple dimensions, while still binding the interface closely to the familiar spreadsheet (which for most of the business world is still Excel).

So where does SQLite come in all this?

At first glance, SQLite lacks the sophisticated join functionality to support star-queries, but of course, if the dataset is small then a full-table scan of a fact table, or better still, loading the fact table into memory negates any such short-comings.

In fact, all traditional ROLAP engines have problems with dimensional models, particularly when you reach the point of using summary tables or query re-writes, that’s why the emerging SQL-speaking columnar-databases are such a godsend for ROLAP data warehouses.

It was SQLite combined with Excel acting as a data prep platform that was originally my main interest, so for pivoting, Excel’s own pivot table would have to do.  Nevertheless, I felt the tool was incomplete without the ability to directly pivot the underlying SQLite database.

Why not use Palo or Mondrian as a pivot tool? Well yes, where a fixed permanent “solution” is required then the extra moving parts of either approach would be justified and indeed necessary but that is to miss the essence of what I call datasmithing.

Datasmithing is not data warehousing nor is it the provision of solutions (which, for example, Palo superbly enables in multi-user budgeting situations). Datasmithing, as a skill, is of course part of the process of both, but it’s on the edges, at perhaps the planning or consumption stages.

Datasmiths deal in the unknown, in change, in disaster recovery, in systems’ commissioning, in the never-ending barely-repeatable processes thrown up by daily business life.  For that, the toolset required must be as simple as possible (but no simpler), self-contained, document-oriented, secureable (is that a word?) and easily archived and retrieved. Excel and file-based DBMSs such as MSAccess or SQLite fit the bill nicely, server-based technologies such as DBA controlled database servers or IT installed “solutions”, less so.

Jedox has made Palo relatively easy to install (and likewise, Canada’s SQLPower has made Mondrian setup a painless exercise via their excellent Wabit reporting tool), but, the zero-install, email friendly document approach that spreadsheets are famous (and infamous) for, is preferable in many situations. This is something that Microsoft have recognised in their Gemini add-in for Excel 2010, but Excel 2010 is a not here yet and it’s likely to be five years or more before it’s as common as Excel 2003 is today.

The inclusion of FTS full-text searching with SQLite triggered an ah-ah moment with regards to pivot-enabling SQLite.

The usual method that hypercube-like excel-friendly OLAP tools use to return data is via a UDF like so…

=DATA(“CubeName”,”value1″,value2″,…)

…where valueN represents dimensional elements, so…

=DATA(“SalesCube”,”Beer”,”Profit”,”Jan 09″,”Actual”)

…is the Actual Profit for Beer sales in Jan 09. The dimensional elements act as “tags” to locate a particular value, there is of course much more to tools like Palo; hierarchies, intra-cube rules etc. but in essence most OLAP tools are like www.delicious.com for number crunchers. This method of retrieving data fits well with how people use Excel and not just for pivots, but for embedding OLAP aggregated cells in lists.  For example, a CRM scenario; a Sales Rep makes a list of her ‘best’ (subjective) customers, but needs hard (objective) stats, to be placed alongside the list to convince the boss or to track actuals against expectation.

Dimensional elements as tags; FTS3 virtual tables as fact table indexes; the concept of a TAG Cube was born.

In the above example “Profit” would most likely be described as a measure (Palo, a near pure hypercube does not distinguish between Measure and other Dimensional coordinates). Dimensions, measures and attributes are in reality interchangeable (a Customer ID can act as a dimension or an attribute, but by applying a  Count Distinct to it, it’s a measure) but most OLAP solutions treat “Measure Dimensions” as different, and so do TAG Cubes.

By using the default fact table structure (a single-columned table) and querying using the default measure (which translates to the SUM() of that single value) a ‘pure’ approach can be used. But, ROLAP is tightly bound to the concept of a fact table, and since SQLite is relational, TAG Cubes offer the ability to use a wide fact table approach and I think gains considerably in flexibility by so going.

The above example of using Count Distinct, or the simple creation of calculated measures are examples of this flexibility. Another, is a measure based on SQLite’s concat_group aggregate function to provide a drill-down facility, e.g.

=DATA(“SalesCUBE”,”ROWIDList,”Beer”,”Jan 09″,”Actual”)

…where “ROWIDList” would be setup as concat_group(rowid,’,') and will return a comma separated list of the underlying fact table ROWIDs.

A major reason for rolling my own pivot engine was to add a concept of “namespaces” and to separate the implementation of these namespaces from the actual pivot.  When a tag (or a predefined hierarchy of tags) is assigned to a cube, it’s also assigned to a namespace, in many cases namespace and cube would be synonymous, but in some cases a more sophisticated approach is required:

  • Multiple cubes sharing the same set of conformed dimensions would be best served by such cubes sharing a common namespace, and so they can.
  • Different consumers of the pivot may require the use of a different language, be that a spoken language or a different ‘business language’ e.g. Manufacturing Product Codes V Consumer Product Names. Again, easily done.
  • Sometimes identifying data can’t be shared with the datasmith or the numerical analyst working on a problem; in such cases being able to replace  the actual namespace with an obfuscated one can be very useful. Or, for added security, the namespace might only be issued to approved  PCs while the tag index and fact table are stored on a shared drive.  Needs some more work to make managing such scenarios secure and easy to use but the structure is there.

As hinted on above, the three elements of a Tag Cube, the namespace, tag index and fact table can be assigned to different databases (i.e. files). Due to the wonders of SQLite’s ATTACH statement and the backup API’s ability to quickly load/unload databases in/out of memory, it’s possible, for example, to load namespace and tag index (i.e. the ‘dimensions’) into a memory database, while a very large (i.e. too big to fit to memory) fact table remains on disk. Fast and cheap SSDs will add further configuration options.

Although most of the TAG Cube functionality is available only within Excel, I’ve built a C based SQLite Virtual Table (cFact) to allow the tag index to used outside xLite. This means that SQLite drivers for ODBC (for use as a Pivot Table source, for example) or JDBC (for use in  SQLPower Wabit perhaps) can efficiently access data models built using xLite.

I had to revert to using C rather than my preferred Python (did I mention that xLite now embeds Python in Excel, no, well it does, Python the newVBA ?), having failed to get around multi-threading issues with callbacks to Python in both the ODBC and JDBC drivers. I’d make a career promise to myself many years ago, not to having anything to do with printers or threads, and I think I’ll stick with it :)

TAG Cubes are the latest addition (still WIP actually) to be added to xLite, adding to:

  • VBA coded SQLite SQL functions.
  • Worksheet Functions; call out to a ‘function’ built using Excel formula, passing a parameter list and returning a value.
  • Workbook Functions; like Worksheet Functions, but loading a new Workbook, passing in parameters, passing back a value (or tables) and closing the Workbook when finished.
  • XLiteScript; xLite exposes its functionality via VBA coded UDFs, which can be called like any other formula, but data prep activities often require sequential procedural logic, xLiteScript is a table-oriented scripting mechanism offering basic flow-control logic.
  • pyScript; I embedded Python into xLite to take advantage of Python’s speed in developing Virtual Tables, SQL Functions and extensions to SQLite and to tap in the wonderful world of Python code. I’ve also added the ability to use Python from scripts defined within Excel (to indent, tab to the next cell!).
  • Fast load/unload to/from CSV.
  • Load from any ADO source.
  • Remove xLite formulae and rename and save Workbook, very handy when used via Workbook Functions to mass produce Excel “reports”.
  • Other WIP items are; load from SAP, load/unload to/from Amazon S3, use Palo cubes as TAG Cube “facts”, slot in/out Palo for TAG Cubes, auto-generate Mondrian XML based on TAG Cubes, write-back and splash, Python & VBA TAG Cube “rules”.

I’ve started the process of releasing the beta code here …

Why not join me on Twitter at gobansaor?

→ 9 Comments

Posted in ETL, Palo, SQLite, VBA, excel, olap, xLite

Tagged , ,

Palo BI Suite Community Edition

Posted on June 12, 2009 by gobansaor| 5 Comments

Jedox have finally published a roadmap for the Palo BI Suite Community Edition, having caused considerable confusion by pre-announcing its availability last April. See here for the details.  The headline dates are, a beta version due 1st of July with a Release Candidate due 1st September.

Although the announcement in April was essentially vapour-ware (no Worksheet Server V3, no Amazon EC2 images), one very significant actual deliverable was the addition to SourceForge of the Palo Excel Add-in sources (GPL licence).  This, at least for me, is very welcome as it now means that Palo is truly open source.  Prior to this, the server sources were GPL’d but not the main front-end tool used by the vast majority of end-users. In fact, the deep Excel integration offered by the Add-In is Palo’s main attraction to the business-focused “datasmiths” who make up the bulk of the product’s user-base.

The GPL’ing of the Add-in has removed the last barrier that had stopped me, as an independent consultant, from committing to the platform.  While the ‘freeness’ of open source is nice, it’s the source that really attracts me.  With access (and rights) to the source, I have no worries that the terms of use or indeed the product’s core functionality can be arbitrarily changed.  Having the source also means I can delve as deep or as shallow as I need to into the inners of the product, improving my understanding of the technology (both bugs and functionality) as needs dictate.

What has Palo BI Suite to offer, besides being open source?  Well, even if Jedox’s offering consisted of mediocre products, being open source as I explained above is in itself a huge advantage. Having an agnostic FOSS pivot engine that can be shared across many technologies, from Excel to Open Office to a PHP based website, is extremely useful.

However, Jedox’s BI suite is far from mediocre.  Palo is now a very polished and powerful in-memory MOLAP server with excellent integration with Excel (through the Add-In, or if you take out a support contract, via ODBO/MDX powered Pivot Tables).  The addition of a browser delivered spreadsheet (Worksheet Server V3) will add significantly to the product’s street appeal.  Version 3 differs significantly from previous WSS versions; being open source is one, but the entire product was also completely redesigned to meet the challenge posed by web-based products from the likes of Zoho, EditGrid and of course Google Docs (not to mention the ever-present threat of a MS response). Web-based spreadsheets are becoming a commodity, so Jedox’s response was to open source the product but at the same time make it more usable for real-world business analytics.

Current browser-delivered spreadsheets suffer from two shortcomings;

  • Spreadsheets with large numbers of inter-related cells (typical of business models ) tend to perform poorly, in many cases being unusable compared with Excel or Open Office.
  • Only available as hosted SaaS; not a major problem for some businesses, but for others, services outside the corporate firewall, especially for sensitive information such as what-if, budgeting and sales analysis models, are a no-no.

WWS V3 gets around both problems.  Performance is improved by the use of Palo as the spreadsheet’s pivot engine but also by the “lazy calculation” of related cells i.e. a cell that’s not visible, and itself not yet referenced by other visible cells, remains uncalculated, saving on the constant churning that can effect large models.  This combined with a DynaRange concept means templates and models react dynamically and efficiently to the ever changing datasets being presented to the sheets from the Palo OLAP server.   The look’n’feel is very similar to Excel with even array-formulae being fully supported.

The second problem of only-behind-the-firewall access is solved by the open source GPL licence and by the front-end being coded in PHP (very approachable to most in-house support staff and even the odd accountant).  The core (the bit not yet released) is, as far as I know, C++, so is likely to join Palo Server as being highly efficient and well engineered but perhaps beyond the technical skills of most.

The other elements to the BI Suite are the web-based OLAP-centric ETL Server (now, I see, with Groovy and Javascript scripting support) and the Supervision Server (only in paid Enterprise version) which offers fine-tuned access control and monitoring, plus drill-through functionality from Palo cells, back to the ETL fact tables. The Enterprise Edition also offers a multi-core version of the Palo server along with SAP and ODBO/MDX connectivity.

If multi-dimensional analysis and budgeting could benefit your business and spreadsheets are your preferred method of communicating and working with such analysis, you need to check this out.  Palo is a well kept secret (at least outside of Germany), hardly ever mentioned by the mainstream BI community, but don’t let that put you off; this is one of the best solutions out there, it’s open source but also comes with the backup of a professional company that can offer not just technical support but also implementation know-how (Jedox eats its own dog-food, being both a BI consultancy and development house).

Update July 4th 2009:

Beta Community Edition is now available.   I downloaded and installed WWS V3 and gave it a quick test-drive; looks good, Palo interface has the look’n'feel of the Excel Add-in and the general spreadsheet functionality is very Excel-like, incluing CTRL-Shift-Enter to assign array formulae.  Overall, the Palo BI suite offers a intuitive end-user-friendly interface; from download to effective use in less than 60 minutes, how many BI tools could you say that about?

Also, in two weeks time a pivot-table friendly ODBO driver will be included for free with the Palo Excel Add-in (previously only available to those with a Jedox support contract).

Why not join me on Twitter at gobansaor?

→ 5 Comments

Posted in BI, GoogleApps, Groovy, Palo, olap

Tagged , ,

LiteBI, Heavy ETL

Posted on April 24, 2009 by gobansaor| 2 Comments

Although my major BI interest is in micro-BI (or is that  workgroup-BI?)  i.e. data, perhaps cleansed and packaged elsewhere, available locally on a datasmith’s PC,with most likely an in-memory OLAP as the analysis tool; the possibilities of the “cloud” as a BI platform have not escaped me.

From a micro-BI perspective, the ability to act as a backup/mirroring tool or as ETL/marshaling tool (anybody for Hadoop and SQLite?) attracts. I’ve yet to make up my mind on BI delivered as a cloud PaaS but obviously many others believe it has a future.

My main worry with PaaS is not lock-in (which exists equally for in-house proprietary solutions) but the dangers of a Coghead-like lock-out.  My other doubts are more technical; believing, as I do, that in-memory offers significant advantages over traditional ROLAP (simplicity been the main one) and multi-tenant in-memory architectures are not yet a runner.  But last week I had a demo of new Spanish BI PaaS service, LiteBI, which might just change my mind.

Javier Giménez Aznar and his team previously worked on delivering Pentaho based datawarehouses to large Spanish corporations and government agencies, so they have a deep understanding of Mondrian ROLAP and are using that knowledge to build the LiteBI service, but this time with SMBs as the target customers rather than corporates. Pricing starts at €145 per month and is based on number of concurrent users, number of analytical spaces and the data volumes, so it’s not for very small firms more for the Medium in SMB.

Impressions? The cube designer, dashboard builders and the general UI are all very good and I would think would appeal to end-user datasmiths and, as such, will be a major up-front aid to selling this product.  But it was LiteBIs approach to the thorny issue of ETL and data loading that impressed me and also helped ease some of my Coghead-induced-fears.

BI technology stacks consist of three elements:

  • The “fancy” front-end; graphs,animated dashboads and so on.
  • The pivot engine; ROLAP or MOLAP or both.
  • The ETL process.
  • (Many would say there’s an important 4th, the data-warehouse, but not every BI effort requires one, but that’s another issue)

LiteBI is continuing to build yet more functionality into their UI and this “fancy” front-end is essential as it’s their “shop window”.

Mondrian provides their pivot engine, and again they continue to work on optimisations such as column-based datastores to increase speed and automate responsiveness tuning (end-users are very unforgiving of slow pivots).

But it’s in the 3rd area, that of the ETL process, that you realise the LiteBI team has real-world BI experience.  Data is loaded into LiteBI via an API, but with the ETL process itself happening on the customer side.

“Well,so what?” you may ask. The extraction of data has to obviously happen customer-side (even though not in the case of data being sourced from the likes of SalesForce.com). Yes, but it’s the transformations and data cleansing that adds true value to the ETL process and subsequently determines the quality and usefulness (as opposed to the speed or the “prettiness” of delivery) of the solution.

Part of the process of adopting LiteBI, is an ETL consultancy stage where a LiteBI partner company will provide on-site services to build this ETL layer, handling not just transformations but initial load and automating the subsequent delta uploads.

So the cost mounts up, but in reality you can’t do BI without this investment; there’s no ETL magic bullet.  Even still, Javier says the typical go-live time for a LiteBI project would be in the order of 3-4 weeks rather than the 3-4 months of similar on-site Pentaho projects.

The end-user ‘owning’ the ETL process makes the prospect of a service lock-out slightly less worrying as, at least, one would still have a good starting point for moving to another provider or back in-house. What I would really like to see would be the option to self-host LiteBI, which I guess would involve open sourcing large parts of the service (the automated optimisation strategies could, for example, be excluded from this open source version).

The load API comes packaged as a plugin to Kettle (aka PDI) and the intention is to offer a similar add-on for Talend in the near future. LiteBI also offers a white-label offering whereby 3rd party OLTP solution providers can use the service as their product’s BI suite.

Like the Skibbereen Eagle keeping its eye on the Czar of Russia, I too will be keeping a watchful eye on LiteBI and the march of on-demand BI in general.

Why not join me on Twitter at gobansaor?

→ 2 Comments

Posted in BI, ETL, Talend, cloud, data, kettle, olap

Tagged

SQLite as the MP3 of data

Posted on March 14, 2009 by gobansaor| 19 Comments

… and Excel as its “mixing desk”.

When I tell people that I use SQLite in combination with Excel (via xLite) as my datasmithing platform, many ask why SQLite? (Many others ask why Excel?  but “sin scéal eile”, that’s another discussion – Excel as the iPod of Downloaded Data.) Those that question my use of SQLite tend to cluster into four camps:

  • Pure Excel jocks.
  • MS Access fans.
  • The client server database brigade (SQL Server,Oracle; or if FOSS fans; MySQL, PostrgeSQL).
  • The MOLAP folks (Essbase, TM1, Palo).

Now while I have used and will continue to use/encounter all four ‘approaches’, I’ve come to believe over the last couple of years that SQLite brings something special to the datasmithing game. When I look back over nearly 30 years in the data handling business I keep thinking – “If only I had SQLite then, how much easier/quicker/cheaper that task would have been!”.

Just as “fractional horsepower” electrical motors revolutionised manufacturing and eventually all our lives (car starter-motors, fridge motors, washing machines etc.), “fractional horsepower” databases can do the same for data. Distributing data to where it is needed.

As operational local caches, this use of SQLite is already far advanced. SQLite is embedded in lots of every day software tools, everything from McAfee anti-virus to TweetDeck Twitter clients (best one IMHO). But my interest is more in SQLite’s potential as a micro-BI (or maybe more correctly a distributed-BI) platform. A sort of MP3 format for distributed structured data, if you like.

But why SQLite (and in particular SQLite in combination with Excel) as my datasmithing tool rather than the four other approaches?  First, what’s a datasmith?

Managing and manipulating datasets has become an integral part of many people’s job, not just accountants (the original of the species) but marketing executives, sales staff, pricing analysts, process engineers; different job titles, different roles but using a skill that they’ve likely never been formally trained in, a skill without a name; a skill I call datasmithing. I like to think of  myself as a master datasmith, or a datamith’s datasmith.

If you consider yourself a datasmith then most likely the tool you use to manage your datasets is Excel. And before you apologise, don’t. Excel is by far the best and most flexible end-user data manipulation tool out there. Everything from the current financial crisis downwards has at some stage being blamed on Excel, but you know and I know that many tasks would remain undone or under-done were it not for end-user generated spreadsheets.

Spreadsheets are not however optimal for some tasks, linked spreadsheets in particular are data disasters in waiting. While fantastic for data transformations and presentations, as books-of-record they’re rarely suitable. Other tools such as SQL based relational databases and in-memory OLAP offer much better and potentially much more cost-effective data modelling functionality, but also at a cost of extra complexity and ongoing technical support.

MS Access (which like SQLite, is a document-centric, non-client-server database; but unlike it, is also a forms/reporting development environment) would appear to be the natural local store database. My problem with MS Access has been its tendency to try to be all things to all men, ending up not fully satisfying anybody. Professional developers think it’s too limiting, non-techs find it too intimidating, even reporting, where it once showed promise left a big enough opening for Crystal Reports to evolve. It is also limited to Windows which might not seem to be a problem if combining with Excel, but, as it’s often necessary, due to scale or complexity of the data,  to use ‘proper’ ETL tools such as Talend, having an OS agnostic database format than can act as a distribution media (think MP3s again) between “mixing desks” can be very useful.

The big difference to MS Access for me is SQLite’s open source code; code that’s a pleasure to browse and with an approachable API that even I, with my very rusty C skills, can manipulate. Having access to that code allows me to tightly integrate it with Excel, so much so, that I can use Excel functions (built-in functions, VBA user-defined functions and 3rd party add-in functions) directly from SQLite’s SQL; and vice-versa, access SQL functionality via Excel “formula” calls. It is  also possible to  load most datasets into memory using SQLite’s in-memory mode enabling very fast processing  and near zero-latency when passing data to and from Excel/VBA. In the near future, cheap, large SSDs will enable non-memory databases to offer similar speed but also handle extremely large datasets (see this for a glimpse of that future).

What about the big beasts of the data world, the client-server databases? Having spent most of my professional life working with such tools I’m aware of the power of a well designed relational database. If SQLite is the MP3, then these are the master tapes, the DDD recordings. Most of the data that eventual ends up in SQLite for analysis and/or transformation will have originated in data-warehouses or be directly sourced  from OLTP systems built using relational technology. But for close-up analysis and transformation, the pure simplicity and convenience of SQLite is hard to beat. That simplicity is primarily due to its Excel-like ‘document’ nature, all code and data can be housed in a single folder (or true-crypt container for added security), ensuring that the ‘problem domain’ can be easily archived and/or shared with others without the need for professional IT resources.

And yes, I hear you, isn’t that the basis of Excel-hell? Yes it is, but over the years I’ve found that this is rarely a problem for datasmiths, they deal day-in day-out with document work-flows, they understand the risks and the benefits (mainly the simplicity) of the approach. Where the nightmare truly happens is when this approach is used as an alternative to an OLTP system i.e. using Excel and other document-like datastores as books-of-record in large multi-user environments – “there be monsters for sure”.

How about MOLAP? Wasn’t Essbase’s name derived from “extended spreadsheet database” and doesn’t Palo offer a truly excel-friendly multi-user database back-end? Again, having worked with Essbase for many years and now being a big fan of the open source Palo MOLAP tool, I fully appreciate the power that such tools brings to analysis and multi-user planning tasks. But for many situations, an Excel Pivot Table is “good enough” and even when it’s not, it is possible by utilising what I call a tOLAP cube (essentially, a fact table indexed via tags enabled by Google’s great addition to SQLite, the FTS3 virtual table) to build and access  powerful, yet simple, cube-like data structures.

By integrating SQLite with Excel, datasmiths can have the best of both worlds, familiar spreadsheet front-end combined with a fast and powerful SQL engine and datastore, in fact, everything that MS Access should have been.

Why not join me on Twitter at gobansaor?

→ 19 Comments

Posted in BI, ETL, Palo, SQLite, VBA, excel, olap

Tagged

I’ll give up Excel Pivot Tables when you take ‘em from my cold, dead hands

Posted on October 10, 2008 by gobansaor| 19 Comments

Jedox, the company behind the open source MOLAP server Palo, has just announced an MDX driver. This means that it’s now possible to access Palo cubes using Excel Pivot Tables or indeed any tool that supports ODBO.  This is excellent news, as MOLAP to most Excel users IS a Pivot Table, and somewhat like the NRA, the NPTA’s (National Pivot Table Association’s) motto is “I’ll give up Excel Pivot Tables when you take ‘em from my cold, dead hands”.

MDX/XMLA is now a de facto standard for OLAP servers, supported not just by MS SQLServer but by SAP BW, Hyperion/ESSBase and by Pentaho’s Mondrian. The new driver is not open source, nor is it for sale but instead comes free to those with Jedox support contracts. I’m sure lots of organisations will be more than willing to enter a support contract (starting at €3000 per server) to get their hands on this; think of the savings in training alone!

UPDATE: 2nd July 2009

Kristian Raue has announced on his blog that the ODBO/MDX driver will now come free with latest Palo BI Suite (both community and enterprise versions). Excellent news!


→ 19 Comments

Posted in BI, Palo, excel, olap

Tagged , , , ,

Why Larry hates the cloud, and my data trinity.

Posted on October 4, 2008 by gobansaor| Leave a comment

Last week Oracle certified Amazon EC2 as a supported platform, that same week Larry Elison attacked the concept of cloud computing as pure hype. Obviously, Larry is not happy with this whole cloud thing, and I think it’s not just the threat it poses to the software industry’s traditional licensing model that worries him, rather, as Robert X. Cringely points out in his “Cloud computing will change the way we look at databases” post, it’s the likelihood that it sounds the death-knell for large-scale traditional databases.

This new database paradigm is memory rather than disk centric, with the disk-based element acting as an archive/backup/restore mechanism which can easily be stored on commodity SAN devices ( e.g. Amazon’s ESB). Using MapReduce technology Google effectively holds the whole Internet in memory, not in one big super computer but in lots of cheap commodity servers.

But it’s not just in the realm of mega datasets that RAM based databases threaten traditional models. Excel is a memory-based database engine, so too in-memory OLAP tools such as Palo. Such products’ ability to handle large volumes of data has increased over the years, with the decrease in RAM costs and the appearance of cheap 64 bit machines (which are no longer limited to 2G/3G process working sets).

That doesn’t mean that we’ll throw away SQL databases in their entirety, SQL and the relational model will continue to be useful. But perhaps of greater use in local datastores/caches that as the building blocks for large scale datastores. For such local caches, less will be more; fewer features, easier to configure, more flexibility. That’s why I like SQLite; long after the dinosaurs of the database world have disappeared, I imagine SQLite databases will continue to survive, embedded in mobile phones, browsers, wherever a local datastore is required. And more than likely operating in memory rather than off disk.

By combining Excel with an in-memory SQLite database, linked to a Palo OLAP in-memory server, it’s possible to take advantage of three powerful data-processing technologies (spreadsheets, SQL, multi-dimensional cubes) all within your PC’s RAM. You could do serious datasmithing with such a combination on a pretty mediocre laptop, with most modern machines providing an excess of CPU power, no need for super fast disks, just as much memory as you can muster. And, with Windows on EC2, these three amigos will soon be capable of being used as a cloud bursting platform.

Excel, SQLite and Palo, my data trinity.

→ Leave a comment

Posted in AmazonAWS, ETL, Palo, SQLite, cloud, excel, olap

Tagged ,

Cloudy skies, cloudy apps…

Posted on August 28, 2008 by gobansaor| 4 Comments

Just back from a break in Clifden, Connemara, summer is nearly over, the kids return to school today, back to work.

Aasleagh Falls, Co. Mayo

Aasleagh Falls, Co. Mayo

Counties Galway and Mayo were like the rest of the country last week, a tad wet, but unlike the developed east of the island, flooding was not a problem; a problematic drainage area is called a lake in the west.

This August has been the wettest and dullest I’ve ever experienced but at least I saw some sunshine earlier in the month thanks to Kristian Raue CEO of Jedox who kindly invited me to visit the company’s offices in Freiburg, Germany.  Freiburg is very green in both senses of the word, surrounded as it is by the Black Forest and its well deserved “eco-city” status.  Its also know as the warmest city in Germany, a reputation it thankfully lived up for this visitor from a rain-soaked Atlantic isle.

August morning, Frieburg Im Breisgau

August morning, Freiburg im Breisgau

If Freburg left a positive impression on my mind, so too did Jedox.  The overall impression is of a company which intends to use a combination of quality, vision and the judicious use of open-source to build the Jedox brand into one associated with best-of-breed products and consultancy.  This vision can be seen in the evolution of Palo, from its “good enough” beginnings to its current near-best-of-breed 2.5 version, and from talking to some of those working on the product, best-of-breed status is not that far off.

Likewise, ETL-Server which is currently a Palo only “loader”, is to be further  developed into a true ETL tool, while continuing to offer MOLAP-centric specialisms.

I also got a glimpse of the next version of Worksheet Server. “Wow!”, is all I can say.

Existing web based spreadsheet products are fine for simple data analysis or basic data capture purposes but cannot compete with their client-based elder cousins when serious datasmithing is required.  Well, from the demo I saw of Worksheet Server in action, that’s about to change.  The look and, more importantly, the feel is similar to that of traditional spreadsheets, its interface with Palo is identical to that of the existing Excel add-in, and here’s the big one, its open source!  Game-changing or what?

But …

That might enable me to move a lot of my spreadsheet applications to the cloud, but what about those applications that are more suited to an MS Access type solution?

Then try out WaveMaker. It’s open source and built on industry standards, Hibernate,Spring and the Javascript Dojo framework but has the ease of GUI database development more usually associated with MS tools. The resulting applications are packaged as a WAR file which can be hosted by any standards based Java server (e.g. Tomcat or Jetty).  The latest version makes developing Ajax-fronted database applications even easier with the addition of layout templates.  Its existing ability to automatically bind interfaces to SOAP web services has been extended to REST web services by means of a new WSDL auto-discover tool.  And Chris Keene CEO of WaveMaker also informs me that …

We are also releasing a cloud-based IDE in October with Amazon – stay tuned…

We launched in February and will be announcing our first 7 figure deal this month. We run on Mac, Linux and Windows and are currently the #1 developer download on Apple.com (http://www.apple.com/downloads/macosx/development_tools/)

Our goal is to make it easy to build rich internet applications without complex coding – kind of a MS Access for the Web.

Jedox and Wavemaker the new breed of open-source businesses…

→ 4 Comments

Posted in BI, ETL, Ireland, Palo, Web2.0, cloud, data, excel, news, olap

Tagged , , ,

Talend + SQLite + Groovy the new Oracle …

Posted on August 2, 2008 by gobansaor| 7 Comments

… well, at least for me.  Let me explain.

For most of my datasmithing career, I’ve had access to corporate Oracle databases and now with the availability of  Oracle10g  Express I can even run my own Oracle instances at home or on EC2.  The combination of a powerful SQL engine, expressive scripting language (PL/SQL) ,OS independence, web front-end (App Express) and the ability to communicate with Excel (via OO4O) made Oracle a natural fit for heavy-duty data manipulation.   But there was always one major problem, Oracle doesn’t play well with other data sources, necessitating a separate ETL bolt-on, which led me to play around with the likes of Kettle and Talend.  But having been seduced by these new shiny (and open source) “toys” I’ve found that rather than just been incidental add-ons they had the potential to totally replace Oracle.  The combination of Talend, SQLite and Groovy, is proving to be particularly magic.

So how will these three tools enable you to leave behind your Oracle past?

Talend (in its Java form) is a superb ETL tool, via JDBC is can access every database type on the planet, it has built-in web-service capability and access to a  multitude of APIs via its Java component for non-database data sources.  The addition of  Groovy makes the use of such Java APIs simpler and quicker and the same Groovy acts as a replacement for PL/SQL when a bit of “if-then-else” logic is required.  And although Talend offers a built-in option to plublish an ETL job as a WAR file exposing a SOAP web service, Java/Groovy also allows for the integration of the powerful, yet simple, Jetty API to embed a web server within Talend itself.  And all this for free, and better than free, open source.

So where does SQLite come in? And, didn’t you say that Excel integration was important, how will Excel communicate with Talend?

As very little corporate data is held in SQLite format, and Talend allows access to every major commercial/free database, the usefulness of SQLite might not be at first obvious.  But if you think of SQLite as a data cache, a fast and efficient local tabular datastore, with a powerful but well understood DSL (i.e. SQL) and a drop-dead-simple setup and backup regime (basically copying and creating files), maybe then you can see its attraction. The ability to extend the DSL by easily creating SQLite user defined functions (UDFs) within Talend using either Java or Groovy is also another powerful feature.

For example…

select customer_id, name,customer, sales_region, getpalodata(“SALES”,customer_id,”All Products”.”Total Sales”,”Euros”,”YTD”)  as customer_YTD, getpalodata(“SALES”,sales_region,”All Products”.”Total Sales”,”Euros”,”YTD”)  as region_total_YTD from list_of_top_customers;

… where getapalodata is a UDF that wraps calls to a Palo cube.

With this type of setup I can easily mix and match list/tabular data with multidimensional data points using SQL (something that Oracle also supports but only if you hand over a large wad of currency). In fact I can create a mini data warehouse, with Palo providing the pivot, ( as SQLite lacks star-query (or even multi-index query) support.)  SQLite would still host the conformed dimensions and the fact tables, but with the fact tables acting as feeds to Palo cubes, supporting finer-grained drill-throughs from cubes or for ad-hoc queries. This is powerful stuff, simple, free, powerful stuff.

… and the spreadsheet access?

A Talend sub-job such as this…

Talend Groovy Jetty web server

Talend Groovy Jetty web server

Example of Groovy code calling Jetty API

Example of Groovy code calling Jetty API

…would provide a simple RESTful (rather than SOAP) web service which could be accessed either with an Excel Web Query or via a VBA macro which would parse the result and allow for more control.  For example …

http://localhost:1234/sqlgateway?sql=select customer_id,name from all_customers&type=HTMLTable

… this would return a list of customers wrapped in an HTML table, or …

http://localhost:1234/job/extractProspects?Rep=JonesTom&Month=JAN&SourceCompany=AXA&type=HTMLTable

…this might call a Talend job called extractProspects, passing in JonesTom, JAN and AXA as context parameters, which would then return a list of prospects extracted from a feed supplied by AXA’s system.

What would the Talend job look like?

The job might operate something like this:

  • It would run either on the client as a service or on a LAN based server (or on a remote server, with a SSH VPN (or Hamachi) to provide security).
  • At start-up, do a bunch of ETL tasks, pulling data from remote sources and databases, transforming and aggregating data etc. Storing the resulting data in local SQLite databases.  It might also build Palo cubes or update larger enterprise databases.
  • The job would then setup a Jetty web server and await requests for data.
  • The requests might be a mixture of raw SQL or requests to run specific Talend transformations which would return a dataset directly to the calling client or maybe just acknowledge the request, queue it up for processing later, sending the resulting dataset by EMail or RSS feed when finished.
  • At a fixed time the service would shut it self down and requeue itself for the next day’s workload.

… or nothing at all like that, and that’s the point, build what you need, add the levels of security (or none at all) that fits your situation, all within a open framework, with zero lock-in (okay, still using Excel, anyone for OpenOffice, Google Apps or Zoho?).  You don’t even need your own server, host it on an EC2 instance, (if you bring up an instance for 10/12 hours every working day, it would cost about $20/$25 a month).

Now tell me that doesn’t make sense?

→ 7 Comments

Posted in BI, EC2, ETL, Groovy, Palo, SQLite, Talend, data, excel, olap

Tagged ,

OLAP Cube as a Mind Map

Posted on July 30, 2008 by gobansaor| 3 Comments

If you’ve worked with OLAP technologies for any length of time you’ll undoubtedly have been in the situation where you’ve had to explain the concept of an OLAP Cube to a “newbie”.  If the person in question has come across Excel pivot-tables, then you can probably short-circuit the conversation some what, explaining that a pivot table is in essence an OLAP cube, maybe highlighting the differences between it and whatever OLAP tool you’re proposing; ragged hierarchies, ability to update cells and ‘spread’ values down hierarchies etc.  Even better if you can show the user an working cube populated with hierarchies and elements that match the user’s business.

But what if the person is a complete novice to the world of analytics and you don’t have a relevant demo cube that you can demonstrate, what then?  I guess, you could start by first trying to fry the victim’s user’s brain by explaining the concepts behind multi-dimensional spaces and/or a quick intro into the world of de-normalised databases, with examples of snow and star schemas. The glazed look in the eyes of your audience may however suggest this doesn’t always work.  And it’s not just business-users who have problems getting their heads around cubes, many programmers also have difficulty the first time they’re exposed to OLAP concepts.

I’ve found that Mind Maps offer a good way to help both me and the client to visualise the domain model that a cube will eventually address; having worked through the mind mapping process it’s then easier to take the user with you as you translate this model to a physical cube or star-schema.  Well Hugo (who’s back blogging after a long absence) promises to take this method a step further, his modifications to FreeMind will allow for the export of a mindmap as a Palo cube!  Check it out …

→ 3 Comments

Posted in BI, Palo, excel, olap

Tagged , ,

Postgres Plus Cloud Edition is boring …

Posted on March 27, 2008 by gobansaor| 2 Comments

… and that’s good. That’s how I like my databases, boring, reliable, consistent, easy to use.

SimpleDB on the other hand is not boring, it’s an exciting new shiny thing that opens up a myriad of new possibilities; but first, I and the rest of the developer community, need to tool up and cast aside some of our cherished database design patterns (oh like, 3rd normal form, strong typing, joins, nothing major) and embrace a slightly different way of thinking, however, as much as I like a challenge, I also like to get things done.

That’s where EnterpriseDB’s new Postgres Plus Cloud Edition comes in, this is an Amazon Ec2/S3 hosted edition of their Oracle compatible PostgreSQL-based product that offers the scalability of SimpleDB but the familiarity of a traditional relational database. The “magic” is supplied by Elastra, who are also offering the same functionality against MySQL and standard PostgreSQL databases.

A Talend ETL job which I had been developing for a client, had been tested against a “normal” EnterpriseDB instance. This ETL job was part of a BI prototype trialling a Postgres Plus Cloud Edition (the new name for EnterpriseDB’s cloud offering) as the back-end database. So, I exported the job as a Java executable, fired up an EC2 instance, copied up the generated JAR files, changed the database’s hostname to that of the Postgres Plus “cloud” database, ran the ETL job and it worked. As I said, boring, nothing to report, it just worked.

Now you may be wondering what’s so special about these Elastra powered databases, surely EC2 is no different from any other Linux virtual machine, why not simply install a standard database? The problem with EC2, and it is a problem to those of us (i.e. practically every IT pro on the planet) who have come to expect highly reliable RAID backed disk storage, is the non-permanence of its disk systems.

When an EC2 instance is powered down or fails, the disk system is wiped!

That, combined with fixed (if generous) disk sizes (160GB, 850GB or 1690GB), means that often a clustered database environment is a necessity, adding considerably to the complexity. It’s this sort of complexity that SimpleDB and Elastra address.

The obvious use-case for both Elastra and SimpleDB is as data stores for OLTP applications but Elastra’s ability to handle S3-backed massive databases means the possibility of using EC2 as a data warehousing platform is also considerably strengthened. Although not obvious at first glance, SimpleDB could also act as an OLAP data store; SimpleDB massively indexed tuples as “sparse dimensions” pointing to S3 objects (SQLite databases?) that hold the fact data combined with dense/”partioning” dimensions (e.g. Time). Possible ? Yes. Fun to do? Yes. A solution that I can apply tomorrow? No, that’s why I’m glad EnterpriseDB and Elastra are delivery such a boring product!

UPDATE Ec2:

The other big EC2 missing – non-permanent IP addresses – has at last been addressed. EC2 now offers “Elastic IP Addresses”, addresses associated with an account not an instance. If the instance fails or is shut down, the IP address can either be immediately re-assigned to a new instance (no more waiting for Dynamic DNS propagation) or “reserved” for future use at a cost of USD0.01c per hour. Also, the new “multiple locations” facility puts the API changes in place to allow for location selection, hopefully a sign that we here in Europe will have “local” EC2 instances to match our European S3 buckets!

UPDATE EnterpriseDB:

It looks like IBM have invested in EnterpriseDB, possibly as a counter-weight against Sun’s acquisition of MySQL (EnterpriseDB’s targeting of Oracle’s customer base would also be an added benefit!).

→ 2 Comments

Posted in AmazonAWS, BI, EC2, ETL, S3, SQLite, SimpleDB, olap

Tagged , , ,

Dublin Bus and PALO ETL – the connection!

Posted on January 26, 2008 by gobansaor| 5 Comments

Dublin buses, as is the norm with most road-based public transport systems in our increasingly car-choked cities, tend to operate on the basis of “no sign of a bus for ages, then two or three arrive at the same time”. Palo MOLAP ETL options appear to be following the same pattern; we’ve been waiting for ETL support for ages and now we see three of them heading down the road towards us. There’s Palo’s own offering, then came Stratebi‘s Kettle Plugin and now Talend Version 2.3.0RC2 is offering a Palo output component.

Mind you, the Talend offering is very basic and I’ve not managed to get the Sratebi plugin to work, leaving Palo’s ETL Server as the front runner at the moment (drill-through capability is a winner in my book).

I’ve also been busy re-factoring my VBA SQLite and Amazon S3 code with the intention of publishing them as an Excel based micro-ETL platform. While cleaning up the Amazon AWS modules I’ve been playing with SimpleDB, I’m impressed, Excel combined with SimpleDB rocks!

I’ve also wrapped the open source XySSL SHA1 HMAC C code in a VBA friendly DLL, as searching for a VBA hmac sha1 hash implementation (essential for Amazon AWS access) has proved fruitless.

Hope to release the lot the end of next month.

UPDATE:

Thanks to Javier and Jorge from Stratebi I’ve managed to get the new Kettle Palo plugin to work. It seems that the TEST facility in the Kettle database connection dialogue throws an exception for Palo connections but the connections work fine in the actual Palo input/output steps. Did a quick test and it looks very easy to use and fits in well with the Kettle “way of doing things”.

→ 5 Comments

Posted in AmazonAWS, BI, ETL, Palo, S3, SQLite, SimpleDB, Talend, VBA, excel, kettle, olap

Tagged , , , ,

PALO ETL-Server and SAP

Posted on January 5, 2008 by gobansaor| 1 Comment

Jedox have just published a roadmap for their open-source ETL-Server, release date of March 2008, same date as the next release of the Palo OLAP Server. In a future release they intend to offer SAP RFC/BAPI and SAP-BW XMLA support, being an old SAP hand this looks very interesting.

There’s also a features page with a good overview of the ETL-Server’s technical architecture.

→ 1 Comment

Posted in ETL, Palo, olap

Tagged , ,

PALO ETL Server, more sightings …

Posted on January 3, 2008 by gobansaor| 1 Comment

First day back after Christmas, snow falling outside.

More additions to the PALO ETL-Server SourceForge project, new version of the core and, a new web server – built using Jetty and Apache Axis. Axis is a SOAP handler so I looked around for the WSDL file to see what services are to be exposed and found a reference to a drillThrough service which I guess is the mechanism by which we’ll soon be able to drill back from a PALO cube to its source data tables. At the moment I’m just wandering through the source code, I’ll need to fire up an EC2 instance to give it a test run as the new server code doesn’t fully support Windows.

Happy New Year!

→ 1 Comment

Posted in ETL, Java, Palo, olap

Tagged , , , ,

New ETL platform for PALO OLAP

Posted on November 28, 2007 by gobansaor| Leave a comment

Jedox have announced that they intend to ship a Palo centric ETL open source server product early next year. This is excellent news and is on top of the new rules engine that was added to Palo this summer. Open source MOLAP has suddenly taken off the training wheels and is getting ready to mess with the big kids. The two things I really like about the new proposed Palo-ETL server are; it’s open-source and it’s designed to enable drill-down from the analysis cube back to the data source.

Drill back is the 2nd most common reason for continuing IT staff involvement in the day-to-day running of BI projects (the 1st is of course the ETL process). The “Where did that figure come from?” question is one of the reasons that the Excel pivot table function is so popular, double-click on a data cell – the rows used to generate that figure are displayed on a new sheet; simple but powerful.

As to what platform the new server is to be built on, I’m guessing Talend or Kettle. Now it is possible that Jedox have rolled their own product from scratch but with two superb open-source products already out there it would seem pointless. Talend actively seeks out technology licensing agreements with other companies and has just opened an office in Germany (Jedox is based in Freiburg) so it would be the most likely contender. But, Pentaho’s Kettle may also be in the running as there’s already prior-art and this Pentaho forum thread. Also, the one major missing from the comprehensive Pentaho stable is lack of an in-memory OLAP tool (their current OLAP offering is the ROLAP Mondrian project).

→ Leave a comment

Posted in BI, ETL, Palo, Talend, kettle, olap

Tagged , , ,

In Memory OLAP

Posted on September 5, 2007 by gobansaor| 9 Comments

The consolidation within the BI market continues, this time with the purchase of Applix by Cognos. As Timo Elliott points out, the interesting bit is the Applix TM1 memory-centric OLAP product. For the vast majority of OLAP users (i.e. the millions of Excel Pivot table jockeys) in-memory OLAP is nothing new, but traditionally big-ticket OLAP was disk-centric, be that ROLAP star-schemas or MOLAP products like Essbase. Until recently, in-memory was a non-runner for large datasets as the cost of memory was high and more importantly 32bit machine programs could only map up to 2G (or with a bit of trickery,3G) of the stuff. But with the appearance of cheap 64-bit servers , and even cheaper RAM, the memory-mapping of large datasets is now possible. While the focus of the big guns in BI turns towards 64-bit servers, the cheapness of RAM makes even a 32-bit machine (i.e. the majority of business PCs) useful for in-memory OLAP especially for SMEs or departmental data-marts – 2 or 3 Gigs of MOLAP data is a lot of data.

This is where tools such as the open source PALO MOLAP server come in to play. Although Palo is both 32 and 64 bit enabled, its status as a free open-source tool, its ability to run client-side, and its close integration with Excel, makes it ideal for adoption in those smaller organisations that previously would not have considered OLAP technologies.

But it’s not just MOLAP that can benefit from cheap client-side RAM, loading data into in-memory relational databases such as SQLite‘s :memory: can also be useful, particularly when that memory is shared with Excel. I’ve already experimented with embedding SQLite into Proto; as Proto’s scripting language is VBA I based it on code I had used in the past for integrating SQLite into Excel. I currently use SQLite/Excel as a micro-ETL tool and I’m looking at the possibility of front-ending a SQLite star-schema with Excel, in effect, a micro-BI environment.

→ 9 Comments

Posted in BI, ETL, Palo, SQLite, VBA, data, excel, olap, xLite

SQLite Star Query Part II

Posted on August 31, 2007 by gobansaor| 1 Comment

In my previous post I looked at simulating a bitmap-join in SQLite using a sub-query and the INTERSECT command. The problem is of course, this is a simulation, SQLite lacks bitmap indices and although the sub-query will read only the fact table’s index B-trees (avoiding accessing the fact table proper) and should be relatively fast, it will not match the efficiency of a bitmap based query. Also, the INTERSECT operation requires the creation of a temporary table which could be very large; admittedly, a bitmap approach would also require temporary storage but the space requirement would be much less and databases such as Oracle10g have highly efficient bitmap set processing algorithms.

Another approach is to apply the method used by Oracle 7/8, i.e. join all the dimensions tables together in a ‘Cartesian product‘ join and then access the fact table using a multi-column index. This can be very efficient when the number of dimensions is low and the dimension tables are small in size. This could easily be implemented using a variation on my VBA based xLITE utilty.

A possible approach would be to load the dimension tables into a :memory: database, either directly from an ATTACHed database e.g.

Select week_no,day,dim_date_id from time_dim where dim_date_id >= '20070801' and < '20070901'

Or, first load the dimensions into excel tables, allow the user to filter the resulting lists and then load these filtered lists into :memory: tables.

Cartesian join the resulting :memory: based tables and then join the resulting dataset to the ATTACHed fact table ensuring to include each dimension’s primary key in the join.

Load the resulting dataset back into Excel and pivot.

The Cartestian join could result in a very large temporary table and although it would contain fewer records than the equivalent ‘bitmap simulation’ table, each record would be much larger ( “primary keys and selected dimensional attribute columns” Vs. “just rowids”).

Both methods would be sub optimal where the query required the reading of a large percentage of the fact table. What that percentage that is varies, but could be as low as 10%.

So, which method to use? Maybe a combination of all three, in effect, writing my own SQLite cost based optimizer! Or, more likely, manually slotting in the required access method depending on the nature of star schema. I’ve got to keep reminding myself that this is micro-BI, the data volumes will be in the megabyte or gigabyte ranges not the terabyte range, with the resulting dataset being in a format suitable for analysing via an Excel pivot table. If a pivot table looks unlikely to be capable of handling the problem (e.g. ragged hierarchies, budgeting read/write/splash requirement or very large datasets) my next port of call would be a Palo cube; in that case the SQLite star schema would simply act as the ETL staging area for subsequent loading into the MOLAP cube.

Part III - http://blog.gobansaor.com/2009/09/29/tag-cubes-sqlite-star-query-part-iii/

→ 1 Comment

Posted in BI, ETL, Palo, SQLite, VBA, excel, olap, xLite

SQLite Star Query

Posted on August 30, 2007 by gobansaor| 7 Comments

While investigating the nature of SQLite‘s query optimisation strategy I came across this 2004 presentation (link dead! try internet archive) by D. Richard Hipp which provides a excellent overview of SQLite’s technical architecture. The reason I’m looking at the optimiser is to figure out if SQLite can handle a star-schema query.

First impressions were not promising as SQLite doesn’t support merge-joins or bitmap-joins, only the classic loop-join is implemented e.g.

Select * from t1,t2;

…is evaluated as;

for each row in t1:
for each row in t2:
output one row of result
end
end

Also, as only one index per table is used in a query this appears to rule out the classic one-index-per-dimension scenario normally applied to fact tables. All is not lost however, by using the INTERSECT command it’s possible to simulate a bit mapped query e.g.

Select sum(sales) from fact_table ft
where ft.rowid in (
select rowid from fact_table ft where ft.customer_dim_id = "35"
INTERSECT
select rowid from fact_table ft where ft.product_dim_id = "56787"
INTERSECT
select rowid from fact_table ft where ft.time_dim_id = "20070823")

Not as slick as an Oracle 10G star transformation but ‘good enough’ for the task at hand. What, I hear you ask, is that task and why would anybody use SQLite to implement a data warehouse star schema? For the last few months I’ve been rabbiting on about micro-ETL, well, I’m about to embark on my micro-BI phase (you’ve been warned) and for a micro-BI environment a micro-datawarehouse is essential. More anon…

Part II - http://blog.gobansaor.com/2007/08/31/sqlite-star-query-part-ii/

Part III - http://blog.gobansaor.com/2009/09/29/tag-cubes-sqlite-star-query-part-iii/

→ 7 Comments

Posted in BI, ETL, SQLite, olap

Palo 2.0 (beta) released.

Posted on August 4, 2007 by gobansaor| 1 Comment

The latest 2.0 version (beta) of Palo has just be released. You can download it here……

The major enhancement over V1.5 is the addition of server-side rules. The rule syntax is similar to that found in TM1 or Essbase e.g. ['Profit%Sales'] = ['Profit']/['Cost of Sales']*100 but Palo rules are restricted to ‘stand alone’ formulae, i.e. each formula must be self contained and not depend on another as the sequence in which a rule fires cannot be determined. We will have to wait until V2.1 for ‘deterministic’ rules to be added to the engine.

There’s also a new demo database showcasing the new rule engine .

→ 1 Comment

Posted in BI, Palo, excel, olap

Talend and Perl

Posted on August 3, 2007 by gobansaor| 2 Comments

I’ve downloaded V2.1 of the open source Talend ETL tool; lots of new connectors added and the Java SQLite connector no longer requires a JNI adapter. I’ve evaluated Talend in the past mainly concentrating on its Java code generating capability, this time I revisited the original Perl generator. Why? Well I know Perl, it has a very active community and I’ve successfully used it in the past as a data hacking language on Unix boxes, but I’ve always found using it on Windows a PITA.

For Perl to be a runner as a micro-ETL tool, Windows deployment had to be as simple (for the end-user) as possible. So I decided to invest some time looking at the options for deployment on Win32 and more particularly the use of the PAR module to generate Windows executables. I eventually managed to get PAR to work and now I have a way of shipping a Talend generated Perl ETL transformation to a Windows box without worrying about installing a Perl environment.

With the wealth of Perl data manipulation modules and examples out there, and the ability to create new Talend components via tFlexPerl, I’m seriously considering using Talend/Perl as my micro-ETL environment. The one thing I’m missing is a Perl wrapper for Palo, but I’ve been looking at using SWIG to generate one, looks easy enough, particularly as I would only be interested in exposing two or three Palo functions (adding data to cubes, adding elements to dimensions and issuing a database save).

I know, I know, I’ve been here,there and everywhere looking for my ideal technology. Where will I land? Which option will I finally go with? Stay tuned, as my business plan for “the country datasmith” becomes more concrete, the technology required to support the business becomes clearer, I call this “the perfect storm”. More anon.

→ 2 Comments

Posted in ETL, Palo, Talend, olap

Tagged