“Talend offers a built-in option to publish an ETL job as a WAR file exposing a SOAP web service”

I have looked at Talend Open Studio but could not find any reference to this. Is this part of their commercial add-on service?

Barak

Also thanks for the tip on scaling. I was talking a long-shot project I might do. I’ll rather build out and keep the basic tool set same. I hope I’ve that problem

Sean

I hope over the next few months do a series of posts, detailing how I would approach this, free time permitting.

As for would it scale? If you happen to work for a major telco or other large utility or bank then I would say no, even though at a departmental level within such organisations, yes. As for millions of rows, if those rows were fact table rows, yes, if they were dimensional rows, no.

With this approach if you did reach a performance bottleneck, you’ve a multitude of escalation options to choose from, some free (replace Groovy with Java, replace SQLite with PostgreSQL), some not (roll back in Oracle or SqlServer, replace Palo with Essbase), But, I think the real test whether an approach like this can scale would lie with whether or not the domain being modelled is amenable to being built out rather than up ,i.e. as you grow you add more servers and instances rather than a more powerful machine with more CPU/RAM plus a more powerful database/MOLAP server.

Tom

Currently, I am using Talend for simpler tasks (read data, transform and output). I have not tried using web services, java, groovy etc. all mashed up together. But this is an excellent tutorial for it. Any chance you can share your sample jobs so I’ll have a starting point?

Also in your estimation, how well will this scale if there are few million rows and complex calculations are involved?

Everyday I work with Informatica which is a great tool. But the ease and flexibility of Talend+JDBC is the best value proposition I’ve seen in a log time. Though in our big company (my day job), this will be unthinkable.

Thanks.