Eclipse Community Forums: ATL » Mapping large source model

Help

Home

Home » Modeling » ATL » Mapping large source model(best practice)

Show: Today's Messages :: Show Polls :: Message Navigator

Mapping large source model [message #1839208]

Wed, 17 March 2021 13:31

jo ber

Messages: 92
Registered: August 2018

Member

Hi

I would to map a source model to a target model where the source model contains a lot of instances such that it is not possible to load all of them at once (i.e. out of memory) when starting the model transformation.

What would the best practices to handle this kind of situation, given the fact (if my understanding of ATL is correct) that the rules apply only to the instances known during the run of the transformation.

Hope my question is clear enough.

Thank you and kind regards,

Report message to a moderator

Re: Mapping large source model [message #1839221 is a reply to message #1839208]

Wed, 17 March 2021 17:59

Dennis Wagelaar

Messages: 581
Registered: September 2012
Location: Belgium

Senior Member

If the model is too large to load in memory, you also need to solve this problem at the level of EMF (or you already have?).

I'll assume you're using regular XMI files as input, and so you're loading entire files into memory when transforming. You'll have to split the model over multiple XMI files in order to load only parts of your model into memory.

Then, you can apply your ATL transformation to only the relevant XMI file(s). As long as your ATL transformation is careful to only navigate the object references that are necessary, EMF won't auto-load unnecessary XMI files into memory.

Cheers,
Dennis

Report message to a moderator

Re: Mapping large source model [message #1839248 is a reply to message #1839221]

Thu, 18 March 2021 08:57

jo ber

Messages: 92
Registered: August 2018

Member

thank you for your answer.

Is there a built-in mechanism in ATL/EMF to lookup instances in database instead of the loaded XMI files, in particular when the resolve function is used?

Kind regards,

Report message to a moderator

Re: Mapping large source model [message #1839287 is a reply to message #1839248]

Thu, 18 March 2021 17:56

Dennis Wagelaar

Messages: 581
Registered: September 2012
Location: Belgium

Senior Member

I see you've already got experience applying ATL/EMFTVM to CDO ;-).

I've never used CDO before, so I cannot speak from experience. However, fetching all instances of a certain EClass may be problematic in EMFTVM, because it just iterates an entire EMF Resource and checks each returned element. That would pull in all rows of all tables in a database, even though you already know what table to look in.

I know a plug-in has been developed for the Epsilon project that allows it to integrate properly with CDO. Most notable, it provides a dedicated CDOModel implementation that provides optimised access to CDO model contents. We would need to develop something similar for EMFTVM that extends ModelImpl. AFAIK, this does not yet exist.

Cheers,
Dennis

[Updated on: Thu, 18 March 2021 17:58]

Report message to a moderator

Re: Mapping large source model [message #1839303 is a reply to message #1839287]

Fri, 19 March 2021 07:27

jo ber

Messages: 92
Registered: August 2018

Member

I completely forgot about my old post ;-).

One thing I'm missing in the Epsilon method signature protected Collection<EObject> getAllOfTypeFromModel(String type) is a list of parameters to filter the instances in the sql statement. Or maybe I misunderstood something, and that's an Epsilon question anyway...

In ATL, I don't have the impression that extending ModelImpl to be a too difficult task.

Thank you for the references, there are quite informative.

Report message to a moderator

Re: Mapping large source model [message #1839410 is a reply to message #1839303]

Sat, 20 March 2021 11:37

Dennis Wagelaar

Messages: 581
Registered: September 2012
Location: Belgium

Senior Member

jo ber wrote on Fri, 19 March 2021 08:27

_ One thing I'm missing in the Epsilon method signature protected Collection<EObject> getAllOfTypeFromModel(String type) is a list of parameters to filter the instances in the sql statement. Or maybe I misunderstood something, and that's an Epsilon question anyway...

To me, this is a matter of diminishing returns: it is fairly easy to reimplement the "getAllOfTypeFromModel" operation, and remove the inefficiency of iterating over the whole database instead of a single table. To go any further than that requires the development of an EOL (or OCL) to SQL transpiler, which is much more effort for (possibly) a similar increase in efficiency.

So it should be much easier to start with a CDO-optimised version of "allInstancesOf" for EMFTVM's ModelImpl. The remaining challenge is how to package things such that not all EMFTVM users are forced to install EMF CDO as well.

Cheers,
Dennis

Report message to a moderator

Re: Mapping large source model [message #1839626 is a reply to message #1839410]

Wed, 24 March 2021 18:29

Dennis Wagelaar

Messages: 581
Registered: September 2012
Location: Belgium

Senior Member

https://bugs.eclipse.org/bugs/show_bug.cgi?id=572248

Cheers,
Dennis

Report message to a moderator

Re: Mapping large source model [message #1839694 is a reply to message #1839626]

Thu, 25 March 2021 20:16

Dennis Wagelaar

Messages: 581
Registered: September 2012
Location: Belgium

Senior Member

Can you try the ATL nightly build from https://download.eclipse.org/mmt/atl/updates/nightly/4.4.0/ ? It includes a first version of CDO support for EMFTVM. I'd like to know if performance improves for large CDO models.

Cheers,
Dennis

Report message to a moderator

Re: Mapping large source model [message #1839889 is a reply to message #1839694]

Wed, 31 March 2021 07:34

jo ber

Messages: 92
Registered: August 2018

Member

Hi

Sorry for the late answer. I didn't thought you would add this feature so quickly in EMFTVM and I move forward with Epsilon.
Of course, I'll evaluate the proposition. I'll keep you posted as soon as I can.

Thank you very much and kind regards,

Thank you and

Report message to a moderator

Previous Topic:	Public2Private Example
Next Topic:	Calling EMFTVM programmatically error

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Sat Nov 11 08:17:34 GMT 2023

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter