[MonetDB-users] underlying storage

newer
[MonetDB-users] problem in extract...

Mag Gam

1 Aug 2009 1 Aug '09

12:47 a.m.

What is the recommended storage for MonetDB? We are planning to buy a Sun storage appliance and were wondering if it would work with MonetDB

Show replies by date

Stefan de Konink

1 Aug 1 Aug

2:03 a.m.

On Fri, 31 Jul 2009, Mag Gam wrote:

...

What is the recommended storage for MonetDB? We are planning to buy a Sun storage appliance and were wondering if it would work with MonetDB

Yup done that :) If you go for the iSCSI based storage think very careful about the choose filesystem. It really matters. Since your network cable mostlikely will be your bottleneck and if more clients are an issue. Think about some efficient way to get the appliance to the switch if you go beyond the 'standard' 4xGe Stefan

Mag Gam

2:59 a.m.

Interesting, what filesystem have you used? We are planning to use a simple NFS filesystem. Would that have any problems? On Fri, Jul 31, 2009 at 8:03 PM, Stefan de Konink wrote:

...

On Fri, 31 Jul 2009, Mag Gam wrote:

...
What is the recommended storage for MonetDB? We are planning to buy a Sun storage appliance and were wondering if it would work with MonetDB

Yup done that :) If you go for the iSCSI based storage think very careful about the choose filesystem. It really matters.

Since your network cable mostlikely will be your bottleneck and if more clients are an issue. Think about some efficient way to get the appliance to the switch if you go beyond the 'standard' 4xGe

Stefan

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Stefan de Konink

10:24 a.m.

On Fri, 31 Jul 2009, Mag Gam wrote:

...

Interesting, what filesystem have you used?

For benchmarking I know that EXT3 is not an option; lets say the OpenStorage sequential output hit so badly that it was 10 times as slow as the input. With 128GB of RAM that is 'kinda' odd.

...

We are planning to use a simple NFS filesystem. Would that have any problems?

I would only use NFS in a shared storage, envirionment not to support databases, serious locking issues and mmaping (like MonetDB does) from NFS is asking for out of memory conditions on Linux (Xen does something in the same way for images, this is not what you want). If you are just going to support your database from storage and you do not share it. I would just create a volume export that over iSCSI and go XFS over it on Linux. That would also allow snapshotting in an easier way than NFS would do. Stefan

Mag Gam

12:51 p.m.

Can you explain to me more about this mmaping? I was really going for the NFS solution. On Sat, Aug 1, 2009 at 1:24 AM, Stefan de Konink wrote:

...

On Fri, 31 Jul 2009, Mag Gam wrote:

...
Interesting, what filesystem have you used?

For benchmarking I know that EXT3 is not an option; lets say the OpenStorage sequential output hit so badly that it was 10 times as slow as the input. With 128GB of RAM that is 'kinda' odd.

...
We are planning to use a simple NFS filesystem. Would that have any problems?

I would only use NFS in a shared storage, envirionment not to support databases, serious locking issues and mmaping (like MonetDB does) from NFS is asking for out of memory conditions on Linux (Xen does something in the same way for images, this is not what you want).

If you are just going to support your database from storage and you do not share it. I would just create a volume export that over iSCSI and go XFS over it on Linux. That would also allow snapshotting in an easier way than NFS would do.

Stefan

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Martin Kersten

1:12 p.m.

Mag Gam wrote:

...

Can you explain to me more about this mmaping? I was really going for the NFS solution. http://en.wikipedia.org/wiki/Memory_mapped_files

On Sat, Aug 1, 2009 at 1:24 AM, Stefan de Konink wrote:

...
On Fri, 31 Jul 2009, Mag Gam wrote:

...
Interesting, what filesystem have you used? For benchmarking I know that EXT3 is not an option; lets say the OpenStorage sequential output hit so badly that it was 10 times as slow as the input. With 128GB of RAM that is 'kinda' odd.

...
We are planning to use a simple NFS filesystem. Would that have any problems? I would only use NFS in a shared storage, envirionment not to support databases, serious locking issues and mmaping (like MonetDB does) from NFS is asking for out of memory conditions on Linux (Xen does something in the same way for images, this is not what you want).

If you are just going to support your database from storage and you do not share it. I would just create a volume export that over iSCSI and go XFS over it on Linux. That would also allow snapshotting in an easier way than NFS would do.

Stefan

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Fabian Groffen

2:10 p.m.

On 01-08-2009 13:12:51 +0200, Martin Kersten wrote:

...

Mag Gam wrote:

...
Can you explain to me more about this mmaping? I was really going for the NFS solution. http://en.wikipedia.org/wiki/Memory_mapped_files

also, google results like these may be interesting for you: http://osdir.com/ml/linux.nfs/2002-12/msg00062.html

Stefan de Konink

2:10 p.m.

On Sat, 1 Aug 2009, Martin Kersten wrote:

...

Mag Gam wrote:

...
Can you explain to me more about this mmaping? I was really going for the NFS solution. http://en.wikipedia.org/wiki/Memory_mapped_files

Linux overcommits memory in a way that you can get an out of memory situation while *caching* big files from NFS. Memory mapping would talk to this cache aswell. If for what reason what so ever you get bigger files than your total amount of memory in your client system, this goes boom. So please test it for yourself :) this was my experience with Xen over NFS with direct mmaping :) Stefan

5586

Age (days ago)

5587

Last active (days ago)

List overview

Download

7 comments

4 participants

participants (4)

Fabian Groffen
Mag Gam
Martin Kersten
Stefan de Konink