Hi All,

Work stuff.

We are looking into hosting a shedload (I mean an awful lot) of meteorological model data on a server that can then be queried by applications hosted on the same box.

The data basically consists of an X (Latitude), Y (Longitude), Z (Height) coordinates, a date/time coordinate and then the data value itself. I.e. it is four dimensional data. Then throw in the fact that at each location in space and time there can be up to 800 different data values (e.g. temperature, wind speed etc.) but usually all the same type, i.e. double precision number. This makes what we conceptually understand to be a five dimensional "data cube".

We are looking for an Intel based solution and a database to handle it. Oracle 9i etc. doesn't cut it, even when using the Spatial Cartridge etc. simply because it is relational and table based. Well, that is from the guy looking at the problem, and I kind of understand his reasoning. The suggestion has been made that a pure object database is the way to consider/go.

We want to be able get at "sub cubes" of data very quickly. By sub cube I mean imagine a cube in three dimensioanl space (X, Y, Z). We would ask for this cube but for a certain date/time coordinate and for one of the 800 odd data values.

We may do the same but for a range of date/time coordinates.

I.e. any combination of dimensional access to the data values in a rapid and easy to query method is what we are after.

So, I'm looking for advice from anybody in the business. This is potentially a very large project to us.

Obviously if we can find a solution without the usual "Get a consultant in, pay loads, not get the solution we wanted" scenario that would make us happy.

By email please: ian.watkins@metoffice.com

Cheers

Ian

[Edited by IWatkins - 4/19/2003 10:03:47 PM]

 

Ian,

Postgres/PostgreSQL is a free object-relational DB with some geometric-type capabilities - don't know whether it'd scale to your requirements.

As a relative novice in the world of database design, what's so special about spatial data? Surely all you're doing is querying x, y, z and time fields with a minimum and maximum value for each?

 

Ian

This isn't the area of databases I specialise in, but I think you probably need to rearrange your database schema from "relational" to a "star", which will then allow easy querying along the lines you want.

I found this after a 30 second Google search:
http://members.aol.com/nraden/str101_e.htm
...which explains the idea. Go forward and back on the hyperlinks at the bottom for more info.

I'm just a bit concerned about having 800 dimensions, but let's worry about that later!

There are some specialised star schema database products, such as Redbrick, or you can make a star schema with any RDBMS and then use a specialised reporting tool, such as Business Objects or Brio(the latter allegedly better and cheaper), which "understands" star schemas.

You normally have to write some ETL (extract, transform and load) software to:
- take the current data into the star schema
- take the new data from the data feed into the star schena

Yes, of course we have data warehouse consultants that you pay loads for AND GET the solution you want! As you are a serious organisation that we already work for I can pass this thread onto someone on Tuesday that can cast an eye over the problem (FOC!) to see if a) the whole thing is feasible b) I'm on the right lines in terms of solution.

Let me know, via this thread, or via the work email I've sent you. If I don't hear, I'll not do anything....but I think we can help you

Regards
Martin

PS I'm pretty sure that an object database is NOT what you want
PPS edit to remember Brio as a reporting tool

[Edited by MartinM - 4/20/2003 11:26:00 AM]

 

Bashful,
Thanks for the pointers. I'm also a fair newbie when it comes to databases, I just write applications instead.

Martin,
Thanks for the email. I'm preping one for return.

Cheers

Ian

 

Big thanks to Martin here. He has helped me resolve some issues and also had a colleage give me a bell and resolved other issues (cheers Nick).

So big round of applause to Martin

Cheers

Ian

 

Ian

No probs.

Just make sure you pay the invoice quickly please

Martin