 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
Natural Science Forum / Physics / Acoustics / December 2005Electronic formats for SLM Data
I have been working in Acoustics for 2 years after having worked in IT for many years. I note that every SLM manufacturer whose equipment I have used has their own propietary PC software. This software will only download data from their own brand instruments and serialise it to the PC in their own propietary format. If one wishes to manipulate the data outside this software (such as in a spreadsheet or database) the propietary software may or may not be of assistance. I have (so far) always found a way of doing it but often only due to my previous experience. Obviously the acoustics world does not need to be like this and manufacturers could agree on common electronic formats for data storage and then implement them, XML would be an obvious choice for creating a format due to its universality and extensibility. I think much noise data would lend itself to XML which likes data to be stored in a structured hierarchical manner. Are there any initiatives within the industry to agree common electronic formats for noise data? > Obviously the acoustics world does not need to be like this and > manufacturers could agree on [quoted text clipped - 3 lines] > itself to XML which likes data to be stored in a structured > hierarchical manner. The problem is that the SLM manufacturers have been doing this for over 20 years. What you see now is what they have all managed to get working in their instruments and into the marketplace likely long before you started in the work force (I think). That you now have a vista that can comprehend their results in retorspect, is not surprising that you can find the commonality and propose a codification. My major SLM was mature before 1990; I still use it almost daily. I suspect that there unfortunately is no funds available to develop to a standard i this field. Rather, what I see happening is that each manufacturer includes software with their hardware that exports the hard data (after being DSP'd by their old tried and true firmware) in Excel form. > Are there any initiatives within the industry to agree common > electronic formats for noise data? The acoustical field is far away from developing any such standard. You are welcome to try, but you will have to beat your oun drum and blaze your own trials in doing so. No one will stop you from doing it, if you can. (There is vastly more enthusiasm for music and entertainment standards; .wav and .mp3 and the like) tha here will ever be for technical acoustics. All above are my opinions... Angelo Campanella > Obviously the acoustics world does not need to be like this and > manufacturers could agree on [quoted text clipped - 3 lines] > itself to XML which likes data to be stored in a structured > hierarchical manner. This is something I tried to do back in 2000 (see <http://www.cross-spectrum.com/audio/articles/acoustic_xml.html> for example) - I was inspired after reading an article in Sound & Vibration magazine about various file formats used in acoustical analyses. I agree that dealing with the various formats is a big pain - especially since most acoustical consultants use a variety of manufacturers' products, and if you do any kind of custom data analysis, you have to deal with all the formats (heck, even if the products come from the same manufacturers, you still have to deal with messy file formats - just look at Larson-Davis products). The biggest problem is that manufacturers just aren't interested - their are some good reasons (memory constraints for example) and bad reasons (lock-in), but everyone I talked with couldn't care less universality or interoperability. Sad but true. > Are there any initiatives within the industry to agree common > electronic formats for noise data? One effort is being spearheaded by IEST, see: <http://www.sandv.com/downloads/0401alle.pdf> I inquired about joining the working group, and the rep that I emailed was receptive, but I got too busy to follow up. There was one other effort I stumbled across a few years ago, but I can't find the info right now - but I do remember that is was very complex, so I didn't pay it much mind. Herb > I agree that dealing with the various formats is a big pain - especially > since most acoustical consultants use a variety of manufacturers' > products, and if you do any kind of custom data analysis, you have to > deal with all the formats (heck, even if the products come from the same > manufacturers, you still have to deal with messy file formats - just > look at Larson-Davis products). Yes, I have; for over 20 years, and I simply live with it. B&K essentially mortgaged the family farm in the 1980's to do this very thing, and they lost that farm. They have not been the same (financially) since. Codifying acoustical data would be like codifying tire sizes for your auto. While I was growing up, both the family auto and the light delivery truck I drove used 6.00-16 tires. Then came the 15" and 14" balloons, then the soft radials, and now we have 16" low profilers; all for the same job; where the rubber meets the road. Codification? Heck, NO. > The biggest problem is that manufacturers just aren't interested - their > are some good reasons (memory constraints for example) and bad reasons > (lock-in), but everyone I talked with couldn't care less universality or > interoperability. Sad but true. I do see a little codification: Most instruments now come with brief software that will Dump ("Export") their instrument's results into an EXCEL spreadsheet. To that extent (Excel) it's codified. But I have to say that even I do not like that result. I prefer to use their Routine (L-D's "TRANS.EXE") that exports a plain ascii file of data in a visually recognizable format that I then condense and massage with an ascii editor like EDIT or Notebook (either works fine) to produce a file that I splice into an Excel spreadsheet for rendering into report pages and graphs. I don't think we can refine (codify) the process any more than that. We professionals are obliged to do some work, and I do not complain. For real time work, tighter connection to the instrument is desirable. I have done that, and LD has cooperated as needed. > One effort is being spearheaded by IEST, see: > <http://www.sandv.com/downloads/0401alle.pdf> > > I inquired about joining the working group, and the rep that I emailed > was receptive, but I got too busy to follow up. I would not waste my time on it. My experience is that I work with what I have; I develop a routine "that gets me there from here", then get on with my life. Angelo Campanella [snip] > Codifying acoustical data would be like codifying tire sizes for your >auto. While I was growing up, both the family auto and the light >delivery truck I drove used 6.00-16 tires. Then came the 15" and 14" >balloons, then the soft radials, and now we have 16" low profilers; all >for the same job; where the rubber meets the road. Codification? Heck, NO. [snip] That's not a good example. Tire sizes are more complex than they used to be, but the nomenclature has kept up. There are uniform codes for aspect ratios, radials, speed rating, etc. I share everyone's pessimism about there ever being a standard format. It's not in the instrument manufacturers interest to do so. Nor in the interest of some consultants. A few years ago, when we were involved with noise in national parks, one of our guys worked out a fairly straightforward device-independent data format. Despite the customer (the Park Service) liking it, other consultants working on the same projects proceded to unilaterally mung the format. Essentailly, a uniform data format did not suit their competitive needs. Incidentally, the suggestion by the OP to base a data format on XML seems to me to be a really bad idea. Ken Plotkin > Incidentally, the suggestion by the OP to base a data format on XML > seems to me to be a really bad idea. Why do you think so? I know that XML has some drawbacks - file size for one (which is why I don't advocate XML for internal device use, but I like it as a target for ASCII translated data). However, one thing that XML has in it's favor (IMHO) is that it's so universally supported nowadays via free/pay toolkits, databases, OS's and now MS Office, it's really easy to integrate XML data into various workflows. Herb > Why do you think so? > > I know that XML has some drawbacks - file size for one (which is why I > don't advocate XML for internal device use, but I like it as a target > for ASCII translated data). I know XML has become the de facto standard for data exchange, but this does not make it the best solution. Lets face it, in recent years, the browser has become the standard user interface client, but it is a massive step backwards in terms of its graphical capability. Browsers still rank way down the list of what is possible from a graphical user interface. > However, one thing that XML has in it's favor (IMHO) is that it's so > universally supported nowadays via free/pay toolkits, databases, OS's > and now MS Office, it's really easy to integrate XML data into various > workflows. Please don't think that just because something is popular and convenient, that it is therefore the best tool for the job. Yes, XML parsers are ten a penny, and yes implementation is very straight forward, and yes they are widely supported; but there are far more efficient ways of communicating a message between two system without having to go through the format/parse cycle every time. XML is horribly verbose; consequently, it has a very poor signal to noise ratio. Non ASCII formats are far more efficient in this respect. Also, because XML is rigidly hierarchical, you have to parse the entire document (irrespective of size) before you know whether the document is required or not. There is no possibility to just "read the header" to find whether or not you should bother reading the rest of the document. That fact means that in a high load situation, XML parsing becomes an inescapable overhead. The parsing process itself is relatively efficient, but the biggest problem comes from the fact (particularly in Java), that there is no effective mechanism for caching parsed objects. Further to this, Java programmers are notoriously bad at considering the memory usage of their programs. This leads to XML parsers being written that work fine for a few large documents, but when they are placed under heavy load, the complex inter-relationship that frequently exists between objects in the parsed tree can lead to the Java garbage collector's runtime rising quadratically with the heap size (Ouch!) I have to use XML frequently in my job, but I must say from experience, that its popularity far exceeds its utility; particularly when a high volume of large documents need to be processed. Chris W  SignatureThe voice of ignorance speaks loud and long, But the words of the wise are quiet and few. --- >I know XML has become the de facto standard for data exchange, but this >does not make it the best solution. Lets face it, in recent years, the [snip] Thanks for the great answer. I was going pretty much on gut feel and experience with various numeric data formats. Generally, the simpler the better. I've had a much better time dealing with unadorned things like TechPlot format, rather than supposed universal formats like NedCDF. Anyone who uses USGS geographic data, and does not want to pay tribute to ESRI, probably has great appreciation for the classic USGS file formats and has absolute hatred for SDTS transfers. Ken Plotkin > Please don't think that just because something is popular and > convenient, that it is therefore the best tool for the job. Whoa there.... as a point of clarification, I never said (and I hope I didn't imply) that XML was the "best" - and furthermore I don't believe the popularity is any indication of quality (let's put it this way: I'm a Mac user ;). [snip, good arguments that XML processing is inefficient] I agree that processing XML can be inefficient, but I see it as a trade-off between efficiency and interoperability. I would argue that most of the arguments you give against XML (particularly the ASCII-based format and the hierarchical structure) are *good* things in that it helps to maintain (and enforce) the structure of the data. I also like ASCII-based file formats because it's much easier (IME) to recover corrupt files. Like I wrote before, the large file size makes ASCII unworkable for local device storage, but it's useful for data interchange. But for me, the issue comes down to being able to work with the data in as many ways as possible on any platform that I chose - interoperability. I've worked with XML data on Windows, Mac, Palm, and a cell phone, and it was easy to do because there are free parsers for each of those platforms (and trust me, I'm no programmer). I'm sure there are more efficient ways to work with data, but can it be done with widely available (and preferably free) toolkits, or do I have to manually work with the bits at each platform? Serious question, if there is something like this out there, I'd like to experiment with it. > I have to use XML frequently in my job, but I must say from experience, > that its popularity far exceeds its utility; particularly when a high > volume of large documents need to be processed. I don't know whether I should envy you or pity you ;) FWIW, I use XML (and Expat) in a couple of custom programs, but with relatively small sets of data. It has it's drawbacks, but again, the wide availability of parsers makes it easy for me to work with the data in different ways on different platforms. That's the part I *really* like. Herb A little OT in that the article isn't specifically about data formats, but the journal Nature just published an editorial about sharing electronic data over the internet: <http://tinyurl.com/ddst2> "A key technological shift that could change this is a move away from centralized databases to what are known as 'web services'. These are published interfaces that serve to simplify access to data and software (for an example of such services in action, see http://www.ebi.ac.uk/xembl/index.html). Until recently the preserve of expert programmers, such interfaces now mean that anyone with even a basic knowledge of programming can automate data processing and analysis." [...] "In biodiversity research, for example, rather than creating centralized monolithic databases, scientists could tap into existing databases wherever the data are held, weaving together all the relevant data on a species, from its taxonomy and genetic sequence to its geographical distribution. Such decentralization also helps to solve the problem that databases are often the fruits of individual or lab research projects that are vulnerable to the vagaries of funding, and to people and labs moving on to pastures new." [...] "Scientists may be justified in retaining privileged access to data that they have invested heavily in collecting, pending publication ‹ but there are also huge amounts of data that do not need to be kept behind walls. And few organizations seem to be aware that by making their data available under a Creative Commons licence (see http://creativecommons.org/license), they can stipulate both rights and credits for the reuse of data, while allowing its uninterrupted access by machines." Herb > "A key technological shift that could change this is a move away from > centralized databases to what are known as 'web services'. These are [quoted text clipped - 4 lines] > basic knowledge of programming can automate data processing and > analysis." Socialism (enforced pooling) didin't work for societies, and it won't work in the long run for data and knowledge. That knowledge that the owner, in his or her wisdom should be shared, should be. That knowledge and data that are either trivial-local, or doubtful, or critical to one cause or another should not be shared, as it causes too much confusion and animosity. Just my opinions. Angelo Campanella |
Reply to this Message |
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2009 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.