Comments
Comment #198 Christian Backe @ 2025-01-20 14:56
## Response to review 2 (Kilian Geiger @ 2024-07-26 19:54)
### Proper requests
- Counter the concern that the presented approach could create additional overhead potentially putting off researchers interested in RDM but unsure where or how to get started. How could approaches like these be used to decrease the entry hurdle into RDM?
Our approach assumes that FAIR data production is necessarily a team effort where someone needs to take on the role of a dedicated FAIR RDM expert. We attempt to contribute to a definition of this role and explain its competing demands. Further, we present tools that intend to facilitate and improve collaborative and iterative RDM. Increased effort is an inherent characteristic of collaboration and interaction; our tools attempt to mitigate and distribute this effort. Lastly, concerning the appeal of FAIR RDM, we argue that data producers are more likely to adopt FAIR practices if they see that their own requirements and challenges are taken into account. Our paper attempts to raise awareness for the producers' perspective in the FAIR data debate.
We have added a paragraph to the conclusion that expresses these points.
- Address other research in the field of RDM and the resulting gap that your paper attempts to close.
We have added a section "Related work" which includes RDM research. The research gap is addressed in the "Motivation" subsection of the revised introduction.
- Increase the resolution of figures 1 and 4 (revised: figures 3 and 7).
Both figures were kindly provided by third parties. Unfortunately, there are no higher resolutions available.
- Increase the font size in most images for easier readability.
For all figures with small text we have increased the font size by 20 percent, either by increasing the font size directly (if we produced the figures ourselves and they had enough internal space) or indirectly by increasing the overall sizes of the figures.
- Truncate lists of references, such as “[11-15]” instead of ”[11] [12] [13] [14] [15]”, to increase readability.
This has been adjusted accordingly.
- Line 17 (revised: 44): Which three parts? It would make sense to name them here.
This line refers to the following three paragraphs. To avoid the confusion, we have broken up and rephrased the entire paragraph, named the sections upfront, and added a matching back reference to each of the subsequent paragraphs.
- Line 22 (revised: 54): Personification of “section”, passive voice would be preferable here. I.e., In this section, base elements […] are developed. Although presented/described would fit better, as the development of the model has already taken place.
We disagree that passive voice would be preferable as it would push the verb to the end of the phrase, thus decreasing readability. Also, it could be argued that the neighboring expressions "Section x discusses/expands/examines ..." are personifications in the same vein and yet have not given cause for objections.
We do agree that our choice of verb was poor: The section *introduces* elements for the modeling of metadata creation. This has been changed in the text.
- Lines 86-87 (revised: 175-176): Where is this mathematical model presented? Do you just yearn for a way to mathematically describe the richness? It sounds like you will present on in Subsection 3.1, which is not the case.
Sorry, these lines appeared by mistake. A mathematical model was layed out in an earlier draft of the paper, but was removed before submission due to space constraints, text flow considerations, and quality issues. The introductory phrase is a relic of this earlier version. We have removed it.
- Lines 103-104 (revised: 95-118 and 56-57): Some more detail on the existing models would help the reader understand why you decided to create your own model. What do they encompass? What distinguishes them?
This remark implies two separate issues which were confounded in our initial draft.
(1) We have moved the discussion of formal knowledge representation in robotics into a dedicated "Related work" section. The breadth and depth of the treatment has been considerably extended. We explain at the end of this section that the usage of formal ontologies/terminologies was out of scope for the present work, but shall be pursued in the future.
(2) Concerning our metadata creation process model, our initial draft did already answer the questions raised by the reviewer, but the answer was buried in a later subsection. Now, we give a brief outlook on the comparison between our model and the M4I ontology in the revised introduction; this also addresses concerns by the other reviewer about our introduction. Further, at the place that inspired the reviewer's remark, we added a reference to the later subsection where the models are compared.
- Line 155 (revised: 251): Why is the input not part of the model?
Input is part of the model. We state in the text that the procedure may receive output of a previous stage as input, and this is depicted in the respective figure. In our view, input and output are just different roles played by the same piece of data.
- Line 158 (revised: 254 and 244-245): How is the passage of time represented in the model? In the image, top to bottom and left to right are both time-dependent.
The vertical and horizontal axes indeed have intended meanings. We failed to make these explicit: The vertical axis shows base data processing, the horizontal axis shows metadata processing. This has been made explicit in the revised text.
In our understanding, the main purpose of a data flow graph is to represent *causal* relationships between the inputs and outputs of processes. As far as there are temporal implications, these were (and are) addressed in the accompanying text ("before ... / after ...").
- Table 8 (revised: Table 6): How can Meta-Metadata, Meta-Meta-Metadata and so on be distinguished? Do they need to be distinguishable? Are they not the same from a model standpoint?
They are all metadata, for sure. Taken to the limit, the same point could even be made for the distinction between data and metadata: If metadata is data, what is the difference?
Though the table shows different metadata manifestations for illustrative purposes, the focus of the entire section (as indicated by its title) is less on the ((meta-)meta-)data itself, but on the process of its creation. Our argument is this: If we take the two assertions from Section 3.1 that metadata is data, and that FAIR data requires metadata, then we get a recursion that could in principle go on indefinitely and therefore needs to be managed. This is further addressed in Section 4.3.
- Lines 180-182 (revised: 276-278): This makes me think that the comparison is not really valid. Both models try to achieve different things. The described model is a meta-model, which could be applied to anything and does not really relate to the specific process class of the M4I model.
Both models collect metadata items around a generic data processing element. Therefore, we think it is fair to compare the two. We agree that both models have different purposes; this is expressed in the noted passage. We also agree that the M4I model is more specific in some sense; this is expressed right below the noted passage ("[The M4I model] specifies a fixed set of attributes, while [our model] is agnostig in this regard."). But we don't think this implies they should not be compared. It just means they are different. Why else compare?
- Table 9 (revised: Table 7): As you mentioned in the text, limited time frames are a very important factor for the data producers.
We state in the text that the table summarizes two particular paragraphs. Time is not mentioned in these paragraphs.
Further, we don't see a general difference between the potential time constraints of data producers and reusers. We do agree that time constraints may have particular consequences for data producers. These consequences are listed in the table.
- Line 249 (revised: 340): Missing comma after “of course”
Has been corrected.
- Lines 249-251 (revised: 340-342): Isn't it possible that the "last" (internal) order always points to an external definition, which is properly defined?
No. In the end, we all rely on our "natural" language understanding, which is conveyed mostly informally and remains in constant flux.
- Lines 253-254 (revised: 344-346): Isn't this exactly what standards and open vocabularies are for?
Yes. This was implied in our original remark. We have made this explicit in the revised text.
- Lines 278-285 (revised: 366-373): Doesn't the circular structure imply this?
We are not sure what "this" refers to, since the paragraph raises four different points: (1) planning is a different kind of activity than production, analysis, storage, access, or reuse. (2) Common data lifecycle models fail to mirror planning with evaluation. (3) (Our) research is iterative. (4) Under iterative conditions, the omission of evaluation from the lifecycle is a missed opportunity to improve the RDM process.
We don't think that any of these points is implied by the circular structure, so it is difficult for us to address the reviewer's concern directly. We have revised the paragraph to improve its overall clarity and hope this also resolves the reviewer's concern.
- Lines 312-315 (revised: 407-412): What are some examples? What steps were taken during the two projects?
As an example, we have added that inconsistency and ambiguity may arise in interdisciplinary settings. Due to overall space constraints, and since this particular section is already quite long and has a condensed style anyway, we trust the reader to remember that DeeperSense and RoBivaL both are in fact interdisciplinary projects, and to conclude for themselves that inconsistencies and ambiguities should be actively identified, addressed, and resolved.
- Line 318 (revised: 421): “, e.g.,”
Has been corrected.
- Line 319 (revised: 422): “units,”
Has been rephrased, so the comma is no longer necessary.
- Line 320 (revised: 423): Who is responsible for this?
Project leadership. We think this is implied. No change.
- Line 324 (revised: 427): “so it is easy to miss important information, e.g., critical failure”
Has been corrected.
- Lines 353-354 (revised: 462-464): Wouldn't this be part of the Evaluation phase?
No. If the adjustment is done during the same cycle it is still part of the execution. But the reviewer is right to point out that the adjustment should be addressed during evaluation.
We have added a corresponding remark to this segment.
- Lines 377-379 (revised: 496-500): How is this different to the planning phase of creation and provision?
We have added a remark to this segment that acknowledges the overlap and explains why this point is highlighted here.
- Line 380 (revised: 500-501): Quality assurance of what? What does this entail?
We have clarified in the text that this is about the quality of (meta)data. Specific examples were already given in the following phrase ("This involves ...") which was not changed.
- Line 427 (revised: 556): “itself”
Has been corrected.
### General feedback and thought-provoking questions that do not necessarily need to be addressed as part of this publication
- Lines 3-6 (revised: 4-8): The general FAIR principles should be adhered to either way. Some broken down requirements for practical implementation could be beneficial though.
Agreed. The revised introduction opens with a reference to a seminal paper which expresses just this view.
- Lines 120-121 (revised: 208-212): Is this “routinely” the case? I feel like this is an area that is still lacking in many cases and tools that facilitate proper RDM would be of great benefit here.
"Routinely" shall indicate that some forms of metadata creation and management are just an innate part of being an effective researcher. In other words, we think that data producers necessarily carry out some form of metadata creation and management *in practice*. This does not imply that they would describe their practice themselves in these terms, or that their practice involves specific tools or methods.
We have expanded this passage to clarify the point.
- Lines 124-125 (revised: 214-216): Shouldn't these be the same? Why not have one set of metadata that satisfies both groups? Otherwise, interoperability would not be satisfied.
This passage is about different demands. These cannot be dictated, but are made by each stakeholder independently. How the demands should be satisfied is a different question. We think that raising the issue of interoperability at this point would detract from the core discussion. Therefore, we haven't changed this passage.
- Lines 135-138 (revised: 230-233): Usually, data producers are data users as well and should have similar requirements towards the reusability of data. I understand that it is often the case, as you stated, that data producers take some implicit knowledge for granted and try to keep the additional effort and overhead to a minimum. From personal experience, I can tell, that this approach often leads to more headaches, including unnecessary iterations and repeated experiments, than one would like to admit.
Our focus here was on describing the actual current practice, rather than suggesting a best practice. It seems we agree on the actual practice. The purpose of this paragraph and the section as a whole is to contrast between the different perspectives of producers and reusers. We feel that the issue of best practices would be a distraction at this point. Therefore, we haven't changed this paragraph.
- Lines 212-215 (revised: 304-307): How can this be adressed?
That depends on the specifics of the discipline, the project, the field, the team, etc.
- Lines 218-220 (revised: 310-312): Why do they need to be in conflict?
We don't say they need to be in conflict. What we mean is: For those requirements that are conflicting it is the data manager's job to mediate. Examples for conflicting requirements are given in Section 3.2 (revised: 4.2) and in the further course of Section 4.2 (revised: 5.2). Therefore, we don't see a reason to change this passage.
- Lines 224-227 (revised: 316-318): Isn't this the problem that should be addressed and that was mentioned in the previous subsection?
Yes, this problem should be addressed. We are not claiming otherwise.
No, this problem was not mentioned in the previous subsection. The problem here is a conflict between producers and reusers. In the previous subsection there were no reusers, therefore no conflict with them. While the conflict depends on points made in the previous subsection (which is stated explicitly at the top of the paragraph), focus and scope are different here.
- Lines 272-277 (revised: 361-365): Are they really that different? Wouldn't it be more beneficial if they were identical?
There are multiple reasons why they aren't and can't be identical: Provision must happen before processing, publication must happen after. Provision applies to all data (including: pretest/proof-of-concept, confidential, erroneous in hindsight, poorly formatted and documented, ...), publication applies to selected and clean data. Provision can be more lax about (or entirely omit) attribution, persistent IDs, explicit references to ontologies, ...
- Lines 290-293 (revised: 378-381): Is this still part of the data management?
We are explicitly listing different data *processing* activities. We are not claiming that these are all data *management* activities. The passage makes a case for renaming a phase of a common data lifecycle model. The existence of this phase is well established. Therefore, we do not see a reason to change this passage.
- Lines 293-295 (revised: 381-383): Is naming really that important that it warrants another standard/model? I understand that your model still excides the goals and border of the NFDI model, but I feel like different naming conventions just increases potential confusion and hesitation people have towards RDM.
In lines 293-295 (revised: 381-383), we explain a term we use. This is standard scientific practice and should hardly be controversial. However, since the reviewer criticizes our attempt to revise naming conventions, we believe their remark goes beyond this limited passage and refers to the entire paragraph 289-295 (revised: 377-383).
If we are to take RDM seriously as a scientific activity then yes: We believe names matter. Precision matters. We are confident that scientists are able to cope with innovation and are willing to entertain suggestions for improvement.
Further, one of the main themes of our paper is to raise awareness for the fact that different groups with a stake in RDM may have very different perspectives. As mentioned in the paragraph: Between the data provision and publication phases, research teams at our institutions perform a broad range of data-related activities which are not data analysis. Therefore, a data lifecycle model that fails to reflect this perspective risks arousing confusion and hesitation just as well -- only in different groups of people than the reviewer might have in mind.
We do agree that too much volatility may stifle the adoption of RDM practices. At the same time, we think the things that get adoped should be the right things.
- Line 310-311 (revised: 404-405): This also needs to be known and properly understood, which can take a lot of effort.
Absolutely.
- Line 336 (revised: 443): Shouldn't all of this happen before creating the data?
Not necessarily. In our case studies, the internal data repository that is used for internal provision is a different resource than the data creation apparatus which has its own preliminary data store.
Further, we don't think that the activities we discuss will or should always happen in the exact order in which they are layed out here. Another way to paint our data lifecycle model would be on two dimensions with (planning, execution, evaluation) on one axis and (creation, ..., reuse) on the other. We opted against this "planar" representation to stay closer to the well-known cycle image and not introduce another factor of contention. We think that each research team has multiple actors who may move simultaneously through different sectors of the 2D-plane. Due to causal dependencies between different intermediate work results, there are restrictions for movements through the plane. Different projects may have different restrictions. It is impossible to map all this complexity onto a single consecutive narration.
We would have liked to integrate these aspects into the text, but did not due to space constraints.
- Lines 360-363 (revised: 471-474): Wouldn't this be a sign of insufficient planning?
This is precisely our point in general: To acknowledge that planning may be insufficient, which is why we suggest to introduce an evaluation phase in order to improve the planning in iterative research.
- Lines 395-398 (revised: 516-519): How is this decided and defined? Could this be declared in the (meta‑)metadata?
We were thinking about the topics discussed in Section 3.2 (revised: 4.2).
Yes, this might be implemented by adding flags to each metadata item indicating its different purposes. The publication could be prepared by filtering metadata items that have a "publication" tag.
- Lines 439-441 (revised: 568-570): Is the evaluation useful before the other "outer" steps have taken place?
We would differentiate between the conceptual model which is sketched here in broad strokes and its detailed implementation within a specific use case.
In practice, at least part of the evaluation will necessarily happen "on the fly" during execution, just by recognizing any mismatch between expectation and reality. This should be documented soon while memory is still fresh. At the end, such raw material can be worked into a summary evaluation. Here, to get a comprehensive picture, it is probably helpful if one can look back on the full context of all project phases.
Comment #197 Christian Backe @ 2025-01-20 14:55
## Response to review 1 (Anonymous @ 2024-07-23 18:45)
### Major issues
- In the introduction, motivate the need for openly available robotics datasets, considering the availability of robotics data is currently a hot topic in AI research with the advent of Robotics Foundation models.
Our revised introduction now motivates the need for open robotics datasets in general. Robotics Foundation Models and other FAIR or open data developments in robotics are addressed in a new section "Related work".
- In the introduction, explain whether and to what extent the presented results go beyond the current state of the art, and refer to similar or related work presenting real-world examples.
The revised introduction (subsection "Outline") addresses the main contributions of each of the three core sections of the paper. Similar or related work is presented in a new section "Related work". This includes work about real-world examples.
- Clearly articulate why the robotics examples were used and how the robotics community benefits from the results presented in this paper. The findings and the way they are presented seem to be applicable to any project that generates research data in multiple steps and iterations.
Our revised introduction starts by declaring the need for and the benefit of domain-specific case studies of FAIR RDM in any domain. We explain why we chose to present RoBivaL and DeeperSense. We clarify that our findings are not supposed to only benefit the robotics community.
- Clarify whether and how the models and concepts presented in Sections 3 and 4 were applied already during the two projects or whether the models and findings were extracted from the experience in the two projects.
We have added the following phrase to the introduction: "The models and concepts presented in these parts were extracted from the experience in RoBivaL and DeeperSense and shall be applied in future projects."
- Major language revision due to extensive use of inserted subordinate clauses and exaggerated use of commas.
Concerning commas, we have reviewed our use and reduced where appropriate.
Concerning inserted subordinate clauses, we do not share the reviewer's point of view. Our use of subordinate clauses might not be to everyone's taste, but it certainly does not warrant a *major* language revision. This issue has never been raised by anyone else, neither by our internal reviewers nor by the other invited reviewer, who finds the text "well written and easy to follow".
Further, we think it is impossible to comply with such a blanket request without specific examples. We assume the reviewer's concern is likely to be tempered by our treatment of some concrete issues that are discussed below.
- Explain the objective of the RoBivaL project in Section 2.1 (revised: Section 3.1) and Table 5 (revised: Table 2). What was the purpose of the evaluation and how was the performance of the different robot types evaluated?
In the revised section , we further explain the objective of the RoBivaL project, state the focus of evaluation, and name two ISO standards that were inspiration for the experiment designs. Given the space constraint and the focus of the paper we don't see an opportunity to elaborate further, and we trust that intested readers will follow the references.
The table is left unchanged, because its purpose is a high-level comparison between the two projects from an RDM perspective. The fact that RoBivaL was considering the concept "robot performance" at all is in itself a major difference to DeeperSense. Going into details about the performance evaluation is neither relevant nor expedient here, because it would detract from the focus of the table. Since these details have been amended to the above section, per the reviewer's request, interested readers have an opportunity to look them up if necessary.
- Complete the sentence on lines 103-104 (revised: 95-113).
The passage has been massively expanded and moved to the new dedicated "Related work" section.
- Support paragraphs ll. 109-118 (revised: 197-206) by a figure or table summarizing the FAIR principles with their attributes, as a service to the reader.
Such a table was added.
- Ll. 131-141 (revised: 221-236): Clarify how these motives were derived. What methods were used to arrive at this conclusion?
We have added clarifying remarks to the revised version.
- Ll. 207-211 (revised: caption of Figure 8): Omit the paragraph, because it is too narrative and exaggerated for the usual neutral tone of scientific writing, and the message is already conveyed by the previous paragraph.
The purpose of the paragraph is to explain why the image below is illustrative of the concepts mentioned in the previous paragraph. The image and its description serve an indispensable role, because they tie these concepts to one of the case studies. Therefore, we think the text cannot be entirely removed. To better reflect its purpose we have transformed it into a caption of the image.
What the reviewer perceives as negative properties of the image description are necessary features: The description appears narrative, because the image shows an unusual situation which requires some explanation, and crucial aspects are either outside of the frame (underwater) or in the minds of the depicted people. It appears dramatic, because the whole point of this passage is to illustrate the stress of the team under field conditions. We do realize, however, that the style of the passage may seem strange in a scientific text. Therefore, during the transformation into a caption the text has been abbreviated and its style attenuated.
- Revise Section 5.2 (revised: Section 6.2): Clarify the connection to Sections 3 and 4 (revised: Sections 4 and 5) and outline where the followed RDM strategies were successful and adequate and in which phases and situations they failed.
Multiple connections to the previous sections have been added or made explicit during revision. These connections have been announced in the revised introduction to this section.
Since this section also serves to illustrate our data lifecycle model more generally, we don't think that every point has to be tied back to the previous sections. This has been clarified in the revised introduction of this section.
Since the section is titled "Lessons learned ...", we believed it was implied that most points were derived from some failure (to execute a task or to anticipate a challenge). We have made this explicit in the revised introduction to this section.
### Minor issues
- Reduce emphasis using italics to a necessary minimum (e.g. l.20 (revised: line 49) "reusable" or l.29 (revised: line 64) "and iterative").
We have removed most italics except for very few select cases.
- L.27 and L.191 (revised: lines 61 and 287): "where" refers to "a FAIR manager", so the singular form should be continued in the sub-clause.
Our usage of "they" is in line with the norm as expressed by, e.g., the Oxford English Dictionary or Merriam-Webster:
"Singular they has become the pronoun of choice to replace he and she in cases where the gender of the antecedent – the word the pronoun refers to – is unknown, irrelevant [...]" (https://www.oed.com/discover/a-brief-history-of-singular-they)
"[...] they has been in consistent use as a singular pronoun since the late 1300s [...]. [...] the traditional singular they [...] is used to refer to a person whose gender isn’t known or isn’t important in the context [...]." (https://www.merriam-webster.com/wordplay/singular-nonbinary-they)
Therefore, the passages were left unchanged.
- Table 1 and 3 (revised: Figures 1 and 2) are actually figures.
Has been corrected.
- Ll. 70-73 (revised: 159-162): The sentence has a very complicated structure and should be split into two parts.
We have simplified the sentence overall and split it into two parts.
- In the case of direct quotations (e.g. ll. 100 – 102 (revised: 187-190)), the references should be given immediately after the quotation.
This has been corrected.
- L. 98 (revised: 186): distinguishes (n missing)
Has been corrected.
- L. 186 (revised: 282): multiple (l missing)
Has been corrected.
- L. 209 (revised: caption of Figure 8): demonstration (n missing)
Has been corrected.
- L. 247 (revised: 338): Why is there a colon before the new paragraph?
This was a typo. The colon has been removed.
Comment #196 Christian Backe @ 2025-01-20 14:53
# Response to reviewers: General remarks
We thank both reviewers for their comprehensive and valuable feedback. Their requests and remarks have greatly helped us make substantial improvements to the paper and have deepened our own understanding of the matter.
Below are our detailed reponses to each request or remark. Some points were isolated from larger paragraphs to facilitate the attribution of our responses. In case of references to specific numbers (of lines, sections, figures, or tables), we have added the respective numbers for the revised draft to facilitate comparison.
Comment #126 Robert H. Schmitt @ 2024-07-30 16:37
I thank the reviewers for their conscientious and detailed reviews.
In accordance with their recommendations, I advise the authors to revise the manuscript incorporating the changes suggested by the reviewers.
The authors are expected to submit a detailed response to the reviewers, in which they make clear how they considered the reviewers' comments.
Invited Review Comment #125 Kilian Geiger @ 2024-07-26 19:54
In the presented work, the authors discuss challenges they faced when trying to implement modern research data management (RDM), based on their experience in two research projects with large focus on in-field experimental campaigns. After a general introduction to the mentioned research projects, they split their discussion into the three dimensions “content”, “social” and “time”, which are in return each separated into multiple subsections. For each dimension, they state the challenges they faced within the practical application of RDM and what models and approaches they developed to address these.
The presented work gives a very interesting insight into the application of RDM in practical applications, which is often lacking in literature. As stated by the authors, many approaches and models exist in theory, but without any practical validation those are of low value to the community. By now, consensus exists in the research community that RDM is important and will be even more so in the future. Most people don’t have issues with the concept itself but lack the necessary knowledge or tools to properly adhere to requirements as they are defined by the FAIR Guiding Principles. The authors are able to show all the downfalls and challenges that arise, when applying these ideas in practice. While doing so, they criticize that some of the existing models might be too general, which might be true to some degree. They also claim that some of the models can be too restrictive, for example in their wording, but by creating meta-models around these generalized models, they create even more restrictions and rules. For the most part, the presented models and approaches seems to be beneficial for the authors and allow them to do some impressive work in the field of RDM. At the same time, I fear that some of the presented could create additional overhead which can potentially put off researchers that are interested in RDM but unsure where or how to get started. By this, I don’t want to convey that the presented models are not valid, on the contrary, they are very interesting and a great contribution to the field. I am just wondering how approaches like these could be used to decrease the entry hurdle into RDM. It sounds like the authors have developed some tools that assist them in applying their theory to the practical field and thereby present RDM as beneficial instead of another annoyance for the researchers that produce the data. It is unfortunate that those tools are neither presented nor shared by the authors. I hope that future publications are planned to do so and am looking forward to those. Additionally, I find the presentation of other research in the field of RDM a bit lacking. While the authors reference plenty of other work, their contents and the resulting gap, which is trying to be addressed within this publication, does not become clear.
The overall presentation is adequate, however, figures 1 and 4 are of low resolution and the font size in most images could be a bit larger for easier readability. Readability could also be increased by truncating lists of references, such as “[11-15]” instead of ”[11] [12] [13] [14] [15]”. The text is, in general, well written and easy to follow while not relying on colloquialisms too often. Minor issues are addressed in the following list of detailed feedback. Mentioned research data is in parts shared via the publicly available zenodo repository. From what I can tell, the provided data is well documented and adheres to the FAIR Guiding Principles. Authors, Affiliations, Contributions and References are provided in a proper manner. No software is provided, which is understandable, as it is not the focus of the presented work.
Line-by-line feedback:
Please note that some of these are more general feedback and thought-provoking questions that do not necessarily need to be addressed as part of this publication. I put a ** before each for easier revision of this publication.
** Lines 3-6: The general FAIR principles should be adhered to either way. Some broken down requirements for practical implementation could be beneficial though.
Line 17: Which three parts? It would make sense to name them here.
Line 22: Personification of “section”, passive voice would be preferable here. I.e., In this section, base elements […] are developed. Although presented/described would fit better, as the development of the model has already taken place.
Lines 86-87: Where is this mathematical model presented? Do you just yearn for a way to mathematically describe the richness? It sounds like you will present on in Subsection 3.1, which is not the case.
Lines 103-104: Some more detail on the existing models would help the reader understand why you decided to create your own model. What do they encompass? What distinguishes them?
** Lines 120-121: Is this “routinely” the case? I feel like this is an area that is still lacking in many cases and tools that facilitate proper RDM would be of great benefit here.
** Lines 124-125: Shouldn't these be the same? Why not have one set of metadata that satisfies both groups? Otherwise, interoperability would not be satisfied.
** Lines 135-138: Usually, data producers are data users as well and should have similar requirements towards the reusability of data. I understand that it is often the case, as you stated, that data producers take some implicit knowledge for granted and try to keep the additional effort and overhead to a minimum. From personal experience, I can tell, that this approach often leads to more headaches, including unnecessary iterations and repeated experiments, than one would like to admit.
Line 155: Why is the input not part of the model?
Line 158: How is the passage of time represented in the model? In the image, top to bottom and left to right are both time-dependent.
Table 8: How can Meta-Metadata, Meta-Meta-Metadata and so on be distinguished? Do they need to be distinguishable? Are they not the same from a model standpoint?
Lines 180-182: This makes me think that the comparison is not really valid. Both models try to achieve different things. The described model is a meta-model, which could be applied to anything and does not really relate to the specific process class of the M4I model.
** Lines 212-215: How can this be adressed?
** Lines 218-220: Why do they need to be in conflict?
** Lines 224-227: Isn't this the problem that should be addressed and that was mentioned in the previous subsection?
Table 9: As you mentioned in the text, limited time frames are a very important factor for the data producers.
Line 259: Missing comma after “of course”
Lines 249-251: Isn't it possible that the "last" (internal) order always points to an external definition, which is properly defined?
Lines 253-254: Isn't this exactly what standards and open vocabularies are for?
** Lines 272-277: Are they really that different? Wouldn't it be more beneficial if they were identical?
Lines 278-285: Doesn't the circular structure imply this?
** Lines 290-293: Is this still part of the data management?
** Lines 293-295: Is naming really that important that is warrant another standard/model? I understand that your model still excides the goals and border of the NFDI model, but I feel like different naming conventions just increases potential confusion and hesitation people have towards RDM.
** Line 310-311: This also needs to be known and properly understood, which can take a lot of effort.
Lines 312-315: What are some examples? What steps were taken during the two projects?
Line 318: “, e.g.,”
Line 319: “units,”
Line 320: Who is responsible for this?
Line 324: “so it is easy to miss important information, e.g., critical failure”
** Line 336: Shouldn't all of this happen before creating the data?
Lines 353-354: Wouldn't this be part of the Evaluation phase?
** Lines 360-368: Wouldn't this be a sign of insufficient planning?
Lines 377-379: How is this different to the planning phase of creation and provision?
Line 380: Quality assurance of what? What does this entrail?
** Lines 395-398: How is this decided and defined? Could this be declared in the (meta‑)metadata?
Line 427: “itself”
** Lines 439-441: Is the evaluation useful before the other "outer" steps have taken place?
Overall, I would recommend this work for publication after minor revision.
Invited Review Comment #124 Anonymous @ 2024-07-23 18:45
Collaborative creation and management of rich FAIR metadata: Two case studies from robotics field research
Summary:
The article „Collaborative creation and management of rich FAIR metadata: Two case studies from robotics field research“ presents lessons learned from implementing research data management in collaborative research projects in robotics field research. It outlines the metadata creation process and the requirements of producers and re-users of research data, and presents a metadata creation process based on the Metadata4ing ontology. The responsibilities of a research data manager towoards the different stakeholders (data production team, re-users, and RDM community) are discussed in this context. Finally, a self-improving data lifecycle workflow is derived from the fact that data production is usually carried out in several cycles, providing an opportunity to improve RDM within a project. Using this data lifecycle, lessons learned from two projects are presented.
Novelty and Originality
The novelty and originality of the paper is rated as moderate or low. The introduction is kept rather short and generic, outlining the general benefits of real-world examples of RDM for practitioners. The authors do not motivate the need for openly available robotics datasets, although the availability of robotics data is currently a hot topic in AI research with the advent of Robotics Foundation models. It is difficult for the reviewer to assess novelty, as no statement is made as to whether and to what extent the presented resutlts go beyond the current state of the art, and no similar or related work presenting real-world examples is mentioned in the text.
The reviewer suggests revising the introduction and concentrating on outlining the novelty of the work.
Major comments:
· It is not clearly articulated why the robotics examples were used and how the robotics community benefits from the results presented in this paper. The findings and the way they are presented seem to be applicable to any project that generates research data in multiple steps and iterations. The reviewer suggest to clarify whether the work should mainly
· From the way the findings are presented, it is not clear to the reviewer whether and how the models and concepts presented in Sections 3 and 4 were applied already during the two projects or whether the models and findings were extracted from the experience in the two projects.
· A major language revision should be undertaken. There is extensive use of inserted subordinate clauses, which is unusual for the English language. In particular, there is an exaggerated use of commas, mostly in places where no comma is required in English.
· The objective of the RoBivaL project is missing in both Section 2.1 and Table 5 (what was the purpose of the evaluation and how was the performance of the different robot types evaluated?) The objective of the RoBivaL project is missing in both Section 2.1 and Table 5 (for which purpose and how was the performance of the different robot types evaluated?).
· L. 103-104: The sentence is incomplete.
· It would be a service to the reader if paragraphs ll. 109-118 were supported by a figure or table summarizing the FAIR principles with their attributes.
· Ll. 131-141: It is unclear to the reviewer how these motives were derived. What methods were used to arrive at this conclusion?
· Ll. 207-211 is too narrative and exaggerated for the usual neutral tone of scientific writing - The message of this paragraph is already conveyed by the previous paragraph. Therefore, the reviewer suggests taht this paragrpah should be omitted.
· Section 5.2 is narrative and the reader does not learn much about the effectiveness and succes of the RDM implemented in the two projects. The reviewer suggests to revise Section 5 making clear the connection to Sections 3 and 4 and outlining where the followed RDM strategies were succesful and adequate and in which phases and situations they failed.
Minor Comments:
· There is quite extensive use of italics (e.g. l.20 "reusable" or l.29 "and iterative"), but it is not obvious to the reader why certain words are emphasised in this way.In addition.The reviewer suggests that this form of emphasis be kept to a necessary minimum.
· L.27 and L.191: "where" refers to "a FAIR manager", so the singular form should be continued in the sub-clause.
· Table 1 and 3 are actually figures.
· Ll. 70-73: The sentence has a very complicated structure and should be split into two parts.
· In the case of direct quotations (e.g. ll. 100 – 102), the references should be given immediately after the quotation.
· L. 98: distinguishes (n missing)
· L. 186: multiple (l missing)
· L. 209: demonstration (n missing)
· L. 247: Why is there a colon before the new paragraph?