Communications of the ACM
Evaluating Research Results for Practical Applications
In this blog post I would like to analyze to what extent the possibility of creating new technologies should be taken into account when carrying out evaluations of research results. This is a problem which is especially acute in the field of computer science. Recently, more and more articles not only do not show the possible applications of the results, but they also lack evidence of any real experiments having been carried out. This situation had led to the fact that only a small part of the research has from the beginning been carried out in order to obtain results intended for advancing the development of technologies.
Currently the principle of citation, which is the basis of scientometrics, is used to assess the work of researchers. In other words, all reference "citations" appearing in a list of published scientific literature is analyzed. Depending on the scope of the number of publication summaries, several different citation systems are then applied. The Google Scholar system analyzes most publications available on the Internet. Scopus is limited to a collection of journals and conferences that focus on technological development. Web of Science is the most conservative index with a focus on traditional scientific fields.
The question thus immediately arises regarding the possibility of using a grading system based on an assessment of citations covering the successful implementation of goals and objectives that are put before scientific research in the modern world. For example, before the Russian scientific community, there is the task of creating cutting-edge products based on the results of research. Even a simple attempt to answer the above question shows that the current citation system is outdated and should be modernized. The only question is in what direction should we proceed.
If we talk about the practical value of the results obtained, then it would be rational to implement the following classification system of achievements in the field of computer science. The highest achievement for researchers is the development of a standard applied as a reference for a whole class of scientific and technical products. The next step down would be the implementation of a prototype of a new device or service. The obtaining of a patent is even one step lower, since patent protection requires additional time and financial costs, and selling a license is quite complicated. The same Elon Musk prefers finished products based on original technologies to patents. The main aspect to consider when evaluating a patent is the degree of readiness of the device or process. The last place on a practice-oriented scale should be held by new algorithms or methods that improve the performance of existing products as a whole or with regard to its component parts.
A new system for evaluating the results of scientific research should take into account practical value, as well as follow tradition and not suddenly break the existing system. Scientific publications should continue to be at the foundation of any system of evaluation of the work of researchers. However, the structure of scientific articles should be revised. Currently the structure of articles must follow the IMRAD format (American National Standards Institute (ANSI) Z39.16-1972). This standard is in need of supplementing with required descriptions of the possibilities for the practical application of research results, in the section where the problem statement is described. Here are possible options:
- which technologies or products can the results of the work be applied to;
- what specific new task will it perform and how will it help to improve or create new technology.
It is important to note that a sign that sets apart fundamental research is the acquisition of new knowledge. The best innovative technologies are based precisely on new knowledge, and not only on the results of applied research. For this class of research, a new definition has been introduced in Russia, and it is now called exploratory studies. It is exactly these kinds of studies that must be supported, first and foremost, in computer science.
Currently there has been a fantastic shift with regard to the financing of pure fundamental research in Russia. The results of the absolute majority of such research will never see applied usage in real life and have no value, except for the fact of publication in high-ranking journals and possible citations. Statements regarding the importance of fundamental research matters also do not stand up to criticism. The authors of those studies, where fundamental groundwork for new technologies is being developed, can always formulate the scope of the application of their results with a perspective of 10-15 years. In the fundamental sciences, however, it is necessary to have a rather narrow circle of researchers that track open publications under very broad thematic spectrums. They also have to highlight promising areas where practical application can be expected within 5-10 years.
Let us return to the structure of scientific publications. In addition to a section within the problem statement, the changes should also affect the conclusion section. In this section, it is necessary to indicate which scientific and practical results are claimed by the author team, according to the classification system proposed above (standard, prototype, patent, algorithm, or method). It also would be desirable to give the specific characteristics of the new product, conduct a comparative analysis with existing analogues, and give an estimate under what timeframe a finished product can be expected.
The proposed changes should also be applied to the application forms for grants. In my opinion, in the near future up to 40% of all grant programs from varied departments should be allocated to support exploratory research in Russia. Such an approach will force a fairly large number of scientists to quickly reorganize, which should lead to greater practical orientation in scientific research.
When moving to a new grading system, it is necessary to determine the exact system for checking the obtained results. It is no secret that the encouragement of exploratory research will cause a surge of unhealthy interest and a desire to pass off research areas as exploratory work. Therefore, it is necessary to plan the verification system in advance, especially since it is possible to rely on relevant international experience.
First, I have already written about the new sections in publications. In addition to these sections, the publication of experimental data (dataset), software source codes, schemes of new nodes, etc. should be encouraged.
Second, at many conferences there is usually a "demo" section, where you need to present your achievements live during practical demonstrations. This is a very important indication of the possibilities for the practical application of the results, especially if the conference is held with the participation of interested organizations. This type of experience should be disseminated everywhere. The demonstration approach is also a characteristic of various exhibitions. Compared to traditional conferences, which are attended mainly by specialists who present reports on certain topics, exhibition visitors are less qualified. Therefore, discussion during conferences is much more productive.
Third, one should not forget about the traditional ways of protecting the results of intellectual activity with regard to copyright issues. In Russia, to assess practical achievements, it is necessary to formulate intellectual activity in the following spheres:
- codes of computer programs
- databases
- inventions
- utility models
- industrial samples
- selected achievements
- topology of integrated circuits
- production secrets (know-how).
Of particular note is the possibility of entering research results into databases such as IEEE DataPort. In such databases, almost any data obtained during experiments is in demand. Experimental studies are always at the middle stage in finding a practical application of the result. Without the experimental verification of an idea, there will be no introduction, and even less possible, a prototype of the new technology. Moreover, only a physical (instrumental) experiment has value; no simulation can replace this step on the way to the practical application of the results.
Unfortunately, in Russia there is clearly a bias towards purely theoretical research, and full-scale experiments are extremely rare. This is due to the fact that it is much more difficult to conduct an experimental study, and the results are not always predictable. While theoretical studies are much simpler, in the vast majority of cases their results cannot be verified. Therefore, experimental studies need just as much support from customers as exploratory ones.
As for the assessment of the possibilities for the practical application of the results of exploratory studies, the simultaneous implementation of any two of the items listed above indicates the highest probability of the possibility of practical implementation. For example, in the case of a key research experiment which was presented at a demo session of a good conference, when the technological prototype is protected by copyright, that is, the results of intellectual activity are formalized.
It seems to me that now, the time has come when it is necessary to start the fight for practical research directions in the field of computer science. Otherwise, an avalanche of passing publications may overwhelm journals, and mechanisms for highlighting studies with practical results will then be required.
Andrei Sukhov is a Senior Member of ACM, and a professor at Sevastopol State University and Samara University. He lives and works in Samara, Russia. e-mail: [email protected]
No entries found