Introduction

The presentation in these pages of a report on the evaluation process, together with the Scientific Program, has now become both an obligation and a tradition carried over from previous meetings of the Spanish Society of Epidemiology (SEE). In our case, we are delighted to accept this obligation for two reasons. Firstly, we hold it essential that the process which culminates in the preparation of the scientific program presented here should be public and, secondly, we believe it is important to make clear the differences between this international congress and the SEE conventions. The decision to hold a joint meeting of the Spanish Society of Epidemiology and the European Federation of the International Epidemiological Association has somewhat complicated the evaluation process compared to prior years, making it necessary to work, read abstracts, write and organize the congress in both Spanish and English. From our point of view, however, this has been a highly enriching experience. Thus, our annual meeting has on this occasion become a gateway for our European colleagues and also for epidemiologists outside the Continent, who have contributed with numerous abstracts and will be both the audience and the readers of our presentations.

Methods

Information and submission of abstracts

For the first time, we have sought to run the whole system exclusively on-line, both publishing presentation rules and evaluation criteria on the website for the XXI Congress and establishing the electronic system as the priority channel for the submission of abstracts. For this reason, it was decided not to mail information to the members of the Spanish Society of Epidemiology. Nevertheless, a letter was sent to the members of the European Federation of the International Epidemiological Association to inform them that the presentation rules and a form for the submission of abstracts would be mailed on request to those who were not equipped with the necessary computer systems.

In the abstracts submission form, the author was requested to choose at least one and not more than three keywords referring to subject areas into which the abstract would fit. Authors were also permitted to choose the language in which they would present their abstracts, which did not have to be in the same language as that in which the abstract was submitted.

Evaluation

The abstracts were evaluated by the ten members of the Scientific Committee together with the external reviewers appointed for this purpose. One significant innovation of this Congress has been the call for external reviewers by internet, with the entry forms posted online. This initiative was enthusiastically received and allowed the Scientific Committee to make use of a total of 74 external evaluators. Each reviewer was required to indicate a preference for abstracts language and to choose one to three areas of expertise.

Each abstract was assigned to two evaluators taking into consideration the subject and the language, while ensuring that the evaluator was neither a member of the same institution as the author nor, of course, one of the authors.

Each reviewer received the abstracts assigned by e-mail without including the names of the authors or the institutions to which they belonged. The evaluator was asked in the first place to read the titles of the papers and to report to the Scientific Committee any error observed regarding the subject or language. When there were any problem, the abstract was reassigned to a different reviewer. The evaluator also received an Excel file for scoring. A period of 15 days was allowed within which he/she was asked to complete the evaluation and send in the scores recorded in the Excel file.

Evaluation was based on five criteria, partially modifying those set out in the Spanish Epidemiological Society Guidelines1 for the Organization of Scientific Meetings. These criteria were decided by the Scientific Committee in such a manner as to provide a score ranged between 10 and 0.

A discrepancy in the evaluation of an abstract was deemed to have occurred when the difference between the two reviewers' scores was greater than or equal to four points, in which case a method for the re-evaluation of the paper concerned would be applied.

Abstracts were sent out for evaluation on April 1 with the deadline for completion on April 14. It was necessary to reassign evaluators for 16 abstracts because the reviewers knew the paper in question or had formed part of the team of authors. One reviewer refused to participate in the evaluation process. Finally, the majority of the evaluations were received within the deadline.

A total of 56 discrepancies were observed out of the 507 abstracts received and evaluated. Three members of the Scientific Committee re-evaluated these abstracts, awarding scores on the basis of the same criteria.

The final score for each paper was established as the average of the two scores obtained, except where a third evaluation was necessary, in which case the final score assigned was the median. The median was chosen in view of the distribution of the scores obtained. It was usually the case with discrepancies that the third reviewer was in agreement with one of the initial two and in disagreement with the other.

The Scientific Committee met on May 9 and 10. In the course of this meeting, the abstracts accepted were selected. A minimum score of 4.5 was established for acceptance. A threshold of 6 points was established for those abstracts applying for oral presentation, and those with a score of between 4.5 and 6 were accepted as posters. In light of the external reviewers' comments, it was observed that certain abstracts intended to be presented as oral communications were very similar in terms of methodology, survey population, etc. and in these cases it was decided to accept the lower-scoring abstract in the poster format, once the similarity was confirmed.

Results

Formal issues concerning the abstracts

A total of 507 abstracts were received. All of them were submitted through the web site, although some authors preferred a double mailing and also sent papers by e-mail or fax, as well as using the online facility.

By language, 220 abstracts were submitted in Spanish (43.39% of the total received) and 287 in English. The distribution by language and presentation type is shown in Table 2.

Table 3 shows the distribution of abstracts per country of origin, defined as the country of the Institution to which the contact author belonged.

The vast majority of the abstracts (91.12%) were, as expected, European in origin. Over 50% were submitted from Spain, and English was chosen as the language of presentation for 66% of these. After Spain, Germany was the country that submitted the most abstracts (6.71%), followed by Poland (5.33%), Portugal (3.94 %) and the United Kingdom (3.35%). Interestingly, papers were also received from such faraway countries as Japan, the Philippines and Taiwan to name but three.

In absolute terms, the number of abstracts (286) submitted from Spain was lower than in previous years (356 and 346 in 2002 and 2001 respectively)2,3. It is no easy task to identify the reasons for this drop, though it may be related with the international nature of the meeting.

Table 4 shows the distribution of the abstracts submitted from within Spain by Autonomous Community (self-governing region). This classification was also prepared on the basis of the Institution of the lead author. Madrid and Catalonia continue to supply the greater part of the papers, and those submitted from these two regions alone represented over 50% of the total. They are followed by Valencia (15.03%) and Andalusia (14.69%). No papers were received from the Canary Islands, Cantabria, La Rioja or Ceuta and Melilla.

We also wished to present a distribution of the abstracts based on the type of institution in which the lead author works, but encountered difficulties in making this classification. As a result, we have only differentiated between three categories: official agencies, universities and schools of public health, and hospitals. The last of these categories was assigned wherever the name of a hospital was given, even if it was a teaching hospital. The data are shown in Table 5. There has been an increase in the percentage of papers submitted from universities and public health schools compared to previous years and a decrease in papers from official agencies. These differences are probably due to the inclusion of papers from countries other than Spain.

Evaluation

The results of the initial review are shown in Table 6. A total of 56 discrepancies requiring review were observed, of which 7 were rejected and 49 accepted in the subsequent review. Despite the difficulties inherent in comparison with previous years, it is interesting to note that the use of a greater number of evaluators did not cause any noticeable dispersion of scoring, as is clear from the low percentage of discrepancies observed.

In the course of both review processes a total of 41 papers were rejected, with the authors opting for presentation in Spanish in 8.63% of cases (19 papers) and 7.67% (22 papers) in English. Rejection rates from previous congresses were 9.8% in Barcelona in 2002, 10.7% in Murcia in 2001 and 5.9% in Madrid in 2000. The final language of presentation and the first choices of authors are shown in Table 7.

Table 8 presents the descriptive results of average scores and differences between evaluators. As shown, the scientific standard of all of the papers submitted may be considered high, with an average score of 7 and a percentile 5 out of 4.

Table 9 shows the accepted abstracts classified by subject and language of presentation. It has not been possible to establish a grouping by subject in accordance with the discussion groups that will be formed because these differ in each of the languages, basically due to the number of abstracts received in the different subjects in each of the two languages. In general, the areas of "epidemiological surveillance" and "outbreaks" make up some 20% of the total, while cancer continues at around 10%, which is in line with prior years2,3. The sharp increase in abstracts dealing with "cardiovascular diseases" is also noteworthy, and to a lesser degree the number of communications given over to "gender and health", "reproductive health" and "child health". The majority of the abstracts in Spanish concern infectious diseases. Thus, the three main areas in Spanish (epidemiological surveillance, outbreaks and tuberculosis and HIV) comprise almost half (46.27%) of the accepted papers. The top three areas for papers presented in English are cardiovascular and other chronic diseases, cancer, and child and reproductive health, making up 42.64% of the total.

External reviewers

Only one of the 74 external reviewers listed on the website between October 24, 2002 and March 17, 2003 did not actually take part, communicating his decision when the abstracts were sent. Forty-three of them were men (59%) and their average age was 44.5 years, falling within a range of between 29 and 72 years. Applying the same classification as above with regard to the institution in which the evaluator works, 40 evaluators are employed in official agencies, followed by 27 working in universities and public health schools. Three of the evaluators work in private companies and one is a retired professional. Fifty-two evaluators (71%) are members of their country's epidemiological society. In connection with the languages reviewed, only five evaluators requested Spanish only, while 47 were able to work in both languages. Table 10 shows the countries of the institutions indicated by the reviewers. Over 50% of them work in Spain. Of these, 16 (40% of the Spanish evaluators) are located in Catalonia and 10 (25%) in Madrid.

Thematic sessions

The Congress website also provided the possibility of submitting proposals for debate in what is call "thematic sessions", although of course using a different form and with other deadlines.

The members of the Scientific Committee evaluated these proposals. A total of nine were received within the deadline, seven of them in English and two in Spanish. Each of them was evaluated by 4 people, who assigned a score of between 0 (session that should not be included in the program under any circumstances) and 5 (highly relevant session: must be included in the program). Only one discrepancy arose between two of the evaluators for one of the proposed sessions, and this was resolved through a second reading by both.

A total of 7 thematic sessions were accepted, all of which scored higher than 3 (sufficient interest and content for inclusion of the session as part of the program). The possibility was provided of presenting the talks included in the sessions that were not accepted as oral communications to be evaluated in the same manner as others.

Comments

In the first place, we believe it is necessary once again to mention the difficulties inherent in organizing a joint congress with the European Federation. These were most noticeable not so much in the evaluation process as in the organization of the program. We are aware that the split between two languages could hinder access to talks that might be of interest to some of the Spanish participants and many guests from other countries. Because of this, we have tried to ensure that all subjects are communicated in both languages in order to provide all participants with the opportunity of finding talks they wish to attend.

For obvious reasons, the number of abstracts received cannot be compared with previous years. We believe that the deadline for the submission of papers, which was very short in comparison to what has been usual in the Congresses of the Spanish Epidemiological Society may have prevented some of our colleagues from sending in their work on time. This situation was the result of our efforts to meet a request by the European Federation of the International Epidemiological Society to the effect that we should respond affirmatively or reject papers from foreign participants at an earlier date than would be required for domestic participants in order to facilitate the required procedures related with leave, expenses and travel.

The whole process for the submission of abstracts and the exchange of information with the authors has been carried out online, and we wish to make clear that it has functioned remarkably well, despite some problems that have come to our attention, and that the information flow has been both fast and fluent or perhaps better flexible.. If at first we considered the use of paper and postal communication for those who were not equipped with the necessary resources, we should point out that nobody actually availed themselves of this possibility.

The opening of the congress to external evaluators has been a significant and innovative experience. We have found this to be a very interesting way of working, and the response from volunteer reviewers has been excellent. It is however true, that this procedure did make it more difficult to resend the papers in those cases where discrepancies arose and it was necessary to opt for a third evaluator from within the Committee. A greater number of discrepancies might, however, have been expected given the disparity between the evaluators. Nevertheless, 11% seems to us a very acceptable rate.

In the course of organizing the program, it was decided to ensure that the thematic sessions did not coincide with oral communications in order to ensure attendance, given the interest that these sessions would generate. It was also decided that the poster sessions should be scheduled to allow the author to spend time beside the poster giving and receiving feedback. The large number of posters and the bilingual presentation mode make it impossible to arrange for specific sessions or "guided tours".

To sum up, we have here explained the pros and the cons of the process employed to prepare the program you have in your hands. We would like to express our appreciation of the efforts made by everybody involved in this program. Thank you for sharing your experience and work with all of the congress participants. We trust that you will not be disappointed with Toledo and that these days will be enriching and serve to establish links both inside Spain and abroad.

References

1. Aibar C, Pérez de Agenda JPA, Borrell C, Fernández S, Benavides FG, Hervada X, Porta M, et al. Guía de Organización de las Reuniones Científicas de la Sociedad Española de Epidemiología. Version 2. January 2000. http://www.websee.org

2. XIX Reunión Científica de la Sociedad española de Epidemiología. Comunicaciones. Gaceta Sanitaria 2001: 15 (Supl 2).

3. XX Reunión Científica de la Sociedad española de Epidemiología. Comunicaciones. Gaceta Sanitaria 2002: 16 (Supl 1).