TSG-SA WG3 (Security) meeting #7		TSGS-WG3#7(99)382
Den Haag, 26-27 October, 1999	
Title:	Report of TSG SA WG3 Ad Hoc Meeting To Review Submissions For Algorithm Evaluation, 25 October 1999
Source:	Vodafone
Opening of the meeting
The meeting was hosted by KPN Research and chaired by Gert Roelofsen in Mike Walker’s absence. The meeting was convened to review submissions for algorithm evaluation received in response to a recent letter sent to candidate external evaluators. 
The following objectives were set:
-	to establish criteria for selection
-	to review submissions
-	to make recommendations to S3#7 plenary
Participants
Name	Company	SDO	
Gert Roelofsen (Chair)	KPN Research	ETSI	
Wael Adi	Bosch Telecom Danmark	ETSI	
Hiroshi Aono	NTT Mobile Comm	ARIB	
Peter Howard (Secretary)	Vodafone	ETSI	
Mike Marcovici 	Lucent	T1	
Ludovic Rousseau	Gemplus	ETSI	
Benno Tietz	Mannesmann Mobilfunk	ETSI	
Temporary documents
The following documents were considered:
-	letter, terms of reference and evaluation criteria sent to list of candidate evaluators (S3-99357)
-	submission from Fred Piper, Royal Holloway
-	submission from Bart Preneel, Katholieke Universiteit Leuven
-	submission from Jacques Stern, Ecole Normale Supérieure
Update on submission process
The letter, terms of reference and evaluation criteria were sent to the following list of candidate evaluators on 4 October (S3-99357):
-	Jacques Stern, Ecole Normale Supérieure
-	Bart Preneel, Katholieke Universiteit Leuven
-	Lars Knudsen, University of Bergen
-	Ueli Maurer, ETH Zentrum
-	Fred Piper, Royal Holloway
-	Bruce Schneier, Counterpane Systems
-	Adi Shamir, Weizmann Institute of Science
-	Eli Biham, Technion
-	Tsutomu Matsumoto
-	Toshinobu Kaneki
The first 8 recipients are taken from the list presented at S3#6 (S3-99278). Tsutomu Matsumoto was added based on the suggestion of Rolf Blom (Ericsson), while Toshinobu Kaneki was added based on the suggestion of Takeshi Chikazawa (Mitsubishi).
There have been three positive responses:
-	Fred Piper, Royal Holloway
-	Bart Preneel, Katholieke Universiteit Leuven
-	Jacques Stern, Ecole Normale Supérieure
Of the other candidates Ueli Maurer (ETH Zentrum) has said that he cannot do the work and Lars Knudsen (University of Bergen) has submitted a joint proposal with Bart Preneel. The other 5 candidates have not responded.
The closing date for submissions as indicated in the letter was Friday 22 November. A reminder was sent on Wednesday 20 October to the 5 candidates that had not yet responded but no further responses have been received.
The original letter indicates that two evaluators will be selected. However, Ericsson have announced that they intend to fund a third evaluator (S3-99371).
Selection criteria
The following criteria were established:
1. Ability to do work in terms of:
Timeframe
Budget (resource/time allocation)
Project plan
2. Qualifications in terms of:
Cryptographic expertise
Balance of team
3. Reputation/profile
4. Working methods
Criteria 2 was judge to be essential. Criteria 3 is not essential but it was acknowledged that selecting a high profile evaluator would help to build confidence. 
Geographical diversity was considered to be a desirable property of the final group of evaluators due to the global reach of 3GPP.
Review submissions
All three submissions were considered to be suitably qualified in terms of cryptographic expertise and balance of team. 
The submission from Bart Preneel is very strong. The team is highly qualified with three members involved in two of the five AES finalists. The team propose to spend 20 person days effort among five cryptographers. This submission is a clear leader.
The submissions from Fred Piper and Jacques Stern are very closely matched on qualifications and reputation/profile. However there are some differences in the methodology and work allocation. Jacques Stern will work with an established colleague, Serge Vaudenay. The pair will spend a proposed total of 20-30 person days. Fred Piper will lead a group of four cryptographers allocating a proposed total of 17 person days.
Based on the above the following recommendations were considered:
1.	Recommend that all three submissions be selected
This recommendation was not adopted because an opportunity still exists to consider another candidate which could potentially improve the balance of the selected groups.
2.	Recommend that one submission be selected (Bart Preneel).
This would allow the other two submissions to be retained and considered together with any further submissions that may be received. However, this recommendation was not chosen because the chance of receiving two high quality submissions within the remaining time frame is low. Furthermore, if two submissions are retained then the risk that one or both of them withdraw their submission before a final decision is made is unacceptably high.
3.	Recommend that two submissions be selected (Bart Preneel plus either Fred Piper or Jacques Stern)
This would allow the other two submissions to be retained and considered together with any further submissions that may be received. This recommendation was chosen because it avoids the disadvantages of the first two recommendations. 
It now remains to select between Fred Piper and Jacques Stern.
As previously mentioned, the submissions from Fred Piper and Jacques Stern are very closely matched on qualifications and reputation/profile. However, after consideration consensus was reached on the Jacques Stern proposal. 
One of the (slight) advantages in favour of the Jacques Stern submission was the fact that the methodology was judged to be complementary to the Bart Preneel submission, whereas the Fred Piper submission adopts a similar methodology to the Bart Preneel submission. 
Recommendations
The submission from Bart Preneel should be selected.
The submission from Jacques Stern should be selected.
The submission from Fred Piper should be retained. If further submissions are received prior to 1 November then they should be considered alongside the Fred Piper submission. Additional candidates should not be contacted due to time constraint. A decision should be made on Monday 1 November with any selection process conducted by email correspondence.
Recommended actions to S3#7 plenary: 
-	the successful candidates should be formally notified that their proposal has been accepted 
-	ETSI should be notified about the successful candidates so that contracts can be drawn up
-	the retained candidate should be informally notified that their proposal is still being considered and that a decision will be made by Monday 1 November
Close of meeting
The meeting was closed.