IT Staff Convention 2006: Evaluation Teams and Supported Products Process Review
From Provider Wiki
Future Directions of Evaluation Teams and the Supported Products Process
Facilitator: Brian Doherty, SAS
Notes: Dan Dougherty, ISC Technology Support Services
Attendees: 21 (including moderator, note taker)
The facilitator briefly outlined a loose agenda that he wanted to follow for the session including the existing structure of the Spring/Fall cycle of evaluation teams with the Fall cycle having special emphasis for LSPs. Next he introduced Mark Aseltine and John Mulhern who were in attendance. A quick poll of the group showed that a high percentage of the attendees had participated in an evaluation team.
The history of the supported products evaluation process, dating back to the reorganization of IT support at Penn, was summarized. Mark indicated that the supported products process and list are managed to try to best serve the decentralized computing support environment. An example is the explicit instruction to users to check with their LSP regarding local standards for database software.
The existing process was outlined, starting with a "call for topics" email in the Spring and Fall seasons and a selection process by IT Roundtable. Typically, applications such as web browsers, anti-virus applications, FTP clients, etc. are evaluated. University members can provide topic suggestions by responding to an email that is sent by John M to Super Users Group (SUG) whose membership accounts for around 550 members according to list-owner, Kristin Nelson. Also, IT-announce listserv receives the call for submission, which is a super list that includes members from SUG, PCNet, MacNet and other groups.
Sometimes the evaluation groups are given their deliverables and scope prior to their first meeting and other times the team determines deliverables and scope. Also most teams are co-lead by a member of ISC and a member from one of Penn’s schools or other centers. Dan Dougherty recounted his participation in eval groups of both kinds, that is, self-directed and scope-specific and briefly outlined the advantages and disadvantages of each.
Eval teams for technologies such as RSS are different from specific applications such as the File Encryption workgroup's scope (FileVault/EFS). It was reported that the workgroup for RSS took two or three weeks to determine evaluation scope. John M stated that this is normal since the group was not expected to specify a supported product. The group talked about some of the challenges that face the more “open ended” groups like RSS and discussed possibilities for better supporting those efforts. The more focused efforts often use the work of past teams as a framework for their efforts, and perhaps a superstructure that is appropriate for the RSS-type efforts could be developed and shared with the co-leads of those teams.
Someone asked if there had ever been an effort to understand the actual installed base – a survey or inventory of the systems and software currently being supported on campus. John M stated that ISC makes all kinds of attempts since support drives acceptance. "Penn is slow to adopt."
There was then some discussion of the desktop recommendations, which are based on a four year life cycle for desktops. John discussed how the life cycle for systems related to support for operating systems and other applications. In some areas, the cycle was somewhat longer and was something of a limiting factor in updating supported products. John M stated that surveys typically only show direction that technologies might be going. Mark said that John is able to look across PennNet and view what OSs are connecting, but that there had not been a broad survey. It was also suggested that surveys would also need to be coordinated within large support staffs otherwise data would be inaccurate. Also that product metrics such as cross platform use affected selection along with accommodating legacy hardware. The example was given of FileMaker Pro being a "second place" application winning over MS Access due to its dual platform support.
John stated that it doesn't take hundreds of submissions to launch an evaluation. Frequently 3 or 4 requests will cause consideration. A suggestion was made that going forward, due to evolving technologies, there was no built-in structure for evaluating products. The environment was changing; there was no enforcement mechanism. Loose areas of evaluation are more common, new technologies, education issues; no longer one-size-fits all. One eval team was recalled that felt like a waste of time in that on the first day the team knew which applications would be promoted. John noted that the Client/Server email evaluation team was a special category and, for the first time in many years wasn't being held this year.
There was a suggestion to have a mechanism for finding out what was being used across campus (i.e. backup software) because one group could benefit from the knowledge in other departments. Mark stated that the ProTools web site and ProNotes Wiki are a resource designed to support this type of information sharing. Because these sites target LSPs rather than end users ISC does not need to be so cautious about the types of information being shared.
Members of WebSIG at times form sub groups of members who share a specific interest in a related product or technology. It was suggested that such a sub group could spawn a product evaluation. One person recounted a time that she needed a product that didn't have a supported product category and was willing to use a solution used elsewhere on campus. She thought that SUG was a great source of information both in detail and reaction time. The group was reminded of the archives of the Special Interest Groups emails. Mark said that SIGs were a good source of information however mechanism for surveying was still necessary.
The topic of where does approval of topics and supported products came up. Mark stated that IT Roundtable was mostly concerned with staying ahead of the technology. A common response from IT Roundtable is "what did SUG say?" Marks stated that getting participation in the eval groups was kind of a reality check on the level of interest in a given technology. It was the team results of eval that mattered.
