Rough-cut evaluation of our GSA test-bed installations

As part of the current reporting cycle for our LSC TIG milestones, the primary funding source for this project, we did three things to evaluate our TFP test-bed installations in Sacramento and Chico. First, we talked to staff in both offices to hear them out about how the basic orientation to enterprise search and the temporary search portal went for them. We’ve already written up some about that here.

Second, in Sacramento we also took the time to observe some staff actually using the temporary search portal to see what their experience was actually like. Why? Well, not only is there is simply no substitute for actually talking to people about an application you are striving to evaluate, more importantly, there is no substitute for observing them while using it. Jakob Neilsen aptly sums up the differences in this regard as the contrast between “what people say” and “what people do.”

Third, we know that Bristow Hardin at LSC is going to want data because he has a Ph.D. in social science (from my alma mater; go Slugs!) and data is his life. So we also used SurveyMonkey.com to conduct a survey with an intentionally light touch to get some basic data, to see how users in our two test-bed locations were dealing with this whole enterprise (pun intended).

We put the survey up for one week and got a modest 17 respondents from the two offices. The survey had 18 questions and participation dropped off from 17 to 11 respondents from the beginning to the end of the survey. Among the results, in the order asked:

• 86% knew how to locate the TFP search portal; 14% did not.
• Asked their reaction to the elements on the temporary search portal page, they said they were “organized” (33%), “well organized” (42%) or “superbly organized” (17%); no one reported the portal page as anything less than organized.
• 92% were confident they understood what the elements on the search portal are and why they are there.
• When asked what they would want added to a search portal, only one person made a suggestion: “Can you add something so I can sort by file type”. (We have already added this filter feature to our next iteration of the GSA search results page.)
• When asked about locating the file servers where shared documents reside, 100% of 12 respondents said they knew how to locate the local file server directories; 11 of 12 said they also knew how to locate the program-wide shared file server.
• 100% knew how to locate their personal user directory and the user directories of others on the shared file server in the local office. (Okay, not a surprising result but we thought we should ask. You never know.)
• Asked how well they understood how the directory structures are organized on the program-wide shared document repository, one person out of 12 respondents said they understood some but not all of the directory names. All other respondent’s said they “get it” (50%); the organization “makes sense but I would do some of them differently” (17%); or the directory structure “makes perfect sense” (25%).
• Asked whether they are familiar with how to search using Google, 100% selected either “yes” (83%) or “are you serious?” (17%). (We have yet to encounter anyone within our organization who does not know what Google is or how to use if for search. Hey, but again, we thought we should ask.)
• We also asked whether the use of the TFP search portal was similar or dissimilar to using Google on the Web. Of 11 respondents, two thought it was “nothing like Google” and two others thought it was “something like Google.” The remainder thought it was “way Googley” (56%) or “exactly like Google” (9%).
• A key question, of course, is whether there is proof in the search pudding. Again, of 11 people who responded, one person felt searching was “not successful at all” and two were only “partially successful.” Of the remainder, four reported their searches were “successful” (37%); two reported their searches were “very successful” (18%); and two felt their searches were “dead-on successful” (18%), with an overall success rate of 73%.
• Finally, we asked staff whether they understood the search results. One person said “not at all.” Two said they “only partially understand” the search results. The rest said they either “do understand the search results” (9%); have a “very good understanding” of the results (55%); or “absolutely understand” the search results (9%), again an overall rate of 73% for those “understanding” the search results.

That’s the rough-cut survey, without any actual polish of our installation. We will be doing other surveys once we get everything in place, including the newly tweaked shared portal, and complete our roadshow to assure everyone knows how to exploit the bells and whistles we will be adding in over the next few months. Stay tuned.

[Update: In response to a question I got after posting this, there are 31 total staff in our local Sacramento and Chico offices. So, about 55% (17/31) of the staff participated in the survey and 35% (11/31) finished it, for a completion rate of 65% (11/17).]