Blue Pages - User Tests 2

After 3 intense weeks Anusha, Monica and I have completed a second round of user tests. This time no mock ups but with the real thing, which of course is the Blue Pages v0.00 :) We did 15 tests with a selection of academics, students, administrators and one research facilitator. Again feedback has proved to be an eye opener, and is helping us on our design.

Tests lasted an average of 45 minutes each and took place mostly in our testers offices. Monica and I (and sometimes Anusha and I) visited them, so lots of buses and taxis. We used our laptop which was setup to access our server. Testers were told that we were using a work in progress version of the software and that they should expect some errors. However we had very little. We used samples of data from 4 areas of the University: Faculty of Philosophy, Department of Cardiovascular Science, Department of Phisiology Anatomy and Genetics and Gray Institute for Radiation Oncology and Biology. That is aproximately 250 people. Anusha had previously added data representing the organisational structure of the University, which includes divisions, departments, faculties, institutes, centres of study and research groups.

Our questionnaire had a similar structure to the one we used in the first round of tests. See below. We altered some of the questions we had in the first 2 or 3 tests as Anusha was adding more data. We also felt the need for repeating the same first 2 questions we had in the first round. These questions tried to capture the testers perceptions of the site struture and look and feel. As we were using the real web-based version of the Blue Pages the look and feel was completely different from the one we had in the mock up version.

v2.0

Introductory Questions 1.Ask tester to look at site’s Homepage for 10 seconds. From looking at this site, what kinds of information do you think you could get from this site? 2.Who do you think this site is designed for? Why? Tasks 3.Find Prof AAA BBB’s profile and display it in the screen. How would you find another person in the Blue Pages who does work in the same department? 4.Obtain a list of/display CCC DDD’s publications 5.Email a list of researchers in the Philosophy department to a colleague. 6.What you would do if you find a misspelling or missing information? 7.You just setup your new project’s website and would like it to be listed in the Blue pages. What do you need to do? 8.Find a list of research projects (activities) under the keyword "cardiology" Briefing 9. Do you think the information displayed in the Blue Pages is useful for Research Work? Would you trust the information provided in this site? 10.What kinds of uses of the Blue Pages can you think of which are related to your work and or studies? 11.Do you think it would be good for you to be in the Blue Pages? What kinds of benefits can you think of? (only for academics) 12.What changes and or additions would you suggest in the Blue Pages?

Yesterday, Anusha, Monica and I met to discuss the feedback. They ended up with a long to-do list. I was able to able to dig out a few outcomes, suggestions for features to add or complement the Blue Pages which are not necessarily within BRII's scope:

• Visual (or textual?) representation of networks of people connected by their collaborations and research interests. Anusha and Monica got excited by this as this could be an interesting challenge from the technical point of view.
  
• A time line of people (and maybe projects) - this involves the changes of people's profiles across time. In this way we can know about their previous roles as they rotate across areas and previous research interests as they progress in their careers. This will also include the display of information of people who have left Oxford. One aspect of this is already covered by the display of research outcomes (e.g. publications) as each item has attached to it a date of production.
  
• An extensive work on research subject ontologies - a sort of backbone of subject matters which Anusha could use to find and highlight keywords across descriptions (as in free text.) These keywords can then be used to interconnect people and activities, enabling the discovery of hidden connections.
• Inclusion of information about funders' calls for funding applications - one of the outcomes of these tests is that academics would mostly use the Blue Pages when they are thinking on starting something: projects and/or collaborations. This is when they would like to find relevant people. However as they are in that stage they would welcome information about possible sources of funding.

BRII Inside O.R.

This week I got a copy of the Inside O.R. magazine - November issue with a short article about BRII. I wrote this article after the OR51 conference (Operational Research) where I presented a paper called Making Sense of Research Activity Information in the Information Systems stream.




Thanks to Rajan Anketell, Inside O.R. editor.

Click here to download whole issue.

Institutional Repositories: acceptance and adoption

I have been reading some literature about institutional (digital) repositories. I am interested in the human and social issues surrounding their development and implementation as well as their embedding. I think this literature is very relevant to BRII as it reports experiences in similar implementations (not necessarily technically but conceptually) in similar kinds of institutions.

Technically speaking BRII is not building a digital repository but a Research Information Infrastructure (so a bit much broader in scope.) However the issues, technical and non-technical, surrounding implementation of institutional (digital) repositories apply. The reasons are:

• BRII has an institutional scope and it is situated within an academic context. It aims to collect information about research in all academic areas and its target audience is everyone within the University who carries out research-related activities.
• BRII deals with Research Activity Data: information about research. So, not information used and generated by research (as in datasets and publications) but descriptions of research (people and activities.) This kind of information is essential to facilitate scholarly communication.
• BRII core users are Researchers and Faculty: as content contributors of research descriptions and as users of the information deposited in the infrastructure.

Being now in the "development stage" of the project we are starting to think on how to make our products more relevant to our core users, so they understand them and use them. Of course to make users understand the Research Information Infrastructure we first need to understand our users. Our work in BRII should not be techocentric only but should expand to reaching out to our users, speaking their own language and observing them in their own working spaces. This can be a very difficult task as we are dealing with heterogeneous groups of academics and administrators, each one with different needs and perspectives (as per their research culture.)

Understanding our users is a crucial first step to achieving acceptance and adoption of the RII.

Research in the area of Institutional Repositories field has reported some issues concerning this.

In a study of implementation of an Institutional Repository (IR) at the University of Rochester, Foster and Gibbons (2005) report a "misalignment between the benefits and services of an IR with the actual needs and desires of faculty." They state that the benefits of Institutional Repositories are attractive only to the institutions which host the repositories. This is because Institutions (including Librarians and IT developers) see IRs as a way to facilitate access, reduce costs and improve efficiencies at collecting and managing information that is produced in the Institution. On the other hand academics see themselves as belonging to research communities (not institutions) and are not concerned about the proccesses of managing all that data they produce. This is, perhaps, because they see the problem as affecting others and not themselves. (Colbertt, 2009)

At the centre of this all is the Scholarly Communication crisis. In 2002 the Scholarly Publishing and Academic Resources Coalition (SPARC) (part of the association of Research Libraries (ARL)) stated that Institutional Repositories could help combat dissatisfactoin with the "monopolistic effects of the traditional and still pervasive journal publishing system" (Crow (2002) quoted in Maness et al (2008)). Rieger (2008) states that to combat this crisis, academic institutions aim at "reducing costs of producing and acquiring publications and gaining control of processes from commercial publishers." Seen from this angle, institutional repositories are good and sensible solutions. Attention therefore is devoted to technical efficiencies and control of the information produced within the institution. A consequence of this is a bias towards computer and librarian approaches to developing IRs, which neglect perspectives from relevant groups’ interests.

From BRIIs Stakeholder analysis we have learned that technical efficiencies, metadata, repositories are terms which are meaningless to academics. This concurs with Foster and Gibbons (2005) view. Seen from this technocentric perspective IRs do not provide benefits to academics. In addition having institutional labels suggests researchers that the IRs will support the needs of the institution and not their own individual needs. Academics are indeed a different crowd:

• Academics think in terms of reading, researching, writing and disseminating.
• Academics have strong ties with the people interested in their own field of research, or with whom they are interested to collaborate. Their geographical location is not important to them.
• Academics are interested in a relatively small subset of research information, that one of their own research field. They have acquired skills to search for and use that information for their own work.

BRII Stakeholder Analysis (Loureiro-Koechlin, 2009)

On the other hand implementers of repositories (IT and library practitioners) possess an institutional perspective, they are interested in developing new forms of scholarly communication, control costs and improve data manipulation.

An example of institutional perspective is the mapping of people and information under the umbrella of their institutions and departments. This is an obvious mismatch as researchers see themselves as belonging to their research communities. Other issues connected to the particular ways in which research is carried out (technical and ethical) also accentuate these differences. One example of this is the differences of methodologies and nature of data collected by natural scientists and social scientists. Carusi and Jirotka (2009) report ethical issues of archiving qualitative data in a digital format, such us consent, anonymity and privacy…They state that reusing research data for other purposes will go against research participants wishes and therefore researchers ideals.

Misalignments like the above are reasons why academics do not feel institutional repositories are relevant to their work. Therefore as Foster and Gibbons (2005) report there is a need to get to know researchers in their own space so we can develop tools which are useful to them as well as to the institution. This improves the chances of IRs to being accepted and adopted by their users.

One example is Maness et al's (2008) study on needs and goals of institutional repository. It reports results which contradicted initial assumptions by IR designers and decision makers. While designers and decision makers assumed users wanted an open-access archive for research outputs, the study found out that in reality users wanted a network to share learning and teaching materials, where collaborators could be identified and where their research could be promoted to institutional colleagues.

Carusi and Jirotka (2009) From data archive to ethical labyrinth. Qualitative Research.
Corbett (2009) The Crisis in Scholarly Communication, Part I: Understanding the Issues and Engaging Your Faculty. Technical Services Quarterly, vol 26 p.125-134.
Foster and Gibbons (2005) Understanding faculty to improve content recruitment for institutional repositories. D-Lib Magazine, Jan 2005.
Maness, Miaskiewickz and Sumner (2008) Using Personas to Understand the Needs and Goals of Institutional Repository Users. D-Lib Magazine, Sept-Oct 2008.
Rieger (2008) Opening up institutional repositories: Social Constitution of Innovation in Scholarly communication. Journal of Electronic Publishing.