Blue Pages - User Tests 2

After 3 intense weeks Anusha, Monica and I have completed a second round of user tests. This time no mock ups but with the real thing, which of course is the Blue Pages v0.00 :) We did 15 tests with a selection of academics, students, administrators and one research facilitator. Again feedback has proved to be an eye opener, and is helping us on our design.

Tests lasted an average of 45 minutes each and took place mostly in our testers offices. Monica and I (and sometimes Anusha and I) visited them, so lots of buses and taxis. We used our laptop which was setup to access our server. Testers were told that we were using a work in progress version of the software and that they should expect some errors. However we had very little. We used samples of data from 4 areas of the University: Faculty of Philosophy, Department of Cardiovascular Science, Department of Phisiology Anatomy and Genetics and Gray Institute for Radiation Oncology and Biology. That is aproximately 250 people. Anusha had previously added data representing the organisational structure of the University, which includes divisions, departments, faculties, institutes, centres of study and research groups.

Our questionnaire had a similar structure to the one we used in the first round of tests. See below. We altered some of the questions we had in the first 2 or 3 tests as Anusha was adding more data. We also felt the need for repeating the same first 2 questions we had in the first round. These questions tried to capture the testers perceptions of the site struture and look and feel. As we were using the real web-based version of the Blue Pages the look and feel was completely different from the one we had in the mock up version.

v2.0

Introductory Questions 1.Ask tester to look at site’s Homepage for 10 seconds. From looking at this site, what kinds of information do you think you could get from this site? 2.Who do you think this site is designed for? Why? Tasks 3.Find Prof AAA BBB’s profile and display it in the screen. How would you find another person in the Blue Pages who does work in the same department? 4.Obtain a list of/display CCC DDD’s publications 5.Email a list of researchers in the Philosophy department to a colleague. 6.What you would do if you find a misspelling or missing information? 7.You just setup your new project’s website and would like it to be listed in the Blue pages. What do you need to do? 8.Find a list of research projects (activities) under the keyword "cardiology" Briefing 9. Do you think the information displayed in the Blue Pages is useful for Research Work? Would you trust the information provided in this site? 10.What kinds of uses of the Blue Pages can you think of which are related to your work and or studies? 11.Do you think it would be good for you to be in the Blue Pages? What kinds of benefits can you think of? (only for academics) 12.What changes and or additions would you suggest in the Blue Pages?

Yesterday, Anusha, Monica and I met to discuss the feedback. They ended up with a long to-do list. I was able to able to dig out a few outcomes, suggestions for features to add or complement the Blue Pages which are not necessarily within BRII's scope:

• Visual (or textual?) representation of networks of people connected by their collaborations and research interests. Anusha and Monica got excited by this as this could be an interesting challenge from the technical point of view.
  
• A time line of people (and maybe projects) - this involves the changes of people's profiles across time. In this way we can know about their previous roles as they rotate across areas and previous research interests as they progress in their careers. This will also include the display of information of people who have left Oxford. One aspect of this is already covered by the display of research outcomes (e.g. publications) as each item has attached to it a date of production.
  
• An extensive work on research subject ontologies - a sort of backbone of subject matters which Anusha could use to find and highlight keywords across descriptions (as in free text.) These keywords can then be used to interconnect people and activities, enabling the discovery of hidden connections.
• Inclusion of information about funders' calls for funding applications - one of the outcomes of these tests is that academics would mostly use the Blue Pages when they are thinking on starting something: projects and/or collaborations. This is when they would like to find relevant people. However as they are in that stage they would welcome information about possible sources of funding.

BRII Inside O.R.

This week I got a copy of the Inside O.R. magazine - November issue with a short article about BRII. I wrote this article after the OR51 conference (Operational Research) where I presented a paper called Making Sense of Research Activity Information in the Information Systems stream.




Thanks to Rajan Anketell, Inside O.R. editor.

Click here to download whole issue.

Institutional Repositories: acceptance and adoption

I have been reading some literature about institutional (digital) repositories. I am interested in the human and social issues surrounding their development and implementation as well as their embedding. I think this literature is very relevant to BRII as it reports experiences in similar implementations (not necessarily technically but conceptually) in similar kinds of institutions.

Technically speaking BRII is not building a digital repository but a Research Information Infrastructure (so a bit much broader in scope.) However the issues, technical and non-technical, surrounding implementation of institutional (digital) repositories apply. The reasons are:

• BRII has an institutional scope and it is situated within an academic context. It aims to collect information about research in all academic areas and its target audience is everyone within the University who carries out research-related activities.
• BRII deals with Research Activity Data: information about research. So, not information used and generated by research (as in datasets and publications) but descriptions of research (people and activities.) This kind of information is essential to facilitate scholarly communication.
• BRII core users are Researchers and Faculty: as content contributors of research descriptions and as users of the information deposited in the infrastructure.

Being now in the "development stage" of the project we are starting to think on how to make our products more relevant to our core users, so they understand them and use them. Of course to make users understand the Research Information Infrastructure we first need to understand our users. Our work in BRII should not be techocentric only but should expand to reaching out to our users, speaking their own language and observing them in their own working spaces. This can be a very difficult task as we are dealing with heterogeneous groups of academics and administrators, each one with different needs and perspectives (as per their research culture.)

Understanding our users is a crucial first step to achieving acceptance and adoption of the RII.

Research in the area of Institutional Repositories field has reported some issues concerning this.

In a study of implementation of an Institutional Repository (IR) at the University of Rochester, Foster and Gibbons (2005) report a "misalignment between the benefits and services of an IR with the actual needs and desires of faculty." They state that the benefits of Institutional Repositories are attractive only to the institutions which host the repositories. This is because Institutions (including Librarians and IT developers) see IRs as a way to facilitate access, reduce costs and improve efficiencies at collecting and managing information that is produced in the Institution. On the other hand academics see themselves as belonging to research communities (not institutions) and are not concerned about the proccesses of managing all that data they produce. This is, perhaps, because they see the problem as affecting others and not themselves. (Colbertt, 2009)

At the centre of this all is the Scholarly Communication crisis. In 2002 the Scholarly Publishing and Academic Resources Coalition (SPARC) (part of the association of Research Libraries (ARL)) stated that Institutional Repositories could help combat dissatisfactoin with the "monopolistic effects of the traditional and still pervasive journal publishing system" (Crow (2002) quoted in Maness et al (2008)). Rieger (2008) states that to combat this crisis, academic institutions aim at "reducing costs of producing and acquiring publications and gaining control of processes from commercial publishers." Seen from this angle, institutional repositories are good and sensible solutions. Attention therefore is devoted to technical efficiencies and control of the information produced within the institution. A consequence of this is a bias towards computer and librarian approaches to developing IRs, which neglect perspectives from relevant groups’ interests.

From BRIIs Stakeholder analysis we have learned that technical efficiencies, metadata, repositories are terms which are meaningless to academics. This concurs with Foster and Gibbons (2005) view. Seen from this technocentric perspective IRs do not provide benefits to academics. In addition having institutional labels suggests researchers that the IRs will support the needs of the institution and not their own individual needs. Academics are indeed a different crowd:

• Academics think in terms of reading, researching, writing and disseminating.
• Academics have strong ties with the people interested in their own field of research, or with whom they are interested to collaborate. Their geographical location is not important to them.
• Academics are interested in a relatively small subset of research information, that one of their own research field. They have acquired skills to search for and use that information for their own work.

BRII Stakeholder Analysis (Loureiro-Koechlin, 2009)

On the other hand implementers of repositories (IT and library practitioners) possess an institutional perspective, they are interested in developing new forms of scholarly communication, control costs and improve data manipulation.

An example of institutional perspective is the mapping of people and information under the umbrella of their institutions and departments. This is an obvious mismatch as researchers see themselves as belonging to their research communities. Other issues connected to the particular ways in which research is carried out (technical and ethical) also accentuate these differences. One example of this is the differences of methodologies and nature of data collected by natural scientists and social scientists. Carusi and Jirotka (2009) report ethical issues of archiving qualitative data in a digital format, such us consent, anonymity and privacy…They state that reusing research data for other purposes will go against research participants wishes and therefore researchers ideals.

Misalignments like the above are reasons why academics do not feel institutional repositories are relevant to their work. Therefore as Foster and Gibbons (2005) report there is a need to get to know researchers in their own space so we can develop tools which are useful to them as well as to the institution. This improves the chances of IRs to being accepted and adopted by their users.

One example is Maness et al's (2008) study on needs and goals of institutional repository. It reports results which contradicted initial assumptions by IR designers and decision makers. While designers and decision makers assumed users wanted an open-access archive for research outputs, the study found out that in reality users wanted a network to share learning and teaching materials, where collaborators could be identified and where their research could be promoted to institutional colleagues.

Carusi and Jirotka (2009) From data archive to ethical labyrinth. Qualitative Research.
Corbett (2009) The Crisis in Scholarly Communication, Part I: Understanding the Issues and Engaging Your Faculty. Technical Services Quarterly, vol 26 p.125-134.
Foster and Gibbons (2005) Understanding faculty to improve content recruitment for institutional repositories. D-Lib Magazine, Jan 2005.
Maness, Miaskiewickz and Sumner (2008) Using Personas to Understand the Needs and Goals of Institutional Repository Users. D-Lib Magazine, Sept-Oct 2008.
Rieger (2008) Opening up institutional repositories: Social Constitution of Innovation in Scholarly communication. Journal of Electronic Publishing.

Data, ideas and more

Yesterday, I met with Luis Martinez Uribe, Digital Repositories Research Coordinator, to talk about research data. Luis is working in the The Embedding Institutional Data Curation Services in Research (EIDCSR) project which looks at research data management and curation challenges. Previously he worked in the Scoping digital repository services for research data management project where he collected very interesting data which forms the foundation for the EIDCSR and hopefully many more similar projects.

Luis and I talked about overlaps in both projects and on how some common themes arised in both our data analyses. Luis has collected information about the processes and needs of researchers regarding the creation and use of research data. These are data that researchers create and use as part of their research activities (e.g., images of the heart. ) He is looking at developing processes for creating metadata and preserving both metadata and research data across time. In this form data can be reused by multiple researchers in multiple ways.

In BRII I have collected information about the nature of research activities across the Scientific - Human spectrum and from the academic, administrative and strategist perspectives and how these activities are reflected in public data such as websites. Part of the information collected reveals the reasons for sharing Rearch Activity Data (e.g., improve visibility) which are the reasons BRII wants to support and enhance across the University and beyond. Of course data have also revealed that there are reasons against making Rearch Activity Data publicly available (e.g. confidentiality issues) as well as technical difficulties which make sharing a complex task.

Anyhow, we found that there were common issues arising from the creation, use and sharing of research related data: Research Data (EIDCSR) and Research Activity Data (BRII) particularly within the context of institutional initiatives (as opposed to individual efforts). We think that these issues deserve some attention as they are influential in the implementation and acceptance of efforts like EIDCSR and BRII by their core users: Researchers.

...So we are outlining a paper which will explore those issues and let's see what happens.

Making Sense of Research Activity Information

This week I attended the Operational Research OR51 conference at the University of Warwick. This is a big conference for the OR people. A very exciting event. I gave a presentation in the Information Systems stream, having previously submitted an abstract. We started conference activities on Tuesday morning, but there was dinner and a wine tasting welcome event on Monday evening. :)

The information systems stream was opened by Nick Davey, IBM Delivery leader with a presentation on Client Relationships.

I followed then. My presentation was titled: Making Sense of Research Activity Information. I talked about the findings of the BRII Stakeholder analysis: needs we need to account for and benefits that a tool like the Research Information Infrastructure RII can bring to researchers, administrators, strategists and disseminators. I was a bit ambitious and in the few last seconds of my presentation I suggested and idea for analysing in depth different stakeholders profiles in different contexts (creation and use of research activity data) by using the Model of Enactment of Technologies-in-practice by Orlikowski (i). This is a useful tool, a practice lens as she calls it, to look at the use of technology and the (social) structures that influence its use. These structures are not necessarily inscribed or embeded in the technology by designers but surround users in their daily activities. See the presentation and abstract below.




Note: the last slide is just an example of how agents and contexts can be modeled by using the model of enactment of technologies-in-practice. By separating the issues in each of the boxes (interpretative scheme, norms and facilities) we can understand better why technologies are used the way they are used. Understanding all these issues within the context of the University of Oxford would help us to design tools that use information about research activities in the ways our users find most relevant and useful.

Abstract

Oxford University is a research-intensive institution. Research is carried out individually or in groups; within one subject field or in cross disciplinary collaborations. It also crosses organisational boundaries to other universities in the UK and abroad. Research expertise, resources and outcomes vary from discipline to discipline. Information about research activities is published in websites using independent systems belonging to academic units, research centres, institutes, projects and individual researchers. These systems and information have been designed to fulfil the individual needs of each area or research activity and accounting for outside needs is beyond their aims. Seen all together they look like fragmented, dispersed systems resembling Oxford’s federated, complex structure. From such heterogeneous sources of information it is very difficult to obtain a clear picture of research done in the University.

Research activity information needs to be visible and easy to access to increase the research impact of the institution, and to boost collaboration and funding. Efforts are being made to create a research information infrastructure to collect research activity data from around the University, to classify and connect those sources and to make them available in a consistent and organised way. This paper explores the perspectives and issues related to the use of research activity information and the systems through which this information can be made visible and kept updated in an accurate and consistent way. The paper is focused on a stakeholder analysis done on a sample of Oxford academic and administrative units involved in research and related activities.

Anyway, conference was good, saw many interesting presentations. We also went on a Shakespeare tour around Stratford upon avon :)



(i) Orlikowski developed her model of enactment of technologies-in-practice based on Giddens' ideas of structuration.

• Orlikowski, W. J., (2000), 'Using Technology and Constituting Structures: A practice Lens for Studying Technology in Organizations', Organisation Science, 11, no.4: 404-428
• Giddens, A., (1984), The Constitution of Society, Cambridge, Polity Press.

JISC SSBR Newsletter

The JISC Institutional Innovation programme has released a new issue of their Support, Synthesis and Benefits Realisation (SSBR) Newsletter. (Issue Six - 1 September 2009.) We are happy to see that we are featured on the front page, or is it the first paragraph? :)

Activity slowed a little over the past few weeks? Looking at the significant progress made by many projects, it appears not. BRII have completed their Stakeholder Analysis report and there are a several assemblies in the pipeline for projects to swap ideas and share resources. In this newsletter, you will also find news of free training, workshops and tools which you may find useful along with key conference and seminar dates.

You can access the SSBR Newsletter from:
http://newsletter.inin.jisc-ssbr.net/2009/09/01/issue-six-1-september-2009/

There is a short article about us titled BRII Stakeholder Analysis Report:

Recently Cecilia Loureiro-Koechlin from the BRII project announced that they had reached a project milestone by completing their Stakeholder Analysis report.

Click on the following link to read the complete article: http://newsletter.inin.jisc-ssbr.net/2009/08/31/briireport/

Beyond the Repository Fringe 2009

Sally and Ben were in Edinburgh last month attending the Beyond the Repository Fringe 2009, an event for repository developers, managers, researchers, administrators and onlookers.

They shared central stage when they gave the opening keynote speech. This is what they said:

Ben O’Steen and Sally Rumsey (Oxford) – “A sneak preview at the A-list stars of future repositories: blockbuster technical developments and the cultural drivers behind them”

Sally opens by explaining that she and Ben will be handing back and forth with Sally looking at the more library view of repositories whilst Ben will be talking about the more technical whizzy end of affairs.

Sir Thomas Bodley set up the library in Oxford and Sally is taking us through the history of the library including a lovely quote from Francis Bacon that the Bodley "is an arc to save knowledge". We're are also looking at search, 1620 style: a paper list.

The original library building fast ran out of space and the Radcliffe Camera, the Radcliffe science library and the new Bodlien library were all built. By 1914 the library received a million items a Year. It continues to grow and grow and Sally shows us a preview of the storage facility in Swindon which will be helping the Bodley deal with the volume of material by 2010.

Source: DataShare Blog

Click here to read the whole transcript.

Where are Repositorie's Going?