Project Evaluation

We just finished our Project Evaluation report. The project's summative evaluation took place on the 19th of February and was directed by Neil Beagrie from Charles Beagrie Ltd. As part of this evaluation Neil ran an online survey with a sample of our interviewees and testers. The results were very positive. We chose people who knew about BRII and who had at least seen a demo or tested the Blue Pages. The reason for this was that we needed respondents who understood the basic concepts of BRII’s entity registry and RAD (Research Activity Data) and and were able to assess the potential benefits that sharing and re-using RAD could have for the University.

For the purpose of the survey we asked respondents to have a look at the working version of the Blue Pages. As this tool is still under development we have limited access only from Oxford networks. However I also compiled a series of screenshots for respondents who were not on campus. You can see the status of the Blue Pages six weeks ago.



and these are the results of the survey (click on + to zoom in):

Uncovering User Perceptions of Research Activity Data

I recently wrote a paper about the user testing of the Oxford Blue Pages. It has been published in Ariadne in their January 2010 issue. Here you have a link to the article in Ariadne www.ariadne.ac.uk/issue62/loureiroKoechlin The title is Uncovering User Perceptions of Research Activity Data. I chose that title to emphasise the aspect of the user testing which we think helped us the most to engage and understand our users. Beside improving software usability the user tests provided us with data which confirmed and complemented the findings of the Stakeholder Analysis. In this article I discuss the concepts of Usability, Perceived Ease of Use and Perceived Usefulness and provide some examples from feedback.

I have also deposited a copy of the article in the Oxford University Research Archive ORA here http://ora.ouls.ox.ac.uk/objects/uuid%3A1d61ea76-1a6a-4595-a011-52a8303c549b
You can also find the ORA record (with the full text) for the BRII Stakeholder Analysis here http://ora.ouls.ox.ac.uk/objects/uuid%3A1df69991-cd37-445b-a4c7-3573ce80c36e


Note: I have added a section titled "BRII Papers, Reports and Presentations" in the right column of this blog (scroll down a bit.) It contains links to relevant outcomes produced by BRIIl.

The BRII Project Use Cases Report is ready

The BRII Project Use Cases report is ready and available in PDF format from here. The Use Cases report contains four short, business-style use cases illustrating the uses and benefits of the BRII project outputs.

Any comments about the report are welcome. (Click on comments below or email me.)

BRII's Products - JISC Trade Fair

These are posters I designed for the Institutional Innovation Projects Trade Fair. The posters explain our three products: a vocab site, an approach to user testing and the Blue Pages.

Click on pictures to get larger versions

We have many more products of course but these are the ones I thought could be more attractive to other projects. In addition to the above, we have produced an object registry containing aggregated research activity data which is actually the core of the project, we are designing processes of data harvesting, we have written a stakeholder analysis which studies the requirements and uses for research activity data at the University of Oxford, we organised a JISC assembly on stakeholder buy-in, etc.

Institutional Innovation Projects Trade Fair

I have been working on the products I will offer in the Projects Traid Fair, part of the JISC Institutional Innovation Exchange event. I have designed a couple of posters to explain my three products. These will be:

• The Vocabulary site storing the new vocabularies created by BRII providing formal descriptions of concepts, terms and relationships within the "research" knowledge domain (funder information, people profiles, research data).
• Experiences and lessons learned from the User Testing of the Blue Pages. I have related some of these experiences in previous posts. I have also written a paper which will hopefully be published at the end of this month in Ariadne. In this paper I describe our approach to user testing, the reasons for choosing that approach and the lessons we have learned from it.
• The Oxford Blue Pages... or at least the concept of the Blue Pages, displaying aggregated Research Activity Data as research objects and the connections between them.

The JISC have asked us to make projects products directly related to your final report and outputs.

I will also have to buy products from other projects attending the event. I asked our JISC contact if we had to buy products which could be used only in our current JISC projects. He said there was not problem with making purchases with related activities. So I can buy products which can be useful in other activities in our office or perhaps other departments in the University.

What is going on in BRII

Since September we have been carrying out user tests of the Oxford Blue pages. These tests have been very useful to improve the Blue Pages usability and to get the perspectives of potential users. These perspectives were not limited to the Blue Pages but to the BRII registry. After seeing the Blue Pages testers asked about the sources of data and the processes of updating these in the registry. Some testers raised issues about accuracy and validity of data, others asked about data coverage (within and outside Oxford) and the processes to update or synchronize these data with changes in the original sources and scope. Regarding this last one, scope, some academics suggested harvesting their personal websites and blogs (i.e., to not only harvest data from official departmental sources.) They said personal sites "reflect more their career and research experience as they are not limited to their activities in Oxford but relate previous experiences.” Other testers suggested ways to organising and presenting information. For example how to present a list of publications. Some people were interested in the abstracts while some others just wanted basic information such as the title, year and author.

All these feedback and questions are food for thought for us. User tests are an ongoing activity. Although were are not doing as many as we did last year we are still running them whenever we get an opportunity.

In addition to the user tests, we have also run some dissemination activities, such as meetings and demos with interested parties. We recently had a meeting with some people in the Social Sciences division. They are working on a divisional system which will cover areas such as teaching, HR and research within their division. Most of our conversation focused on the nature of the data harvested. We discussed if the aggregation of data in the registry and the Blue Pages could change their original nature and intentions. We also talked about how the Blue Pages will make these data, which is already publicly available, more visible. The Blue Pages will enhance all the positive aspects of research within the University. However it will also make some data problems more obvious. Social Sciences participants realised that some inconsistencies could be made more obvious in the Blue Pages. As for example, wrong names or affiliations. Correcting these inconsistencies is of course not a task for BRII but for the people responsible of the original sources. However BRII can potentially help them by letting them know where these problems are, e.g. Blue Pages users can report errors in data.

We have also been working on a new Graduate Opportunities website for the Medical Sciences division. This is a second planned output of BRII. Anne Bowtell is in charge of this. She is using the BRII API and data from the registry to populate this website. These are the data Anusha is harvesting from Medical Sciences sources which of course will also be available from the Blue Pages.

On a different matter, in a couple of weeks I will attend the next JISC Innovation Exchange event. This will take place in Birmingham. One of the main events will be a kind of trade fair where we will have to sell to and buy products from other JISC Institutional Innovation projects. This sounds exciting! Now I need to think on how to sell our BRII products and need to get some updates of the status of other projects so I know in advance what will be on offer then.

JISC MEAoT Assembly

Yesterday I attended a JISC assembly organised by the Modular e-Administration of Teaching MEAoT Project run by CARET - Cambridge University. The aim of the event was to explore ways in which stakeholders and other parties can be encouraged to adopt tools developed in JISC projects. Below you can see my presentation:



The second slide briefly explains the nature of BRII and the Entity Registry we are creating. The registry is a container of Research Activity Data and it is these data which has been semantizised that we are trying to sell around the University. BRII is doing this at two levels. First, being the registry an abstract concept, users find it difficult to understand. Therefore BRII needs to provide tangible examples that use its data. Slide 3 shows this as level 1: providing practical examples of data use, one of which is the Blue Pages. However, as the Blue Pages is a new piece of software BRII needs to find ways to promote it across its user base. Through user tests (level2 ) on the Blue Pages we make sure it fits the needs of our potential users, but at the same time we sell the idea of Research Activity Data being available in our registry for another users.

There were other presentations of course. The MEAoT people have written a summary about the event in their blog:

The Centre for Applied Research in Educational Technologies (CARET) in Cambridge University hosted a JISC assembly for the Modular e-Administration of Teaching (MEAoT) project on Thursday, 10 December 2009.

The list of participants included:

Richard Prager, MEAoT project PI, Cambridge University Engineering Department (CUED)
Laura James, CARET
Anne Clarke, MEAoT project, CARET and CUED
Guy Chisholm, MEAoT project, CARET
Avi Naim, MEAoT project, CARET
Cecilia Loureiro-Koechlin, JISC BRII project, Oxford University
Bridget Taylor, DAISY project, Oxford University
Jenny Mackness, JISC liaison for MEAoT
Rachel Tuley, Teaching Administrator, CUED
Jen Pollard, Computer Officer, Cambridge University English Department
Carmen Neagoe, Teaching Administrator, Cambridge University Judge Business School
Helen Marshall, Teaching Administrator, Cambridge University Physics Department
David Goode, Computer Officer, Cambridge University, Department of Divinity

The programme was designed to make the meeting a pleasant event. It started with a historical overview of the project (by Prof. Prager). This was immediately followed by lunch at a nearby restaurant, which helped break the ice and make everybody more comfortable.

Read more here: http://modular-e-admin.blogspot.com/2009/12/modular-e-administration-of-teaching.html