Wednesday, September 04, 2013

Questioning the Crowd - Data Verification Challenges for Humanitarians

Source: ISN

Questioning the Crowd - Data Verification Challenges for Humanitarians

While crowd-sourcing is widely lauded as an increasingly effective approach to information gathering, it still has its limitations. Today, IRIN considers some of the challenges associated with gathering reliable and verifiable data in complex emergency situations.

Prepared by: IRIN

As evolving technologies multiply the rate of “big data” in emergencies from crowd-sourcing, to crowd-seeding to social media - humanitarian organizations are now focusing on how to ensure it is accurate and thus usable.

There are many plusses to crowd-sourced information. “It addresses a real pain for humanitarian organizations: situational awareness in unstable environments,” Chris Albon, director of data projects at Kenyan crisis mapping organization Ushahidi, told IRIN. “The data gathered by crowdsourcing technologies can provide humanitarian organizations with new and powerful streams of intelligence about the area in which they are, or will soon be, operating.”

However, information that comes from multiple, on-the-ground sources is no guarantee of its accuracy. In a crisis situation information can be limited, unreliable and poorly analysed. As Paul Currion, an IT and humanitarian coordination specialist and consultant for the International Council of Voluntary Agencies (ICVA - a Geneva-based NGO network on humanitarian developments), put it: “Crowd-sourcing is almost never going to be representative, because the crowd is by definition self-selected. This isn't a huge problem as long as it is recognised, and as long as crowd-sourcing is not the only source of data.”

For example, Somalia Speaks, set up by Al Jazeera in 2011 ahead of the 2013 London Conference on Somalia, asked Somalis, using text messages, to send in their views and questions for their government about the conference. Although they received more than 3,000 replies, the International Telecommunications Union estimates that mobile penetration within Somalia is only 7 percent. Therefore the views texted in are unlikely to be widely representative of all Somali people.

“It's therefore a mistake to talk about "speaking to" people in the context of crowd-sourcing, because what you're really doing is "soliciting from" people,” Currion told IRIN.

Networking solutions company Cisco estimated that in 2012, IP traffic reached 43.6 Exabyte per month (1 Exabyte is equivalent to about 10 billion magazine copies, according to the Economist). Between 2012 and 2017, Cisco predicts that mobile data will grow 13-fold.

In a crisis, information sharing shoots up. In the day following Japan’s earthquake in 2011, 572,000 new Twitter accounts were created and 177 million tweets were sent - some 1,200 tweets a minute from Tokyo alone. Following the Haiti earthquake, over 80,000 text messages were sent to the 4636 short code that had been set up to receive requests, according to the UN Office for the Coordination of Humanitarian Affairs (OCHA). About 90 percent of these were repetition, or “white noise”.

“Increasingly, it is the information processing requiring time and personnel, which is creating a bottleneck,” Eric Gujer wrote in a 2011 report for the ICT4Peace Foundation. There is a risk of overloading systems that aren’t equipped to handle and sift through the sheer quantity of information, much of it repeated, that comes in during a crisis.

The technology used to crowd-source information may be inappropriate. During the May 2013 Boston bombings, the public became actively engaged in attempting to identify the perpetrator of the attack, and unfortunately, initially pinned the crime on the wrong person. Through websites such as Reddit, a social news and entertainment service in which users submit content that is rated “up” or “down” by the crowd, the rumours went viral.

Analysing veracity

Patrick Meier, director of social media at Qatar Foundation’s Computing Research Institute (QCRI) and a leading expert in the field of using new technologies for crisis response, argues that a significant problem with websites such as Reddit, Twitter and Facebook is that they allow people to re-transmit the information without personally assessing its veracity.

“Reddit is not designed for critical thinking. People posted all sorts of dubious information [regarding the Boston bombings] and it snowballed,” Meier told IRIN.

Meier argues that if a technology could force people to analyse the truth of what they were reporting, crowdsourcing could be better harnessed within crisis situations.

Meier is part of a team creating a tool called, an online platform to collate and verify information in a crisis. “People can post verification requests in the form of yes/no questions. There would be a very simple sharing button, to get people to share it with their network,” he told IRIN. In addition to simply sharing or posting new information, users would be required to write a sentence explaining why information that they were claiming to be true or false was in fact the case.

“One of the reasons works is the idea that you’re only a few degrees removed from the person close to the disaster,” said Meier. “You want to get confirmation from disaster-affected communities to verify if the information is accurate or not.”
Ushahidi, in its verification guide, recommends that developers of crowd-sourced-based platforms for emergency response take extra steps to ensure the information they put up is accurate, that it distinguishes between “unverified” and “verified” responses, and is sensitive to the possibility of “poisoned data” - information posted to deliberately mislead or distract.

Keeping it simple
“Crowdsourcing is not plug-and-play,” said Albon. “Setting up a successful crowdsourcing system involves far more than simply installing software like the Ushahidi platform. It also requires training staff, conducting risk evaluations, and developing a model that can effectively turn the information gathered into real action on the ground.”

Benjamin Davies, deputy director of the signal program on human security and technology at Harvard’s Humanitarian Initiative, believes it is easier to trust the crowd if is given relatively straightforward tasks using a platform they are already familiar with, given decision-making after a crisis “is very erratic”. For instance, after the 2012 US Hurricane Sandy, there was a petrol crisis around the city of New York so Google created an app next to their maps, enabling people to put a “thumbs up” or “thumbs down” relating to petrol availability.

Similarly, during the 2012 Typhoon Bopha in the Philippines, the government endorsed a Twitter hashtag, asking people to tweet information relating to the typhoon. First responders were then able to identify who needed help, as well as distribute early-warning information. Meier believes that steps like these are crucial to reducing confusion, and getting timely, useful information in crises.

“Getting people to give geographic information is one of the biggest things that governments can do,” Meier said. “Asking people during a disaster to switch on their GPS on social media,” will help to find people quickly, and to get rapid information, he said. “If you’re doing that you’re basically doing disaster philanthropy.”

Teaming up with the tecchies
OCHA has teamed up with tecchies and media providers to crisis map in five emergencies, including Syria most recently. In Libya in 2011, it created a crisis map partnering with a "standby taskforce" of some 250 digital volunteers who collaborated to create a map of events in two months.

For Imogen Wall, OCHA coordinator of communications with affected communities, it's all about partnerships. In a recent online webchat she said: “How do we work with big data? That's about partnerships (see Flowminder in Haiti). How do we restore telco and internet networks? That's about partnerships with the private sector, as they are the service providers. How do we prep to mitigate the impact of crises on networks? That means partnering on preparedness issues. It's not just about money. It's about shared resources (data, capacity, skills, technology), shared visions, shared work on the ground.”

“The institutional humanitarian sector is increasingly irrelevant, overtaken by a range of other actors, including the military, private sector, diaspora and disaster-affected communities themselves,” noted Currion. Therefore to reform, he argued that humanitarians need to engage with a wider group of actors, and strive to preserve and promote humanitarian principles, not institutions, within this environment.

Satellite technology
Triangulating information from multiple sources, using a variety of technologies, is one way of boosting its accuracy. Davies is involved in the Satellite Sentinel Project at Harvard’s Humanitarian Initiative, which used satellite technology to collect images of what was going on between the border of South Sudan and Sudan from December 2010 until mid-2012. They collated information from open-sourced materials (newspapers, reports etc.) and sources on the ground, looking for reports of “incident spots” which would then inform where they placed their satellites. “We had a living situational report of what was happening at the time,” Brittany Card, data analysis coordinator of the project, told IRIN.

“Satellites offer you a unique lens on the face of the earth that can help decision-making,” Davies said. “An individual homeowner can see that their garage has caved in, but their roof is intact.”

But the massive knowledge gap that currently exists between the humanitarian community and those developing these technologies remains an impediment to making these tools work for humanitarians and disaster-affected people.

Bridging the gap
OCHA has recommended several ways to bridge this gap, in its recent report: Humanitarianism in the Network Age. These include setting up deployable field teams made up of humanitarians, tecchies and others; creating a neutral forum for people in the technology community, humanitarians and volunteers to share ideas before and after a disaster takes place; setting data standards; and creating more room for research and development within humanitarian agencies.

After Typhoon Pablo, OCHA created the Digital Humanitarian Network, a group of technical experts and digital volunteers, which can be “activated” during a crisis (thus far in Sudan, the Philippines, India, Samoa and the Democratic Republic of Congo). By getting humanitarians and technical experts together, technology is used more effectively to communicate with affected communities.

For Wall, there are "very few answers. The world is changing too fast for that. Get into the field, apply the principles and see what effectiveness looks like where you are. It will, in practice, look different everywhere. That's because communication is a social and cultural activity, no matter what tool you use, and is different everywhere.”

For additional reading on this topic please see:
Crisis Mapping in Switzerland
Focal Report 7: Risk Analysis
Lessons from Haiti and Beyond: Report from the 2010 International Conference on Crisis Mapping

For more information on issues and events that shape our world please visit the ISN's Dossiers and the ISN Blog.

This article was originally published by IRIN.